I can't provide a competent review of the blist changes on the fly, but the description makes sense. I'd offer to do more swap testing, but you've tested to about the same extent I meaningfully can. Do we have a tool which can meaningfully exercise swap for test purposes (maybe using mdconfig(8)?)? I have a ton of swap allocated, but don't touch it because I also have an abundance of RAM. Anything useful I can do? Thanks for digging in to this!!

I would wonder if it makes sense to implement these in terms of cmpset operations on register-sized quantities in C with the appropriate arithmetic shifts, rather than doing the assembly sort of half by-hand. That's maybe less optimal, but do we have performance-critical 8- and 16-bit atomics in performance-critical areas? I've always been skeptical of smaller-than-register atomics and tend to resist them, so that may just be a personal bias. I can't speak to correctness beyond that.

In D18593#440643, @jah wrote:

To answer my own question: no, we can't repurpose entry 0. We wire it down in switch.S to guarantee we won't take TLB faults on access to the kernel stack. Makes sense, esp. if we don't have something like a boot stack or exception stack to fall back on.

I don't have any exceptionally helpful comments; the code looks fine, but I haven't given it the review it deserves.

In D16852#359003, @pkelsey wrote:

To avoid that, the bucket needs to be sized such that the bucket depth divided by the average packet size through such a burst dequeue is less than the number of slots in the hardware ring. Even though we could with some work (less now that iflib is around) know the hardware ring size (and then assume it is stationary), I don't think in general 'average packet size over a burst dequeue' can be considered to be a stationary parameter, so I don't think we can come up with a formula that dials this in much better.

This seems straightforward, reasonable, and sufficient. Longer-term, I'd wonder about some sort of arithmetic approach here rather than this kind of hand-scaling?

Fixes from first round appear complete.

Did we come by that as a NetBSDism? Looks good to me.

Please destroy it, and while there double-check usedness of various other support.S functions if you haven't already. The amount of cruft has steadily fallen throughout the port, and the odds of unused things is probably higher than when I last did a pass. Thank you so much!

I take full blame, and acknowledge spending too much time in code with -G0, and thus being somewhat lazy about gp's treatment.

Strong approve; this is a great improvement. Thank you!

I'd concur, and I had already pared down from NetBSD some in what's in that file when fixing various ABIs, IIRC, and I think it's reasonable to remove other things that are wrong or misleading.

In D13875#292023, @jhb wrote:

I'm not quite sure what you would like to do for makecontext. It is already using the right value (16 for newabi), but I don't know if it's really safe to expose STACK_ALIGN to userland (perhaps I can include <machine/md_var.h> to get it in this file?)

Also, lib/libc/mips/gen/makecontext.c uses its own constants, and should likely be pulled out of the conditional, after subtracting enough registers' worth o' room.

Also fix alignment in cpu_set_upcall in vm_machdep.c?

This seems acceptable and correct within what's currently present. I feel like a unified macro for these sets of flags is really needed, to simplify the code and ensure consistency in checks, but as an incremental improvement this looks good. Thanks!

Awesome, thank you for implementing it in the right place! And we're confident this doesn't produce any referential loops, right? Where builtin_ctz would generate a call to ffssi2?

This looks reasonable, but I share the confusion about correctness. It seems like this should be in libgcc generally. I know we spoke about this, but I've forgotten; can you remind me why libgcc isn't providing this symbol?

In D5017#196604, @julian wrote:

I think that "collision" is ok.

Of course it IS a new behaviour, but that is what we are trying to get. new behaviour.

Added @pkelsey as a reviewer because I didn't realize it would *assign* to him, and rather wanted to invite his comments. Given the work he's done on promiscuous sockets, with a fairly widely-deployed codebase, I think his insights would be meaningful on these changes. The principle seems about right to me here, but I'm not sure about some of the specifics.

This looks good to me, though it'd be nice to see the default ISA thing resolved or confirmed. I also do favour collapsing the target emulation stuff more as possible, but it's not worth holding this up.

I've done this for years in a proprietary product based on FreeBSD and can't imagine going back. The ill effects are few and mostly easy to diagnose, being only about as strange as the things you encounter when you're pretty sure /usr/local is in the search path, but it turns out that it isn't for obscure reasons. Or when badly-behaved configure scripts, etc., won't let you add it. While I sort of morn the loss of more strange BSD conservatism, this is such an obvious reality that I cannot possibly bring myself to tolerate the idea that we should keep going. Users should not have to care about how things were in the mid '90s. An increasing number of them weren't even born yet. System compilers search the default location of local additions and packages.