I think the correct way to do this is to modify vmxnet3_attach_post() such that when RSS is defined, if rss_gethashalgo() is not TOEPLITZ, the VMXNET3_FLAGS_RSS flag will not get set in sc->vmx_flags. When that flag is not set, vmxnet3_reinit_rss_shared_data() will never be called, and thus can be written assuming that rss->hash_func should always be set to TOEPLITZ.

The way it is written now, we have to wonder whether the vmxnet3 virtual device implementation will tolerate RSS enabled with hash_func set to NONE. We have no documentation for this virtual device implementation and no reference code that tries this mode. We do however know (via the successful operation of both the FreeBSD and Linux drivers) that the virtual device works when RSS is disabled in the virtual device requested features, and also when it is enabled in the requested features with the hash_func sent to TOEPLITZ.

Another idea that I've just got is to regress to using M_HASHTYPE_OPAQUE_HASH for all received packets when rss_gethashalgo() is not TOEPLITZ while still configuring the "hardware" to use TOEPLITZ. With that, we still get the benefit of distributing incoming packets between queues / CPUs, but the software RSS will not expect that hardware hashes are the same as software hashes.

My understanding is that with that approach, the benefit would be that the work of the hashes in software would then be distributed across multiple cores (it would be done on the core handling the given receive queue prior to dispatching the packet to the core indicated by the computed hash). If that's the case, seems reasonable to me.

I think you mean for this to be #endif, and for the #endif below to go away.

Oops!

This could be placed under #ifdef RSS to leave the code out as VMXNET_FLAG_SOFT_RSS can't be set when RSS is not defined. Probably does not make much of a practical difference given this driver is not run on very constrained platforms. I'll leave that up to you as to whether to do it.