add NUMA support to powerpc
ClosedPublic
Actions

Authored by jhibbits on Nov 9 2018, 9:46 PM.

Details

Reviewers

markj
jeff
kib
alc
jhb
nwhitehorn
mmacy

Commits

rS346174: Add NUMA support to powerpc

Summary

Perfunctory NUMA support:

associate CPU with domain
associate memory ranges with domain
identify domain for devices
limit device interrupt binding to appropriate domain

Additionally fixes a bug in the setting of Maxmem which led to only memory attached to the first socket being enabled for DMA

A pmap variant can opt in to numa support by by calling numa_mem_regions at the end of pmap_bootstrap - registering the
corresponding ranges with the VM.

Diff Detail

Lint

Lint Skipped

Unit

Tests Skipped

Event Timeline

mmacy created this revision.Nov 9 2018, 9:46 PM

Herald added a subscriber: imp. · View Herald TranscriptNov 9 2018, 9:46 PM

Harbormaster completed remote builds in B20731: Diff 50236.Nov 9 2018, 9:46 PM

mmacy edited the summary of this revision. (Show Details)Nov 9 2018, 9:53 PM

markj added inline comments.Nov 9 2018, 9:55 PM

sys/powerpc/powerpc/mp_machdep.c
186	Rather than being conditional on options NUMA, it should be conditional on vm_ndomains > 1. One can disable NUMA support at boot-time with a tunable, and this breaks that.
sys/powerpc/powerpc/platform.c
121	Extra newline.
136	It'd probably be nice to say which.
167	The locality table is optional, you can just pass NULL.
sys/vm/vm_page.c
785 ↗	(On Diff #50236)	This shouldn't be necessary. We shouldn't be registering zero-length segments at all - they impose a runtime overhead on every operation that walks the segment array, and the array size is fixed.

kib added inline comments.Nov 10 2018, 7:13 AM

sys/powerpc/ofw/ofw_machdep.c
261	This looks like a for() loop.
sys/vm/vm_page.c
785 ↗	(On Diff #50236)	They might appear as a consequence of sloppy phys_avail[] manipulations, FWIW. At least x86 startup does a lot of surgery on it for early allocations. Might be the forthcoming KPI would solve this.

markj added inline comments.Nov 10 2018, 5:54 PM

sys/vm/vm_page.c
785 ↗	(On Diff #50236)	I was thinking of the vm_phys_segs[] array rather than phys_avail[] when I wrote that comment. Which KPI are you referring to?

kib added inline comments.Nov 10 2018, 9:51 PM

sys/vm/vm_page.c
785 ↗	(On Diff #50236)	D17431 (not committable)

Will take this to completion.

Fix the CPU domain binding. Only cpu0's domain was being bound, for all CPUs. Apply NUMA to HPT.

I just ran a test building llvm with ninja (3701 jobs):

With NUMA (144 threads): 25:32
Without NUMA (144 threads): 31:30
With NUMA (cpuset to socket 0, 72 threads): 24:35

This proves that the numa patch provides a significant performance boost. There is likely heavy contention in the pmap layer slowing things down below the single-socket case. I plan to measure and see what I can fix in that regard.

This revision was not accepted when it landed; it landed in state Needs Review.Apr 13 2019, 4:03 AM

Closed by commit rS346174: Add NUMA support to powerpc (authored by jhibbits). · Explain Why

This revision was automatically updated to reflect the committed changes.

Cool. +1