math/symfpu: fix build on non-x86
Only supply x86-specific CXXFLAGS on i386 and amd64.
Approved by: portmgr (build fix blanket)