In D26035#577636, @araujo wrote:

The configuration file is not intuitive at all, how I will know what pci slot my virtio-blk shall live?
Why not simplify it instead of complicate more? The configuration file supposed to be something to simplify the command line, and instead is getting more complex than bhyve cli itself. Perhaps we will need another io<something> wrapper to simplify the creation of bhyve configuration files.

Could we use a similar approach of the configuration file syntax like everybody else is using? I know make things simple is hard.

The config file here is for bhyve internal, it is accurate but complex.
Maybe other hypervisor manager, such as vm-bhyve will read their simple config file and output this config style for bhyve, then launch the bhyve process.

In D26035#577644, @wanpengqian_gmail.com wrote:

In D26035#577636, @araujo wrote:

The configuration file is not intuitive at all, how I will know what pci slot my virtio-blk shall live?
Why not simplify it instead of complicate more? The configuration file supposed to be something to simplify the command line, and instead is getting more complex than bhyve cli itself. Perhaps we will need another io<something> wrapper to simplify the creation of bhyve configuration files.

Could we use a similar approach of the configuration file syntax like everybody else is using? I know make things simple is hard.

The config file here is for bhyve internal, it is accurate but complex.
Maybe other hypervisor manager, such as vm-bhyve will read their simple config file and output this config style for bhyve, then launch the bhyve process.

In D26035#577644, @wanpengqian_gmail.com wrote:

In D26035#577636, @araujo wrote:

The configuration file is not intuitive at all, how I will know what pci slot my virtio-blk shall live?
Why not simplify it instead of complicate more? The configuration file supposed to be something to simplify the command line, and instead is getting more complex than bhyve cli itself. Perhaps we will need another io<something> wrapper to simplify the creation of bhyve configuration files.

Could we use a similar approach of the configuration file syntax like everybody else is using? I know make things simple is hard.

The config file here is for bhyve internal, it is accurate but complex.
Maybe other hypervisor manager, such as vm-bhyve will read their simple config file and output this config style for bhyve, then launch the bhyve process.

Maybe you are not familiar enough with the bhyve configuration fie attempt implementations!!!
I will drop these slides here: https://docs.google.com/presentation/d/15s9rWCe8ILJB-wTP_cZFFWZrAPI0IzG3jKvi_2zHgss/edit?usp=sharing

In D26035#577645, @araujo wrote:

Maybe you are not familiar enough with the bhyve configuration fie attempt implementations!!!
I will drop these slides here: https://docs.google.com/presentation/d/15s9rWCe8ILJB-wTP_cZFFWZrAPI0IzG3jKvi_2zHgss/edit?usp=sharing

I apologize I didn't know there is a history of bhyve configuration.
Now I have read the slides you wrote before. and I think your proposal and this one has no conflict at all.

I have done some modifications to pci_ahci.c and pci_nvme.c before, and that is a pain, every file has its own parser.
And for virtual machine hypervisors, we have to provide more closed syntax than real computer topology.
while this MIB-like syntax is the best for bhyve(hypervisor) ever in my opinion. when it stores value to nvlists, the consumer(each soft controller) will be happy.
while it can do some modifications without affecting others.

Making syntax simple, that requires bhyve do more computing on runtime. while changing the pci slot/func etc. will affect the running virtual machine.(For example, changing a nic slot number, will make Windows guests recognize it as new nic, and lose the static IP setting.) so It is better to fine control this by the user.

As your UCL like configuration RFC, it can provide more friendly to users while this one provides more friendly to bhyve internal functions(programmer).
We will be happy if both can cooperate together in the future.

Agreed it's a shame this isn't UCL, it could be much closer to jail.conf (which should be made UCL someday, but it's very nearly compatible on a surface level). It doesn't look like any features unique to libnv are used.

The config file overriding options passed on the command-line seems backwards. I'd expect to be able to override config file settings explicitly on the command line.

To all those harping about it not being foo or bar or zed, this *is* an underlying frame work that is very flexible and allows one to implement what ever foo, bar or zed configuration you want on top of it. This leaves json, xml, or any other implementation out of the base layer code and those can be implemented in a layer above this. This has been discussed for a some time extensively by those that attend the semi-monthly bhyve call.

In D26035#577636, @araujo wrote:

The configuration file is not intuitive at all, how I will know what pci slot my virtio-blk shall live?
Why not simplify it instead of complicate more? The configuration file supposed to be something to simplify the command line, and instead is getting more complex than bhyve cli itself. Perhaps we will need another io<something> wrapper to simplify the creation of bhyve configuration files.

This is much more complex than bhyve cli itself.
pci.0.0.0.device=hostbridge
pci.0.2.0.device=virtio-net
pci.0.2.0.backend=%(tap)
pci.0.3.0.device=virtio-blk
pci.0.3.0.path=/dev/zvol/zroot/bhyve/%(name)
pci.0.4.0.device=virtio-rnd
pci.0.31.0.device=lpc
lpc.com1.device=stdio
lpc.com2.device=/dev/nmdm%(name)2B
lpc.bootrom=/usr/local/share/uefi-firmware/BHYVE_UEFI.fd

Could we use a similar approach of the configuration file syntax like everybody else is using? I know make things simple is hard.

To be clear, one can build a UCL parser (I think I mentioned this on the list) that uses rules to write a backing config. I explicitly mentioned there that the UCL syntax could auto-compute the PCI slots to use while having an optional keyword to force a specific PCI slot if desired. Reading the config file is just a means of translating the config the UCL file likes into a set of configuration variable settings. That is, you could very much do something like:

disk {
      path = "/foo";
      type = "virtio-blk"
}

And the UCL parser would use an internal routine in bhyve to say "give me the next free PCI slot" and use that to pick the right "pci.<b>.<s>.<f>" node to hang its explicit variables off of. Some users of bhyve have expressed a desire for a flat config store (and the ability to set arbitrary options via '-o' that override what it is in a base config file), so the backing store for the config needs that level of detail, but you can populate the config in various ways.

In D26035#577801, @freqlabs wrote:

Agreed it's a shame this isn't UCL, it could be much closer to jail.conf (which should be made UCL someday, but it's very nearly compatible on a surface level). It doesn't look like any features unique to libnv are used.

The config file overriding options passed on the command-line seems backwards. I'd expect to be able to override config file settings explicitly on the command line.

Please read the mail on virtualization@. It is very feasible to add a UCL config file syntax (and that is even a desired plan, just not currently implemented). Using libnv is more convenient than writing trees by hand in C.

I don't follow your last comment. The example I gave uses command line options to augment what is in the config file as it had "template" values. However, in general the rule is that each command line argument is processed in sequence and each command line argument results in zero or more modifications to the configuration. An option occurring before a '-f <file>' is processed first and can be overridden by the config file, and an option after the '-f' is processed later and will override settings in the config file. You can also have multiple '-f' files if you wanted to (or multiple UCL files in the future).

In case the template part isn't clear, the same config file can be used for different VMs, e.g.:

bhyve -f uefi.cfg -o tap=tap0 head
bhyve -f uefi.cfg -o tap=tap1 fbsd12
bhyve -f uefi.cfg -o tap=tap2 fbsd11

Where each of these VMs uses /dev/zvol/zroot/bhyve/{head,fbsd12,fbsd11} as its disk and /dev/nmdm{head,fbsd12,fbsd11}2B for com2 (which I use for kgdb over serial) without having to have separate config files. Instead, the config accepts parameters. Note that the template thing would also work with a UCL config file, e.g.

disk {
      path = "/dev/zvol/zroot/bhyve/%(name)";
}

In D26035#578018, @jhb wrote:

In D26035#577801, @freqlabs wrote:

Agreed it's a shame this isn't UCL, it could be much closer to jail.conf (which should be made UCL someday, but it's very nearly compatible on a surface level). It doesn't look like any features unique to libnv are used.

The config file overriding options passed on the command-line seems backwards. I'd expect to be able to override config file settings explicitly on the command line.

Please read the mail on virtualization@. It is very feasible to add a UCL config file syntax (and that is even a desired plan, just not currently implemented). Using libnv is more convenient than writing trees by hand in C.

Certainly, I guess I'm not subscribed to that list yet, I'll fix that :)
libucl is unsavory to work with, I know. libnv also has its shortcomings. I have somewhere stashed code to convert between the two, but the features don't map very well so it wasn't a particularly useful exercise; educational, though. But like I said, it doesn't look like anything specific to libnv is being used. If the long-term goal is to eventually support a more canonical config file format in addition to this format, I'm all for it. My main concern was that introducing a strange new config format might preclude us from that. It doesn't sound like that will be the case. May be as simple as interpreting *.cfg one way and *.conf another way.

I don't follow your last comment. The example I gave uses command line options to augment what is in the config file as it had "template" values. However, in general the rule is that each command line argument is processed in sequence and each command line argument results in zero or more modifications to the configuration. An option occurring before a '-f <file>' is processed first and can be overridden by the config file, and an option after the '-f' is processed later and will override settings in the config file. You can also have multiple '-f' files if you wanted to (or multiple UCL files in the future).

The description in the summary confused me, then. I got the impression that configuration parsing happened after command line options, but you've cleared up that misunderstanding for me, thanks.

In D26035#578022, @freqlabs wrote:

In D26035#578018, @jhb wrote:

In D26035#577801, @freqlabs wrote:

Agreed it's a shame this isn't UCL, it could be much closer to jail.conf (which should be made UCL someday, but it's very nearly compatible on a surface level). It doesn't look like any features unique to libnv are used.

The config file overriding options passed on the command-line seems backwards. I'd expect to be able to override config file settings explicitly on the command line.

Please read the mail on virtualization@. It is very feasible to add a UCL config file syntax (and that is even a desired plan, just not currently implemented). Using libnv is more convenient than writing trees by hand in C.

Certainly, I guess I'm not subscribed to that list yet, I'll fix that :)
libucl is unsavory to work with, I know. libnv also has its shortcomings. I have somewhere stashed code to convert between the two, but the features don't map very well so it wasn't a particularly useful exercise; educational, though. But like I said, it doesn't look like anything specific to libnv is being used. If the long-term goal is to eventually support a more canonical config file format in addition to this format, I'm all for it. My main concern was that introducing a strange new config format might preclude us from that. It doesn't sound like that will be the case. May be as simple as interpreting *.cfg one way and *.conf another way.

I was probably thinking more like -F vs -f or the like. Perhaps different letters if necessary to avoid being too confusing. I also wouldn't mind UCL being the "preferred" format for end-users and the flat style being intended more for orchestration frameworks that want to generate configs.

Given the config file syntax will likely change but the "-f" option will be around forever, can a magic number or first-line metadata comment be used to distinguish this key/value file format from a future version ?

In D26035#578006, @jhb wrote:

In D26035#577636, @araujo wrote:

The configuration file is not intuitive at all, how I will know what pci slot my virtio-blk shall live?
Why not simplify it instead of complicate more? The configuration file supposed to be something to simplify the command line, and instead is getting more complex than bhyve cli itself. Perhaps we will need another io<something> wrapper to simplify the creation of bhyve configuration files.

This is much more complex than bhyve cli itself.
pci.0.0.0.device=hostbridge
pci.0.2.0.device=virtio-net
pci.0.2.0.backend=%(tap)
pci.0.3.0.device=virtio-blk
pci.0.3.0.path=/dev/zvol/zroot/bhyve/%(name)
pci.0.4.0.device=virtio-rnd
pci.0.31.0.device=lpc
lpc.com1.device=stdio
lpc.com2.device=/dev/nmdm%(name)2B
lpc.bootrom=/usr/local/share/uefi-firmware/BHYVE_UEFI.fd

Could we use a similar approach of the configuration file syntax like everybody else is using? I know make things simple is hard.

To be clear, one can build a UCL parser (I think I mentioned this on the list) that uses rules to write a backing config. I explicitly mentioned there that the UCL syntax could auto-compute the PCI slots to use while having an optional keyword to force a specific PCI slot if desired. Reading the config file is just a means of translating the config the UCL file likes into a set of configuration variable settings. That is, you could very much do something like:

Definitely that is what will happen, somebody else will write a wrapper for this configuration file!!! The problem I see here is the standard(syntax and etc), it does not look like any configuration file I have ever see, and you know after you commit it, it will live there for forever, because we need to respect POLA. See bhyvectl, supossed to be a cli but it is so complex that we call it as a developer tool, because nobody knows how to use half of its features.

Have you take a look on the OpenBSD VM configuration file?

We should try to unify and simplify the hundreds of different configuration files syntax we have in the system, but no, we are adding another band-aid to the Frankenstein. I have no rights to blame the work you have done, however I'm the one that spent a good chunk of time going around the bhyve configuration file and failed to get a standard accepted. I feel frustrated to see the configuration file going in this direction masked by a nvlist implementation. The nvlist implementation is nice, but the configuration file syntax really sucks(Sorry my French).

disk {
      path = "/foo";
      type = "virtio-blk"
}

And the UCL parser would use an internal routine in bhyve to say "give me the next free PCI slot" and use that to pick the right "pci.<b>.<s>.<f>" node to hang its explicit variables off of. Some users of bhyve have expressed a desire for a flat config store (and the ability to set arbitrary options via '-o' that override what it is in a base config file), so the backing store for the config needs that level of detail, but you can populate the config in various ways.

In D26035#578062, @grehan wrote:

Given the config file syntax will likely change but the "-f" option will be around forever, can a magic number or first-line metadata comment be used to distinguish this key/value file format from a future version ?

I would be fine with using a more obscure option for the flat config file (if we did long opts it could be a --flat-config or some such without a short letter even, but we don't do any long opts so not quite as straightforward to pick an obscure one). I don't really expect the syntax of that to change, but I think having '-f' be used for a more friendly UCL syntax might be more useful and I would be happy to leave "-f" free for the UCL thing to use. The UCL thing might itself include a version number as a field for the UCL parser to use that I think would permit some level of compatibility in the future.

Definitely that is what will happen, somebody else will write a wrapper for this configuration file!!! The problem I see here is the standard(syntax and etc), it does not look like any configuration file I have ever see, and you know after you commit it, it will live there for forever, because we need to respect POLA.

I think you are conflating a few things. First, the syntax to me at least seems rather straight forward and even rather common (e.g. Windows INI files, the OpenSSL config file, even rc.conf) all use 'name=value' lines. It doesn't have [group] lines. I suppose if you really wanted that you could do that for nesting instead of a MIB name, but that seems equally unreadable.

Also, to be clear, the main thrust here is not really having a config file (that is more thrown in as a PoC of how one can combine both files and command line options to build a config), but it is really the nvlist tree and separating config parsing from acting on the config. That removes lots of dubious string parsing code from device models and also makes it much easier (IMO) to add new config variables in the future with minimal code changes. One of the things this model permits is adding config options to USB devices with previously were not permitted by the limited syntax of '-s' options used for XHCI controllers. The "flat" config file is basically the simplest type of config file format one could have (direct key=value without additional implicit knowledge). Right now we already require people using '-s' to know about which PCI slot devices live on, so the "flat" config file isn't worse, just equally non-ideal in that regard.

I think your actual concern is not the syntax, but the fact that the file maps 1:1 onto the set of config values and the stability of how the config tree is managed over time. Devices in particular I think are one of the hardest to reason about. I had originally toyed with trying to do a device namespace that looked a bit more like this:

device.0.type="ahci"
device.0.location="pci0:1:0"
device.0.path="disk.img"

etc. However, while more abstract, it is harder to deal with this sanely in bhyve itself as there may be parent-child dependencies this doesn't capture. Even the current 'pci.<bus>.<slot>.<function>' scheme isn't really ideal if we ever have support for PCI-PCI bridges (though it is doable, you do pci bus 0 first and do a depth-first traversal using the bridge bus numbers to look for devices instead of the current iteration over all possible values).

The other scheme I considered would be to have the device nodes be an actual tree reflecting their layout much like ACPI or FDT. USB devices are already following this somewhat by being hung under the xhci node, whereas lpc devices are not (they are not children of the lpc device). For PCI devices this would be more invasive if PCI-PCI bridges were added since that would then become more of a tree rather than the current flat layout it is now. However, this would also be more complex than the current layout which more naturally fit the existing device model initialization.

In D26035#578286, @jhb wrote:

In D26035#578062, @grehan wrote:

Given the config file syntax will likely change but the "-f" option will be around forever, can a magic number or first-line metadata comment be used to distinguish this key/value file format from a future version ?

I would be fine with using a more obscure option for the flat config file (if we did long opts it could be a --flat-config or some such without a short letter even, but we don't do any long opts so not quite as straightforward to pick an obscure one). I don't really expect the syntax of that to change, but I think having '-f' be used for a more friendly UCL syntax might be more useful and I would be happy to leave "-f" free for the UCL thing to use. The UCL thing might itself include a version number as a field for the UCL parser to use that I think would permit some level of compatibility in the future.

-k ? ("k"ey-value)

I really like the changes related to option handling.

Great work! Will 'config.dump' or another flag do a dry-run dump of the bhyve command to be executed? This would be very helpful. Apologies if I am overlooking it.

In testing this against 13.0-BETA1, stderr redirection does not appear to work:

bhyve -k win.cfg 2>/dev/null

The boot string that the configuration file is based on works fine with stderr redirection.

Suggestion: Unless it is required by the parser or something, remove the period in "config.dump=1" or replace it with a dash to be more like other bhyve parameters and perhaps safer parsing with /bin/sh. Do also consider adding it to the manual page as it is very useful.

Great work!

In D26035#641414, @editor_callfortesting.org wrote:

In testing this against 13.0-BETA1, stderr redirection does not appear to work:

bhyve -k win.cfg 2>/dev/null

This is a bit unusual. Can you show which errors you are getting on stderr that you expect to not be getting?

The boot string that the configuration file is based on works fine with stderr redirection.

Suggestion: Unless it is required by the parser or something, remove the period in "config.dump=1" or replace it with a dash to be more like other bhyve parameters and perhaps safer parsing with /bin/sh. Do also consider adding it to the manual page as it is very useful.

The period is basically required in that the config variables are a MIB like sysctl names where '.' is the delimiter. It is also documented in bhyve_config(5).

In D26035#642819, @jhb wrote:

In D26035#641414, @editor_callfortesting.org wrote:

In testing this against 13.0-BETA1, stderr redirection does not appear to work:

bhyve -k win.cfg 2>/dev/null

This is a bit unusual. Can you show which errors you are getting on stderr that you expect to not be getting?

The boot string that the configuration file is based on works fine with stderr redirection.

This is my bhyve command for Windows Server 2019 which produces a SAC> prompt on stdio without the stderr messages redirected:

/usr/sbin/bhyve -c 2 -m 4G -H -w -P \

-l com1,stdio \
-l bootrom,/root/windows-bhyve-2021/bcran-UEFI-CFT/BHYVE_UEFI.fd \
-s 0:0,hostbridge \
-s 2:0,e1000,tap0 \
-s 3:0,ahci-hd,/root/win.raw \
-s 30:0,fbuf,tcp=10.0.0.14:5900,w=1024,h=768 \
-s 31:0,lpc \
windows 2> /dev/null

Adding the config dump flag gives:

acpi_tables=false
memory.size=4G
x86.vmexit_on_hlt=true
x86.strictmsr=false
x86.vmexit_on_pause=true
cpus=2
#config.dump=1
lpc.com1.path=stdio
lpc.bootrom=/root/windows-bhyve-2021/bcran-UEFI-CFT/BHYVE_UEFI.fd
pci.0.0.0.device=hostbridge
pci.0.2.0.device=e1000
pci.0.2.0.backend=tap0
pci.0.3.0.device=ahci
pci.0.3.0.port.0.type=hd
pci.0.3.0.port.0.path=/root/win.raw
pci.0.30.0.device=fbuf
pci.0.30.0.tcp=10.0.0.14:5900
pci.0.30.0.w=1024
pci.0.30.0.h=768
pci.0.31.0.device=lpc
name=windows

Redirecting this to win.cfg can be launched with:

bhyve -k win.cfg 2> /dev/null

Results in stdio flooded with messages like "rdmsr to register 0x198 on vcpu 1" where previously it was a clear SAC> console.

Perhaps I have the syntax wrong?

Hmm, I tested with bhyveload (rather than the UEFI ROM), and stderr was blocked ok for me. I used a FreeBSD guest and used 'cpucontrol -m 0x198 /dev/cpuctl0' after 'kldload cpuctl' to provoke the error. One thing to check perhaps is getting 'procstat -f' output of the running bhyve process so we can see what fd 2 is set to.