I wonder if using make(1) is really a good choice for this. Doing things like computing the RAM size of the VM in make feels awkward. There's also no good way to see what parameters are available to developers, and the framework is not friendly to user error (e.g., misspelling a variable name on the command line means that make will silently do the wrong thing). There is no namespacing for parameters (parameters for the build, VM boot, and the test itself are all mixed together).

For the purpose of running canned smoke tests, I think what you have is sufficient, and it's simple and easy to understand. But if this make(1) target is going to expand and become more flexible to suit various developer workloads, not just running some static smoke test in a CI pipeline, I'm worried that this code will become very hard to use and maintain. It would be helpful to understand what future directions you are planning for this work. How do you see this fitting into developer workflows? For instance, I think this needs to run as root, but this is not actually required (ignore bhyve for now) - do you plan to remove that constraint?

In D43786#1000186, @markj wrote:

I wonder if using make(1) is really a good choice for this. Doing things like computing the RAM size of the VM in make feels awkward. There's also no good way to see what parameters are available to developers, and the framework is not friendly to user error (e.g., misspelling a variable name on the command line means that make will silently do the wrong thing). There is no namespacing for parameters (parameters for the build, VM boot, and the test itself are all mixed together).

make is definitely not the best tool but this is what was discussed by many when we initially planned this work.

For the purpose of running canned smoke tests, I think what you have is sufficient, and it's simple and easy to understand. But if this make(1) target is going to expand and become more flexible to suit various developer workloads, not just running some static smoke test in a CI pipeline, I'm worried that this code will become very hard to use and maintain. It would be helpful to understand what future directions you are planning for this work. How do you see this fitting into developer workflows? For instance, I think this needs to run as root, but this is not actually required (ignore bhyve for now) - do you plan to remove that constraint?

Truly speaking I haven't given the thought about the root user case but in the future when src or kernel developers need testing they can run the full tests either in their local hardware or through CI Pipelines in various repositories like Github or Cirrus-ci or also leveraging direct Cloud resources. The main purpose for working on this was to facilitate the developers with the same suite of tests we run in our CI platform but at pre-commit stage. As in the current workflow the tests or CI are done after committing or merging the codes and there is no easy way for a developer to go through the extensive testings we go through at our CI. At this stage a smoke test only really feels vague but after lots of delay I think we should push this and more stuffs gradually. When we will have the entire test suite added here which we run in our CI pipeline it will be more beneficial for the developers and this will make more sense.

I will definitely need to write some sections for the Committers Guide at some stage but for most of the developers I think this is only like `cd /usr/src/tests/ci && make CITYPE=<smoke|default|full> TARGET=<TARGET> TARGET_ARCH=<TARGET_ARCH> ci'

I am still not sure whether if META builds will be helpful for developers or not. If there is a demand I will also add support for META.

In D43786#1000229, @bofh wrote:

In D43786#1000186, @markj wrote:

I wonder if using make(1) is really a good choice for this. Doing things like computing the RAM size of the VM in make feels awkward. There's also no good way to see what parameters are available to developers, and the framework is not friendly to user error (e.g., misspelling a variable name on the command line means that make will silently do the wrong thing). There is no namespacing for parameters (parameters for the build, VM boot, and the test itself are all mixed together).

make is definitely not the best tool but this is what was discussed by many when we initially planned this work.

I'm still reading other parts, but I think I can response this one first.

I think the goal is having several make targets for CI operations, and when things getting well, adding a target to src root, such as 'make ci' for let people can run everything with a single command. This doesn't mean that we have to write everything in make. Similar to what is done in ports/Mk and src/releases, we can introduce several helper scripts written in other languages. These script interpreters may not even need to be part of the base system (in such cases, the Makefile would check for dependencies and prompt for their installation if necessary). For now I think the current Makefile isn't too overly complex so I can accept it. However, I will definitely support extracting the complex logic out of Makefile when we are adding more feature.

In D43786#1002265, @lwhsu wrote:

In D43786#1000229, @bofh wrote:

In D43786#1000186, @markj wrote:

I wonder if using make(1) is really a good choice for this. Doing things like computing the RAM size of the VM in make feels awkward. There's also no good way to see what parameters are available to developers, and the framework is not friendly to user error (e.g., misspelling a variable name on the command line means that make will silently do the wrong thing). There is no namespacing for parameters (parameters for the build, VM boot, and the test itself are all mixed together).

make is definitely not the best tool but this is what was discussed by many when we initially planned this work.

I'm still reading other parts, but I think I can response this one first.

I think the goal is having several make targets for CI operations, and when things getting well, adding a target to src root, such as 'make ci' for let people can run everything with a single command. This doesn't mean that we have to write everything in make. Similar to what is done in ports/Mk and src/releases, we can introduce several helper scripts written in other languages. These script interpreters may not even need to be part of the base system (in such cases, the Makefile would check for dependencies and prompt for their installation if necessary). For now I think the current Makefile isn't too overly complex so I can accept it. However, I will definitely support extracting the complex logic out of Makefile when we are adding more feature.

So far the way I have divided the Makefiles there won't be any too much complexity in the Makefile itself. The entire complexity will add up in the freebsdci rc script. However if you want me to mimic everything from the jenkins ci scripts that will be real trouble. My point of view is we should stay as close as possible on how the re@ generates the images so that the problems are reproducible by multiple teams. The only complex thing that I will need to add in the Makefile is a method of archiving the images or separating the environments. But I won't bother too much with that now. Will think about the bridge when we need to cross the river. :D