@ngie , it seems we could land it, and copyright topic could be revised for entire project with separate commit(s). What do you think?

Throw std::system_error instead of std::runtime_error upon kldload failure

Explicitly describe execenv involvement in "kyua help debug"

Make it skip writing stdout/stderr to tmp files, as "debug -p" does

Communicate kldload issues via exceptions

In D52642#1205688, @markj wrote:

This works almost exactly how I want, thank you. I have one "complaint": when I use this mode, kyua 1) drops me in the shell, 2) waits for the shell to exit, 3) prints the debug output, e.g., executed commands and the test failure string. The -p mode is basically the same. It is more useful to do 3) first, then 1) and 2). Is it hard to make this change?

Resolved review comments

Hi @ngie. It seems your time budget is very limited. Thank you for all the comments you provided for this patch, it helped polishing the things and making it better. I'm going to land it soon so that we could start actual testing of the idea. Fortunately, it's an opt-in helper tool for a developer and nothing depends on it. Anyway, before it can be useful we need to continue labelling the tests with required_kmods declarations.

LGTM. The same testing passed on my side, with the same build and test env. The only obstacle I had is this:

In file included from netlink_netlink_snl_route_parsers.c:1:
In file included from /home/igoro/src/sys/netlink/netlink_snl_route_parsers.h:30:
/home/igoro/src/sys/netlink/netlink_snl.h:122:12: error: comparison of integers of different signs: 'int' and 'uint32_t' (aka 'unsigned int') [-Werror,-Wsign-compare]
  122 |         if (delta > lb->size - lb->offset)
      |             ~~~~~ ^ ~~~~~~~~~~~~~~~~~~~~~
1 error generated.
*** [netlink_netlink_snl_route_parsers.o] Error code 1

Implement x/y flaky spec

In D54148#1237012, @jhb wrote:

Igor, are you able to test this again on your armv7 setup btw?

LGTM. That's an important point regarding the other data allocated from the snl_state.

It seems to be ready for landing to slowly start actual testing and adoption.

In D53697#1226388, @jhb wrote:

@igoro would you be able to test this on your workload (armv7) to ensure it still does the correct thing?

Now I remember that the latest fix's key goal was to add missing nw->size maintenance not to realloc again-and-again when it's actually not needed, i.e. after each realloc we still think that nw->size is only 256. And the motivation was to avoid overlapping buffers appeared to be there as a consequence of missing nw->size maintenance.

In D52910#1211188, @ivy wrote:

i'm not opposed to this, but i don't understand why it's required. is this an upstream bug, or are we building libkrb5 wrongly?

Thanks for your time and review.

Tiny improvements

This seems to be revealed with the recent switch to MIT version. For example, 15-CURRENT back in March had this:

# ldd /usr/lib/libkrb5.so
/usr/lib/libkrb5.so:
        libasn1.so.11 => /usr/lib/libasn1.so.11 (0x36e32d570000)
        libcom_err.so.5 => /usr/lib/libcom_err.so.5 (0x36e309956000)
        libcrypt.so.5 => /lib/libcrypt.so.5 (0x36e30a44a000)
        libcrypto.so.30 => /lib/libcrypto.so.30 (0x36e30a600000)
        libhx509.so.11 => /usr/lib/libhx509.so.11 (0x36e339780000)
        libroken.so.11 => /usr/lib/libroken.so.11 (0x36e30ac5a000)
        libwind.so.11 => /usr/lib/libwind.so.11 (0x36e33bec0000)
        libheimbase.so.11 => /usr/lib/libheimbase.so.11 (0x36e30834e000)
        libprivateheimipcc.so.11 => /usr/lib/libprivateheimipcc.so.11 (0x36e30ba76000)
        libc.so.7 => /lib/libc.so.7 (0x36e308e00000)
        libmd.so.7 => /lib/libmd.so.7 (0x36e30c885000)
        libthr.so.3 => /lib/libthr.so.3 (0x36e30d741000)
        libsys.so.7 => /lib/libsys.so.7 (0x36e318150000)

The prototype has the following logic currently:

Always attempt to use $SHELL

In D51136#1196294, @ngie wrote:

In D51136#1170880, @igoro wrote:

In D51136#1170613, @ngie wrote:

What was the behavior prior to this change?

It's turned out that kyua test catches issues with temporary dir/files cleanup but does not propagate it -- that's obvious from the code where test_result is not used actually. As a result, a test is marked as passed leaving garbage behind (tmp dirs/files), while by design and code comments it is expected to be broken instead.

Will this cause false positives with cleanup failures?

Rebase onto the latest kyua changes and rework from rr to prepare

A minuscule improvement to get the following upon atf_check_equal failure:

Error Message:
1 != $(jexec alcatraz sysctl -n net.dummymbuf.hits) (1 != 2)

instead of this:

Error Message:
1 != 2 (1 != 2)

LGTM

Thank you very much for identifying the root cause. I've been monitoring the false positives on the CI since Oct-2024 -- 25 of them so far. And my TODO list is too deep.

In D51136#1170613, @ngie wrote:

What was the behavior prior to this change?

LGTM. And I had a successful testing on my side.

LGTM

In D51035#1165778, @ngie wrote:

Please submit this upstream. We need to work through the porting issues with Linux/MacOS in order to land this to avoid having multiple different forks/mutations of this (ATF) code.

LGTM

The combinatorics seems to be as follows:

Improve the man page wording

It looks fine that atf-check wants to verify that its temporary dir has enough access rights as it needs to store output files beneath. Why does it add so many code to handle it explicitly? Probably, a usual noperm file system error does not propagate nicely and leaves a test result reader guessing about the actual root cause. Why doesn't it "escape" the current umask instead (as this patch proposes)? Maybe, there was another reason to obey the current umask when ATF was just a separate project before Kyua introduction and it was invoked directly. I do not see answers in the code parts I've read.

In D49463#1143637, @ngie wrote:

This feature can be refined, but I don’t want to get in the way of the utility of this change for folks like @kp, who need the debuggability. Let’s go forward with this change and refine things/sand down rough edges on GitHub.

Fix the man page format

I remember ngie@ agreed in some past cases to go directly to the src while the expected vendor path could go in parallel at its own pace. The vendor branch seems to be non-updated since its import back in 2020, so the vendor path may face obstacles and could take time. I mention this just in case if there is a need to hit src sooner, so that an exception agreement could be considered.

In D49798#1135647, @ziaee wrote:

for format/typo changes, I don't bump the date.

Sorry for the oversight, this is correct. We should not bump the date here.

Many thanks for review! It was a rush to meet the deadline, and there weren't enough rounds of self-check.

Fix format and style

It seems that as a temporary solution it is good to have them explicitly "tagged" to easily find the ones which need revising to avoid external dependencies.

In D49594#1131294, @lwhsu wrote:

I am happy to see allow_network_access feature has been added to the test framework. I believe it would be useful in some cases that network access is mandatory. However, in the case of DNS resolving, I suggest we just using the built-in local unbound for a simple local resolver to serve the test needs.

In D49594#1131294, @lwhsu wrote:

Sorry for being late to this (well, the window is quite short and I have been busy with other things recently...)

In D49594#1130770, @igoro wrote:

While ideally tests should not depend on external things as much as possible to have more reproducible nature, some of them may require to reach external resources, let's say to fetch something or to talk to the world services like DNS, etc.

Indeed and that's also why I am still hesitated to enable internet access on the jails in builder (when doing builds with the patches come from external) and when running the tests in VM. It's not only the security reason but also the matter of tests stability and reliability.

In D49594#1130777, @ziaee wrote:

We can do it later, but if you mark these variables up as It Va instead of It, they become searchable; e.g. apropos Va=allow_network.

While ideally tests should not depend on external things as much as possible to have more reproducible nature, some of them may require to reach external resources, let's say to fetch something or to talk to the world services like DNS, etc.