Test queue and test selector reference

The queue

• One queue per account: <nWave home>/.nwave/test-queue/, where the nWave home is NWAVE_AGENTS_HOME when set, else the account's home directory. Runs of every project and every worktree of the account wait in it. Two accounts on one machine have two queues and do not coordinate; neither do two nWave homes.
• Order: the classes the project declares, the first class first; inside a class, arrival order. A run of a later class that has waited aging_seconds goes before every run that has not, and nothing behind it starts first.
• Places: the lowest of the declared ceiling max_concurrent, the usable cores divided by cores_per_run, and the available memory divided by memory_per_run_mb; never fewer than one. When cores or memory cannot be read on the platform, one run at a time. A running run of a project with a lower ceiling keeps the lower ceiling for as long as it runs.
• The time limit starts when the command starts. The wait in the queue is measured apart and never counted in it. At the limit the command and every process it started are stopped.
• A run whose process dies leaves the queue: the next run that looks removes its ticket.
• A run whose process is alive keeps its place however long it runs: the queue never stops a run. The wait for a place is bounded instead, by the declared max_wait_seconds. A run that reaches it leaves the queue, runs nothing and ends indeterminate with exit 75 (see below).
• A run started inside a running run (the environment carries NWAVE_TEST_QUEUE_TICKET) starts at once: it already holds a place.

A waiting run prints one line on standard error:

[nwave tests] waiting: ticket 12, class long, 1 running, 2 waiting; gives up after 1800 s

Each ticket of the queue directory (tickets/<n>.json) records the class of the run, the process that holds it (pid), its command, when it arrived (enqueued_at) and when it started (started_at).

A run that waits too long

When the wait reaches max_wait_seconds the run leaves the queue and prints, on standard error, an INDETERMINATE: line and then WHAT:, WHY: and HOW::

INDETERMINATE: this run is neither a pass nor a failure
WHAT: the run waited 1800.0 s for a place, reached the limit queue.max_wait_seconds of 1800 s and did not start; ahead of it: ticket 12, pid 4242, running for 2710 s: <command>
WHY: no test ran, so this is neither a pass nor a failure; a run that never ends keeps its place, and the queue does not stop another run
HOW: if a run ahead is stuck, stop its process (pid 4242): its place is free as soon as that process is gone; if it is only long, declare a larger queue.max_wait_seconds; then run this command again

WHAT names each run ahead: its ticket, its process, whether it is running or waiting and for how long, and its command on one line. The runs ahead are the running runs; when nothing runs, the run at the head of the line. The run ahead is not touched. The figures above are an example.

A run whose process is killed

A run does not start its command itself. It starts a supervisor process, in a session of its own and held by a pipe, and the supervisor starts the command in a process group of its own. When the pipe closes, the supervisor stops that group. The pipe closes when the command's time limit is reached, when the run is asked to end (interrupt, termination), and when the run's process disappears, whatever killed it and at whatever moment: before, while or after the command starts.

When the supervisor itself disappears, the run stops the command's group and ends indeterminate with exit 75: the exit of the command is unknown, so it is neither a pass nor a failure.

Limits of this:

• it needs process groups (POSIX). Elsewhere the run starts the command itself and prints NOTE: this platform has no process groups: if the process of this run is killed, its command keeps running; it prints the same note, with the reason, when the supervisor cannot be started;
• a command that moves itself to another process group or session is not stopped;
• when the run and its supervisor are both killed, nothing stops the command;
• the place of the killed run is free as soon as its process is gone, which can be an instant before its command has ended.

The receipt

Every run ends with one line on standard error, [nwave tests] receipt: {...}, and the same JSON document in the --receipt file when one is named.

Exit codes

Refusals

A refusal prints three lines, WHAT:, WHY: and HOW:, and runs nothing. The run is refused when the declaration is malformed (every defect is named at once), when the class or the named command is not declared, when the time limit is not a positive number, when the queue directory cannot be made or locked, and when the tool under test resolves outside the tree:

• the project declares tool_resolution.argv, a command that prints the path or paths the tool under test resolves to;
• a path outside the project tree refuses the run (the known case: an editable installation that points at another checkout), and HOW carries the declared repair;
• a check that cannot run, exits non-zero or prints nothing also refuses: a check that cannot decide does not count as a pass.

The dependents command

Which files depend on which is answered by a command the project declares, in selection.dependents: {"argv": [...], "covers": [...]}. argv runs from the project root, without a shell. covers lists the path patterns the command answers for: a pattern with a slash matches the whole path, one without matches the file name.

The selector asks the command in rounds, and only when a path needs dependents: each round names the files it must answer for. The command gets one JSON object on standard input, {"files": [...], "targets": [...]}: the tree-relative paths of the project and the targets. It prints one line per dependency on a target, TARGET<TAB>DEPENDENT, where DEPENDENT depends directly on TARGET. The selector follows those lines through any number of files, up to the depth of the use.

Exit status 0 is an answer. Any other status, a command that cannot start, or a line that is not of that form is could-not-verify: the affected paths are INDETERMINATE, with what, why and how on standard error, and the widest group is listed. A changed file that no covers pattern, assets prefix or test root decides is undecided: the selector does not know its language and does not guess.

This repository declares its own Python command, scripts/hooks/python_dependents.py (imports, modules started with -m, paths named in text, the implicit files). The selector names no language; another repository declares its own command, in any language, and needs no change to the tool.

Each call of the command is bounded by selection.dependents.timeout_seconds: default 120 seconds, ceiling 600. A call still running at its bound is killed, with the whole process group it started, and gives no answer: every path that needs it is INDETERMINATE with a TIMEOUT refusal, and the widest group is listed. A timeout is reported as a timeout, not as a failure of the command's answer.

The command answers for every listed file, and refuses (non-zero exit, so the paths that need it are INDETERMINATE) when a listed file cannot be read, or when a target names no path of the tree. A deleted target is not refused: its dependents are found by its name. A non-Python file is a dependent when it names the target by its path, or by the target's file name inside the target's own directory.

What the command does not see, accepted limits: a name built at run time (a dynamic import, a path assembled in a string), and symbolic links (none exist in this tree). A file that names a target only in a way the command does not match is not a dependent, and nothing says so: the selection is then short. nWave's declaration covers *.py only; every other changed file is mapped (assets) or declared no_tests, by search of the test sources, and any other file widens loudly.