Skip to content
Learning Outcomes
  • Read and hand edit tests/expected-results.yaml.
  • Choose the right comparison for each output.
  • Waive an output you cannot usefully compare.
  • Know how a capture records a file name that contains glob characters.
  • Know which of your choices survive a re-capture and which values refresh.

Process Guide

Expected Results

Capturing a run writes a file called tests/expected-results.yaml into the process. You never have to write it by hand, but it is an ordinary file in the process and editing it is how you say what "still works" means for each output.

The shape of the file

The document has three top level keys, version, outputs and waived. Only the first two are required. Any other key is refused when the file is read.

version: 1
outputs:
  - port: counts
    compare: exact
    file: "counts/*.tsv"
    checksum: { algo: sha256, value: 71c4... }

  - port: log
    compare: text
    normalize:
      timestamps: true
      paths: true
      numbers: { relative_tolerance: 1.0e-6 }
    expected: |
      Total reads: 1000000
      Mapped: 987654

  - port: bigwig
    compare: exists
    bytes: { min: 1024 }

waived:
  - port: plots
    reason: the layout is randomised per run, so no comparison is meaningful

Every entry needs a port, which must not repeat and normally names one of the process's declared output ports. An entry for a port the process no longer declares is not refused, but it is still checked on every comparison, as What a re-capture keeps explains. file is optional and is the pattern used to find the port's file, matched first as a literal name and then as a glob. compare chooses how the file is checked.

outputs must not be empty. A document that asserts on nothing used to read as a passing test, so it is now refused outright when the file is read:

expected-results.yaml declares no outputs, so it constrains nothing. Remove the file or give it an output.

The three comparisons

exact requires the file to be byte for byte identical, recorded as a checksum. The only supported algo is sha256. Use it for output that is genuinely deterministic.

text compares content after normalising the noise that changes between runs. normalize accepts timestamps and paths as flags, and numbers with a relative_tolerance for values that drift in the last decimal places. The content is stored inline under expected, which is capped at 256 KB.

exists only checks that the port produced a file, optionally within a size range given as bytes with min and max. Use it for output that is large or legitimately different every time, where the useful assertion is that the step produced something plausible.

Each comparison accepts only its own keys, so a checksum on a text entry or a normalize block on an exists entry is refused rather than ignored. That is deliberate, because a silently ignored key is an assertion you think you have and do not.

Waiving an output

Some outputs cannot be usefully compared at all. A plot with a randomised layout, or a report that embeds the machine it ran on, will differ on every run whichever comparison you choose. Listing such a port under waived records that you decided not to assert on it.

waived:
  - port: plots
    reason: the layout is randomised per run, so no comparison is meaningful

A waiver takes exactly two keys, port and reason, and the reason must not be empty. That is deliberate. A waiver is a recorded decision, and one without a reason cannot be told apart from a port somebody forgot.

A port cannot appear in both outputs and waived. The two say opposite things about whether the output is tested, so a document that does both is refused rather than one of them being honoured silently. The same goes for waiving one port twice.

A capture never captures a waived port, even when the run published a file for it. The port is listed in the capture result with your reason, it is not offered in the Match outputs to files dialog, and its waiver is written back unchanged. To start asserting on a waived port, delete its waiver first and then capture.

Not available in Foundry Connect 2.22.0 or earlier

Waivers are read from the release that follows 2.22.0. Earlier releases refuse a waived key as an unknown field.

Waivers are what makes this worth writing down rather than just deleting the entry. A waived port counts as covered, so a required output that can never be compared stops blocking publication. See Cover every output before publishing.

What a fresh capture chooses for you

When a port has no authored entry, the capture picks a comparison from the file itself. Anything over 256 KB becomes exists with a size floor. Text like extensions, which includes .log, .txt, .tsv, .csv, .json, .yaml, .yml, .stats and .md, become text with timestamp and path normalisation already switched on. Everything else becomes exact with a checksum.

The file value written is the name the run actually published, never a guessed pattern. If you want a pattern that survives a changing filename, author the glob yourself.

Names that contain glob characters

file is always read back as a pattern, so a recorded name that happens to contain glob characters is a pattern by accident. Bracketed sample numbers such as data[1].tsv are the ordinary case, because [1] is a character class. While that exact file exists, the literal match finds it. On a later run where it is missing, the pattern could match a different file and grade the wrong output.

A capture therefore escapes those characters when it writes a name, so the value can only ever match the file it came from:

  - port: table
    compare: text
    file: table/data\[1\].tsv

Each of *, ?, [, ], (, ), {, } and \ gets a backslash in front of it, and so does a ! or # at the very start of the name. A name with none of those is written exactly as published.

Only names the capture records are escaped: a file it matched on its own, and a file you pick in the Match outputs to files dialog. A file you authored is a pattern you chose, so a re-capture keeps it exactly as you wrote it. Documents written before names were escaped keep working, because an exact file name is always tried before the pattern. If you hand write a literal name that contains these characters, escape it the same way.

Not available in Foundry Connect 2.22.0 or earlier

Names are escaped from the release that follows 2.22.0. Earlier releases write the name exactly as it was published and rely on the literal match alone.

What a re-capture keeps

Re-capturing an existing document does not throw your decisions away. It refreshes the measurements and leaves the intent alone.

Kept as you wrote it Refreshed from the new run
the file pattern checksum.value for exact
the compare choice the inline expected content for text
normalize flags and tolerances
the bytes range on exists

There are two deliberate exceptions. An authored text entry whose output has grown past the 256 KB cap is changed to exists with a freshly measured size floor, because the content can no longer be stored inline. And if you use the Match outputs to files dialog to point an entry at a different file, an exists size floor is re-measured against the file you chose.

The first exception is never silent. The capture result carries a note on that port saying it was captured as exists rather than text, and why.

One kept value earns a note too. An authored bytes range that this run's file falls outside is kept as written, because the range is your decision, but the note says so and warns that the next comparison of this run will fail. Widen the range, or point the port at a different file. Read the capture result shows both notes as they appear.

Entries are never quietly dropped. A port that produced nothing this time, or a port the process no longer declares, is carried through unchanged so that fixing the process later restores the assertion rather than requiring you to write it again.

An entry for a port the process no longer declares is not inert, though. Every entry in the document is checked on every comparison, so while the run publishes no matching file, that entry reports absent and fails the whole comparison. The capture result lists each such entry so you can find it, and delete it once the port is gone for good. Coverage ignores it either way.

Waivers ride through a re-capture on the same rule, for a sharper reason. Dropping one would silently reopen a coverage gap you had closed by hand, and the revision would quietly stop being publishable with nothing on the page saying why. Every waiver is written back as it was. That includes a waiver for a port the process no longer declares, which the capture result also lists and which no longer affects coverage.

A re-capture also keeps the order of your document. Each entry is refreshed where it already sits, and only ports new to the document are added, at the end, in the order the process declares them.

Not available in Foundry Connect 2.22.0 or earlier

The notes, the list of kept entries and the preserved order arrive in the release that follows 2.22.0. Earlier releases make the same changes without a note, and write the captured entries ahead of the ones they keep.

A document that does not parse stops the capture

If tests/expected-results.yaml cannot be read, the capture is abandoned and reports the problem instead of overwriting the file. Your hand edits are never destroyed by a capture that could not understand them, so fix the reported error and capture again.

See also

  • Testing. Capturing, matching and comparing runs, and the output coverage a revision needs before it can publish.
  • Process Outputs. Declaring the ports these entries name.