Learning Outcomes

• Capture a finished run as the expected result for a process.
• Read what a capture reports about waived ports, kept entries and the entries it had to change.
• Understand how output ports are matched to the files a run published.
• Use the Match outputs dialog when a file's name does not match its port.
• Compare a later run against what you captured.
• Cover every declared output, or waive it, so the revision can publish.
• Read stored validation and test run evidence through the API.

Process Guide

Testing

A process test pins down what a process is supposed to produce. You take one finished run, capture its outputs as the expected result, and every later run can then be compared against it. That turns "did my change break anything?" into a question you can answer in one click.

Both actions live on the Test tab of a run.

The Test tab only appears on a run of a single process

The tab is shown when the run is a standalone process run, which is what the Run button inside the process editor creates. It does not appear on pipeline runs, because the expected results belong to one process rather than to a whole pipeline. See Run a Process on Its Own for how to make one.

The tab is available whether or not the process already has expected results. If it has none, only capturing will do anything useful.

Capture a run as the expected result

Open a finished process run, go to the Test tab, and click Capture this run as expected.

Foundry Connect works out which published file belongs to each of the process's declared output ports, then writes them into a file called tests/expected-results.yaml inside the process. On success the tab reports:

Captured this run into tests/expected-results.yaml. Later runs are compared against it.

Underneath, it lists one line per port, naming the file each was captured from, so you can confirm the match was the one you intended before relying on it.

Any port that could not be captured is listed separately with the reason, for example when the run produced no file matching that port. The rest of the capture still succeeds, but that port is left with no expected result, and if the process declares it as required, the gap will stop the revision publishing. See Cover every output before publishing.

Read the capture result

Besides the captured and uncaptured ports, the result reports everything else the capture did to tests/expected-results.yaml.

Not available in Foundry Connect 2.22.0 or earlier

These reports arrive in the release that follows 2.22.0, along with waivers themselves. Earlier releases list only the captured and uncaptured ports.

Notes on a captured port. When a capture changes something you authored, or keeps something you authored that this run contradicts, a note in amber appears under that port's line. There are two. The first is for an authored text entry whose output is over the inline limit:

Captured as "exists", not "text": the output is 300000 bytes, over the 262144 byte (256KB) inline limit, so its content is no longer compared.

The second is for an authored exists byte range that this run's file falls outside:

The authored byte range (min 1024) excludes this run's 512 bytes and was kept as written, so the next comparison of this run fails. Widen the range, or pick a different file.

Waived ports. A port you waived is never captured, even when the run published a file for it. It is listed with your own reason:

plots is waived: the layout is randomised per run, so no comparison is meaningful. It was not captured.

Entries kept for ports the process no longer declares. These are listed in amber, because such an entry is still checked and can fail every later comparison:

summary No longer a declared output of this process. Its entry was kept unchanged, and later comparisons still check it: while the run does not publish a matching file it reports absent and fails the whole comparison. Remove it from tests/expected-results.yaml when the port is gone for good.

A waiver kept for such a port is listed the same way, with a reason saying it no longer affects coverage.

What a re-capture keeps explains why each of these is kept rather than removed.

How ports are matched to files

You do not have to rename anything for a first capture to work. A port matches a file by the first of these rules that applies:

1. The file pattern authored for that port in tests/expected-results.yaml, matched first as a literal name and then as a glob.
2. A published file whose name, or whose name without its extension, is the port's identifier.
3. Any file inside a publish directory named after the port.

Rule 3 is what makes a first capture work out of the box, because a process that publishes into port named folders is already organised the way the matcher expects.

If more than one file qualifies for a port, the matcher orders the candidates and takes the first, then tells you which one it used in the per port list. It is worth reading that list rather than assuming, because a process that publishes several files into one port folder will have had one of them chosen for it.

Match outputs to files

When some ports would be left out, the capture does not silently drop them and it does not dead end. It first checks what would match, and if anything would be missed it opens a dialog titled Match outputs to files:

Some output ports did not match any of the files this run published. Choose which file each port should be captured from, or leave it out.

Ports that already matched are listed at the top, each naming the file it will be captured from. The dialog does not change how they match. If capturing one of them will produce a note, the note appears under it here, before anything is written.

Each unmatched port then gets a dropdown listing the files the run actually published, with each file's size beside its path, because size is often the only thing that distinguishes two similarly named outputs. Pick the file that belongs to that port.

Every dropdown also offers Leave this port uncaptured, which is the default. A port left this way is not written into the expected results at all, so nothing will be checked for it on later runs.

That is a real choice with a cost. Leaving a required port uncaptured is what stops the revision reaching Ready to publish, and the way to say "not tested, on purpose" is a waiver rather than an omission. See Cover every output before publishing.

Waived ports are not offered at all. A waiver says the port is deliberately not asserted on, so there is no file to pick for it. To start testing a waived port, delete its waiver from tests/expected-results.yaml first. If the document gained a waiver after the dialog opened, capturing a file for that port is refused:

Port "plots" is waived in tests/expected-results.yaml ("the layout is randomised per run, so no comparison is meaningful"), so it cannot be captured from a file. Reload the run's files, or remove the waiver first if the port should now be asserted on.

A port you pick a file for here shows no note in the dialog. If capturing it produces one, the note appears in the capture result instead.

If the same file ends up chosen for more than one port, a warning appears directly above Capture chosen files. A port that already matched counts as well as one you picked, so choosing a file an already matched port uses is caught too:

The same file is selected for more than one output: counts/summary.tsv (ports counts, stats). Confirm only if that is intended.

The warning never blocks the capture, because one file can deliberately serve two ports. It disappears as soon as no file is chosen twice, and when several files are each chosen more than once, a single warning lists them all.

Not available in Foundry Connect 2.22.0 or earlier

Leaving waived ports out of the dialog, the notes and the duplicate file warning arrive in the release that follows 2.22.0.

Click Capture chosen files to capture with your choices applied, or Cancel to change nothing.

You are picking from real file names

The point of the dialog is that you choose by looking at what the run actually produced, rather than by editing tests/expected-results.yaml by hand and guessing at a pattern.

Compare a later run

Once a process has expected results, run it again and click Compare against expected results on the new run's Test tab. Each captured port is checked using the comparison recorded for it, and the tab reports whether the run matched.

A passing comparison is not a correctness proof

The tab says this on every run, and it is worth repeating:

A passing comparison means this run matches the results captured for this process. It does not mean the results are scientifically correct.

A test can only tell you that a process still does what it did when you captured it. If the captured run was wrong, a passing test says the process is reliably wrong.

If the process has nothing captured yet, the comparison is refused and tells you so, because there is no baseline to compare against.

When a capture is refused

If none of the run's files could be attached to any declared port, nothing is captured and the tab explains every way to fix it:

None of the run's output files could be matched to a declared output port, so nothing was captured. A port matches a produced file named after it, any file inside its port-named publish directory, or the "file" glob its entry in tests/expected-results.yaml declares. Rename the output or author a glob for each port, and promote again.

In practice one of three things is true. The process published its files under names unrelated to its ports, in which case renaming the outputs or authoring a file glob fixes it permanently. Or the run genuinely published nothing, which is reported separately and means the run itself needs looking at first. Or the ports you expected are not actually declared on the process, which you can check on Process Outputs.

When some of the process's ports are waived, the refusal says so instead, because a waived port is never captured. If every declared port is waived:

Every declared output port of this process is waived in tests/expected-results.yaml, so nothing was captured. Remove a waiver first if a port should now be asserted on, and promote again.

If only some are, and none of the rest matched a file:

2 of this process's declared output ports are waived in tests/expected-results.yaml, and none of the rest could be matched to a produced file, so nothing was captured. Remove a waiver if a waived port should now be asserted on, or rename an unwaived output or author a glob for it, and promote again.

Cover every output before publishing

A passing comparison and a complete test are two different things. The comparison grades the entries the document has. Coverage asks a separate question: do those entries account for the outputs the process declares at all? A run can match every entry it carries and still assert nothing about half the process.

Publication reads both. A revision reaches Ready to publish only on a passing run whose coverage is complete.

Not available in Foundry Connect 2.22.0 or earlier

Coverage is recorded and checked from the release that follows 2.22.0. Before that, a port left uncaptured simply went unchecked, and publishing did not consider coverage at all.

Coverage is complete when every required declared output port is either asserted on in outputs or listed in waived, and at least one port is actually asserted on. That last clause carries its weight: without it, a process declaring no required outputs would count as fully covered while testing nothing.

State	What it means
complete	Every required port is asserted on or waived, and at least one is asserted on.
partial	At least one port is asserted on, but some required port is neither asserted on nor waived.
empty	None of the declared ports is asserted on.

Entries naming a port the process no longer declares are ignored when coverage is worked out. They stay in the file, because removing them is your editorial decision rather than a side effect of capturing, but a stale entry never satisfies a live port. The comparison is a different matter. It checks every entry in the document, so a stale entry can fail it, which is why a capture lists each one.

What you see

When coverage is anything other than complete, the Test tab says so underneath the comparison result. On a document that covers only some ports:

This process's expected results do not cover every declared output, so this run does not count as publication evidence.

On a document that asserts on nothing:

The expected results assert on none of this process's declared outputs, so this comparison tested nothing.

Either way, each port at fault is named on its own line:

counts has no entry in the expected results and is not waived.

Publication refuses separately, and reports its tested step as unmet with:

the passing test run does not have complete output coverage

The revision does not move backwards

A revision that had already reached Ready to publish keeps that status. What changes is that its tested step now reports unmet, and publishing is refused when you attempt it. Coverage is worked out fresh at publication time rather than read from a stored verdict, so fixing the document and running the test again is all that is needed.

Closing the gap

There are two remedies, and they say different things about the output.

Add an entry for the port when you can assert something about it. That is the ordinary answer, and Expected Results covers choosing the comparison.

Waive the port when you genuinely cannot. Some outputs differ on every run no matter which comparison you pick, and a waiver records that you decided not to assert on that one. It requires a written reason, which is the point of it: an unexplained gap cannot be told apart from an oversight. A waiver is also the only way a process with a genuinely untestable required output ever reaches complete coverage.

Runs recorded before this release do not count

Coverage is recorded at the moment a comparison runs, so runs recorded before this release carry no coverage verdict at all. Unknown is not treated as complete, which means a revision resting on an older passing run needs one new passing run before it can publish. Nothing else about the revision changes.

What capturing does to the revision

Capturing writes a file into the process, so it changes the process's content. Two consequences follow.

A revision that is already published or deprecated cannot be captured into, because its content is frozen. Create a new revision and capture into that.

If you have already reserved a container digest to publish a revision, capturing afterwards invalidates that reservation, because the content it was taken against has moved. Capture first and reserve second. The working order is to capture the expected results, then reserve, then run the comparison that records the evidence, then publish.

Read the stored evidence through the API

Validating a process (POST /v1/process/{processId}/validate) and comparing a run (POST /v1/process/{processId}/runs/{runId}/compare) both record evidence, and a compare that recorded evidence returns its evidenceId. Two read only routes fetch that evidence back, for example to audit what a publication decision rested on.

Not available in Foundry Connect 2.22.0 or earlier

Both routes arrive in the release that follows 2.22.0.

Route	Returns
GET /v1/process/{processId}/validations	Every validation recorded for the process, newest first. Paged with take (1 to 100, default 20) and skip.
GET /v1/process/{processId}/test-runs/{evidenceId}	The one test run evidence record a compare response named.

A validation record carries its result, its findings and the contract content hash it is bound to. A test run record carries the run it was observed from, the compare's verdict, the output coverage at that moment, and the hashes and container digest the comparison ran under. Coverage is complete, partial or empty, or null for a run recorded before coverage was tracked, which does not count as complete.

Evidence that no longer describes the process's current content is not deleted. It is returned with invalidatedAt, invalidatedBy and invalidatedReason filled in, and all three are null while the record still describes current content.

Both routes require permission to edit the process, because evidence exposes what a publication decision rests on. An evidenceId that belongs to a different process is reported as not found, exactly like one that does not exist.

The full request and response schemas are in the interactive API reference at <your instance>/api-docs/#/, under process. See API Overview.

See also

• Run a Process on Its Own. How to create the standalone run that the Test tab needs.
• Expected Results. The format of tests/expected-results.yaml, how to waive an output, and what a re-capture keeps.
• Process Outputs. Declaring the output ports that a capture matches against.