Skip to content

Organization Run Environment Connections

This page is written for Foundry Connect site administrators. It explains the two ways a platform managed organization run environment can connect to a head node, how to switch between them, and what happens when a switch is refused.

Organization connections are not available in Foundry Connect 2.22.0 or earlier.

1. What a connection mode is

A platform managed organization run environment connects in one of two modes:

  • Inherited. The environment uses the platform's shared head node. This is how every run environment works today, and it stays the default for existing environments.
  • Organization. The environment connects to the organization's own head node instead of the shared one.

2. What organization mode guarantees

  • Runs launch only while the connection is active.
  • The host, port, SSH account, SSH key, work directory and publish directory all come from the organization's own run environment. None of them ever falls back to the shared template.
  • Each run keeps the connection it launched with, even if the environment's connection changes later.
  • Every SSH and SFTP connection to the organization's head node checks the head node's pinned host key and refuses to connect on a mismatch.

3. Before you switch

Have these ready before you switch a run environment to organization mode:

  • A head node reachable over SSH from the Foundry Connect server.
  • An SSH key owned by the user who owns the organization's run environment. Generate it as that user, in Profile, Credentials, "Generate Keys for Me", then install its public half on the head node.
  • The head node's host public key, obtained out of band. For example, the content of /etc/ssh/ssh_host_ed25519_key.pub read on the instance, or its fingerprints from the EC2 console output. Never trust a first connection to supply it.
  • No run of that environment still in flight. A run in a launch blocking status (queued or in progress) blocks a switch. A run whose status is Aborted does not block one: Aborted means the head node could not be reached, so waiting on it would lock out the very switch that repairs an unreachable head node.
  • For an AWS environment using an assume role credential, the organization role's maximum session duration set to 12 hours (43200 seconds). Activation checks this only when an AWS credential is already attached to the environment, so attach the credential first.

4. Switching

Switching is a site administrator only operation.

GET /api/admin/organization/{organizationId}/run-environment/{profileClusterId}/connection returns the current mode, state, connection fields, and the pinned host key's fingerprint. {profileClusterId} is the run environment's id.

PUT the same path with a body of:

{
  "mode": "organization",
  "hostname": "<head node host name>",
  "username": "<account on the head node>",
  "port": 22,
  "sshId": <id of an SSH key the environment's owner holds>,
  "defNextflowWorkdir": "s3://<bucket>/work",
  "defPublishdir": "s3://<bucket>/pub",
  "hostKey": "<head node's public host key line>"
}

Foundry Connect saves the connection as pending, connects to the head node over SSH verifying the pinned host key, checks the 12 hour AWS session when the environment names an assume role credential, and only then marks the connection active. Compare the fingerprint the response returns with the one you read on the head node yourself; do not take it on trust.

Attach the AWS credential first, with PATCH /api/admin/organization/{organizationId}/run-environments/{runEnvironmentId} and a body of {"amazonCreId": <credential id>, "awsUseInstanceRole": false}, so activation checks it. Once the environment is in organization mode, that same PATCH refuses a different amazonCreId: to change the AWS credential of an organization connection, switch the environment back to inherited mode, change the credential, then switch to organization mode and activate it again.

Field formats. hostname accepts letters, digits, dots and hyphens, and cannot start with a hyphen. username accepts letters, digits, underscores, dots and hyphens, and cannot start with a hyphen. defNextflowWorkdir and defPublishdir must be absolute paths, or an s3://<bucket>/... or gs://<bucket>/... location, with no spaces and no shell characters. A value outside these formats is refused with HTTP 400 before Foundry Connect attempts to connect anywhere.

5. When the switch is refused

A refusal is returned as HTTP 409, with a reason and a remedy. Some refusals happen before Foundry Connect writes anything, and leave the run environment exactly as it was before you started the switch. Others happen after Foundry Connect already wrote the environment as organization mode and pending, and leave it that way: a run on it then refuses to launch with "This run environment is still being set up" until a later PUT both revalidates and activates the connection. The "Environment after the refusal" column below names which applies to each reason.

Reason Meaning Environment after the refusal Remedy
not_platform_managed The run environment named in the request is not the organization's platform managed environment. Only that one environment's connection can be switched this way. Unchanged. Switch the organization's platform managed run environment instead. An environment the organization defines for itself already connects with its own settings.
runs_in_progress The run environment has a run in a launch blocking status (queued or in progress), and switching its connection while that run depends on it would strand the run. An Aborted run does not count. Unchanged. Wait for the run to finish, or terminate it, then switch again.
host_key_malformed The hostKey value is not a supported OpenSSH public host key line. Unchanged. Paste the head node's public host key line, for example the content of /etc/ssh/ssh_host_ed25519_key.pub.
ssh_key_not_owned_by_environment_owner The sshId does not name an SSH key owned by the run environment's owner. This is also what you see when sshId names no SSH key at all, so the response never reveals whether the id belongs to another user. Unchanged. Choose an SSH key the run environment's owner holds.
aws_credential_missing The run environment names an AWS credential that no longer exists, so its runs could not reach AWS. Unchanged. Attach an existing AWS credential to the run environment, or clear its AWS credential, then switch again.
ssh_validation_failed Foundry Connect could not run a command on the given host and port, as the given user, with the given SSH key, while verifying the pinned host key. Pending. Check the host name, port and user name, that the key's public half is authorized for that user on the head node, and that the pinned host key is the head node's own key, then try again.
session_identity_unreadable This server could not read its own AWS identity, so the 12 hour session check could not run. Pending. Check this deployment's AWS identity, then try again.
session_chained_source_identity This server's own AWS identity is itself an assumed role session, and AWS caps a session assumed from one at one hour, short of the 12 hours a run needs. Pending. Give this deployment an AWS identity that is not an assumed role session, then try again.
session_trust_or_permission Either the assume role credential has no role ARN recorded, or this server could not assume the organization's role at all, not even for 15 minutes. Unchanged if the credential has no role ARN recorded; pending if the assume attempt itself failed. Attach an assume role credential that names the organization's role. If it already does, check that the role's trust policy allows this deployment's AWS identity with the credential's external id, then try again.
session_max_duration_insufficient This server can assume the organization's role, but not for the 12 hours (43200 seconds) a run needs. Pending. Raise the role's maximum session duration to 43200 seconds, then try again.
changed_during_validation The run environment's connection changed while it was being validated: its host name, user name, port, SSH key, work directory, publish directory, host key, AWS credential, connection mode, connection state, or its owner; or the run environment itself was deleted while its connection was being validated. Pending, if the environment still exists. Gone, if the environment itself was deleted. Review the run environment's connection, then switch again. If instead the environment was deleted, review the organization's run environments, then switch again.

A pending environment refuses to launch runs with that same "still being set up" message until a switch to organization mode succeeds. Fix the cause the table names, then PUT again: from unchanged, that repeats the switch to organization mode from the start; from pending, it revalidates and activates the connection Foundry Connect already wrote.

6. Switching back

PUT the connection path with {"mode": "inherited"} returns the environment to the shared head node. This does more than change the mode: it replaces the environment's host name, SSH account, port, SSH key, work directory and publish directory with the shared template's own values, and clears the environment's pinned host key. The organization's own head node values are not kept anywhere. If you switch the environment back to organization mode later, you need to re-enter its host name, SSH account, port, SSH key, work directory and publish directory from scratch.

Like a switch to organization mode, switching back is refused while a run in a launch blocking status depends on the environment. It is also refused if the shared template run environment itself does not exist: nothing is written, and the environment keeps its current connection.

7. Changing connection fields elsewhere is refused

Once a run environment is in organization mode, saving it through the regular run environment settings (POST /api/v1/user/credentials, the run environment form) refuses any change to its host name, SSH account, port, SSH key, work directory, publish directory, or AWS credential fields (the attached AWS credential and whether it uses an instance role). The save fails with HTTP 403 and a message naming exactly which fields were refused. Submitting the current, unchanged values back is fine and saves normally, as an edit form does.

Those fields change only through the site admin connection endpoint described above. Every other field on the run environment form, such as its name or description, still saves normally.

8. What users see

A run environment whose connection is pending or incomplete refuses to launch runs, with one of two reasons and remedies:

  • organization_connection_pending: "This run environment is still being set up and cannot launch runs yet." Remedy: wait for an administrator to finish activating the organization's connection, then launch again.
  • organization_connection_incomplete: "This run environment has no [the missing fields] on record, so it cannot launch runs." Remedy: an administrator must complete the organization's connection before runs can launch.

File browsing, SSH uploads and report downloads on an organization environment also verify the pinned host key, and refuse with the same two reasons while the connection is pending or incomplete.

A run that launched before its environment switched to organization mode is a special case. It already ran on the previous head node, and Foundry Connect no longer knows how to reach that host: polling, stopping, or fetching reports for the run all refuse with attempt_connection_unpinned, "This run started before its run environment switched to the organization's own head node, so Foundry Connect cannot reach the host it ran on." Remedy: "Check the job on the previous head node directly. Foundry Connect cannot poll, stop, or fetch reports for this run."