Pools
Pools. A pool is the set of computers that runs your work, and you choose the pool for each job. On a hosted pool, TALOS Defence operates the computers in AWS Sydney, and you pay from prepaid credit. On a customer-controlled pool, the computers are in your own AWS account, and you pay for a licence and support, and you pay AWS for the instances.
1. The pool object
GET /v1/pools lists the pools your organisation can see, and GET /v1/pools/{id} gives one, by id or by name.
| Field | What it holds |
|---|---|
id, name | The id of the pool, and the name of the pool, such as hosted-cpu-syd. |
kind | hosted, operated by TALOS Defence, or customer_controlled, in the customer's own environment. |
state | available, registered (a customer-controlled pool that waits for a licence and keys) or retired. |
platforms | The image platforms that the pool runs, such as linux/arm64. |
gpu | none, h100 or trn2. |
region | The AWS region of the pool. Each certificate from the pool shows the region. |
placement | on_demand (a fresh instance for each attempt), slot (inside a booked slot) or local (the customer's own runtime). |
release_jobs | true when the pool runs releases. |
identity | The identity that every return shows: provider, site, owner_nationality, jurisdiction and assurance (the assurance statement of the pool). |
organisation | The owning organisation, for a customer-controlled pool. |
deployment | For a customer-controlled pool: provider (aws), account and region. |
keyset | For a customer-controlled pool: the digest and issuer of the key set that signs the returns of the pool. |
licence | For a customer-controlled pool: the id, serial, state and end of the licence. |
revocation_policy | For a customer-controlled pool: max_age_days and on_stale (Section 3.7). |
Every certificate and every return shows the identity of the pool. A reader of a return sees where the job executed and under which jurisdiction, without asking Hangar.
2. The hosted pools
| Pool | What it runs | Placement |
|---|---|---|
hosted-cpu-syd | Jobs of a release and CPU sessions, for linux/arm64, on AWS Graviton in Sydney. | on_demand: each attempt uses one fresh instance for that attempt alone. Hangar terminates the instance when the attempt ends. |
hosted-h100-syd | Booked GPU jobs on one NVIDIA H100, for linux/amd64 with CUDA. | slot: inside the hours that you booked on a night. |
TALOS Defence Pty Ltd, which is 100% Australian-owned and controlled, operates both pools in AWS Asia Pacific (Sydney). The published key set of TALOS Defence signs the returns and certificates of both pools. You can verify them offline.
3. The customer-controlled form
The customer-controlled form runs the same releases inside the customer's own environment: the same package, release object, envelope and return, on capacity that the customer owns. TALOS Defence offers this form from 2027. Pool registration, licences, the local records contract and the offline bundles in this section are part of the Hangar control plane now. The runtime in the customer's environment is hangar-api --profile customer. The customer runtime explains how the runtime starts, checks the licence, imports releases and signs returns.
3.1 What the customer controls
| The customer controls | In practice |
|---|---|
| The environment | The customer's own AWS account in ap-southeast-2: a network with no inbound path from the internet, the local control plane behind the customer's internal load balancer, a PostgreSQL database, and storage buckets that are versioned and locked against deletion. |
| The capacity | The customer's own On-Demand or Spot instances, on the customer's own AWS bill. |
| The identity | The roles in the customer's account. The roles trust principals in that account alone. |
| The keys | The records-signing root at the customer's own key site, the day keys that the customer attests, and every data protection key. |
| Admission | Local approval of each release and each widening, by a named person in the customer's organisation. |
| The data | Every input, output, configuration value and secret, in the customer's own account. |
3.2 Key custody
TALOS Defence keeps two keys for this form, and only two. The release signing key signs runtime releases and package acceptances. The entitlement key signs licences. The customer holds every other key: the root that signs the records, trees and returns of the customer, the day keys under the root, and the keys that protect the data of the customer. The device identity of each instance comes from the hardware of the instance.
The owner decides what runs, and TALOS Defence keeps only the proof of what TALOS Defence shipped and licensed. A return from a customer-controlled pool shows the organisation of the customer as the issuer. The keys of the customer sign the return. Anyone can verify the return offline against the published key set of the customer and the published release key set of TALOS Defence.
PUT /v1/pools/{id}/keyset records the key set of the customer for a registered pool. Hangar accepts a key set that its own root signs and that names the organisation of the customer as the issuer. Hangar refuses these key sets with keyset_issuer_refused: the record key set of TALOS Defence, and any set that shares a root with the record, release or entitlement key sets of TALOS Defence. After a pool has a key set, Hangar accepts a new key set only as a rotation: a root of the old set must sign the new set. Otherwise Hangar answers keyset_invalid.
3.3 Who operates what
The contract takes one of three forms. In every form, the keys and the data stay with the customer.
| A: customer-operated | B: operated by TALOS Defence | C: operated by a partner | |
|---|---|---|---|
| Installs and updates | The customer's people | TALOS Defence, in the customer's account | The vendor or an Australian managed service provider, in the customer's account |
| Access to the account | The customer's own | A role the customer issues per session: time-bound, approved by the customer, logged in the customer's audit trail, and denied every data bucket and every key operation | As B |
| Releases and documentation | TALOS Defence | TALOS Defence | TALOS Defence, through the partner |
| Support | From diagnostics exports and the customer's description | As A, with operation in the session the customer approves | As B |
| Keys and data | The customer | The customer | The customer |
3.4 Registering a pool and its licence
hangar pool register --name harbour-aws --aws-account 123456789012
hangar pool licence harbour-aws
An owner registers a customer-controlled pool with POST /v1/pools. The request gives the name, the provider aws, the 12-digit AWS account id, the region ap-southeast-2, the platforms and, optionally, the revocation policy. The pool is registered until TALOS Defence issues the licence and the customer records the key set.
The licence is an entitlement file of the format hangar-entitlement-v1. The file is canonical CBOR, signed by the entitlement key of TALOS Defence. The licence lists:
- the customer organisation;
- the permitted deployments, as AWS account ids;
- the supported capability: the installation profiles, the number of packages and the number of tenants;
- the support level,
standardorextended; - the update channel;
- the term.
The price of the licence depends on organisational scale, permitted deployments, supported capability and lifecycle commitments. GET /v1/pools/{id}/licence gives the fields of the licence. GET /v1/pools/{id}/licence/file gives the signed file.
The local runtime checks the licence offline, at start and once a day. The runtime reports nothing to TALOS Defence. No usage record leaves the customer's account for licensing, and a renewal is a new file. A licence lists its deployments, so the runtime refuses at start a licence for another account. A changed byte fails the signature.
When the term ends, the runtime keeps running the releases that are installed, shows the lapse on every return, and takes no new release import. After a further 90 days of grace, the runtime takes no new tenant.
3.5 The local records contract
A customer-controlled pool keeps the records, tree heads and returns of the pool in the customer's account. The customer chooses what to share. POST /v1/pools/{id}/returns imports one return that the customer's runtime signed. Hangar can then list the return beside hosted returns. Hangar verifies the return against the key set of the pool and the release key set of TALOS Defence, and stores the return with the verifier report. If the return does not verify, Hangar refuses the import with return_import_invalid. The import contains only the return.
Every return from a customer-controlled pool shows the licence of the runtime and the age of the revocation list that the runtime used. The verifier requires both. The import judges both against the licence of the pool that was in force when the last attempt of the job started, and against the revocation policy of the pool. The newest licence whose term had begun is the licence in force. The licence state must be the state that the term gives: valid, or lapsed, before or after the grace. The age of the list must meet the revocation policy of the pool. The import also refuses a return in three cases: the pool identity on the return is not the full identity of the pool; the last attempt starts after the return was issued; or the release was revoked before the list that the return says the runtime held. The import answer and the kept record show both results. A pool with no issued licence imports nothing (licence_not_found).
A vendor usually supplies the release that a customer-controlled pool runs. Hangar serves the release bundle (Section 3.6) to the vendor, and to an organisation that holds an issued licence for a customer-controlled pool. The release must be accepted and not revoked.
3.6 The release bundle and the site snapshot
A disconnected pool needs two documents. Each item in the two documents verifies offline on its own.
| Document | Route | What it holds |
|---|---|---|
| Release bundle | GET /v1/releases/{id}/bundle | The signed release with the acceptance, the signed revocation list, the release key set of TALOS Defence, and the package manifest, byte for byte. The signed release lists the digest of the manifest. |
| Site snapshot | GET /v1/pools/{id}/snapshot | The licence of the pool, the signed revocation list, the release key set and the entitlement key set. |
The vendor ships the image layers, as an OCI layout on the customer's media. The runtime checks every layer against the digests in the signed release.
3.7 The disconnected operating policy
With no outside connection, everything in the customer profile works: jobs, the compile, returns, approvals and the local log. Each imported item allows one more thing:
| Item | Arrives as | What it allows |
|---|---|---|
| A package release | A release bundle and the vendor's OCI layout, on the customer's media | Jobs of that release, after the customer's local approval. |
| A runtime update | A signed runtime release, as an offline bundle | An update, after the customer's acceptance jobs and approval (Section 3.8). |
| The release revocation list | Inside each bundle and snapshot | New jobs while the list is younger than the policy's maximum age. |
| The licence | The licence file, or a site snapshot | Release imports and new tenants for the licence's term. |
| The key sets | Inside each bundle and snapshot | Offline checks of releases, acceptances and licences. |
The revocation policy sets the maximum age of the list, from 1 to 365 days (30 by default). It also sets what happens after that age:
record: jobs continue, and each return shows the age of the list;refuse: new jobs are refused withrevocation_list_staleuntil a fresh list is imported.
Time comes from the time source of the customer. Each return shows the error bound of the clock, measured against the time source. The gateway connects to the destinations that the network of the customer connects to, and refuses every other destination.
3.8 Updates and rollback
A runtime update installs only when the customer accepts the update:
- The customer imports a signed runtime release, from the channel that the licence lists or as an offline bundle.
- The local control plane installs the new release beside the running release.
- The control plane runs the conformance suite that ships with the release, and the acceptance jobs of the customer. The acceptance jobs are a list that the customer keeps of package jobs with their expected output checks.
- A named person in the customer's organisation approves.
- The control plane switches.
Every step is a record. A failed check stops the update and leaves the running release in place. A rollback returns to the previous release within the rollback window. The database migrations of each release are additive for one release, so the previous release still works on the new schema. Package releases follow the same path. The customer imports a release that TALOS Defence accepted and approves the release locally. The widening rule (Releases and jobs, Section 5) applies, and the customer is the approver.
3.9 The diagnostics export
When the customer wants support, the customer exports a diagnostics bundle. The customer previews every field before export. The customer signs the bundle with the day key of the customer and sends the bundle through the customer's own channel. The bundle contains:
- the versions and digests of every installed artefact;
- the configuration key names, with a digest of each value in place of the value;
- health, and the operating metrics;
- error codes, with their counts;
- attempt and switchover timings;
- record counts by kind;
- tenant ids, each replaced by a keyed hash;
- the redacted operational log.
3.10 Support boundaries
| Party | Supports |
|---|---|
| TALOS Defence | The runtime release as released, on the supported profiles, until the end-of-life date of 5 November 2031. Support works from diagnostics exports and the description of the customer. |
| The vendor | The package and the application of the vendor. |
| The customer | The account, identity, network and keys of the customer, and the changes that the customer makes to the installation. TALOS Defence supports a modified installation again after the installation report passes. |
Support works from diagnostics exports and descriptions. It never asks for payloads, secrets or screen sharing.
3.11 The runtime release
The release root of TALOS Defence signs each runtime release. The manifest of the runtime release has the format hangar-runtime-release-v1 and lists:
| Field | What it holds |
|---|---|
| version | The runtime version. |
| control plane | The digest of the local control plane built with the customer profile. |
| worker and gateway | The digests of the worker image and the gateway image, each built twice and compared, with the SBOM digest and the boot measurements of each image. |
| installation | The digest of the installation template of each environment. |
| migrations | The database migrations, in order, by digest. |
| package runtime range | The package runtime versions this release runs. |
| rollback | The previous release that the customer can roll back to. |
The installation verifier reads the deployed account through the read-only role of the customer. It writes a signed installation report. The report covers every artefact digest against the manifest, every role and the trust policy of the role, the network paths, the bucket settings, the key-site link, the licence and the boot measurements. hangar verify checks the report offline.
4. The portability report
tools/portability_report.py compares two returns of one job: one release, the same inputs and narrowing, for one tenant, on two pools. The tool verifies each return with hangar verify --json, then lists:
- the fields that must be equal: the release id, the image, platform and command, the companions, the destinations and dropped items, the data scopes, secrets, model routes and person-present steps, the compute and failure policy, the inputs, the values digest, the tenant, the descriptor result of each output, and every other field of the compiled envelope (including the sources of the envelope), each shown by path;
- the fields that may differ: the pool, the attempts and instances, the times (including the window), the issuer, and the configuration digest, which is a declared difference;
- each output digest, equal or different, against the determinism that the package declares.
Only a return that hangar verify passes counts. Any other answer, or no hangar on the search path, fails the report. --no-verify compares the fields alone, and the report then never says that the returns are portable.
A package that declares deterministic promises equal output bytes for equal inputs and configuration, so a different digest fails the report. A package that declares envelope promises results within the declared contract, and the report lists any different digest.
tools/portability_report.py hosted.cert harbour.cert \
--keys-b harbour_keyset.cbor --release-keys release_keyset.cbor
5. Errors
| Code | Status | When |
|---|---|---|
pool_not_found | 404 | No pool of that name or id. |
pool_registration_invalid | 422 | The registration lacks a name, the provider aws or a 12-digit account id. |
pool_runs_locally | 409 | A job on a customer-controlled pool starts from the runtime of the pool. |
keyset_invalid | 422 | The key set is not valid, or is not signed by its own root. |
keyset_issuer_refused | 422 | The key set does not name your organisation as issuer. |
licence_not_found | 404 | No licence is issued for the pool. |
licence_invalid | 422 | The licence is not signed by TALOS Defence's entitlement key, or names another deployment. |
return_import_invalid | 422 | The return does not verify against the pool's key set and the release key set. |
revocation_list_stale | 409 | The pool's revocation list is older than its policy allows. |
not_in_this_runtime | 404 | The call is served by the hosted Hangar service at api.gethangar.au. |
runtime_only | 404 | The call is served by a customer runtime. |
profile_not_supported | 422 | Hangar does not install the deployment profile. Use the AWS installation profile. |