Jobs and the API

Jobs and the API. The Hangar API lets a program start jobs, follow them and download the results. Each job that a request starts uses your prepaid credit, and each request key has its own spend limit.

The OpenAPI document docs/api/openapi.json lists every route, request and answer. docs/api/errors.json lists the three parts of every error (what failed, the next step and what to do if that fails) and says whether a client may retry the request. Every error body also contains reference, the id of the request. When support is the next step, the third part of the error shows reference.

Three pages go further. Releases and jobs explains packages, releases, channels, the envelope and the return. Pools explains where a job executes, and the customer-controlled form. The Hangar SDK explains request keys, tenants and the Python package gethangar.

Calling the API

• The CLI and the apps send a bearer token with a Hangar-Proof header on every request. The page "Proof of possession: the Hangar-Proof header" explains it.
• The website sends its session cookie.
• Your own server, a vendor's or any organisation's, sends Authorization: Hangar-Key hgk_... with a Hangar-Signature made with its request key, and Hangar-Tenant for a tenant. The routes marked x-hangar-sdk in the OpenAPI document take a request key as well as a session. The Hangar SDK page explains the signature, the capabilities and the tenants.
• Every request that changes state needs an Idempotency-Key header with a new UUID. To retry, send the same key again. Hangar returns the first answer and does the work once.
• Times are Melbourne local, as an object with at (RFC 3339 with the offset) and label (for example Sat 24 Oct 22:50 AEDT).

Jobs

A job is one container command inside a booking. hangar run submits one job. A booking can contain several jobs. The jobs of a booking use one instance, in the order of their seq numbers.

Jobs started by a request key

A request key with the jobs capability starts a GPU now, as a person does. A CI job or an agent can start a GPU with no person signed in.

The rules:

• A key spends credit only through the sessions that the key starts. Hangar meters those sessions as it meters every session, inside the limits of your organisation: the sign-up checks, the current terms, the new-account limit of one live GPU at a time, and credit for the first ten minutes. A key of an organisation with a held or declined sign-up starts no session (account_not_active).
• A key may also have its own spend limit, in Australian cents including GST (spend_limit on POST /v1/keys, or PUT /v1/keys/{id}/spend-limit by an owner). Each time the key starts a session, Hangar takes two amounts from the limit: what the jobs of the key were charged (less refunds), and what the running sessions of the key may still be charged. If less than one minute at the price of the GPU is left, Hangar refuses the start with key_spend_limit_reached (402). Hangar starts nothing and charges nothing. The error detail contains limit_cents, spent_cents and held_cents. Otherwise ends_by of the session is the time when the rest of the limit runs out. The stop after ends_by takes up to the interval of the meter, the three-minute grace period and the wipe. For each session, the spend of the key can pass the limit by the cost of that time.
• GET /v1/keys and GET /v1/keys/{id} show the spend_limit of each key, spent_cents, jobs_started and images_pushed.
• A key never books, holds, confirms or cancels a booking, tops up, approves, invites, changes a policy or makes a key. Those routes answer a key with not_signed_in.
• The start record, the session, the booking and each job show which key started them. The job and the session contain started_by: {"kind": "key", "id": ...} with the key's id as GET /v1/keys lists it, or {"kind": "person", "id": ...}.
• A revoked key is refused from its next request (key_revoked). A session that the key started continues until the session ends, or until a person or another key stops the session.
• These routes cover the jobs of your organisation. Hangar refuses a key that belongs to a tenant on these routes (tenant_mismatch), and a key without jobs gets key_capability_missing.

Images pushed by a request key

A request key with the uploads capability sends the image that the jobs of the key use. A CI job can then build, send and start with no person signed in.

The rules:

• The push credential, the records and the image show who pushed the image. Each image has pushed_by: {"kind": "key", "id": ...} or {"kind": "person", "id": ...}, the person or key that last pushed, registered or tagged the image.
• Hangar checks an image that a key pushes before the image runs, as Hangar checks every image. Only a person can approve a job of an image that your policy does not list, change the policy, delete an image or do anything that pays. A key gets not_signed_in for these actions.
• The registry is your organisation's and has no tenant, so a key that belongs to a tenant is refused these routes (tenant_mismatch). A key without uploads gets key_capability_missing.
• Hangar refuses a revoked key from the next request. The push credentials of the key stop working.
• The spend limit applies to jobs, never to uploads or images.

Uploads

hangar upload PATH and the Python package send a file in chunks. Each chunk is checked, sealed under the file's own key and stored in Sydney. Hangar then checks the digest of the whole file and stores the file once.

The chunk size is per upload. The size is 8 MiB while a file needs at most 9,000 chunks of 8 MiB (70.3 GiB). A larger file gets larger chunks, in whole MiB, so the file stays at 9,000 chunks or fewer: 43 MiB for 400 GB, and 59 MiB for 512 GiB. Use the chunk_bytes that POST /v1/uploads gives.

Hangar adds each chunk to the digest of the whole file as the chunk arrives in order. When the chunks arrive out of order, the chunk that fills a gap adds the chunks after the gap. So when you complete the upload, Hangar has usually checked every byte already.

POST /v1/uploads/{id}/complete answers in one of these ways:

• When a chunk is missing, Hangar answers upload_incomplete (409) at once. detail.missing_chunks lists the chunk numbers. Send those chunks, then complete again with the same Idempotency-Key.
• With the header Prefer: respond-async, Hangar answers 202 at once with the state of the upload, and the header Preference-Applied: respond-async. Follow the upload with GET /v1/uploads/{id}.
• Without the header, Hangar holds the request while it checks and stores the file, for up to 300 seconds (limits.completion_hold_secs). When the file is stored in that time, the answer is 201 with the object. When it is not, the answer is idempotency_in_progress (409): send the same request with the same Idempotency-Key, and the final answer for that key is the 201 object.
• When the file is already stored, the answer is 201 with the object. A second request, with any key, never starts a second check.

GET /v1/uploads/{id} gives:

The errors at the end:

• upload_digest_mismatch (422): the bytes did not match the digest. Hangar removes the chunks and stores nothing. Upload the file again.
• upload_completion_failed (503): Hangar could not store the file after three tries. Hangar keeps the chunks. Complete the upload again, with a new Idempotency-Key. Hangar checks and stores the file again, and you send no chunk twice. hangar upload does this when you run the same command again.
• tenant_suspended or tenant_closed: the tenant of the upload changed state before Hangar stored the file. Hangar removes the chunks and stores nothing.

A person, or a request key with uploads, calls these routes. A key that belongs to a tenant sees the uploads of its tenant only. The upload of another organisation or another tenant is upload_not_found. The movement record of the receipt names the person or the key that completed the upload.

Jobs of a release

A job of a release runs a vendor's accepted application release for one tenant, on the CPU pool hosted-cpu-syd. Before the job, TALOS Warrant (the part of Hangar that decides what a job may do) compiles the envelope, the set of limits that the job cannot leave. hangar run --release or hangar run --package --channel starts the job. hangar status, hangar logs, hangar pull, hangar stop and hangar verify work on a job of a release as they work on any job. The Releases and jobs page explains the whole contract.

Following a job

GET /v1/jobs/{id}/events?after=&wait= follows a job as it prepares and runs. Each answer gives the position of the job now and the events since your last call:

• preparation: a step of the preparation of the booking, and the time of the step. The steps are 0 the fresh instance, 1 the measured boot, 2 the reset and GPU checks, 3 the image and data staged, and 4 the job started.
• status: the state of the job, each time the state changes.
• log: one chunk of the log, with the lines of the chunk, the stream and the time Hangar received the chunk. The stream is stdout or stderr for the output of the job, and system for the lines that Hangar writes about the job: the start of each attempt, a stop signal and why, each checkpoint sealed and stored, the exit code and the outputs sealed.

The position shows the phase of the job, the preparation step and the elapsed time. It also shows what the worker reports:

• checkpoint: the last checkpoint sealed in the booking, with its time, SHA-384 digest and size, and checkpoint_count. A checkpoint is sealed and stored in Sydney when an attempt ends.
• gpu_util_percent, gpu_memory_bytes and gpu_memory_total_bytes: the GPUs' use at the last heartbeat, with gpu_reported, the time the worker read them.
• envelope: the egress decisions of the booking's proxy so far, as refusals and connections, with the time of the last refusal. Each refusal is also a signed record on the certificate.

GET /v1/jobs/{id} shows the same parts in position, with exit_code: 0 for a job that finished by itself, and the code of the process for a job that ended with another code or was stopped. certificate_signed is the time Hangar signed the certificate of the booking. That certificate covers the job.

Pass next from the answer as after in the next call. With wait (up to 25 seconds), Hangar delays the answer until new data exists or the job ends. When complete is true, the job has ended and you have the whole log.

Logs and outputs

• GET /v1/jobs/{id}/logs?after=&max= gives the log lines of the job in order. Each line has the number of its chunk (seq), its place in the chunk (n) and its stream: stdout, stderr, or system for lines from Hangar. The three streams share one sequence, so the log shows the lines in the order that they were written. To read the next lines, pass next_seq as after. When truncated is true, Hangar holds more lines than the call returned.
• GET /v1/jobs/{id}/outputs lists what the booking of the job wrote, by SHA-384 digest. Each object shows its class, size, retention date and the time of writing. Download an output with POST /v1/objects/{digest}/download.

Hangar stores each log chunk in Sydney, sealed under the key of the job.

Certificates

Every booking ends with a job certificate, signed by Hangar and anchored in the public log. One certificate covers every job in the booking.

The report comes from the same verifier as hangar verify --json. The verifier checks the certificate bytes against the published key set and tree heads. To check a certificate yourself, download the certificate and run the offline verifier. The offline verifier uses only the file and the published keys.

A certificate's public link

Each certificate has a public page at {public_origin}/c/{page_key}. The list shows the address as page. The download of a stored certificate shows the address in the hangar-certificate-url header. The page key is 32 hexadecimal characters, chosen at random when Hangar signs the certificate. The link is the permission. Anyone who has the link can read the certificate without a sign-in. To show a certificate to an assessor, give the assessor the link.

These two routes find a certificate only by the page key, never by the id. A page key that matches no certificate gets not_found. Each answer has the headers Cache-Control: private, no-store and X-Robots-Tag: noindex. No shared cache keeps a copy, and no search engine lists the certificate.

Your account

Status and open nights

These two routes need no sign-in.

• GET /v1/status gives the service status: the state and release of the API, the latest status line of each component, the incidents of the last 14 days by reference, and the latest signed tree head of the public log.
• GET /v1/public/nights lists the open nights. Each night shows the bookable window, the free minutes, whether the night is open, filling or full, and the price for each hour including GST.

Records

Every change that you make through the API becomes a signed record in the record chain of your organisation, before Hangar changes a row in the database. A removed passkey, an ended session and an invitation are records too.

Read this page as Markdown