# How a job is isolated Every slot runs on its own freshly launched instance, so your job never shares a machine with another customer's job. The instance runs inside Hangar's capacity block in Sydney, from the signed Hangar image. The switchover between two customers is the security boundary, so this page sets out the switchover step by step. Each step is a signed record, and the records are on the job certificate. ## The switchover, step by step 1. **The previous job ends.** Its process tree and container stop. Hangar seals outputs and checkpoints to Sydney storage and records their digests. 2. **The instance is terminated, not reused.** Each slot runs on a freshly launched instance from the Hangar image. The previous customer's operating system, filesystem, kernel state and GPU driver state end with the machine. Sometimes a relaunch inside the block is not available in time. Hangar then discards the working disk and reboots the same machine, and the certificate says so. 3. **The working disk ends with the instance.** During a job, customer data is on the instance's local NVMe disk. A second encryption layer protects the data. The key of that layer exists only in memory. The hardware encryption sits under that layer, and AWS destroys its keys at termination. The root volume contains only the signed image, and Hangar deletes the root volume with the instance. Hangar never writes customer data to EBS. 4. **The new instance proves what it is.** The image boots under UEFI Secure Boot with Hangar's keys, from a read-only verified root. NitroTPM measures the boot. At slot start, the instance returns a Nitro-signed attestation over a nonce that the control plane chose. The control plane compares the boot measurements with the signed reference values for the release. The attestation digest goes on the certificate. 5. **The GPU is checked and scrubbed** before any customer code runs, in the order below. 6. **The network and the credentials are fresh, and egress follows the slot's allowlist.** The slot's network and credentials work as follows: - Each slot has its own security group with no inbound access, as the outer wall. - Inside the instance, a job's container reaches exactly the names on the account's egress allowlist, over HTTPS. The worker answers the container's name lookups for listed names only. The worker receives every HTTPS connection and checks the server name against the list. The worker resolves the name itself and connects to the destination. The worker passes the job's encrypted traffic through unchanged. - The worker refuses every other connection and lookup. Each refusal is a signed record on the certificate. - Instance metadata answers version 2 requests from the host only. - The instance has no role. After the attestation passes, the worker receives credentials scoped to the slot's storage and image for 15 minutes. The worker reaches the control plane only over TLS 1.3, to the control plane's pinned certificate, with the per-slot credential. 7. **The wipe record is written and signed** before Hangar mounts the next customer's data. The record shows the instance, the attestation, the scrub result, the release manifest digest and the times. The record is part of the certificate. ## The GPU check and scrub, in order The worker runs these steps on every slot, in this order, and records each step with every value that the worker compared. Any mismatch, pending row remap or uncorrected memory error stops the slot before customer code runs. Hangar then alerts the operator and returns the customer's credit. 1. `gpu_identity`: the worker compares each GPU's UUID and serial, VBIOS, InfoROM image and driver version with the block's signed acceptance record and the release. Uncorrected ECC errors must be zero, MIG must be off, and no row remap may be pending or failed. 2. `no_process`: no process holds any GPU. 3. `scrub`: the worker allocates all free device memory and counts its non-zero words. The worker fills the memory with zeros and reads the memory back. The worker then writes a test pattern and reads the pattern back, to prove that the reading counts. NVIDIA documents that freed device memory is not cleared, so the read-back is the evidence. 4. `diagnostic`: NVIDIA's short diagnostic (`dcgmi diag -r 1`) passes and the health check reads healthy. 5. `persistence`: persistence mode starts on every GPU. 6. `fabric`: on eight-GPU instance types, the NVLink fabric reads complete. On single-GPU types, the record says that the step does not apply, and why. Once for each block, before the first slot, the first instance runs NVIDIA's long diagnostic (`dcgmi diag -r 3`, about 15 minutes). The worker records each GPU that the diagnostic accepts. The worker verifies the driver and firmware reserved memory, the on-chip state and the firmware images against the signed manifest. The scrub covers all device memory that the driver gives to programs. ## The Neuron check, in order On a Trainium2 pool, the worker runs these steps on every slot, in this order, and records each step with every value that the worker compared. The only reset between two customers is a fresh instance, launched for the slot. The worker refuses a reboot in place in code, because the published Neuron documentation gives no call that clears device memory. Any mismatch stops the slot before customer code runs. Hangar then alerts the operator and returns the customer's credit. 1. `accelerator_identity`: the worker compares these values with the block's signed acceptance record and the release: the device count, each device's id and serial, the device name, the logical cores, the logical NeuronCore configuration, the memory size, the firmware build and the driver version. Uncorrected memory errors must be zero, and no component may read unhealthy. 2. `fresh_instance`: the instance is the instance that Hangar launched for this slot, and no program has run on any NeuronCore since the driver loaded. 3. `no_process`: no process uses any Neuron device. 4. `scrub`: the worker allocates the device memory that programs can reach and counts its non-zero words. The worker fills the memory with zeros and reads the memory back. The worker then writes a test pattern and reads the pattern back, to prove that the reading counts. The fresh instance is the boundary, and the scrub is the evidence. 5. `diagnostic`: one neuron-monitor report reads healthy. 6. `interconnect`: on an instance with one device, the record says that the step does not apply, and why. ## Where the boundary sits Hangar's reset covers everything from the hypervisor up: the operating system, the driver, the GPU's memory, the disks, the network and the credentials. AWS operates the physical hosts and the hypervisor in its Sydney Region, under AWS's own assurance. ## How long it takes Hangar publishes the switchover as 15 minutes for every instance type. The switchover sits inside Hangar's time, not the customer's time. Hangar measures every switchover: stop and flush, termination, launch, attestation, the checks above, and staging of the next image and data.