# Agent Substrate: Full Documentation Bundle (llms-full.txt) > Complete Markdown compilation of Agent Substrate's Get Started guides, Concepts, Tutorials, How-to Guides, Reference, Security, and Project documentation. > Canonical index: https://ate.dev/llms.txt ================================================================================ SECTION: GET STARTED PAGE: Overview URL: https://ate.dev/docs/get-started/overview/ MARKDOWN: https://ate.dev/docs/get-started/overview/index.md SOURCE: docs/get-started/overview.md ================================================================================ # Overview > A high-density, low-latency execution runtime for stateful agent workloads. **Agent Substrate** is a secure-by-default agent execution runtime engineered to run millions of stateful sandboxes at **10x+ higher density** than standard container runtimes. Autonomous agents spend most of their lifecycle idle waiting on user input, model responses, or external events; Substrate reclaims compute during those idle periods without losing session state. Substrate decouples the **Actor** (a stateful agent or workload instance) from the **Worker** (a pre-warmed sandbox in a `WorkerPool`). Idle actors are checkpointed to object storage and evicted from compute; when a request arrives at the router, the control plane assigns a ready worker and the node-level supervisor restores the actor's full memory and filesystem state in **under 500ms**—taking the Kubernetes control plane out of the critical path. * **Core Platform Terms at a Glance:** Each actor lives in an `atespace` (`/`), traffic enters via `atenet-router`, node supervision and snapshot streaming are handled by `atelet`, and sandbox execution inside each warm `Worker` is managed by `ateworker` (internally prefixed `ate` across binaries and CRDs). See the **[Glossary](https://ate.dev/docs/reference/glossary/index.md)** for complete definitions. * **Framework Agnostic:** Runs standard OCI containers inside zero-trust sandboxes—supporting ADK, LangChain, Hermes, OpenClaw, MCP servers, and stateful coding environments out of the box. * **Ready to try it?** Jump straight to **[Quickstart](https://ate.dev/docs/get-started/quickstart/index.md)** to launch your first actor locally on [kind](https://kind.sigs.k8s.io/), follow the **[GKE Installation Guide](https://docs.cloud.google.com/kubernetes-engine/ai-ml/install-overview-substrate)** to deploy on Google Kubernetes Engine, or explore the **[Architecture](https://ate.dev/docs/concepts/architecture/index.md)**. ================================================================================ SECTION: GET STARTED PAGE: Quickstart & Installation URL: https://ate.dev/docs/get-started/quickstart/ MARKDOWN: https://ate.dev/docs/get-started/quickstart/index.md SOURCE: docs/get-started/quickstart.md ================================================================================ # Quickstart & Installation > Deploy Agent Substrate on a local kind cluster and invoke your first stateful actor. Deploy the Agent Substrate control plane and the stateful **Counter** sample application on a local [kind](https://kind.sigs.k8s.io/) cluster. > [!NOTE] > **Prerequisites:** Ensure [Go](https://go.dev/doc/install), [`kubectl`](https://kubernetes.io/docs/tasks/tools/), and [Docker](https://www.docker.com/) are installed ([`kind`](https://kind.sigs.k8s.io/) is managed automatically via Go). A Linux host that will run more than about a thousand sandboxes also needs a larger kernel neighbor table (`net.ipv4.neigh.default.gc_thresh1/2/3` = 4096/16384/32768); `hack/create-kind-cluster.sh` warns when the host is at the default of 1024. ## 1. Clone the repository Clone the repository (if you haven't yet) and `cd` to it: ```bash git clone https://github.com/agent-substrate/substrate.git cd substrate ``` ## 2. Deploy Agent Substrate & Create an Actor Run the following commands from the repository root to provision a local `kind` cluster, install Agent Substrate, and create a `counter` actor: ```bash # 1. Create a local kind cluster and image registry (IPv4; IP_FAMILY=dual|ipv6 overrides) hack/create-kind-cluster.sh # 2. Target the local kind cluster in every shell that runs ate-setup export ATE_INSTALL_KIND=true # 3. Install Agent Substrate, PostgreSQL, and snapshot storage (rustfs) go run ./cmd/ate-setup deploy ate-system --credential-provider='{"name":"k8s.io"}' \ --ingress-allowed-spiffe-ids=spiffe://cluster.local/ns/ate-system/sa/atenet-client # 4. Deploy the Counter sample application go run ./cmd/ate-setup deploy demo counter # 5. Install the kubectl-ate CLI plugin and create a stateful actor go install ./cmd/kubectl-ate kubectl ate create actor my-counter-1 -a ate-demo-counter --template counter # 6. Serve a local proxy on port 8000 that sends requests to the actor through the router kubectl ate proxy actor my-counter-1 -a ate-demo-counter ``` > [!IMPORTANT] > `{"name":"k8s.io"}` deploys the bundled credential provider with an empty, default-deny namespace policy. Before an actor can have a credential injected, grant its atespace the namespaces whose Secrets it may read; see [Egress Credential Injection](https://ate.dev/docs/guides/egress-credential-injection/index.md#enable-it). **Teardown:** When finished, remove the local cluster and registry with: ```bash ./hack/delete-kind-cluster.sh ``` ## 3. Snapshot & Telemetry Configuration In `kind`, snapshots are stored in the in-cluster S3-compatible `rustfs` service (`BUCKET_NAME=ate-snapshots`), and telemetry exports to the local OpenTelemetry Collector (`manifests/ate-install/kind/ate-otel-config.yaml`): ```yaml apiVersion: v1 kind: ConfigMap metadata: name: ate-otel-config namespace: ate-system data: OTEL_EXPORTER_OTLP_ENDPOINT: http://opentelemetry-collector.otel-system.svc:4317 OTEL_METRIC_EXPORT_INTERVAL: "10000" OTEL_METRIC_EXPORT_TIMEOUT: "10000" OTEL_LOGS_EXPORTER: otlp --- # ActorTemplate snapshotConfig from the counter demo snapshotConfig: preferredFidelity: SNAPSHOT_FIDELITY_MEMORY ``` The template sets no `storageLocation`, so its snapshots go under ate-api-server's `--snapshot-location`. The kind profile points that at the local `rustfs` bucket, `gs://ate-snapshots/`. ## 4. Verify Your Running Actor With `kubectl ate proxy actor` running in your first terminal, open a **second terminal** and send an HTTP `POST` request to wake and increment your actor: ```bash curl -X POST -i http://localhost:8000/ ``` Run the `curl` command several times—each request routes to `my-counter-1` and increments its stateful in-memory counter. When the actor becomes idle, Substrate checkpoints its memory and filesystem state to storage and releases the worker Pod; the next `curl` request automatically resumes the actor in under 500ms with its counter state intact. > [!TIP] > **Running on Google Kubernetes Engine (GKE)?** Follow the official [GKE Installation Guide](https://docs.cloud.google.com/kubernetes-engine/ai-ml/install-overview-substrate). ================================================================================ SECTION: CONCEPTS PAGE: Architecture URL: https://ate.dev/docs/concepts/architecture/ MARKDOWN: https://ate.dev/docs/concepts/architecture/index.md SOURCE: docs/concepts/architecture.md ================================================================================ # Architecture > The components of Agent Substrate and how actors, workers, and the control plane interact. ## Overview Agent Substrate is a security-hardened, zero-trust runtime for sandboxed agents, engineered to safely run millions of agents and tools with 10x higher density than standard containers. Purpose-built for the era of autonomous agents, Substrate delivers fast resume operations in less than 500ms, with more than 500 activations per second. Agents get isolated compute, identity, networking, and storage. Sensitive credentials are kept out of reach from untrusted code. It supports multiple sandbox technologies, including microVMs and gVisor, while enabling a consistent lifecycle and management of all sandboxes. At its core, Agent Substrate maps a larger set of “actors” (applications such as agents) onto a smaller set of ready “workers”, relying on the fact that agent-like applications tend to be idle most of the time to achieve heavy multiplexing. It provides functionality to manage an actor’s lifecycle (e.g. create/destroy, suspend/resume), to assign actors to workers in real time, and to route incoming traffic to them. ## Problem Statement Agents and "agent-like" workloads are generally very bursty, spending most of their time waiting for input or events, then handling those events, then going back to waiting. The time they spend actually doing work is often very short, and the time they spend waiting can be unbounded. Because they often run untrusted logic, they are often run in sandboxes, which means they are usually single-tenant instances, and there are a great many of them. ## Core Concepts and Approaches ### Terminology: "Actor" Agent Substrate seeks to solve these problems for "agent-like" workloads. Often we simply say "agents", but it's important to clarify that "agent-like" workloads are not necessarily literally AI agents. In most of the docs we instead use the term "actor" to refer to an instance of an agent-like workload. ### Decoupling the Actor Lifecycle from Workers (Kubernetes Pods) Since many agents run untrusted code, we need to run them in sandboxes of some sort. Agent Substrate is designed to support multiple sandbox technologies with [gVisor](http://gvisor.dev) and microVMs such as [Kata Containers](https://katacontainers.io/) being popular options. Both of those happen to support a notion of suspend and resume which is core for the Agent Substrate functionality. Agent Substrate starts with suspend and resume. When an actor is idle, we suspend it, which frees up the resources it was consuming. When an event comes in that needs to be handled, we resume the actor. To avoid the latency of going through the Kubernetes scheduler, we pre-start long-running "worker" Pods, which are just sandboxes waiting for work. When an event comes in, we assign it to a worker, which resumes the actor in that worker. We can now multiplex a larger number of actors onto a smaller number of workers, which allows us to achieve higher density and efficiency, while also reducing latency. ### A Focused Control Plane Agent Substrate includes a small, focused control-plane component which focuses on high scale, QPS and low-latency suspend/resume operations for actors (Agent Sandboxes). It relies on the Kubernetes control plane for infrastructure and worker (Pods) provisioning which are low-frequency operations that align well with Kubernetes' strength in the infrastructure lifecycle management and workload resource isolation. ### Agent-Aware Routing In order to achieve on-demand resuming of actors, we need to be able to trap and inspect network traffic. Agent Substrate includes a lightweight, substrate-aware network proxy which can inspect incoming traffic and trigger the resumption of actors as needed. ### Trade-offs and Design Considerations Of course, there's no such thing as a free lunch, and these approaches have trade-offs. First, we trade one problem (many idle Pods consuming resources) for another. Doing suspend and resume at this scale introduces a large data management problem. We need to store the state of millions of actors, which can update frequently. Agent Substrate balances fast local reactivation against durable remote storage: `atelet` keeps a node-local copy of an actor's latest snapshot after suspend so subsequent resumes on that node avoid an object-store round-trip, but local copies are ephemeral (held only until `--snapshot-detach-delay`) and asynchronous suspends risk state loss if the node fails before background upload completes (see [Actor Lifecycle](#actor-lifecycle)). Second, a specialized control plane must integrate cleanly with Kubernetes while enforcing its own security boundaries. `ate-api-server` authenticates callers via mTLS and OIDC and authorizes every RPC per atespace with an embedded OpenFGA model, while `atecontroller` reconciles declarative `WorkerPool` CRDs into Kubernetes Deployments. Third, with actors multiplexed onto and off of workers, operators need actor-level and pool-level visibility. Agent Substrate emits OpenTelemetry metrics, structured logs, and lifecycle events across its components so operators can monitor worker capacity, track suspend/resume latency, and correlate activity across an actor's lifetime through container logs (see the [Observability](https://ate.dev/docs/guides/observability/index.md) guide). ## North Star Metrics Agent Substrate is an ambitious effort. To keep us focused on the right problems, we have identified a few north star metrics and targets. * Activation Latency: The time from when a wakeup event is received to when the agent can receive traffic. - Target: 100ms at 95th percentile * Scale: The total number of agents, active and idle, that can be supported in a single cluster. - Target: 1 billion * Throughput: The number of wakeup events that can be handled per second in a single cluster. - Target: 1000/second ## Personas There are a few different personas that interact with the system: 1) **Cluster admins**: These are the people who own the Kubernetes cluster and are responsible for its overall health and performance. If the Agent Substrate needs more capacity, these are the folks who manage things like cluster autoscaling, node provisioning, and so on. They probably don't need to interact directly with the Agent Substrate much, if ever. 2) **Substrate admins**: These are the people who set up and own the Agent Substrate instance(s) in a Kubernetes cluster. They are obviously aware that it is running in a Kubernetes cluster, and are responsible for configuring the Kubernetes resources for it (such as WorkerPools). 3) **Agent developers**: These are the people who deploy agents into a substrate for users or higher-level systems to consume. They might have to be aware that they are using Kubernetes (some concepts are represented as CRDs, such as WorkerPools), or they might be using a higher level API which itself uses Agent Substrate under the hood. ActorTemplates are managed through the substrate API (`kubectl-ate`), not through Kubernetes. 4) **Agent users**: These are the people who interact with agents running in the substrate. They might be end-users of an application that is built on top of Agent Substrate, or they might be higher-level systems that are using Agent Substrate as a building block. They should not need to be aware that they are using Kubernetes at all. ## Relationship to Kubernetes Kubernetes is the industry standard platform for running modern workloads. It is built to support a wide array of workloads, and it can scale to very large clusters. Agent Substrate leverages Kubernetes for infrastructure provisioning and worker lifecycle management (Kubernetes Pods). It builds on top of Kubernetes features like Pods and Pod autoscaling, while Agent Substrate provides agent-specific scheduling and control to achieve lower latency. Using Kubernetes as the underlying system enables consistent infrastructure management across all workloads types that are required for end to end agentic deployments and allows holistic infrastructure optimizations for RL scenarios that span agentic, inference and training cycles. ### Why do we need a specialized control plane for Agent Substrate? * Idle Pods still consume resources. While Kubernetes is very scalable, compute capacity is finite and has real costs. Whether we're talking about CPU time, memory space, or just the number of Pods per node, agent-like workloads are terrible for efficiency. * The Kubernetes API server is not designed to handle millions of resources. It excels at reconciling resources asynchronously across many controllers, but it is not so good at storing very large numbers of discrete resources, or for handling a huge volume of write traffic. * Scheduling a workload on Kubernetes requires several asynchronous processes to converge, plus several network hops, plus image pulling and other steps, which can add up. Getting a Pod running in a couple of seconds is great when that Pod will run for hours or days, but for a workload that will run for milliseconds to low-digit seconds, that latency is unacceptable. * State management is difficult. Kubernetes provides an API for managing state (PersistentVolumes), but it is not designed for millions of volumes, with widely varying amounts of data, being attached and detached at high Speed. Using a specialized Agent Substrate control plane for Actor allows us to achieve high scale and low latency control. We still rely on Kubernetes for infrastructure provisioning, such as worker pods, and workload isolation, which is what Kubernetes excels at. ## High-Level Design A substrate admin deploys the Agent Substrate into a Kubernetes cluster. They configure the WorkerPool(s) that will be used to execute actors and prepare the substrate's control-plane for use. Worker Pods are started, waiting for assignments. An agent developer defines an ActorTemplate (an ate API resource) which describes what it means to instantiate that actor - what OCI image to run, how much memory it needs, behavioral parameters, etc. The Agent Substrate uses that ActorTemplate to create a "golden snapshot" of the actor, which will be used to fast-start instances of that actor in the future. An agent user (or a higher-level system) makes a request to instantiate an actor. They specify which ActorTemplate to use and other parameters. The Agent Substrate creates an Actor record in its control-plane storage, which tracks the state of that actor (suspended or running). When a request comes in for that actor instance, the Agent Substrate's proxy intercepts the request. It looks in the control-plane to determine if that actor is currently running or not. If not, it assigns the actor to a worker, which involves telling a node-level component (the "atelet") to restore the actor's most-recent snapshot into the worker on that node. The request is then forwarded to the actor. Eventually, the user (or the higher-level system) is done with the actor, or it has been idle for a while. It can request the Agent Substrate to suspend the actor, taking a snapshot and freeing up the worker. The next time a request comes in for that actor, it will be resumed again, possibly on a different worker. ## API Resource Models Agent Substrate categorizes resources into two groups based on their persistence requirements and the frequency of state transitions. ### System Configuration (Declarative) These resources define the intended state of the system. They are used for administrative operations and actor environment definitions. * **WorkerPool** (Kubernetes CRD): Defines a pool of "warm" compute capacity. It manages a fleet of standby worker pods initialized and ready to receive resumed actor states. Optional `spec.template` fields configure worker pod node selection, tolerations, priority class, and node affinity. * **SandboxConfig** (Kubernetes CRD): A cluster-scoped resource that lists the sandbox runtime versions and assets for a sandbox class (`gvisor` or `microvm`). Each `WorkerPool` resolves a `SandboxConfig` for its class. * **ActorTemplate** (ate API resource): An immutable definition of an actor-version. It encapsulates the container image, configuration, and environment required to generate a "golden" snapshot. ActorTemplates are created and managed through the substrate gRPC API (e.g. `kubectl ate create actor-template`) and stored in the control-plane state store; they are not Kubernetes objects. ### Dynamic Instance State (Database-based) These resources represent the high-frequency, ephemeral state of individual actors and workers. They are stored in a high-performance, low-latency state store (PostgreSQL) to support real-time operations. * **Actor**: A specific instance of an ActorTemplate. An Actor record tracks its globally unique identifier, physical location (Worker IP), lifecycle state (such as `RUNNING` or `SUSPENDED`), and version-specific state metadata. * **Worker**: A representation of a worker pod in a WorkerPool. It tracks the worker's unique identifier, status (`ACTIVE`, `UNAVAILABLE`, or `DRAINING`), capacity and allocated resources, and the Actors it currently hosts (each represented by an `ActorAssignment` subresource). ### Architectural Rationale Agent Substrate utilizes this dual-layer model to optimize for both Reliability and Performance: 1. **Scalability**: Offloading the high-frequency, high-scale management of millions of actors to a specialized state store prevents overwhelming the primary cluster control plane with thousands of updates per second. 2. **Latency**: Achieving 100ms resumption requires low-latency state lookups and atomic worker assignments that bypass the eventual consistency and variable latency of standard Kubernetes API servers. 3. **Governance**: Keeping infrastructure resources (WorkerPools, SandboxConfigs) as Kubernetes objects allows platform teams to apply familiar RBAC, auditing, and policy enforcement to the underlying infrastructure, while workload definitions (ActorTemplates) are managed through the substrate API instead, which authenticates and authorizes callers itself (see [authentication.md](https://ate.dev/docs/guides/authentication/index.md) and [authorization.md](https://ate.dev/docs/guides/authorization/index.md)). ### Resource Model The Kubernetes resources and control-plane records described above, with their relationships and multiplicities (UML class diagram): ```mermaid classDiagram namespace kube-apiserver { class WorkerPool { <> } class SandboxConfig { <> } class Deployment class WorkerPod { ateworker } } namespace ate-api-server { class ActorTemplate { <> } class Actor { <> status.state status.snapshots } class Worker { <> status.state status.ips } class ActorAssignment { <> } } ActorTemplate ..> WorkerPool : worker_selector (labels) WorkerPool ..> SandboxConfig : sandboxClasses WorkerPool ..> Deployment : reconciled by atecontroller Deployment "1" *-- "*" WorkerPod : manages Actor "*" --> "1" ActorTemplate : actor_template Worker "1" *-- "*" ActorAssignment ActorAssignment "0..1" --> "1" Actor : actor Worker "1" --> "1" WorkerPod : maps to ``` ## System Components ### Control Plane (`ate-api-server`) The brain of the system. It exposes a gRPC API for the data plane and CLI to manage actor lifecycles. * **State Store**: Tracks the mapping of Actors to Workers in a PostgreSQL store. * **Scheduler**: Selects a ready worker for a resumption request. * **Workflow Engine**: Orchestrates the multi-step Resume/Suspend sequences (lock acquisition, storage download, sandbox restore). * **Authentication & Authorization**: [Authenticates](https://ate.dev/docs/guides/authentication/index.md) every RPC and [authorizes](https://ate.dev/docs/guides/authorization/index.md) it per atespace with an embedded OpenFGA model. ### Node Supervisor (`atelet` + `ateworker`) The node-level subsystem manages the physical execution of sandboxes and the movement of snapshots. * **atelet**: A lightweight supervisor running on each node as a DaemonSet. It manages local worker pods and communicates with the Control Plane. * **ateworker**: The worker container image (`ateworker-gvisor` or `ateworker-microvm`, one per sandbox class) that runs inside each physical worker pod. It provides a gRPC interface for `atelet` to trigger `RunWorkload`, `CheckpointWorkload`, and `RestoreWorkload` operations. This separation ensures that the physical pod lifecycle remains decoupled from the sandboxed agent process. * **Lifecycle Management**: The `ateworker` process invokes the sandbox runtime to checkpoint or restore processes within the physical pod boundaries — `runsc` for gVisor, or the Kata + Cloud Hypervisor stack for microVMs. (Note: the gVisor backend currently requires a `runsc` version with the `--allow-connected-on-save` flag to work around a bug in networking resumption during checkpointing.) * **Storage Mover**: External snapshots move through an *object-store plugin*, a sidecar that serves the `objectstore.v1` gRPC API over a Unix socket in its pod. `atelet`'s sidecar (`NodeProvider`) uploads and fetches a snapshot's files; `ate-api-server`'s sidecar (`ControlProvider`) copies and deletes whole snapshots. Neither main process talks to object storage for snapshots. The in-tree plugin, `cmd/objectstore-plugin`, backs the API with GCS or S3; `ATE_STORAGE_BACKEND` on the sidecar (unset or `gcs`, or `s3` with the `AWS_*` variables) selects which, see `manifests/ate-install/atelet.yaml`, `ate-api-server.yaml`, and the kind overlay for an S3 example. The same `atelet` sidecar serves `AssetProvider`, which fetches every sandbox asset; `atelet` reads a `gs://` asset anonymously only when the plugin cannot serve it, which covers public buckets such as `gs://gvisor`. The plugin stages each download where `atelet` cannot read it, and copies it into `atelet`'s cache only if its sha256 matches; it refuses any URL under a snapshot location. `atelet` verifies the sha256 again and has no object storage credentials or settings of its own. * **Sandbox Asset Cache**: `atelet` keeps one node-local, content-addressed cache of the sandbox binaries every actor runs on and gives each actor hard links into it, prewarms each local worker's default runtime and keeps it linked under the worker's directory, and evicts to a byte budget without taking anything an actor or a worker still uses. See the [Sandbox Asset Cache](https://github.com/agent-substrate/substrate/blob/main/docs/concepts/sandbox-asset-cache.md) page. ### Sandbox Classes A `WorkerPool` selects a **sandbox class** (`spec.sandboxClasses[].name`), and each class has a matching `ateworker` image. The sandbox binaries themselves are not baked into the worker image. They come at runtime from the cluster-scoped [`SandboxConfig`](https://ate.dev/docs/reference/api-guide/index.md#3-sandboxconfig-the-sandbox-itself) the `WorkerPool` resolves for its class (an explicit `configRef`, else the class default). Each snapshot records the sandbox runtime it was taken on, and a memory restore runs on a worker that still offers that runtime, so restores stay reproducible across runtime upgrades. For workload-visible differences between the classes, see [Sandbox Classes](https://ate.dev/docs/reference/sandbox-classes/index.md). * **gVisor** (`ateworker-gvisor`, the default): Runs the workload under `runsc` for kernel-level sandboxing. Suspend and resume use gVisor's native checkpoint/restore of the sandboxed process tree. `DurableDir` volumes and each container's rootfs overlay are saved as separate filesystem checkpoint bundles (`fs/` and `rootfs/`). A `MEMORY` snapshot splits them out of the process checkpoint. `VOLUMES` and `ROOTFS` snapshots save only the bundles they need with `runsc fscheckpoint`, without memory, and restore them on a cold boot. * **microVM** (`ateworker-microvm`): Runs the workload inside a [Kata Containers](https://katacontainers.io/) guest on the [Cloud Hypervisor](https://www.cloudhypervisor.org/) VMM. Suspend and resume capture a memory-only VM snapshot and restore it on-demand using `userfaultfd` memory demand-paging. Container rootfs writes are host-backed: the overlay is assembled on the host (read-only OCI image lower plus a per-actor writable upper) and served to the guest over the single virtio-fs share, so they cost reclaimable host page cache rather than guest RAM, and `ROOTFS` and `MEMORY` snapshots ship the upper as its own tar (a `ROOTFS` restore puts it back and cold-boots on it). `DurableDir` volumes travel over that same share and are likewise shipped as a tar, so a `VOLUMES` snapshot can capture them without any guest memory. Each volume is a subdirectory of the share, so an actor can have several at no extra cost in devices. ### Networking Stack (`atenet` + `atunnel`) Handles actor-aware ingress routing, on-demand resumption, and policy-enforced egress. * **Ingress Routing**: `atenet-router` runs Envoy with an `ext_proc` external processor. A higher-order system connects to the router and supplies the Actor target in `ate-target-actor` as `/`. The ext_proc calls the Control Plane to resume the Actor and resolve its current worker assignment. `Host` remains application authority and does not select the Actor. * **Worker Tunnel**: After resolving the assignment, `atenet-router` opens an mTLS `CONNECT` tunnel to the worker's `atunnel` listener on port 8443. `atunnel`, hosted by `ateworker`, forwards the request to the active Actor over its private veth interface. Worker pod port 80 is not a direct Actor ingress path. With `--upstream-connect=false`, the Envoy router instead reaches `atunnel` over HTTPS on port 443. * **Arbitrary-Port Ingress**: A client reaches a port on the Actor other than its default (80) with an HTTP `CONNECT` request naming the port in the authority (e.g. `CONNECT :9090`), rather than a separate header or field. `atenet-router` terminates the `CONNECT` on a dedicated listener and re-enters the tunneled traffic through the same ingress path as ordinary requests, so each request inside a long-lived tunnel still resumes the Actor and re-routes it independently if it moves workers. Only HTTP(S) traffic over the tunnel is supported today; raw TCP or other non-HTTP protocols on a non-default port are not yet reachable this way. * **Egress Filtering and Credential Injection**: Outbound traffic from an Actor is tunneled through `atunnel` to `atenet-egress`, which enforces destination policy and resolves credential replacement effects without placing ambient secrets in the sandbox (see the [Network Egress Contract](https://ate.dev/docs/guides/network-egress/index.md) and [Egress Credential Injection](https://ate.dev/docs/guides/egress-credential-injection/index.md)). * **Latency**: The data plane is optimized for low-latency activation by bypassing Kubernetes' eventual consistency and performing atomic physical assignments. ### System Infra Isolation System infra components, such as ate-api-server, atenet-router, atenet-egress and postgres, can be kept on nodes of their own, away from the workers, so a burst of worker or actor load cannot starve them. Under `--cordon-system-infra-components`, these workloads select a node label and tolerate the matching `NoSchedule` taint: * `ate.dev/workloadType=ate-system-infra`: a small pool shared by every system infra workload except postgres. Each workload's replicas are spread across its nodes on a best-effort basis. * `ate.dev/workloadType=ate-postgres`: a one-node pool for postgres alone, in the zone of the postgres volume. ## Actor Lifecycle The lifecycle of an actor follows a state-driven sequence. A request reaches an actor through the networking stack, which resumes it onto a worker if it is suspended (UML sequence diagram): ```mermaid sequenceDiagram actor Client participant Gateway as atenet-router participant API as ate-api-server participant Atelet as atelet participant Tunnel as ateworker / atunnel participant A as Actor participant Store as snapshot storage Client->>Gateway: HTTP request (ate-target-actor) Gateway->>API: ResumeActor(atespace, actor name) API->>Atelet: Restore Store-->>Atelet: download snapshot Atelet->>Tunnel: RestoreWorkload Note over Tunnel: restore sandbox and activate Actor network Tunnel-->>Atelet: ready Atelet-->>API: worker pod IP API-->>Gateway: worker assignment Gateway->>Tunnel: mTLS CONNECT tunnel to worker port 8443 Tunnel->>A: forward request to Actor port A-->>Tunnel: response Tunnel-->>Gateway: response Gateway-->>Client: response Note over API,Store: later: an explicit SuspendActor checkpoints the actor and frees the worker ``` An Actor's `state` moves through these states (UML state machine diagram): ```mermaid stateDiagram-v2 [*] --> SUSPENDED : CreateActor SUSPENDED --> RESUMING : ResumeActor RESUMING --> RUNNING : restore / boot complete RUNNING --> SUSPENDING : SuspendActor SUSPENDING --> SUSPENDED : checkpoint complete SUSPENDED --> [*] : DeleteActor ``` Both synchronous and asynchronous `SuspendActor` pass through `SUSPENDING`, leave the actor in `SUSPENDED`, and keep a local copy of the snapshot on the node (`locality`). They differ in when the snapshot is uploaded to object storage: * After an asynchronous suspend (the default), `SuspendActor` returns as soon as the local checkpoint is written, with a `PENDING` `durableSnapshot`. `atelet` uploads the snapshot in the background (`--snapshot-upload-delay`) and reports it, which marks the `durableSnapshot` `COMPLETED`. * After a synchronous suspend, `atelet` uploads the snapshot before `SuspendActor` returns, so the `durableSnapshot` is immediately `COMPLETED`. * While the latest snapshot has `locality`, `ResumeActor` restores from the local files on that node. `--snapshot-detach-delay` after the suspend (or during `ResumeActor` when that node has no room), `atelet` deletes the local files (uploading the snapshot first if needed), clearing `locality` and `status.assignedNode`. `SuspendActor` on a `SUSPENDED` actor returns it as it is, in either mode. ### Phase 1: Creation (`CreateActor`) A user or framework calls `CreateActor` with a unique ID and a reference to an `ActorTemplate`. * **State**: The actor is registered in the database with state `ACTOR_STATE_SUSPENDED`. * **State**: The record is initialized in the database with the metadata and **Golden Snapshot** (Version 0) reference derived from the associated ActorTemplate. This ensures the actor can be instantly hydrated into a warm worker upon its first request. ### Phase 2: Activation (`ResumeActor`) Triggered by an inbound request at the Gateway or an explicit API call. 1. **Trigger**: The Gateway pauses the request and asks the Control Plane for the actor's location. 2. **Assignment**: The Control Plane claims a warm worker from the `WorkerPool`. 3. **Hydration**: The `atelet` supervisor coordinates with the `ateworker` process inside the worker pod to restore the ActorTemplate's golden external snapshot (for first-run) or the Actor's own external snapshot (for recurring runs) into the sandbox. 4. **State**: State transitions to `ACTOR_STATE_RUNNING`. The actor now has an active Worker IP. 5. **Response**: The Control Plane returns the worker assignment to the Gateway. The Gateway opens an authenticated tunnel to `atunnel` on that worker, which forwards the original request to the active Actor. ### Phase 3: Hibernation (`SuspendActor`) Triggered by an explicit `SuspendActor` call. 1. **Checkpoint**: The `atelet` instructs `ateworker` to freeze the process and capture a memory+disk snapshot. 2. **Persistence**: The `atelet` keeps a local copy of the snapshot on the node and uploads it to durable storage (e.g., GCS). With `DURABILITY_MODE_SYNCHRONOUS` the upload finishes before `SuspendActor` returns. With the asynchronous default `SuspendActor` returns once the local checkpoint is written, and `atelet` uploads it in the background. 3. **Reclaim**: The physical worker is wiped and returned to the `WorkerPool`. 4. **Release**: A synchronous suspend deletes the external snapshot the Actor held before it from object storage. An asynchronous suspend keeps that snapshot, since its own is not yet durable. It is deleted by the first suspend after a newer snapshot is durable, or by `DeleteActor`. A snapshot borrowed from a tag is left alone, since the tag owns it. 5. **State**: State transitions back to `ACTOR_STATE_SUSPENDED`, and the Actor's `status.snapshots` records the external snapshot it resumes from. Snapshots may be given tags owned and addressed by an Atespace. The same tag name may exist in different Atespaces. A tag is an immutable alias and retention pin: it holds its own copy of the external snapshot, made at creation, so it outlives the Actor that took it, and publishing it permits reuse from other Atespaces without changing its `atespace/name` address. Deleting a tag deletes that copy; an Atespace with tags cannot be deleted until they are. ### Phase 4: Recovery (`RevertActor`) and Deletion (`DeleteActor`) An actor in `ACTOR_STATE_RUNNING` or `ACTOR_STATE_CRASHED`, or an `ACTOR_STATE_SUSPENDED` actor with a node-local latest snapshot, can be reverted back to `ACTOR_STATE_SUSPENDED` at its durable snapshot via `RevertActor`. Reverting terminates any active sandbox, discards any node-local checkpoint or partial in-progress snapshot, and preserves the actor's last completed external snapshot so a subsequent `ResumeActor` restores from that checkpoint. An actor in any state (such as `ACTOR_STATE_RUNNING`) can be deleted from the Control Plane. The workflow terminates the running containers on the worker, detaches mounted volumes, and frees the worker assignment before deleting the record. After deletion, the state of the actor (i.e., memory+disk snapshots) is garbage collected. ## State Management & Persistence A snapshot captures up to three cumulative layers of actor state, controlled by `snapshotConfig.preferredFidelity` on the `ActorTemplate`: 1. **`VOLUMES`**: The contents of `DurableDir` volumes mounted into the actor's containers. 2. **`ROOTFS`**: `VOLUMES` plus files written to each container's writable rootfs overlay since boot. 3. **`MEMORY`**: `ROOTFS` plus the running process tree and RAM state (`MEMORY` is the default). `MEMORY` and `ROOTFS` state is tied to the `ActorTemplate` (`actorTemplateUid`). `MEMORY` state is also tied to the `sandboxRuntime` version it was captured under. Changing an actor's template while suspended keeps its `VOLUMES` data, and its next resume cold-boots the containers. Resume falls back to `ROOTFS` and cold-boots under the default runtime if no worker that offers the snapshot's runtime version has room. Snapshots are stored durably in object storage (GCS or S3) through `objectstore-plugin`. This allows worker capacity in the `WorkerPool` to be reclaimed while an actor is suspended. ## Security & Isolation See the [Security Overview](https://ate.dev/docs/security/security/index.md) and [Threat Model](https://ate.dev/docs/security/threat-model/index.md) for how Agent Substrate isolates workloads, authenticates and authorizes callers, and enforces network and credential boundaries. ================================================================================ SECTION: CONCEPTS PAGE: Request Parking URL: https://ate.dev/docs/concepts/request-parking/ MARKDOWN: https://ate.dev/docs/concepts/request-parking/index.md SOURCE: docs/concepts/request-parking.md ================================================================================ # Request Parking > How the atenet router holds requests for suspended actors while they resume. ## Summary **Request parking** lets the `atenet` router hold ("park") an inbound request whose target actor cannot be served *yet* because of transient worker-pool saturation, retrying the resume until the actor becomes routable or a bounded wait elapses — instead of immediately returning `503` to the client. ## Motivation When a request arrives for a suspended actor, the router resumes it before routing: ``` Envoy --(ext_proc RequestHeaders)--> router.handleRequestHeaders --> ActorResumer.ResumeActor --> ateapi ResumeActor (gRPC) ``` `ateapi`'s `AssignWorkerStep` claims a worker from the actor's `WorkerPool`. In an oversubscribed system — the core premise of Substrate, where many actors multiplex onto few workers — a burst of traffic can momentarily exhaust the pool. `AssignWorkerStep` then returns `ResourceExhausted: "no worker has room for the actor"`. Previously the router mapped that straight to an HTTP `503` and failed the request. But such saturation is usually momentary: another actor suspends within milliseconds and frees its worker. Failing fast turns a sub-second blip into a user-visible error. ## Behavior With parking enabled (the default), the router treats `ResourceExhausted` and `Unavailable` from `ResumeActor` as **retryable** conditions (alongside the existing `Aborted` concurrent-resume conflict) — a parked request rides out transient pool saturation and control-plane blips (e.g. an ateapi rolling restart) alike. The request is *parked*: the resumer keeps retrying with exponential backoff until either - the resume succeeds (the actor is `RUNNING` and has a worker IP) — the request is then routed normally; or - the **park budget** (`--parked-request-budget`, default `5s`) elapses — the underlying capacity error is returned, surfacing as `503 "actor unavailable: no worker has room for the actor"`. **The budget bounds retries, not a committed resume.** When the budget elapses the router stops starting new resume attempts, but an attempt already in flight is **never canceled**: by then the control plane has committed work to it, and canceling would discard an in-progress restore. Instead the router waits for that attempt's real result: a restore that overshoots the budget (routine under node contention) is **served late** rather than failed, and a late retryable error still surfaces as the capacity `503`. The attempt is bounded by the control plane's own server-side RPC deadline, and Envoy's ext_proc message timeout (`budget + 5s`) remains the ceiling on how long a client is held either way. **Worst-case occupancy.** A parked request's lot slot is held until its caller stops waiting; with the Envoy dataplane that is bounded by the ext_proc message timeout (`budget + 5s`). The resume attempt itself can outlive every caller: it carries no client-side deadline and runs until ateapi's server-side maximum RPC deadline, holding that actor's flight entry. New requests for the same actor during that window do not start another control-plane call — they join the in-flight attempt, and if it has not resolved by their own stream deadline they are ended by the dataplane's timeout rather than a router verdict. To bound resource use and provide backpressure, parked requests are admitted to a **parking lot** of fixed capacity (`--parked-request-max`, default `1024`). A request occupies a slot only from the moment it actually parks — its resume flight's first retryable failure. A request resolved on the flight's first attempt (the actor was already running) never occupies a slot. When the lot is full, a request reaching its park transition is shed with `503 "actor unavailable: router at capacity"` rather than queueing without bound — at the cost of exactly the one resume attempt that revealed it would have to wait. Every parked request holds one ext_proc stream — one active request against Envoy's ext_proc cluster — for its entire wait, while ordinary requests hold one only for a millisecond-scale header exchange. The cluster's circuit breaker is therefore the hard ceiling on concurrent parked requests. By default the router **derives** it as twice `--parked-request-max` (minimum `1024`), so the lot always fits and an equal share of **fast-path headroom** remains — a saturated lot cannot starve requests to already-running actors, at any lot size. The lot upholds the same guarantee on its side: admission happens at the park transition, so fast-path requests never compete for slots (#1081). `--extproc-max-requests` overrides the derivation; explicit values are validated `>= --parked-request-max` at startup, because a breaker below the lot would silently truncate it — Envoy would reject the overflow itself, with 503s that never reach the lot and never count in `parking.rejected`. Concurrent requests for the *same* actor are de-duplicated by the resumer's per-actor flight registry: they share a single in-flight `ResumeActor` call and all park on its result, so a hot actor consumes N parking slots but only one control-plane RPC. **The park budget is per-flight, not per-request.** The budget clock starts when a flight's first caller begins the resume; every later request for the same actor joins that flight and shares its remaining budget and outcome. A request that joins late may therefore see `budget_exhausted` after waiting far less than a full budget itself — the accepted cost of collapsing a hot actor's requests into one control-plane call. (`parking.wait.duration` records each request's *own* parked time, so sub-budget `budget_exhausted` samples are expected under sustained saturation.) ### What is *not* parked Only transient conditions — capacity (`ResourceExhausted`), concurrency (`Aborted`), and control-plane unavailability (`Unavailable`) — are parked. Errors that will not resolve by waiting are returned immediately (fail fast): | Resume result | Behavior | | -------------------------------------- | --------------------------------- | | `OK` | Route to worker | | `Aborted` (concurrent resume) | Retry (always) | | `ResourceExhausted` (no room) | **Park & retry** (when enabled) | | `FailedPrecondition` (e.g. CRASHED) | Fail fast → `503` | | `Unavailable` (control-plane blip) | **Park & retry** (when enabled) | | `NotFound` | Fail fast → `404` | | `DeadlineExceeded` | Fail fast → `504` | | `PermissionDenied` / `Unauthenticated` | Fail fast → `403` / `401` | When parking is **disabled** (`--parked-request-max=0`), the router fails fast: `ResourceExhausted` and `Unavailable` are returned immediately, there is no admission cap, and only `Aborted` (concurrent-resume) conflicts are retried, within a `15s` budget. ### Parked requests survive router shutdown A request parked when the router pod receives SIGTERM is **not** reset: the shutdown sequence keeps the ext_proc server (and, via a preStop handshake, the Envoy sidecar) alive until in-flight streams finish, and the ext_proc drain deadline (`--drain-timeout`) defaults to a value derived from `--parked-request-budget` and is validated at startup to be `>=` the budget — so a parked request always gets its full budget and a normal verdict (routed `200` or capacity `503`) even mid-termination. See the graceful-shutdown knobs (`--drain-delay`, `--drain-timeout`) in `manifests/ate-install/atenet-router.yaml`. ## Configuration | Flag | Default | Meaning | | -------------------------------- | ------- | ------------------------------------------------------------------ | | `--parked-request-budget` | `5s` | Park budget per resume *flight*; requests de-duplicated onto an in-flight resume share its remaining budget (see Behavior). | | `--parked-request-max` | `1024` | Max concurrent **parked** requests (a slot is taken at the park transition, never for a first-attempt lookup); requests parking beyond it are shed (503). `0` disables parking. | | `--parked-request-retry-interval` | `100ms` | Delay before a parked request's first resume retry. | | `--parked-request-retry-factor` | `1.1` | Multiplier applied to the retry delay after each attempt (>= 1). | | `--parked-request-retry-jitter` | `0.1` | Random fraction in `[0, 1)` added per retry to de-synchronize parked requests. | | `--extproc-max-requests` | `0` (auto) | Envoy circuit-breaker `max_requests` for the ext_proc cluster. `0` derives twice `--parked-request-max` (min `1024`); explicit values must be `>= --parked-request-max` (enforced at startup). The excess is fast-path headroom (see Behavior). | The retry backoff deliberately has no cap and no attempt limit: the budget alone bounds the wait. ## Observability **Metrics** (OpenTelemetry, meter `atenet-router`): - `atenet.router.parking.active` — up/down counter: requests currently parked. - `atenet.router.parking.wait.duration` — histogram (seconds) of time spent parked. Recorded **exactly once per parked request**, at the moment its wait ends; never recorded for requests served on their flight's first attempt (those never park), for shed requests (those only increment `parking.rejected`), nor when parking is disabled. The `outcome` label says how the park ended: | `outcome` | When it is set | | ------------------ | --------------------------------------------------------------------------- | | `served` | The resume succeeded and the request was routed to its worker. | | `budget_exhausted` | The park budget elapsed while the resume was still blocked on a retryable condition (pool saturated, a concurrent operation holding the actor, or the control plane unavailable) — the signal that capacity, not a fault, is the bottleneck. | | `canceled` | The client disconnected while parked (request context canceled). | | `timeout` | The request's own deadline expired while parked (distinct from the park budget). | | `error` | The resume failed with a non-retryable error (`NotFound`, `PermissionDenied`, ...). | - `atenet.router.parking.rejected` — counter: requests shed because the lot was full. On `atenet.router.route.duration`, a shed request reports `no_capacity` when its resume failed with `ResourceExhausted`, and `unavailable` otherwise. The `ate.router.outcome` label of this counter has the same value. Because `no_capacity` combines fast shed requests with requests that waited out the parking budget, use `atenet.router.parking.rejected{ate_router_outcome="no_capacity"}` to count the shed requests in it. **Status page** (`/statusz`): a "Request Parking" card shows whether parking is enabled, the current vs. maximum parked count, and the max wait. ================================================================================ SECTION: TUTORIALS PAGE: Stateful Counter URL: https://ate.dev/docs/tutorials/counter/ MARKDOWN: https://ate.dev/docs/tutorials/counter/index.md SOURCE: demos/counter/README.md ================================================================================ # Stateful Counter > Preserve in-memory, root filesystem, and durable volume state across actor suspend and resume cycles. This demo runs a stateful HTTP server (`counter.go`) on Agent Substrate and shows how actor snapshots preserve state across suspend and resume cycles. On every HTTP request, `counter.go` increments three counters: - `preserved memory count` in process memory - `preserved file counter` in `/home/counter/a.txt` on the `data` `durableDir` volume - `preserved rootfs counter` in `/rootfs-counter.txt` on the container root filesystem The demo `ActorTemplate` sets `preferredFidelity` to `SNAPSHOT_FIDELITY_MEMORY`, which captures process memory, root filesystem writes, and durable volumes when an actor suspends. When the actor resumes, all three counters continue from where they left off. Under `SNAPSHOT_FIDELITY_ROOTFS`, the durable volume and root filesystem counters survive while the in-memory counter resets to `1`. Under `SNAPSHOT_FIDELITY_VOLUMES`, only the durable volume counter survives. The demo creates two resources: - `WorkerPool` (`counter`), a Kubernetes custom resource in the `ate-demo-counter` namespace. - `ActorTemplate` (`counter`), a Substrate API resource in the `ate-demo-counter` **atespace**, managed through `kubectl ate`. ## Prerequisites - A Kubernetes cluster with Agent Substrate installed as in the [Quickstart](https://ate.dev/docs/get-started/quickstart/index.md) or [Operator Install Guide](https://ate.dev/docs/guides/operator-install/index.md). On [kind](https://kind.sigs.k8s.io/), keep `export ATE_INSTALL_KIND=true` set in your shell so `ate-setup` targets the local cluster and registry. - `atenet-router` configured with `--ingress-allowed-spiffe-ids=spiffe://cluster.local/ns/ate-system/sa/atenet-client` (`ATE_INGRESS_ALLOWED_SPIFFE_IDS`) so `kubectl ate proxy actor` can authenticate to the router. - A snapshot location set at install (`SNAPSHOT_LOCATION`). `ATE_INSTALL_KIND=true` sets this automatically to `gs://ate-snapshots/`. ## How to Run on Agent Substrate This walkthrough deploys the default `gvisor` sandbox variant (`gvisor-default`). See [Running on microVM](#running-on-microvm) for the `microvm` variant. The [Quickstart](https://ate.dev/docs/get-started/quickstart/index.md) already deploys this demo, creates `my-counter-1`, and starts its proxy on port `8000`. If you ran it, skip to [How to Use](#how-to-use). ### 1. Build and Deploy > [!NOTE] > Do not manually edit `demos/counter/counter.yaml.tmpl` or `demos/counter/counter-template.yaml.tmpl`. `ate-setup` fills in their `${...}` placeholders during deployment. Use `ate-setup` to build the image and deploy the demo to your cluster: ```bash go run ./cmd/ate-setup deploy demo counter ``` This command: - Builds the counter server image using `ko`. - Applies [`counter.yaml.tmpl`](https://github.com/agent-substrate/substrate/blob/main/demos/counter/counter.yaml.tmpl) to create the `ate-demo-counter` namespace and the `counter` `WorkerPool`, then waits for the worker rollout. - Creates the `ate-demo-counter` atespace. - Creates the `counter` actor template through the Substrate API from [`counter-template.yaml.tmpl`](https://github.com/agent-substrate/substrate/blob/main/demos/counter/counter-template.yaml.tmpl). The manifest uses the message's protojson form, which matches `kubectl ate get actor-template counter -a ate-demo-counter -o yaml`. - Waits until the template's golden snapshot is ready. Inspect the deployed template: ```bash kubectl ate get actor-templates -a ate-demo-counter kubectl ate get actor-template counter -a ate-demo-counter -o yaml ``` ### 2. Create a Counter Actor Install the `kubectl-ate` CLI plugin and create an actor named `my-counter-1` in the `ate-demo-counter` atespace from the `counter` template: ```bash go install ./cmd/kubectl-ate kubectl ate create actor my-counter-1 -a ate-demo-counter --template counter ``` ### 3. Proxy to the Actor Start a local proxy that authenticates with `atenet-router` and forwards local port `8000` to port `80` on the actor: ```bash kubectl ate proxy actor my-counter-1 -a ate-demo-counter ``` ## How to Use When you send an HTTP request through the proxy, `atenet-router` resumes the actor onto an available worker pod and routes the traffic. 1. In a separate terminal, send an HTTP POST request to increment the counters: ```bash curl -X POST http://localhost:8000 ``` Expected output: ```text preserved memory count: 1 | preserved file counter: 1 | preserved rootfs counter: 1 ``` 2. Verify that the actor is in `ACTOR_STATE_RUNNING` and assigned to a worker pod: ```bash kubectl ate get actor my-counter-1 -a ate-demo-counter ``` 3. Manually suspend the actor: ```bash kubectl ate suspend actor my-counter-1 -a ate-demo-counter ``` The snapshot stays on the worker's node and uploads to snapshot storage in the background. Pass `--mode=sync` to wait for the upload before returning. Repeat the `curl` command from step 1: ```bash curl -X POST http://localhost:8000 ``` Expected output: ```text preserved memory count: 2 | preserved file counter: 2 | preserved rootfs counter: 2 ``` The actor resumes from its snapshot, potentially on a different worker pod, and all three counters continue from `1` to `2`. Run `kubectl ate get actor my-counter-1 -a ate-demo-counter` again and check the `WORKER POD` column to see which worker pod it resumed on. ## Reaching a non-default port `counter.go` also listens on a second configurable port (`--extra-port=9090` in this demo's template) to exercise arbitrary-port ingress through `atenet-router`. Pass `[local-port:]remote-port` arguments to `kubectl ate proxy actor` to forward additional ports. Stop any earlier proxy on port `8000` first: ```bash kubectl ate proxy actor my-counter-1 8000:80 9090:9090 -a ate-demo-counter ``` Send a request to local port `9090`: ```bash curl http://localhost:9090/ ``` Expected output: ```text hello from extra port 9090 ``` This reaches the same actor's second listener and resumes the actor if it is suspended. **Current limitation**: only HTTP(S) traffic over the tunnel is supported. Raw TCP or other non-HTTP protocols on a non-default port are not reachable through `atenet-router` yet. Delete the actor when finished: ```bash kubectl ate delete actor my-counter-1 -a ate-demo-counter ``` ## Running on microVM You can also run the counter demo on the `microvm` sandbox class (`ateworker-microvm`, which runs a Kata guest on Cloud Hypervisor and requires `/dev/kvm` on worker nodes). This variant uses: - [`demos/counter/counter-microvm.yaml.tmpl`](https://github.com/agent-substrate/substrate/blob/main/demos/counter/counter-microvm.yaml.tmpl): the `WorkerPool` for the `microvm` sandbox class. - [`demos/counter/counter-microvm-template.yaml.tmpl`](https://github.com/agent-substrate/substrate/blob/main/demos/counter/counter-microvm-template.yaml.tmpl): the `counter-microvm` `ActorTemplate` in the `ate-demo-counter-microvm` atespace. `deploy ate-system` stages the microVM guest assets and applies the cluster-wide `microvm` `SandboxConfig` by default (`--sandbox-classes=gvisor,microvm`), so on a cluster installed as in the [Quickstart](https://ate.dev/docs/get-started/quickstart/index.md), deploy the demo with: ```bash go run ./cmd/ate-setup deploy demo counter-microvm ``` Inspect the deployed template: ```bash kubectl ate get actor-templates -a ate-demo-counter-microvm kubectl ate get actor-template counter-microvm -a ate-demo-counter-microvm -o yaml ``` Create, proxy, suspend, and delete the microVM actor in the `ate-demo-counter-microvm` atespace: ```bash kubectl ate create actor my-counter-1 -a ate-demo-counter-microvm --template counter-microvm kubectl ate proxy actor my-counter-1 -a ate-demo-counter-microvm ``` In a second terminal: ```bash curl -X POST http://localhost:8000 kubectl ate get actor my-counter-1 -a ate-demo-counter-microvm kubectl ate suspend actor my-counter-1 -a ate-demo-counter-microvm curl -X POST http://localhost:8000 kubectl ate delete actor my-counter-1 -a ate-demo-counter-microvm ``` ## Attaching an External Volume Pass `--with-external-volume` to attach a CSI-backed external volume mounted at `/external-data`. That flag sets `--validate-existing-file-path=/external-data/test.txt` on `counter.go`, so `curl` returns `failed to read test file: open /external-data/test.txt: no such file or directory` until `/external-data/test.txt` exists on the volume. > [!IMPORTANT] > External volumes are a **preview** feature and require the `ExternalVolumes` preview feature gate plus an adapted CSI driver with a `CSIDriverConfig` (see [CSI Volumes](https://ate.dev/docs/guides/csi-volumes/index.md) and [Deploying CSI Drivers](https://ate.dev/docs/guides/csi-deployment/index.md)): ```bash # 1. Deploy ate-system with the ExternalVolumes preview gate enabled go run ./cmd/ate-setup deploy ate-system \ --preview=ExternalVolumes \ --credential-provider='{"name":"k8s.io"}' \ --ingress-allowed-spiffe-ids=spiffe://cluster.local/ns/ate-system/sa/atenet-client # 2. Install a bundled CSI driver and its CSIDriverConfig: # - NFS (creates StorageClass csi-nfs-sc, requires the nfsd kernel module on nodes): go run ./cmd/ate-setup setup csi nfs # - Or hostpath on kind clusters (creates StorageClass csi-hostpath-sc): # go run ./cmd/ate-setup setup csi hostpath # 3. Delete any existing counter demo (templates are immutable), then deploy with the external volume go run ./cmd/ate-setup delete demo counter kubectl wait --for=delete namespace/ate-demo-counter --timeout=60s go run ./cmd/ate-setup deploy demo counter \ --with-external-volume \ --storage-class=csi-nfs-sc \ --preview=ExternalVolumes # 4. Create the actor with --preview enabled on kubectl ate kubectl ate --preview create actor my-counter-1 -a ate-demo-counter --template counter ``` ## Settings The installer reads these settings in addition to those in [docs/guides/operator-install.md](https://ate.dev/docs/guides/operator-install/index.md#settings), through the same flag, environment variable, and configuration document channels. | Key | Environment variable | Flag | Default | Applies to | |-----------------------------------|-----------------------------------------|--------------------------|------------|-----------------------| | `demo.counter.storageClass` | `STORAGE_CLASS` | `--storage-class` | `standard` | `deploy demo counter` | | `demo.counter.withExternalVolume` | `ATE_DEMO_COUNTER_WITH_EXTERNAL_VOLUME` | `--with-external-volume` | `false` | `deploy demo counter` | ## How to Uninstall Remove the counter demo resources from your cluster: ```bash go run ./cmd/ate-setup delete demo counter # Or for the microVM variant: go run ./cmd/ate-setup delete demo counter-microvm ``` This deletes the actors created from the demo templates, the templates themselves (along with their golden actors and golden snapshots), the demo atespaces, and the worker pools. ================================================================================ SECTION: TUTORIALS PAGE: Interactive Sandbox URL: https://ate.dev/docs/tutorials/sandbox/ MARKDOWN: https://ate.dev/docs/tutorials/sandbox/index.md SOURCE: demos/sandbox/README.md ================================================================================ # Interactive Sandbox > Execute interactive shell commands inside an isolated Alpine container with stateful snapshots. This directory contains a demo of a stateful sandbox execution environment running on Agent Substrate. It runs commands in a sandboxed, isolated container (Alpine Linux) and preserves container filesystem and memory state across suspends and resumes. `sandbox-client` sends each command as `sh -c` to `/process`, which spawns a fresh subprocess per request, so working-directory changes (`cd`) and shell variables reset between commands while files written inside the container persist across requests and snapshots. > [!WARNING] > **Security Disclaimer:** This demo is not secured by any authorization checks, and the sandbox actor will execute any client-provided commands with no validation. Do not deploy this configuration in production or expose it to untrusted networks. ## Components 1. **Sandbox Server (`main.go`)**: The application that runs inside the Agent Substrate actor. It exposes a `/process` endpoint that runs each command in a fresh subprocess, while files in the container persist across requests. 2. **Sandbox Client (`client/`)**: A CLI REPL tool that allows you to interact with the sandbox actor interactively. ## Prerequisites - A Kubernetes cluster with Agent Substrate installed as in the [Quickstart](https://ate.dev/docs/get-started/quickstart/index.md) or [Operator Install Guide](https://ate.dev/docs/guides/operator-install/index.md). - `atenet-router` configured with `--ingress-allowed-spiffe-ids=spiffe://cluster.local/ns/ate-system/sa/atenet-client` (`ATE_INGRESS_ALLOWED_SPIFFE_IDS`); see [Ingress to actors](https://ate.dev/docs/guides/operator-install/index.md#ingress-to-actors). - A snapshot location (`SNAPSHOT_LOCATION`, see [Snapshot storage](https://ate.dev/docs/guides/operator-install/index.md#snapshot-storage)) and image repository (`KO_DOCKER_REPO`, or `ko.dockerRepo` in `--config`) for `ate-setup deploy demo`. On [kind](https://kind.sigs.k8s.io/), `export ATE_INSTALL_KIND=true` configures local defaults (`gs://ate-snapshots/` and `localhost:5001`) automatically. - `kubectl-ate` CLI installed (`go install ./cmd/kubectl-ate`). ## How to Run on Agent Substrate ### 1. Build and Deploy Use `ate-setup` to build the image and deploy the demo to your cluster: ```bash go run ./cmd/ate-setup deploy demo sandbox ``` This command will: - Build the sandbox server image based on Alpine Linux. - Create the `ate-demo-sandbox` namespace and `WorkerPool` (`sandbox.yaml.tmpl`). - Create the `ate-demo-sandbox` atespace and the `sandbox-template` actor template in it (`sandbox-template.yaml.tmpl`, applied with `kubectl ate create actor-template`). - Wait until the template's golden snapshot is built. Inspect the template with: ```bash kubectl ate get actor-template sandbox-template -a ate-demo-sandbox ``` ### 2. Create a Sandbox Actor Create the sandbox actor in the demo's atespace with a chosen name (e.g., `my-sandbox-1`). `--template` names the template in the actor's atespace: ```bash kubectl ate create actor my-sandbox-1 -a ate-demo-sandbox --template sandbox-template ``` ### 3. Proxy to the Actor Start a local proxy to the actor in a separate terminal: ```bash kubectl ate proxy actor my-sandbox-1 -a ate-demo-sandbox ``` ## How to Use the Client Build and run the client REPL: ```bash go build -o bin/sandbox-client ./demos/sandbox/client ./bin/sandbox-client --atenet=localhost:8000 --atespace=ate-demo-sandbox --name=my-sandbox-1 ``` Once in the `sandbox>` prompt, you can run commands: ```text sandbox> ls -la sandbox> pwd sandbox> echo "Hello" > test.txt sandbox> cat test.txt ``` Type `exit` to leave. This will automatically trigger the suspension of the actor. To resume the actor non-interactively and verify `test.txt` persisted across the snapshot, pipe commands into `sandbox-client`: ```bash printf 'cat test.txt\nexit\n' | \ ./bin/sandbox-client --atenet=localhost:8000 --atespace=ate-demo-sandbox --name=my-sandbox-1 ``` To permanently delete the suspended actor: ```bash kubectl ate delete actor my-sandbox-1 -a ate-demo-sandbox ``` ## How to Uninstall To remove the sandbox demo resources (actors, template, atespace, workerpool, and namespace) from your cluster, run: ```bash go run ./cmd/ate-setup delete demo sandbox ``` ================================================================================ SECTION: TUTORIALS PAGE: Jupyter Notebook URL: https://ate.dev/docs/tutorials/jupyter/ MARKDOWN: https://ate.dev/docs/tutorials/jupyter/index.md SOURCE: demos/jupyter/README.md ================================================================================ # Jupyter Notebook > Run an unmodified Jupyter Notebook container that checkpoints on suspend and resumes on browser access. This demo runs a Jupyter Notebook on Agent Substrate. It shows how an unmodified container image (`quay.io/jupyter/base-notebook`) runs as a Substrate actor that checkpoints on suspend and resumes when accessed through your browser. ## Prerequisites - A Kubernetes cluster with Agent Substrate installed as in the [Quickstart](https://ate.dev/docs/get-started/quickstart/index.md) or [Operator Install Guide](https://ate.dev/docs/guides/operator-install/index.md). - `atenet-router` configured with `--ingress-allowed-spiffe-ids=spiffe://cluster.local/ns/ate-system/sa/atenet-client` (`ATE_INGRESS_ALLOWED_SPIFFE_IDS`); see [Ingress to actors](https://ate.dev/docs/guides/operator-install/index.md#ingress-to-actors). - A snapshot location (`SNAPSHOT_LOCATION`, see [Snapshot storage](https://ate.dev/docs/guides/operator-install/index.md#snapshot-storage)) and image repository (`KO_DOCKER_REPO`, or `ko.dockerRepo` in `--config`) for `ate-setup deploy demo`. On [kind](https://kind.sigs.k8s.io/), `export ATE_INSTALL_KIND=true` configures local defaults (`gs://ate-snapshots/` and `localhost:5001`) automatically. ## Deploy the Jupyter Notebook Demo ```bash go run ./cmd/ate-setup deploy demo jupyter ``` ## Creating and Accessing Your Jupyter Actor Once the infrastructure and template are deployed, you can create instances (actors) of Jupyter Notebook. ### 1. Create the Actor First, ensure you have the `kubectl-ate` CLI installed: ```bash go install ./cmd/kubectl-ate ``` The deploy created the `jupyter` actor template in the `ate-demo-jupyter` atespace. Create your `jupyter-notebook` actor there (`--template` names the template in the actor's atespace): ```bash kubectl ate create actor jupyter-notebook -a ate-demo-jupyter --template jupyter ``` ### 2. Access Jupyter via the Proxy 1. **Start the local actor proxy:** Keep this running in a separate terminal: ```bash kubectl ate proxy actor jupyter-notebook 8888:80 -a ate-demo-jupyter ``` 2. **Verify from the command line or open your browser:** Query the Jupyter Server `/api` endpoint to wake the actor and confirm it is serving (`HEAD /lab` via `curl -I` returns `405 Method Not Allowed`): ```bash curl -s http://localhost:8888/api ``` Or open your web browser and visit `http://localhost:8888/lab`. The JupyterLab interface opens without a password because the template passes `--ServerApp.token=''` and `--ServerApp.password=''`. Create a new notebook and try running: ```python print("hello world") ``` ### 3. Suspending and Resuming the Notebook When you are finished using the notebook, checkpoint and suspend the actor: ```bash kubectl ate suspend actor jupyter-notebook -a ate-demo-jupyter ``` Check the actor status to confirm it is suspended: ```bash kubectl ate get actor jupyter-notebook -a ate-demo-jupyter ``` The `STATE` column shows `ACTOR_STATE_SUSPENDED`. To **resume** the notebook, you can either explicitly resume it via CLI: ```bash kubectl ate resume actor jupyter-notebook -a ate-demo-jupyter ``` Or refresh `http://localhost:8888/lab` in your browser (or run `curl -s http://localhost:8888/api`) while the actor is suspended. `atenet-router` automatically resumes the actor from its snapshot and serves the request. ## Clean up Stop the `kubectl ate proxy actor` process in your proxy terminal, then uninstall the demo deployment. This deletes the demo's actors, the `jupyter` actor template, the `ate-demo-jupyter` atespace, the `jupyter` `WorkerPool`, and the `ate-demo-jupyter` namespace: ```bash go run ./cmd/ate-setup delete demo jupyter ``` ================================================================================ SECTION: TUTORIALS PAGE: Multi-Template WorkerPool URL: https://ate.dev/docs/tutorials/multi-template/ MARKDOWN: https://ate.dev/docs/tutorials/multi-template/index.md SOURCE: demos/multi-template/README.md ================================================================================ # Multi-Template WorkerPool > Share a single WorkerPool across multiple ActorTemplates and atespaces using label selectors. This demo shows that **two different actor templates running two different binaries can share a single `WorkerPool` — even when the templates live in different atespaces**. Each template gates on the pool via `workerSelector`, a label selector matched against the pool's labels — pool selection is cluster-wide, not scoped by atespace or namespace. ## Prerequisites - A Kubernetes cluster with Agent Substrate installed as in the [Quickstart](https://ate.dev/docs/get-started/quickstart/index.md) or [Operator Install Guide](https://ate.dev/docs/guides/operator-install/index.md). - `atenet-router` configured with `--ingress-allowed-spiffe-ids=spiffe://cluster.local/ns/ate-system/sa/atenet-client` (`ATE_INGRESS_ALLOWED_SPIFFE_IDS`); see [Ingress to actors](https://ate.dev/docs/guides/operator-install/index.md#ingress-to-actors). - A snapshot location (`SNAPSHOT_LOCATION`, see [Snapshot storage](https://ate.dev/docs/guides/operator-install/index.md#snapshot-storage)) and image repository (`KO_DOCKER_REPO`, or `ko.dockerRepo` in `--config`) for `ate-setup deploy demo`. On [kind](https://kind.sigs.k8s.io/), `export ATE_INSTALL_KIND=true` configures local defaults (`gs://ate-snapshots/` and `localhost:5001`) automatically. ## How to Run on Agent Substrate ### 1. Build and Deploy ```bash go run ./cmd/ate-setup deploy demo multi-template ``` This command will: - Build the `counter` and `fspersist` images using `ko`. - Create one `WorkerPool` (`shared-pool`) in the `ate-demo-multi-template-pool` namespace (`multi-template.yaml.tmpl`). - Create 2 atespaces — `ate-demo-multi-template-counter` and `ate-demo-multi-template-fspersist` — and an actor template in each: `counter` (`counter-template.yaml.tmpl`) and `fspersist` (`fspersist-template.yaml.tmpl`), both selecting the pool via the same `workerSelector` label and applied with `kubectl ate create actor-template`. - Wait until both templates' golden snapshots are built. ### 2. Create one actor per template Each actor goes in its template's atespace (`--template` names the template, resolved in the actor's atespace): ```bash # Install the CLI as a kubectl plugin if not already installed go install ./cmd/kubectl-ate # Create two actors from different templates, one per atespace. kubectl ate create actor c1 -a ate-demo-multi-template-counter --template counter kubectl ate create actor f1 -a ate-demo-multi-template-fspersist --template fspersist ``` ### 3. Proxy to the actors Start a local proxy for each actor: ```bash kubectl ate proxy actor c1 8000:80 -a ate-demo-multi-template-counter & PROXY_C1=$! kubectl ate proxy actor f1 8001:80 -a ate-demo-multi-template-fspersist & PROXY_F1=$! ``` ## How to Use When you send an HTTP request through the proxy, `atenet-router` automatically activates (resumes) the actor onto an available worker pod and routes the traffic: ```bash # counter binary curl -s http://localhost:8000 # -> preserved memory count: 1 | preserved file counter: 1 | preserved rootfs counter: 1 # fspersist binary curl -s http://localhost:8001 # -> pod: # --- history --- # pod= | count=0 | time= ``` Confirm both actors landed on workers in the one `shared-pool`: ```bash kubectl ate get workers ``` The `counter` actor increments its in-memory, durable-volume (`/home/counter/a.txt`), and rootfs (`/rootfs-counter.txt`) counters on each request, while `fspersist` prepends a line to its history file on each request. Suspending and re-requesting an actor preserves that state across the snapshot/restore cycle: ```bash kubectl ate suspend actor f1 -a ate-demo-multi-template-fspersist curl -s http://localhost:8001 # history persists; count keeps climbing ``` ## How to Uninstall Stop the background proxy processes and remove the demo. This deletes the actors, the templates, both atespaces, and then the pool and its namespace: ```bash kill "$PROXY_C1" "$PROXY_F1" go run ./cmd/ate-setup delete demo multi-template ``` ================================================================================ SECTION: TUTORIALS PAGE: Request Parking URL: https://ate.dev/docs/tutorials/parking/ MARKDOWN: https://ate.dev/docs/tutorials/parking/index.md SOURCE: demos/parking/README.md ================================================================================ # Request Parking > Hold and retry inbound requests during momentary WorkerPool saturation instead of failing with 503. This demo shows the **request parking** feature of the `atenet` router: when the `WorkerPool` is momentarily saturated, the router *holds* (parks) an inbound request and retries the resume until a worker frees up — instead of failing fast with a `503`. The setup is deliberately **oversubscribed**: a 2-worker pool with several actors running the `counter` binary from the counter demo. Run `kubectl ate get actors -a ate-demo-parking` to see which worker pod each actor is assigned to. See [docs/concepts/request-parking.md](https://ate.dev/docs/concepts/request-parking/index.md) for the design. ## Prerequisites - A k8s cluster with Agent Substrate installed as in the [Quickstart](https://github.com/agent-substrate/substrate/blob/main/README.md#quickstart-development) (with `ATE_INSTALL_KIND=true` exported in your shell) or [GKE Quickstart](https://github.com/agent-substrate/substrate/blob/main/README.md#gke-quickstart). - A snapshot location set at install (`SNAPSHOT_LOCATION`). The demo stores its snapshots there. ## How to Run on Agent Substrate ### 1. Build and Deploy ```bash go run ./cmd/ate-setup deploy demo parking ``` This command will: - Build the `counter` workload image using `ko`. - Create the `ate-demo-parking` namespace and a **2-replica** `WorkerPool` (`parking`, from `parking.yaml.tmpl`). - Create the `ate-demo-parking` atespace and the `parking` actor template in it (`parking-template.yaml.tmpl`, applied with `kubectl ate create actor-template`). - Wait until the pool is rolled out and the template's golden snapshot is built. ### 2. Create more actors than workers Actors live in the demo's **atespace** (`ate-demo-parking`). `--template` names the template, resolved in the actor's atespace: ```bash # Install the CLI as a kubectl plugin if not already installed go install ./cmd/kubectl-ate # 4 actors share a 2-worker pool -> oversubscribed. for id in p1 p2 p3 p4; do kubectl ate create actor "$id" --atespace ate-demo-parking --template parking done ``` ### 3. Proxy to the actors ```bash kubectl ate proxy actor p1 8001:80 -a ate-demo-parking & kubectl ate proxy actor p2 8002:80 -a ate-demo-parking & kubectl ate proxy actor p3 8003:80 -a ate-demo-parking & ``` ## How to Use Parking is **on by default** (`--parked-request-budget=5s`, `--parked-request-max=1024`), so the cluster you just deployed already parks. ### A. Watch a 503 become a served request Fill both workers by requesting two actors, leaving them `RUNNING`: ```bash curl -s http://localhost:8001 curl -s http://localhost:8002 kubectl ate get workers # both workers are now bound to p1 and p2 kubectl ate get actors -a ate-demo-parking # p1,p2 RUNNING; p3,p4 SUSPENDED ``` Now request **p3** with timing. The pool is full, so this request **parks** — the `curl` hangs while the router retries the resume: ```bash curl -s -w '\n-> HTTP %{http_code} in %{time_total}s\n' http://localhost:8003 ``` While that is hanging, in a **second terminal** free a worker by suspending p1 (within the 5s park budget): ```bash kubectl ate suspend actor p1 --atespace ate-demo-parking ``` Back in the first terminal, the parked request now completes with **`HTTP 200`**, and `time_total` shows how long it waited for the worker. With parking disabled, that same request would have returned **`503`** immediately (see section D). ### B. See it under load `load.sh` drives one concurrent request→suspend loop per actor. Because there are more actors than workers, the pool stays saturated; the suspend at the end of each loop frees a worker for a competitor (standing in for an actor going idle). The tally shows parking absorbing the contention: ```bash ./demos/parking/load.sh # 30s, actors p1 p2 p3 p4 # ==> results # total requests : 142 # 200 OK : 142 # 503 unavailable: 0 # 200 latency : avg 0.43s, slowest 6.12s <- parked requests sit here # => 0 failures under saturation: parking absorbed the contention. ``` ### C. Observe parking state The router's `/statusz` page has a **Request Parking** card. Port-forward the status port and read it (run this while `load.sh` is generating load to see a non-zero `active`): ```bash kubectl -n ate-system port-forward deployment/atenet-router 4040:4040 curl -s 'http://localhost:4040/statusz?format=json' | jq .parking # { "enabled": true, "active": 3, "max_parked": 1024, "max_wait": "5s" } ``` The parking metrics are also exported on the router's metrics endpoint (`--metrics-listen-addr`, container port `9090`): `atenet.router.parking.active`, `atenet.router.parking.wait.duration` (labeled by `outcome`), and `atenet.router.parking.rejected`. ### D. Compare with parking disabled Turn parking off to see the old fail-fast behavior. Add the flag to the router container's args: ```bash kubectl -n ate-system patch deployment atenet-router --type=json \ -p='[{"op":"add","path":"/spec/template/spec/containers/0/args/-","value":"--parked-request-max=0"}]' kubectl -n ate-system rollout status deployment/atenet-router ``` Re-run the load test — now transient saturation surfaces as `503`s: ```bash ./demos/parking/load.sh # 503 unavailable: 37 # => 37 requests were shed with 503 (parking off, ...). ``` Re-enable parking by removing that flag again: ```bash kubectl -n ate-system rollout undo deployment/atenet-router ``` > [!TIP] > You can tune parking instead of disabling it: add `--parked-request-budget=10s` or > `--parked-request-max=512` to the same args list. ## How to Uninstall Remove the demo — this deletes the demo's actors (suspending running ones first) and then the template, atespace, pool, and namespace: ```bash go run ./cmd/ate-setup delete demo parking ``` ================================================================================ SECTION: TUTORIALS PAGE: Pluggable Egress Networking URL: https://ate.dev/docs/tutorials/egress/ MARKDOWN: https://ate.dev/docs/tutorials/egress/index.md SOURCE: demos/egress/README.md ================================================================================ # Pluggable Egress Networking > Tunnel and authenticate outbound actor traffic over mTLS using Envoy or agentgateway. This demo shows an Actor's outbound traffic being **transparently tunneled through an egress gateway** and **authenticated by actor identity**, end to end. The same demo runs with either Envoy or [agentgateway](https://agentgateway.dev/). The Actor is a tiny service that accepts `{"url":"..."}`, performs an HTTP `GET`, and returns the upstream response. The Actor believes it is dialing plain HTTP directly — but its egress is intercepted and carried over mTLS to a gateway that verifies who is making the request. ## What it demonstrates ```text ┌─────────────────── worker pod ────────────────────┐ │ Actor (gVisor) │ │ GET http://:80/ (plain HTTP) │ │ │ │ │ ▼ nftables REDIRECT │ │ atunnel egress ──(mTLS with the actor's own │ │ │ certificate + bare CONNECT) │ └────────┼────────────────────────────────────────────┘ ▼ ┌──────────── atenet-egress pod ───────────────────┐ │ Egress Gateway │ │ • downstream mTLS, trusted by actor-id CA │ │ • terminates HTTP CONNECT │ │ • verifies Actor identity with the ATE API │ │ • forwards to the CONNECT authority │ │ │ │ └───────────┼───────────────────────────────────────┘ ▼ real destination (the CONNECT authority, an IP:port) ``` 1. **Guide 1 - gateway accepts CONNECT + mTLS.** `atenet-egress` terminates the actor's mTLS `CONNECT` and tunnels to the requested destination. 2. **Guide 2 — transparent interception.** `nftables` REDIRECTs actor TCP egress into `atunnel`, which wraps it in mTLS + `CONNECT`. 3. **Guide 3 — HTTP-only actors, identity carried by the certificate.** The Actor only dials plain HTTP. atunnel presents the actor's own certificate — minted per actor by ateapi off the actor-identity CA, carrying an `ActorIdentity` X.509 extension — and sends a bare `CONNECT` with no actor routing header at all. 4. **Identity authentication.** The gateway requires a client certificate signed by the actor-identity CA, so a non-actor client is refused at the handshake. It authorizes the certificate against the ATE API and rejects it unless the certified **UID** matches a real, `RUNNING` actor. 5. **Policy authorization.** What goes through the tunnel is checked against the Actor's `EgressPolicy`. A request the gateway can read (cleartext HTTP, or TLS it terminates) is decided per request: the rules in order, over its `Host` and the address the Actor dialed, first match wins, and the request is sent to what that rule checked. TLS to a name no `https` rule covers is allowed only by a matching `tls_passthrough` rule. An Actor with no policy gets no tunnel at all. ## Choose a dataplane The dataplane is selected when installing `ate-system`, not when installing this demo. Install as in the [Quickstart](https://github.com/agent-substrate/substrate/blob/main/README.md#quickstart-development) or [GKE Quickstart](https://github.com/agent-substrate/substrate/blob/main/README.md#gke-quickstart); for agentgateway, add `--atenet-dataplane=agentgateway` to `deploy ate-system`. The Actor, ActorTemplate, worker pool, test, and manual walkthrough are otherwise the same. | | Envoy | agentgateway | | --- | --- | --- | | Select with | `--atenet-dataplane=envoy` (default) | `--atenet-dataplane=agentgateway` | | Egress routing | Dynamic forward proxy | Dynamic backend from CONNECT authority | | Actor authentication | Co-located atenet `ext_proc` | Built-in `substrateEgress` policy | | Configuration | Envoy bootstrap in `atenet-egress.yaml` | Static agentgateway ConfigMap overlay | | Access log | Text beginning with `[egress]`, including actor SAN | Structured log including `substrate.connect.authority` | The experimental additional egress `ext_proc` service currently requires Envoy; the installer rejects that option with agentgateway rather than silently omitting it. ## Components - **Egress app (`main.go`)** — the Actor: `POST /` fetches an HTTP(S) URL and returns the status, body, and observed protocol. `POST /websocket` probes a `ws` or `wss` origin; `POST /grpc` makes unary and streaming gRPC calls, described below; `POST /connect` sends a forward proxy an HTTP CONNECT, in the clear or over TLS, and returns its answer. - **Egress gateway** — the `atenet-egress` Deployment. Envoy uses a co-located atenet `ext_proc` container started with `--mode=egress`; agentgateway uses its built-in `substrateEgress` policy and does not need that sidecar. The installer renders the matching configuration and container. - **Egress opt-in** — `ate-api-server --default-egress-gateway-address=atenet-egress.ate-system.svc:443` (set in `manifests/ate-install/ate-api-server.yaml`). ateapi stamps the address onto every atelet `Run`/`Restore`, which turns on tunneled egress cluster-wide. - **Egress policy** — the gateway denies by default, so the demo Actor needs an `EgressPolicy` before its fetches succeed. `kubectl ate create egress-policy` creates one from a manifest (step 3 below) and `kubectl ate get egress-policy` reads it back; the e2e suites create theirs with `e2e.EnsureEgressPolicy`. An `all` rule reproduces the pre-policy behavior. - **Actor-identity trust** — the gateway mounts the `actor-id-ca-certs` Secret, a cert-only copy of the actor-identity CA root that `ate-setup` derives from `actor-id-ca-pool` (which also holds the CA signing key and is deliberately *not* mounted here). ## Prerequisites - A k8s cluster with Agent Substrate installed as in the [Quickstart](https://github.com/agent-substrate/substrate/blob/main/README.md#quickstart-development) (with `ATE_INSTALL_KIND=true` exported in your shell) or [GKE Quickstart](https://github.com/agent-substrate/substrate/blob/main/README.md#gke-quickstart), with the dataplane [chosen above](#choose-a-dataplane). Egress is enabled by the ateapi flag above. - `kubectl` and `kubectl-ate` (`go install ./cmd/kubectl-ate`). ## Deploy the demo fixture ```bash go run ./cmd/ate-setup deploy demo egress ``` The install applies the worker pool, creates the `ate-demo-egress` atespace and the `egress` ActorTemplate (a substrate resource, not a CRD) through the ate API, and blocks until the template's golden snapshot is built: ```bash kubectl ate get actor-template egress -a ate-demo-egress ``` ## Run the automated test (easiest) ```bash ./demos/egress/test-egress.sh ``` It detects the deployed dataplane, deploys an in-cluster HTTP target, creates and resumes an Actor, then asserts: - **positive** — a real Actor's egress reaches the target (`HTTP 200`) *through the gateway* (the target sees the gateway's IP as its client), and the gateway logs the CONNECT. Envoy's log includes the actor certificate SAN; agentgateway's structured log includes the authority; - **negative** — a pod holding a valid *pod* identity but no actor certificate cannot open a tunnel at all: the gateway trusts the actor-identity CA, so the mTLS handshake is refused before any CONNECT is answered. Add `--cleanup` to remove everything the script created. ### HTTPS and WebSocket E2E tests Both supported egress dataplanes intercept HTTPS policy traffic. The Actor trusts the gateway through its projected CA bundle; the gateway verifies each local test origin against the service-DNS CA. The [egress protocol test runbook](https://github.com/agent-substrate/substrate/blob/main/docs/dev/mitm-egress-test-runbook.md) sets up this trust on Kind and runs the HTTP, HTTPS, WebSocket, and delayed-restore tests. ## Manual walkthrough ```bash # 1. An in-cluster target the Actor will fetch (any HTTP server works). kubectl create namespace egress-target kubectl -n egress-target create deployment whoami --image=traefik/whoami kubectl -n egress-target expose deployment whoami --port=80 TARGET_IP=$(kubectl -n egress-target get svc whoami -o jsonpath='{.spec.clusterIP}') # 2. Create and resume an Actor in the demo's atespace: --template # resolves the template by name within the actor's own atespace. kubectl ate create actor egress-demo -a ate-demo-egress --template egress kubectl ate resume actor egress-demo -a ate-demo-egress # wait for ACTOR_STATE_RUNNING # 3. Allow the Actor's egress; without a policy the gateway denies everything. # Any host or address: cleartext HTTP on any port, and HTTPS on 443, # intercepted by the gateway. kubectl ate create egress-policy egress-demo -a ate-demo-egress -f - <<'EOF' rules: - http: {hostnames: ["*"], ports: {all: {}}} - https: {hostnames: ["*"]} EOF # 4. Drive the Actor's egress through the ingress gateway. The gateway caches a # missing policy as deny for 10s (--egress-policy-cache-ttl), so a fetch tried # before step 3 keeps failing for up to that long after the policy appears. kubectl ate proxy actor egress-demo 8000:80 -a ate-demo-egress & curl -s -X POST http://localhost:8000/ \ -H 'Content-Type: application/json' \ -d "{\"url\":\"http://${TARGET_IP}:80/\"}" ``` The URL in the JSON body selects the Actor's egress destination. ### What to observe With Envoy: ```bash # The egress gateway logs each tunneled CONNECT against the verified peer certificate: kubectl -n ate-system logs deploy/atenet-egress -c envoy | grep '\[egress\]' # [egress] authority=:80 peer_san=spiffe://substrate-actor.local/worker-for-actor/ate-demo-egress/egress-demo … code=200 … # The co-located ext_proc sidecar logs the identity decision, including the UID it authorized on: kubectl -n ate-system logs deploy/atenet-egress -c ext-proc | grep -i 'egress tunnel opened\|egress denied' # egress tunnel opened: an address rule allows the destination leg=egress actor=ate-demo-egress/egress-demo actorUid=… destination=:80 rule=0 ``` With agentgateway: ```bash # The structured access log includes the CONNECT authority: kubectl -n ate-system logs deploy/atenet-egress -c agentgateway \ | grep 'substrate.connect.authority' ``` The `whoami` body shows `RemoteAddr: ` — proof the request egressed *through* the gateway rather than directly. ## gRPC over the same tunnel `POST /grpc` with `{"target":":","message":"hello","streamCount":2,"bidiCount":2}` makes the Actor dial that address as a cleartext-HTTP/2 gRPC server and return what came back: ```json { "message": "hello", "stream": [{"message":"hello","index":0},{"message":"hello","index":1}], "bidi": [{"message":"hello-0","index":0},{"message":"hello-1","index":1}], "code": "OK" } ``` One RPC per streaming shape, because each fails differently over a network path: - **unary `Echo`** — always run. Its status arrives in trailers, *after* the response body, which is what `code` reports and what a path that dropped trailers or downgraded to HTTP/1.1 could not produce. - **server-streaming `EchoStream`** — when `streamCount` is positive. Many frames over one held-open connection. - **bidirectional `EchoBidi`** — when `bidiCount` is positive. The Actor sends each message only after reading the response to the previous one, then half-closes the request direction while the response direction is still open. A path that serialized the two directions hangs here rather than answering short. The gateway terminates the `CONNECT` and relays opaque TCP, so all of this crosses it untouched. A failed RPC answers `502` and still carries its gRPC code in the same field. `internal/e2e/suites/networking` drives this endpoint against the `testserver` fixture's `grpc` subcommand (`internal/e2e/fixtures/testserver`, deployed by `e2e.DeployServerPod` from the shared server-pod manifest) in `TestActorEgressGRPC`; the same fixture, or any h2c gRPC server reachable from the cluster, works for a manual run. ## Notes / limitations - The gateway **authenticates** identity (is this a real, running actor?) and **authorizes** destinations against the Actor's `EgressPolicy`. The same `ext_proc` can also replace a placeholder request header with an upstream credential on HTTPS an `https` rule allows — see [docs/guides/egress-credential-injection.md](https://ate.dev/docs/guides/egress-credential-injection/index.md). - Identity comes entirely from the actor certificate: the atespace, actor name, and UID are read out of the `ActorIdentity` extension and the UID is matched against the live actor, so a certificate cannot survive its actor being deleted and recreated under the same name. Nothing the actor can write into the CONNECT contributes to the decision. ## Cleanup ```bash ./demos/egress/test-egress.sh --cleanup go run ./cmd/ate-setup delete demo egress ``` ================================================================================ SECTION: TUTORIALS PAGE: Claude Code Multiplexing URL: https://ate.dev/docs/tutorials/claude-code-multiplex/ MARKDOWN: https://ate.dev/docs/tutorials/claude-code-multiplex/index.md SOURCE: demos/claude-code-multiplex/README.md ================================================================================ # Claude Code Multiplexing > Multiplex three Claude Code agents onto two worker pods with automatic suspend and resume. A demo of three Claude-Code-driven agents sharing two Agent Substrate pods. The dashboard resumes an agent when it gets a task and suspends it when the task ends, so the cluster runs *fewer pods than agents*. **Walkthrough video (150s):** https://storage.googleapis.com/yojowa-claw-demo-screenshots/multiplex-demo-2026-05-18-captions.webm The video shows the three-agent rotation end-to-end: queued → running → completed, with the third agent suspending while the other two run, and substrate cycling pod ownership as tasks complete. > [!NOTE] > This demo intentionally provisions **two pods for three agents** to exercise substrate's suspend/resume path. The same pattern scales — ten agents on three pods, a hundred agents on twenty. ## What this shows - Three Claude Code agents (`luna`, `mars`, `orion`) registered as Substrate actors. - A `WorkerPool` of two pods. - A small web UI that drives "give a task" against random idle agents, renders the queued/running/completed badge state per agent, and resumes and suspends the agents to match. - Substrate handles the hard parts: state snapshot on suspend, scheduling decisions, resume-correctness when a pod becomes available. This demo uses the ateapi control plane to manage Actors and does not send workload HTTP traffic through `atenet-router`, so it does not use the `ate-target-actor` routing header. ## Audience This guide assumes you know Kubernetes and the general shape of agent runtimes (autonomy + LLM API access). It does **not** assume prior Substrate experience. ## Prerequisites - A k8s cluster with Agent Substrate installed as in the [Quickstart](https://github.com/agent-substrate/substrate/blob/main/README.md#quickstart-development) (with `ATE_INSTALL_KIND=true` exported in your shell) or [GKE Quickstart](https://github.com/agent-substrate/substrate/blob/main/README.md#gke-quickstart). - `kubectl` configured against that cluster (the dashboard uses the operator's kubeconfig via [`client-go`](https://github.com/kubernetes/client-go) for pod-log reads). - Network reach to the substrate **api** gRPC service (`api.ate-system:443`). From outside the cluster, the dashboard port-forwards to it by itself. If you run the dashboard as a pod in the cluster, label its namespace `ate.dev/api-access=true`, because ate-system's NetworkPolicies admit API clients only from labeled namespaces (see [`ate-system-networkpolicies.yaml`](https://github.com/agent-substrate/substrate/blob/main/manifests/ate-install/ate-system-networkpolicies.yaml)). - An **Anthropic API key** (the agents call Claude). - A snapshot location set at install (`SNAPSHOT_LOCATION`). The agents' snapshots are stored there. - The image registry (`KO_DOCKER_REPO`) that your Substrate install uses. The deploy step builds and pushes the workload image to that registry with a sha256-pinned reference. - `docker buildx` (the deploy function builds the workload image — a Dockerfile-based `node:20-slim` + Claude Code wrapper, not a Go binary, so `ko` doesn't apply for the workload itself). `docker` also needs push access to that registry; for `gcr.io`, run `gcloud auth configure-docker gcr.io`. ## Components | Path | Purpose | |---|---| | `demos/claude-code-multiplex/claude-code-multiplex.yaml.tmpl` | Namespace and WorkerPool manifest | | `demos/claude-code-multiplex/agent-*-template.yaml.tmpl` | One protojson ActorTemplate per agent, created through the ate API with `kubectl ate create actor-template` | | `cmd/ate-setup/internal/demos/claudemultiplex/` | The installer's demo package, behind `deploy demo claude-code-multiplex` / `delete demo claude-code-multiplex` | | `demos/claude-code-multiplex/workload/` | The agent container image source (Dockerfile + entrypoint that wires Claude Code; built and pushed by the deploy step) | | `demos/claude-code-multiplex/ui/` | Static dashboard (`index.html` + `server.go`) that talks to the cluster | ## How to Run ### 1. Deploy the demo From the repo root, with your Anthropic key in the environment: ```bash ANTHROPIC_API_KEY=sk-ant-... \ go run ./cmd/ate-setup deploy demo claude-code-multiplex ``` This creates the `claude-multiplex-demo` namespace and its 2-pod `WorkerPool`, then the `claude-multiplex-demo` atespace and three actor templates in it named `agent-luna`, `agent-mars`, `agent-orion`, waits for each template's golden snapshot, and creates the three actors (`luna`, `mars`, `orion`), each with an egress policy that lets it reach `api.anthropic.com:443`. The egress gateway passes that TLS through, and denies all traffic from an actor without a policy. The actors start suspended; the dashboard resumes them. Under the hood, the deploy function builds the workload image with `docker buildx`, pushes it to `${KO_DOCKER_REPO}/claude-multiplex-demo-workload`, resolves the pushed sha256 digest, and substitutes the digest-pinned reference and `ANTHROPIC_API_KEY` into the agent templates at apply time. ### 2. Start the dashboard The dashboard is a small Go HTTP server that reads worker and actor state from the substrate **ateapi** gRPC service (mirroring the pattern in `demos/sandbox/client/main.go`) and pod logs from the Kubernetes API via [`client-go`](https://github.com/kubernetes/client-go), and resumes and suspends the agents as their tasks start and end. No `kubectl` process invocations. From the repo root: ```bash cd demos/claude-code-multiplex/ui PORT=8090 go run . ``` Or build a binary: ```bash cd demos/claude-code-multiplex/ui go build -o ui-server . PORT=8090 ./ui-server ``` Either way, the UI is served on `http://localhost:8090` (or whatever `PORT` you pick). Env vars: | Var | Default | Purpose | |---|---|---| | `PORT` | `8080` | TCP port the dashboard binds (pick `≠ ATEAPI_ADDR`'s port when both are on the same host). | | `ATEAPI_ADDR` | empty | Address of the substrate ateapi gRPC service. Empty: the dashboard port-forwards to `svc/api` in `ate-system` itself. | | `DEMO_NAMESPACE` | `claude-multiplex-demo` | Kubernetes namespace the dashboard filters to and reads pod logs from. | `GET /healthz` reports whether the kube client picked up a cluster context (`logs:true|false`) — useful for quick smoke-tests after starting the server. ### 3. Drive the demo Click "Give a task". The UI picks a random idle agent and gives it a task for a fixed time. The task and its timing live only in the dashboard; a resumed agent runs `claude --print` in a loop (see [`workload/run.sh`](https://github.com/agent-substrate/substrate/blob/main/demos/claude-code-multiplex/workload/run.sh)), and its log shows the replies. Watch: - The badge shows `queued` until the dashboard has resumed the agent, then `running`. If both pods hold agents with running tasks, the task stays `queued` until one of them finishes. - When the time is up, the badge flips to `completed` and the dashboard suspends the agent, snapshotting its state and freeing its pod for the next task. With three agents and two pods, at most two agents run at once, and the third task waits for a pod. ## Settings The installer reads this setting in addition to those in [docs/guides/operator-install.md](https://ate.dev/docs/guides/operator-install/index.md#settings), through the same environment variable and configuration document channels. | Key | Environment variable | Flag | Default | Applies to | |------------------------|----------------------|-------------|------------|------------| | `demo.anthropicAPIKey` | `ANTHROPIC_API_KEY` | — (no flag) | — (secret) | (global) | It is a secret, so it has no flag and the run report prints `` for it. ## Upstream blockers worked around for this demo This demo currently applies workarounds at runtime for two Substrate issues. See the linked issue threads for details and workarounds. - **`#189`** — Atelet OCI bundle gaps (`Args`, `Secret`, symlinks). - **`#197` Bug 3** — Atelet symlink resolution. ## Teardown From the repo root: ```bash go run ./cmd/ate-setup delete demo claude-code-multiplex ``` This removes the demo's actors, templates, atespace, and the `claude-multiplex-demo` namespace with everything the deploy step created in it. You can also stop the port-forward and the dashboard processes in their respective terminals. ================================================================================ SECTION: HOW-TO GUIDES PAGE: Operator Install URL: https://ate.dev/docs/guides/operator-install/ MARKDOWN: https://ate.dev/docs/guides/operator-install/index.md SOURCE: docs/guides/operator-install.md ================================================================================ # Operator Install > Configure the installer with a settings document, and how it resolves flags, environment, and config. Every setting the installer reads can be supplied three ways: a flag, an environment variable, or a configuration document. This page describes the document, because it is the only one of the three an operator owns as a file, reviews in a change, and reuses across runs. ## The document ```yaml apiVersion: install.ate.dev/v1alpha1 kind: SubstrateInstall namespace: ate-system atenet: dataplane: envoy images: repo: registry.example.com/substrate tag: v0.4.1 ``` Pass it with `--config`, or name it in `ATE_CONFIG`: ```bash ate-setup deploy ate-system --config substrate-install.yaml ATE_CONFIG=substrate-install.yaml ate-setup deploy ate-system ``` A worked example is at [docs/examples/substrate-install.yaml](https://github.com/agent-substrate/substrate/blob/main/docs/examples/substrate-install.yaml). ### Keys are nested; the table below is flat A key written `ateapi.postgres.schema` is nested in the document: ```yaml ateapi: postgres: schema: public ``` An unrecognized key is rejected rather than ignored, naming the file: ``` Error: install.yaml: unknown setting "ateapi.postgres.schemas" ``` A typo would otherwise leave the install running on the default with nothing on screen to say so. ## Precedence Lowest to highest: ``` default < environment < configuration file < flag ``` The file outranks the environment deliberately: the file is the artifact an operator wrote and can review, and ambient state left in a shell must not silently outrank it. A flag outranks both because it was typed for this run. A channel that does not mention a setting leaves the one below it in place, so a document need only name what differs from the defaults. ## Empty values A setting a channel mentions is set, whatever it is set to. An empty value is a value: ```yaml namespace: "" ``` ```bash export ATE_NAMESPACE= # the same thing ate-setup deploy ate-system --namespace= ``` All three clear the setting rather than falling through to the default. That is how a setting is removed from an installation: write its zero value and re-apply. Only an *absent* setting falls through. In the shell that distinction is the difference between `export FOO=` and `unset FOO`, and the two do not mean the same thing here: | Shell | Meaning | |---------------------------------|----------------------------------| | `unset ATE_NAMESPACE` | not set; the layer below applies | | `export ATE_NAMESPACE=` | set to empty | | `export ATE_NAMESPACE=ate-prod` | set to `ate-prod` | A script that exports an empty variable to mean "unset" is configuring the setting. The report names the channel each value came from, so when that happens it is visible rather than mysterious. For a setting that is not a string, an empty value is not a value of its kind and is reported as such -- `ATE_INSTALL_PODCERT_WORKERS_PER_SIGNER=` is rejected, naming the variable, rather than quietly ignored. ## Every run reports what it resolved Every setting is listed with the channel that supplied it: ``` configuration: 6 set, 35 default ateapi.expectedJWTIssuer "" (from default) ateapi.postgres.readWrite.connectionString (from ATE_API_POSTGRES_READ_WRITE_CONNECTION_STRING) ... atenet.dataplane agentgateway (from ATE_ATENET_DATAPLANE) benchmark.sandboxClass microvm (from --sandbox-class) benchmark.workerCount 7 (from config.benchmark.workerCount) clusterSize size0 (from default) context kind-kind (from the --kind profile) namespace ate-prod (from config.namespace) ``` An install that succeeds with the wrong value never produces an error, so this report is the only place an inherited variable or a forgotten file becomes visible -- and a value that came from a default is worth seeing for the same reason, because next release it may be a different default. A setting marked secret in the table below prints as `` when something supplied it, with the channel still named so you know where to change it. One nothing supplied prints its default, which is empty: no credential appears either way. To list only what was set: ```bash ate-setup deploy ate-system --no-report-defaults export ATE_NO_REPORT_DEFAULTS=1 # the same, for a shell that runs it repeatedly ``` ``` configuration: 6 set, 35 default ateapi.postgres.readWrite.connectionString (from ATE_API_POSTGRES_READ_WRITE_CONNECTION_STRING) atenet.dataplane agentgateway (from ATE_ATENET_DATAPLANE) benchmark.sandboxClass microvm (from --sandbox-class) benchmark.workerCount 7 (from config.benchmark.workerCount) context kind-kind (from the --kind profile) namespace ate-prod (from config.namespace) ``` ## Errors name the channel ``` Error: atenet.dataplane must be envoy or agentgateway, got "nginx" (from config.atenet.dataplane) Error: at least one of ATE_API_POSTGRES_READ_WRITE_CONNECTION_STRING or config.ateapi.postgres.readWrite.connectionString must be set, got none ``` ## Command-scoped settings Most settings apply to every command and are flags on the root. A few belong to one command and are flags only there — but they are still reachable from the environment and the file, so one document can describe a whole install: ```yaml csi: setup: nfs benchmark: workerCount: 4 ``` ## Install runs record what they used A `deploy` command that installs Substrate writes the settings it resolved to a file, so a later run can reproduce it without anyone remembering what was typed. `deploy demo` and `deploy benchmarks` write nothing: they deploy on top of the system with only their own settings. ``` recorded this configuration in ~/.cache/ate-setup/installs/prod.yaml ``` Records go under the user cache directory, or wherever `record.dir` points. Nothing reads them on its own -- a record only takes effect when you pass it to `--config`. | Outcome | Path | Retention | |-----------|--------------------------------------------------|------------------------------------------------------------| | Succeeded | `/installs/.yaml` | replaced each run: a cluster has one current configuration | | Failed | `/failed/-.yaml` | kept: each attempt is its own artifact | ### Retrying a failed install A run that fails prints the record it wrote and the command that replays it: ``` Error: while reading kubeconfig: context "prod" does not exist the settings this run used were written to ~/.cache/ate-setup/failed/prod-20260929T231438Z.yaml retry with: ate-setup deploy atenet --config ~/.cache/ate-setup/failed/prod-20260929T231438Z.yaml 1 secret setting(s) were not written and must be supplied again: ateapi.postgres.readWrite.connectionString ``` The record is ordinary `--config` input. Nothing has to be stripped from it first: the `cluster:` block holding the timestamp and outcome is skipped when the file is read back. **A retry is not a complete replay.** Secrets are never written, so the last line above lists what to supply again: ```bash export ATE_API_POSTGRES_READ_WRITE_CONNECTION_STRING='postgres://...' ate-setup deploy atenet --config ~/.cache/ate-setup/failed/prod-20260929T231438Z.yaml ``` A failed run changes nothing about later runs. Its record is inert until you name it -- a value that caused the failure must not become the default that reaches the cluster on the next success. ### Reusing a successful install The same applies to the success record, which is the one to keep: ```bash # Rebuild a cluster the way it was, or set up a second one to match. ate-setup deploy ate-system --config ~/.cache/ate-setup/installs/prod.yaml ``` Point `record.dir` at a checkout to keep these under version control, which is how several operators share one configuration and how an install survives a cluster that no longer exists: ```bash export ATE_RECORD_DIR=~/ops/substrate-installs ``` Records hold no credentials, so they are safe to commit. They are written readable only by you regardless. ### What a record looks like ```yaml apiVersion: install.ate.dev/v1alpha1 kind: SubstrateInstall cluster: context: prod outcome: succeeded substrateVersion: v0.4.1 writtenAt: "2026-09-29T23:14:38Z" writtenBy: ate-setup v0.4.1 atenet: dataplane: agentgateway context: prod namespace: ate-prod ``` `cluster.substrateVersion` is the Substrate release the run installed: the version stamped into the binaries it built, or the image tag of a prebuilt install. `cluster.writtenBy` is the `ate-setup` that wrote the file. Only settings something supplied are recorded. A setting left on its default is left out on purpose, so it keeps following the default if a later release changes it. ## Settings Defaults are also shown by `ate-setup --help` and by `--help` on the owning command. A setting with no default is unset unless something supplies it. The `ateapi.authz.*` settings are described in [authorization.md](https://ate.dev/docs/guides/authorization/index.md), and `ateapi.expectedJWTIssuer` in [authentication.md](https://ate.dev/docs/guides/authentication/index.md#configuring-jwt-providers). Settings for the demos, benchmarks, and e2e tests are listed with them: the [counter](https://ate.dev/docs/tutorials/counter/index.md#settings) and [claude-code-multiplex](https://ate.dev/docs/tutorials/claude-code-multiplex/index.md#settings) demos, the [benchmarks](https://github.com/agent-substrate/substrate/blob/main/docs/guides/benchmarking.md#settings), and the [e2e tests](https://github.com/agent-substrate/substrate/blob/main/internal/e2e/README.md#settings). | Key | Environment variable | Flag | Default | Applies to | |----------------------------------------------|-------------------------------------------------|----------------------------------------------------|-----------------------|------------------------------------------| | `actorJWT.algorithm` | `ACTOR_JWT_ALGORITHM` | `--actor-jwt-algorithm` | `ES256` | (global) | | `actorJWT.issuer` | `ACTOR_JWT_ISSUER` | `--actor-jwt-issuer` | — | (global) | | `ateapi.authz.bootstrapOwners` | `ATE_API_AUTHZ_BOOTSTRAP_OWNERS` | `--ateapi-authz-bootstrap-owners` | `system:serviceaccount::ate-client` | (global) | | `ateapi.authz.egressGatewayExtraSPIFFEIDs` | `ATE_EGRESS_GATEWAY_EXTRA_SPIFFE_IDS` | `--egress-gateway-extra-spiffe-ids` | — | (global) | | `ateapi.expectedJWTIssuer` | `EXPECTED_JWT_ISSUER` | `--ateapi-expected-jwt-issuer` | — | (global) | | `ateapi.logLevel` | `ATE_API_LOG_LEVEL` | `--ateapi-log-level` | — | (global) | | `ateapi.postgres.cloudsql.gsa` | `ATE_API_POSTGRES_CLOUDSQL_GSA` | `--ateapi-postgres-cloudsql-gsa` | — | (global) | | `ateapi.postgres.cloudsql.iamAuth` | `ATE_API_POSTGRES_CLOUDSQL_IAM_AUTH` | `--ateapi-postgres-cloudsql-iam-auth` | — | (global) | | `ateapi.postgres.cloudsql.instance` | `ATE_API_POSTGRES_CLOUDSQL_INSTANCE` | `--ateapi-postgres-cloudsql-instance` | — | (global) | | `ateapi.postgres.cloudsql.ipType` | `ATE_API_POSTGRES_CLOUDSQL_IP_TYPE` | `--ateapi-postgres-cloudsql-ip-type` | — | (global) | | `ateapi.postgres.owner.connectionString` | `ATE_API_POSTGRES_OWNER_CONNECTION_STRING` | — (no flag) | — (secret) | (global) | | `ateapi.postgres.owner.role` | `ATE_API_POSTGRES_OWNER_ROLE` | `--ateapi-postgres-owner-role` | `substrate_owner` | (global) | | `ateapi.postgres.poolMaxConns` | `ATE_API_POSTGRES_POOL_MAX_CONNS` | `--ateapi-postgres-pool-max-conns` | — | (global) | | `ateapi.postgres.readWrite.connectionString` | `ATE_API_POSTGRES_READ_WRITE_CONNECTION_STRING` | — (no flag) | — (secret) | (global) | | `ateapi.postgres.readWrite.role` | `ATE_API_POSTGRES_READ_WRITE_ROLE` | `--ateapi-postgres-read-write-role` | `substrate_readwrite` | (global) | | `ateapi.postgres.schema` | `ATE_API_POSTGRES_SCHEMA` | `--ateapi-postgres-schema` | — | (global) | | `ateapi.postgres.serverCAFile` | `ATE_API_POSTGRES_SERVER_CA_FILE` | `--ateapi-postgres-server-ca-file` | — | (global) | | `ateapi.postgres.storageClass` | `ATE_API_POSTGRES_STORAGE_CLASS` | `--ateapi-postgres-storage-class` | — | (global) | | `atenet.dataplane` | `ATE_ATENET_DATAPLANE` | `--atenet-dataplane` | `envoy` | (global) | | `atenet.egress.additionalExtprocService` | `ATE_ADDITIONAL_EGRESS_EXTPROC_SERVICE` | `--experimental-additional-egress-extproc-service` | — | (global) | | `atenet.egress.credentialProvider` | `ATE_CREDENTIAL_PROVIDER` | `--credential-provider` | — | (global) | | `clusterSize` | `ATE_INSTALL_CLUSTER_SIZE` | `--cluster-size` | `size0` | (global) | | `context` | `KUBECTL_CONTEXT` | `--context` | — | (global) | | `cordonSystemInfraComponents` | `ATE_INSTALL_CORDON_SYSTEM_INFRA_COMPONENTS` | `--cordon-system-infra-components` | `false` | (global) | | `csi.setup` | `SETUP_CSI` | `--setup-csi` | `none` | `deploy ate-system` | | `docker.buildFlags` | `DOCKER_BUILD_FLAGS` | `--docker-build-flags` | — | (global) | | `fakeDataPlane.enabled` | `ATE_INSTALL_FAKE_DATA_PLANE` | `--fake-data-plane` | `false` | (global) | | `gcp.clusterLocation` | `CLUSTER_LOCATION` | `--gcp-cluster-location` | — | (global) | | `gcp.clusterName` | `CLUSTER_NAME` | `--gcp-cluster-name` | — | (global) | | `gcp.projectID` | `PROJECT_ID` | `--gcp-project-id` | — | (global) | | `images.repo` | `ATE_IMAGE_REPO` | `--image-repo` | — | (global) | | `images.tag` | `ATE_IMAGE_TAG` | `--image-tag` | — | (global) | | `ingressAllowedSPIFFEIDs` | `ATE_INGRESS_ALLOWED_SPIFFE_IDS` | `--ingress-allowed-spiffe-ids` | — | (global) | | `kindCluster.enabled` | `ATE_INSTALL_KIND` | `--kind` | `false` | (global) | | `kindCluster.name` | `KIND_CLUSTER_NAME` | `--kind-cluster-name` | `kind` | (global) | | `ko.defaultPlatforms` | `KO_DEFAULTPLATFORMS` | `--ko-default-platforms` | — | (global) | | `ko.dockerRepo` | `KO_DOCKER_REPO` | `--ko-docker-repo` | — | (global) | | `kubeconfig` | `KUBECONFIG` | `--kubeconfig` | — | (global) | | `namespace` | `ATE_NAMESPACE` | `--namespace` | `ate-system` | (global) | | `otlpEndpoint` | `ATE_OTLP_ENDPOINT` | `--otlp-endpoint` | — | (global) | | `podcert.workersPerSigner` | `ATE_INSTALL_PODCERT_WORKERS_PER_SIGNER` | `--podcert-workers-per-signer` | `0` | (global) | | `preview` | `ATE_PREVIEW` | `--preview` | — | (global) | | `publish.binariesDir` | `ATE_PUBLISH_BINARIES_DIR` | `--out` | `bin/release` | `publish release-binaries` | | `publish.overwrite` | `ATE_PUBLISH_OVERWRITE` | `--overwrite` | `false` | `publish release-images` | | `record.dir` | `ATE_RECORD_DIR` | `--record-dir` | — | (global) | | `rolloutTimeout` | `ATE_INSTALL_ROLLOUT_TIMEOUT` | `--rollout-timeout` | `1m0s` | (global) | | `sandboxClasses` | `ATE_INSTALL_SANDBOX_CLASSES` | `--sandbox-classes` | `gvisor,microvm` | `deploy ate-system`, `deploy sandboxconfig` | | `storage.bucketName` | `BUCKET_NAME` | `--storage-bucket-name` | — | (global) | | `storage.snapshotLocation` | `SNAPSHOT_LOCATION` | `--snapshot-location` | — | (global) | | `storage.transferGSAName` | `SNAPSHOT_TRANSFER_GSA_NAME` | `--snapshot-transfer-gsa-name` | — | (global) | ## Snapshot storage ate-api-server stores every actor and tag snapshot under a cluster-wide `--snapshot-location`, an object-storage URI such as `gs://BUCKET/PREFIX/`. Its path segments may contain only ASCII letters, digits, `_` and `-` (see [Snapshot Storage Layout](https://ate.dev/docs/reference/api-guide/index.md#snapshot-storage-layout)). Set it with `SNAPSHOT_LOCATION` (or ate-setup's `--snapshot-location`). It has no default, and ate-setup fails before deploying without it unless the `PerTemplateSnapshotLocation` preview gate is enabled. Even with the gate, the demos and benchmarks need it, since they do not set `snapshotConfig.storageLocation`. So do e2e runs, which deploy the demos. If you deploy ate-api-server some other way, pass the flag yourself; without the gate, the server refuses to start without it. On GCS, ate-api-server's credential exchange runs snapshot transfers as a dedicated Google service account, so GCS audit logs name that account. Set its name with `SNAPSHOT_TRANSFER_GSA_NAME` (or ate-setup's `--snapshot-transfer-gsa-name`), and ate-setup builds its email in `gcp.projectID`. ate-setup refuses to deploy ate-api-server on GKE without one. The exchange impersonates the account, so ate-api-server's identity needs `roles/iam.serviceAccountTokenCreator` on it, and the account needs object admin and bucket viewer on every snapshot bucket. `tools/setup-gcp create iam` in [substrate-gke](https://github.com/ai-on-gke/substrate-gke) creates it under that name, `ate-snapshot-transfer` by default, with both grants. kind does not use it. The exchange verifies the actor JWT with keys it fetches from `actorJWT.issuer` through OpenID Connect discovery. The default issuer is the install's ate-idp-server. A custom issuer must route to that ate-idp-server, which serves the keys the install signs actor JWTs with. It must also be reachable from the ate-api-server pod and serve a certificate that the install's service DNS CA trusts. For an issuer whose host does not end in `.svc`, a certificate the system roots trust works too. The exchange fetches the keys once at start, and exits if the issuer's certificate does not verify. Then ate-api-server does not start either. With `--kind`, ate-setup defaults `storage.snapshotLocation` to `gs:///` (`gs://ate-snapshots/`, the bucket the in-cluster rustfs creates) and ignores an ambient `SNAPSHOT_LOCATION`. On GKE, ate-setup does not create the bucket or grant access to it: run substrate-gke's `tools/setup-gcp create bucket` and `create iam` first, or `bootstrap` for both (see [tools/setup-gcp](https://github.com/ai-on-gke/substrate-gke/blob/main/tools/setup-gcp/README.md) in substrate-gke). substrate-gke's setup-gcp also creates an asset bucket, `-assets`, where `ate-setup` stages microVM sandbox assets, and gives atelet read access to it. atelet gets no access to the snapshot bucket: every snapshot transfer runs on a token scoped to its actor, issued by ate-api-server's credential exchange. A GCP project set up before the asset bucket existed keeps atelet's old grants, since substrate-gke's setup-gcp only adds bindings. Remove them as [tools/setup-gcp](https://github.com/ai-on-gke/substrate-gke/blob/main/tools/setup-gcp/README.md#moving-to-per-actor-snapshot-credentials) in substrate-gke shows, which also covers recreating a cluster with microVM workers. With the `PerTemplateSnapshotLocation` preview gate, an ActorTemplate may set `snapshotConfig.storageLocation` to store its snapshots elsewhere; templates that leave it empty use `--snapshot-location`. Grant ate-api-server's identity and the snapshot transfer service account object admin and bucket viewer on any other bucket a template names, as substrate-gke's setup-gcp does on the snapshot bucket. Without the gate, ate-api-server rejects templates that set `snapshotConfig.storageLocation`. See [Snapshot Storage Layout](https://ate.dev/docs/reference/api-guide/index.md#snapshot-storage-layout). ## Ingress to actors > [!IMPORTANT] > atenet-router lets a client reach actors only if the client's SPIFFE ID is > in `ingressAllowedSPIFFEIDs` (`--ingress-allowed-spiffe-ids`). The setting > has no default, so an install that leaves it unset accepts no ingress > traffic. List every client that calls atenet-router. If all traffic reaches actors through one frontend or gateway, list that gateway's identity. If several services call the router, list each of them. Any listed client may reach any actor. ## Secrets Credentials should come from the environment, not from a document that gets committed: ```bash export ATE_API_POSTGRES_READ_WRITE_CONNECTION_STRING='postgres://...' ate-setup deploy ate-system --config substrate-install.yaml ``` These settings have no flag. A command line is visible in `ps` to every user on the host and is written to shell history, so a credential is supplied through the environment or through a file you can permission. The run report prints `` for these rather than the value. ================================================================================ SECTION: HOW-TO GUIDES PAGE: Database Configuration URL: https://ate.dev/docs/guides/postgres/ MARKDOWN: https://ate.dev/docs/guides/postgres/index.md SOURCE: docs/guides/postgres.md ================================================================================ # Database Configuration > PostgreSQL roles and connections for the bundled database and for bring-your-own databases. ## Overview Substrate supports separate connections for schema ownership (DDL) and normal reads and writes (DML). Migrations and partition maintenance use the owner connection; application queries use the read/write connection. Each connection assumes its configured PostgreSQL role. The bundled development database uses fixed owner and read/write roles and separate login users. `ate-setup` creates them from [`pkg/postgressetup/setup.sql`](https://github.com/agent-substrate/substrate/blob/main/pkg/postgressetup/setup.sql) before starting `ateapi`. The script has no built-in identity defaults; `ate-setup` supplies the development values from `postgressetup.DefaultConfig`. The bundled login users have no database passwords; they authenticate with certificates. External databases are never provisioned by `ateapi` or `ate-setup`. Their operators must create the database identities and schema described below before deploying Substrate. ## Bundled database authentication The API server connects without a password using two Kubernetes projected Pod Certificates from `postgres.podcert.ate.dev/identity`. This signer only issues client certificates to pods using the `ate-api-server` service account in the namespace configured by the controller's `--postgres-client-namespace` flag (default: `ate-system`). Set this flag to the API server's namespace when relocating an installation; the certificate controller runs in a separate namespace. Each projection sets `userAnnotations.postgres.podcert.ate.dev/username` to either `substrate_owner_user` or `substrate_readwrite_user`; the signer rejects missing annotations, unknown keys, and every other username. It sets the certificate Common Name (CN) to that authorized login. Each connection pool presents its own certificate. The namespace and service account are the authorization boundary. Kubernetes validates those identity fields in each PodCertificateRequest; the username annotation only selects between the two logins authorized for that identity. Anyone allowed to create pods using `ate-api-server` in that namespace can request either login, so operators must restrict that capability to trusted control-plane administrators. Both credentials are available to the API-server process; separate logins do not isolate the owner role from a compromised API server. The bundled PostgreSQL server trusts only this signer's dedicated CA for client certificates. Its `hostssl ... cert` rule requires the certificate CN to match the requested database username. The API server uses `sslmode=verify-full` to verify the server's service DNS certificate. New database connections reread the projected client credentials and server trust roots. The bundled PostgreSQL TLS-reloader checks its server credential bundle and client trust bundle every 60 seconds and requests a configuration reload when they change. Existing database connections remain open. Pod Certificates require a minimum lifetime of one hour; the signer's default is 24 hours. Local Unix-socket access inside the database pod remains trusted for bootstrap, health checks, and TLS reloads. The installer stores the signer's CA pool in the `postgres-ca-pool` Secret in `podcertificate-controller-system`; the API pod receives its client private key and certificate through a projected volume rather than a username/password Secret. The owner and read/write logins retain their respective role grants from the bootstrap script. The `postgres` administrator is restricted to local connections inside the database pod. ## BYO DB Configuration A custom database can be provided to Substrate. `ATE_API_POSTGRES_SCHEMA` (or `--postgres-schema` when running `ateapi` directly) selects the schema, defaulting to `substrate`. The operator must provision the schema and its grants before starting Substrate. `ateapi` then uses the schema for both connection pools and migrations, overriding any `search_path` in the connection strings. ### Operator Provisioned BYO DB For production systems, the recommendation is to fully provision an external Postgres Database and provide credentials to Substrate to interact with that database. The `ATE_API_POSTGRES_OWNER_CONNECTION_STRING` and `ATE_API_POSTGRES_READ_WRITE_CONNECTION_STRING` are used to configure Substrate's owner and runtime connection pools. The external database must contain: - A database selected by both connection strings. Substrate does not require a particular database name. - A schema owned by the owner role. The default schema name is `substrate`. - A stable `NOLOGIN` owner role. The default is `substrate_owner`. - A stable `NOLOGIN` read/write role. The default is `substrate_readwrite`. - One or two login users. The owner login must be a member of the owner role, and the runtime login must be a member of the read/write role. One login may be a member of both roles when separate credentials are unavailable. Operators using ordinary PostgreSQL login users may run [`pkg/postgressetup/setup.sql`](https://github.com/agent-substrate/substrate/blob/main/pkg/postgressetup/setup.sql) with an administrator connection. It is a `psql` script and requires all of these variables: | Variable | Bundled development value | |---|---| | `substrate_schema` | `substrate` | | `substrate_owner_role` | `substrate_owner` | | `substrate_owner_user` | `substrate_owner_user` | | `substrate_owner_password` | Empty (certificate authentication) | | `substrate_readwrite_role` | `substrate_readwrite` | | `substrate_readwrite_user` | `substrate_readwrite_user` | | `substrate_readwrite_password` | Empty (certificate authentication) | The login users come from the database provider: they may be ordinary PostgreSQL users, IAM identities, or another provider-managed identity. Substrate does not create them or require specific login names. Create an owner role with schema ownership and a read/write role with schema usage. Grant the runtime role SELECT, INSERT, UPDATE, and DELETE on tables; USAGE, SELECT, and UPDATE on sequences; EXECUTE on routines; and USAGE on types. Configure the owner's default privileges so newly migrated objects receive these grants. Grant the respective roles to the login users. The [Cloud SQL setup example](https://github.com/ai-on-gke/substrate-gke/blob/main/tools/setup-gcp/cloud-sql.md#2-one-time-roles-and-schema-privileges) in substrate-gke shows the SQL for one login with both roles; use separate logins for stronger isolation. Apply these grants before the first `ateapi` startup. The owner connection creates and updates tables through migrations; the read/write connection cannot create schema objects. Set `ATE_API_POSTGRES_OWNER_ROLE` and `ATE_API_POSTGRES_READ_WRITE_ROLE` to the roles you created, and leave role settings out of the connection strings: `ateapi` runs `SET ROLE` after connecting. User credentials are passed as connection strings through `ATE_API_POSTGRES_OWNER_CONNECTION_STRING` and `ATE_API_POSTGRES_READ_WRITE_CONNECTION_STRING`. When running `ateapi` directly, pass the DSNs as flags or use `@env` flags to read these environment variables; the installer manifest already uses `@env`. `ATE_API_POSTGRES_POOL_MAX_CONNS` (or `--postgres-pool-max-conns` when running `ateapi` directly) sets the read/write pool limit after the connection string is loaded. When unset, a `pool_max_conns` value in the DSN or the pgxpool default applies. This setting does not affect the owner, watch, or readiness probe pools; they are capped at 2, 3, and 1 connections, respectively. `ateapi` uses the same `@env` flag pattern for role names and schema. ### CloudSQL Configuration CloudSQL setup information can be found in the dedicated guide [here](https://github.com/ai-on-gke/substrate-gke/blob/main/tools/setup-gcp/cloud-sql.md) in substrate-gke. ================================================================================ SECTION: HOW-TO GUIDES PAGE: Authentication URL: https://ate.dev/docs/guides/authentication/ MARKDOWN: https://ate.dev/docs/guides/authentication/index.md SOURCE: docs/guides/authentication.md ================================================================================ # Authentication > How clients authenticate to ate-api-server over SPIFFE mTLS or OIDC JWTs. Every call to `ate-api-server` must carry a credential, which authentication turns into a principal ID. [Authorization](https://ate.dev/docs/guides/authorization/index.md) then decides what that principal may do. - **Client certificate.** A client certificate must chain to the `podidentity.podcert.ate.dev/identity` trust bundle, or the TLS handshake fails. The principal is its first URI SAN, a SPIFFE ID. A bearer token sent alongside it is ignored. - **Bearer JWT.** A client that sends no certificate needs an `authorization: Bearer ` header with a token from a [trusted issuer](#configuring-jwt-providers). The principal is the token's `sub`. - A call with no certificate and no valid token gets `Unauthenticated`. ## Which credential to use | Caller | Credential | Principal | |---|---|---| | [A person running `kubectl ate`](#people-running-kubectl-ate) | `--token-file` with a token for your own ServiceAccount, or from your identity provider | `system:serviceaccount::`, or the token's `sub` | | [A pod in the Substrate cluster](#pods-in-the-substrate-cluster) | A projected ServiceAccount token, or a pod certificate | `system:serviceaccount::`, or `spiffe://cluster.local/ns//sa/` | | [A pod in another cluster](#pods-in-another-cluster) | A projected ServiceAccount token from that cluster | `system:serviceaccount::` | | [A CI job](#ci-jobs) | The CI system's OIDC token | The token's `sub` | Then grant the principal a role in an [access policy](https://ate.dev/docs/guides/authorization/index.md#managing-access). ## People running `kubectl ate` `kubectl ate` needs a kubeconfig for the Substrate cluster even when you pass `--token-file`. It uses it to: - list ClusterTrustBundles, to verify `ate-api-server`'s serving certificate; - without `--endpoint`, get the `api` Service, list pods and port-forward in `ate-system`; - without `--token-file`, create a token for the shared `ate-client` ServiceAccount, warning that it is acting as a shared identity; - for `logs actor` and `top worker`, read worker pods' logs and metrics, and for `admin`, read and write the Secrets it manages. A person who isn't a cluster admin needs these roles for the first two: ```sh kubectl apply -f - <<'EOF' apiVersion: rbac.authorization.k8s.io/v1 kind: Role metadata: name: ate-api-client namespace: ate-system rules: - apiGroups: [""] resources: [services] resourceNames: [api] verbs: [get] - apiGroups: [""] resources: [pods] verbs: [list] - apiGroups: [""] resources: [pods/portforward] verbs: [get, create] --- apiVersion: rbac.authorization.k8s.io/v1 kind: ClusterRole metadata: name: ate-api-client rules: - apiGroups: [certificates.k8s.io] resources: [clustertrustbundles] verbs: [list] EOF kubectl -n ate-system create rolebinding alice-ate-api-client --role=ate-api-client --user=alice@example.com kubectl create clusterrolebinding alice-ate-api-client --clusterrole=ate-api-client --user=alice@example.com ``` `--user` is the person's Kubernetes user name, which on GKE is their Google account email. RBAC can limit port-forwarding only by pod name, and `ate-api-server`'s pod names change on every rollout, so in practice the grant covers every pod and port in `ate-system`. NetworkPolicy doesn't filter it. It reaches ports that listen only on loopback, including the router's `/statusz`, which lists recent requests to every atespace, and the unauthenticated Envoy admin APIs of the router and egress gateway, which can drain or stop them. Grant it only to people you'd trust to operate Substrate. Without `--token-file`, everyone is the same principal, and `ate-setup` makes it a bootstrap owner by default (see the warning in [authorization.md](https://ate.dev/docs/guides/authorization/index.md)). That suits a cluster only you use. On a shared cluster, give each person their own identity in one of the two ways below. `kubectl ate` rereads `--token-file` on every call but never refreshes it, so get a new token when it expires. With `--token-file=-` it reads the token once from stdin. ### Your own ServiceAccount Create each person's ServiceAccount in a namespace set aside for them, such as `ate-users`, and let each person create tokens only for their own ServiceAccount: ```sh kubectl create namespace ate-users kubectl -n ate-users create serviceaccount alice kubectl -n ate-users create role alice-token --verb=create --resource=serviceaccounts/token --resource-name=alice kubectl -n ate-users create rolebinding alice-token --role=alice-token --user=alice@example.com ``` Alice then mints a token and passes it: ```sh kubectl create token -n ate-users alice --audience api.ate-system.svc > alice.token kubectl ate --token-file alice.token get actors -a team-a ``` The principal is `system:serviceaccount:ate-users:alice`. The audience must be one the `kubernetes` provider lists; the config `ate-setup` writes lists `api.ate-system.svc`. RBAC grants only add up, so these roles don't stop anyone else from acting as alice. Anyone who can create pods or tokens in `ate-users` can, and the built-in `edit` and `admin` roles allow both. The same rights in `ate-system` get the `ate-client` token. On GKE a call is allowed if either IAM or RBAC allows it, and an IAM grant covers every namespace: Kubernetes Engine Developer can create a token for any ServiceAccount, and the basic Editor role can create pods anywhere. Give these people only Kubernetes Engine Cluster Viewer in IAM, and grant the rest through RBAC. `ate-api-server` checks a token's signature and expiry without asking Kubernetes whether its ServiceAccount still exists. Deleting alice's ServiceAccount doesn't revoke tokens she already holds: they work until they expire, plus a five-minute allowance for clock skew. To cut her off at once, remove her from access policies. ### A token from your identity provider If your identity provider issues OIDC ID tokens, [add it as a JWT provider](#configuring-jwt-providers) and pass its token. With the Google provider from that section: ```sh gcloud auth print-identity-token | kubectl ate --token-file=- get actors -a team-a ``` The principal is the token's `sub`. For Google that is the numeric account ID, not the email; [bootstrap owners](https://ate.dev/docs/guides/authorization/index.md#bootstrap-owners) shows how to find it. Any Google account can get a token for the Cloud SDK's client ID, so this provider authenticates every Google account, and access policies decide what each may do. Any service you send one of these tokens to can replay it here until it expires, so prefer a provider that issues tokens for an audience of your own. ## Pods in the Substrate cluster Apart from Substrate's own components, the install's NetworkPolicy admits connections to `ate-api-server` only from pods in namespaces labeled `ate.dev/api-access=true`, and not from worker pods. Label the client's namespace: ```sh kubectl label namespace ate.dev/api-access=true ``` Then dial `api.ate-system.svc:443` and verify the serving certificate against the `servicedns.podcert.ate.dev/identity` trust bundle, with server name `api.ate-system.svc`. The Service is headless, so DNS returns one address per replica; use round-robin load balancing to spread calls across them. Go clients can use the generated stubs in `api/ateapi/v1`. Authenticate with a ServiceAccount token or a pod certificate. The kubelet updates all of these files, so read them from the volume instead of caching them: ```yaml spec: serviceAccountName: my-app containers: - name: app volumeMounts: - {name: ate-api, mountPath: /var/run/ate, readOnly: true} volumes: - name: ate-api projected: sources: - clusterTrustBundle: signerName: servicedns.podcert.ate.dev/identity labelSelector: matchLabels: podcert.ate.dev/canarying: live path: ca.crt # Either a ServiceAccount token, sent as a bearer token: - serviceAccountToken: audience: api.ate-system.svc path: token # or a pod certificate, presented as the TLS client certificate: # - podCertificate: # signerName: podidentity.podcert.ate.dev/identity # keyType: ECDSAP256 # credentialBundlePath: credential-bundle.pem ``` - **ServiceAccount token.** The principal is `system:serviceaccount::`. The config `ate-setup` writes already trusts the cluster's issuer and this audience. To use `kubectl ate` from the pod: ```sh kubectl ate --endpoint=api.ate-system.svc:443 --token-file=/var/run/ate/token get actors -a ``` It still reads the trust bundle through the Kubernetes API, which default RBAC lets every ServiceAccount do. - **Pod certificate.** The principal is `spiffe://cluster.local/ns//sa/`. The credential bundle is one PEM file holding the private key followed by the certificate chain; reread it on each TLS handshake. `kubectl ate` can't present a client certificate. ## Pods in another cluster A pod in another cluster sends a ServiceAccount token from its own cluster. That cluster's pod certificates aren't trusted, and it has no copy of the Substrate cluster's `servicedns.podcert.ate.dev/identity` trust bundle, so the pod's projected volume holds the token and a copy of the serving CA from step 3: ```yaml volumes: - name: ate-api projected: sources: - configMap: name: ate-api-ca - serviceAccountToken: audience: api.ate-system.svc path: token ``` 1. **Trust only a cluster whose admins you'd trust as owners here.** Principals don't name the cluster ([principals are not qualified by issuer](https://ate.dev/docs/guides/authorization/index.md#known-limitations)). Anyone who can get a token for `/` in the other cluster acts as `system:serviceaccount::` here, whether or not that ServiceAccount exists here. That includes `ate-system:ate-client`, the default [bootstrap owner](https://ate.dev/docs/guides/authorization/index.md#bootstrap-owners), so replace it first, and run the caller as a ServiceAccount that no one else in either cluster can get a token for. 2. **Trust the other cluster's issuer.** Add a [JWT provider](#configuring-jwt-providers) whose `audiences` include the audience the pod requests. As a cluster admin of the other cluster, run `kubectl --context get --raw /.well-known/openid-configuration | jq -r .issuer` to print its issuer; on GKE it is `https://container.googleapis.com/v1/projects//locations//clusters/`. `ate-api-server` fetches the issuer's discovery document and keys over HTTPS, so they must be reachable from the Substrate cluster. GKE serves them publicly. 3. **Reach and verify `ate-api-server`.** The install doesn't expose it outside its cluster, and its NetworkPolicy admits only the in-cluster peers described [above](#pods-in-the-substrate-cluster). Exposing it and admitting that traffic is up to you; the [threat model](https://ate.dev/docs/security/threat-model/index.md) (T-09) says not to expose it directly to the internet. Its serving certificate names only in-cluster Service names such as `api.ate-system.svc`, so use that as the TLS server name whatever address you dial. Copy the serving CA into the other cluster, and copy it again when the CA rotates: ```sh kubectl --context get clustertrustbundles -l podcert.ate.dev/canarying=live -o json \ | jq -r '.items[] | select(.spec.signerName == "servicedns.podcert.ate.dev/identity") | .spec.trustBundle' \ > ca.crt kubectl --context -n create configmap ate-api-ca --from-file=ca.crt \ --dry-run=client -o yaml | kubectl --context apply -f - ``` `kubectl ate` works there only with a kubeconfig for the Substrate cluster, which it needs to read the trust bundle and, for `logs actor` and `top worker`, worker pods' logs and metrics. ## CI jobs Trust the CI system's OIDC issuer and send its token. Any workflow on GitHub can request a GitHub Actions token for any audience, so this provider authenticates every workflow on GitHub, and access policies decide which of them may do anything. Use an audience unique to this install, so that a token minted for it can't be replayed against another install that trusts GitHub: ```yaml - name: github-actions issuer: https://token.actions.githubusercontent.com audiences: - ate-api.prod.example.com ``` and request a token in the job: ```yaml permissions: contents: read id-token: write steps: - run: | curl -sSf -H "Authorization: bearer $ACTIONS_ID_TOKEN_REQUEST_TOKEN" \ "$ACTIONS_ID_TOKEN_REQUEST_URL&audience=ate-api.prod.example.com" \ | jq -r .value > "$RUNNER_TEMP/ate.token" kubectl ate --token-file "$RUNNER_TEMP/ate.token" get actors -a ci ``` The principal is the token's `sub`. Grant only a `sub` that names the repository by ID and either a protected branch or an environment that only protected branches can deploy to, such as `repo:my-org@123456/my-repo@456789:ref:refs/heads/main`. Repositories created before July 15, 2026 use names instead of IDs until they opt in or are renamed or transferred, and a name can pass to a new owner. Don't grant a `sub` ending in `:pull_request`, which every pull request job without an environment shares. GitHub's [OIDC reference](https://docs.github.com/en/actions/reference/security/oidc) lists the forms; decode a token before granting it. The job also needs a kubeconfig for the Substrate cluster. On GKE, get one with `google-github-actions/auth` and `google-github-actions/get-gke-credentials` using a service account. As for people [above](#your-own-serviceaccount), give it only Kubernetes Engine Cluster Viewer in IAM, and bind the `ate-api-client` roles [above](#people-running-kubectl-ate) to it; its `--user` is the service account's email. ## Configuring JWT providers `ate-api-server` reads the issuers it trusts from the file passed to `--authentication-config`, which the install mounts from the `ate-api-authentication` ConfigMap: ```yaml jwtProviders: - name: kubernetes issuer: https://kubernetes.default.svc.cluster.local audiences: - api.ate-system.svc certificateAuthorityFile: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt discoveryTokenFile: /var/run/secrets/kubernetes.io/serviceaccount/token - name: google issuer: https://accounts.google.com audiences: - 32555940559.apps.googleusercontent.com ``` - Provider names and issuers must be unique. `issuer` must be an HTTPS URL and `audiences` must be non-empty. - `ate-api-server` checks a token with the provider whose `issuer` equals its `iss`. The signature must verify against keys from the issuer's OpenID discovery document, and `aud` must contain one of the provider's audiences. - `certificateAuthorityFile` and `discoveryTokenFile` are for an in-cluster issuer such as `https://kubernetes.default.svc`, and `ate-setup` sets them for one. `certificateAuthorityFile` replaces the system roots for that provider. `discoveryTokenFile` requires it, and sends the token in that file to the issuer and to any `/openid/v1/jwks` URL. The example points it at `ate-api-server`'s own ServiceAccount token, so set it only for the Substrate cluster's own issuer. `ate-api-server` re-reads both files when they change. - The Google values are what the Google Cloud CLI issues today: issuer `https://accounts.google.com`, and the Cloud SDK's shared client ID as the audience. They are examples, not built-in defaults; check the claims your identity provider issues. The principal ID doesn't record which provider issued the token, so only configure providers you trust not to issue a token whose `sub` is another principal's ID. `ate-setup` creates the ConfigMap on first install with one provider, `kubernetes`, for the cluster's ServiceAccount issuer and the audience `api.ate-system.svc`. Later runs leave it alone, so providers you add survive them. If it detected the wrong issuer, set `ateapi.expectedJWTIssuer`, run `ate-setup create api-authentication-config`, which rewrites the file and drops any providers you added, then restart `ate-api-server` as below. To add a provider, edit the ConfigMap and restart `ate-api-server`, which reads it only at startup: ```sh kubectl -n ate-system get configmap ate-api-authentication \ -o jsonpath='{.data.authentication\.yaml}' > authentication.yaml # Add the provider to authentication.yaml, then: kubectl -n ate-system create configmap ate-api-authentication \ --from-file=authentication.yaml \ --dry-run=client -o yaml | kubectl replace -f - kubectl -n ate-system rollout restart deployment/ate-api-server kubectl -n ate-system rollout status deployment/ate-api-server ``` If the rollout doesn't finish, the new pod (the one restarting in `kubectl -n ate-system get pods -l app=ate-api-server`) logs what is wrong with the config. Fix the ConfigMap promptly: the old pods mount it too and read the bad file if they restart. `ate-api-server` caches signing keys and refreshes them periodically, and when a token names an unknown key ID. ## Not yet supported - **Signing in.** `kubectl ate` can't run a command to fetch or refresh a token; people obtain tokens themselves. - **`kubectl ate` without Kubernetes access** to the Substrate cluster, or, from outside the cluster, without port-forward rights in `ate-system`. - **Reaching `ate-api-server` from outside its cluster.** The install doesn't expose it. - **JWT providers at install time.** `ate-setup` configures only the cluster's own ServiceAccount issuer; add others to the ConfigMap afterwards. ================================================================================ SECTION: HOW-TO GUIDES PAGE: Authorization URL: https://ate.dev/docs/guides/authorization/ MARKDOWN: https://ate.dev/docs/guides/authorization/index.md SOURCE: docs/guides/authorization.md ================================================================================ # Authorization > How ate-api-server checks every call against an OpenFGA model, and how to manage roles and access. > [!WARNING] > **Do not keep `ate-client` as the bootstrap owner outside development.** > `ate-setup` makes the shared `ate-client` ServiceAccount the default > bootstrap owner so the demos, benchmarks and e2e tests work. Anyone who can > mint a token for it, including every `kubectl ate` user who doesn't pass > `--token-file`, is then a global owner. For a shared or production install, > set [bootstrap owners](#bootstrap-owners) to your operators' own principals. `ate-api-server` checks every API call against an [OpenFGA](https://openfga.dev) model before running it. OpenFGA is embedded in `ate-api-server` and stores its data in the same PostgreSQL database. For how callers authenticate, see [authentication.md](https://ate.dev/docs/guides/authentication/index.md). ## How a call is checked 1. **Authentication** turns the credential into a principal ID, or rejects it with `Unauthenticated`. 2. **Authorization** allows the call if the principal is a bootstrap owner, holds a role from an access policy that permits it, or is a Substrate component with the system role it needs. Otherwise, or if the RPC has no rule, it returns `PermissionDenied`. ## Settings Authorization is always enforced. `ate-setup` reads the setting below and passes it to `ate-api-server` as the `--authz-bootstrap-owners` flag on its container; an install that doesn't use `ate-setup` sets that flag in its manifest. `ate-api-server` won't start without a bootstrap owner. A change rolls `ate-api-server`. | Setting | Environment variable | Example | |---|---|---| | `ateapi.authz.bootstrapOwners` | `ATE_API_AUTHZ_BOOTSTRAP_OWNERS` | `alice@example.com,bob@example.com` | If you don't set `ateapi.authz.bootstrapOwners`, `ate-setup` uses the shared `ate-client` ServiceAccount (`system:serviceaccount::ate-client`), so the demos, benchmarks and e2e tests work. See the warning at the top of this page. ## Principals A principal is the ID authentication produces. Access policies name it as `user:`; bootstrap owners list the bare ``. | Caller | `` | |---|---| | Kubernetes ServiceAccount token | `system:serviceaccount::` | | Other JWT provider, such as Google | The token's `sub` (for Google, a numeric account ID, not the email) | | Pod certificate (mTLS) | Its SPIFFE ID, e.g. `spiffe://cluster.local/ns/benchmarking/sa/default` | A denied call names the principal in the error with `:` percent-encoded (`user:system%3Aserviceaccount%3A...`); decode it before using it. ## Bootstrap owners Bootstrap owners always hold `owner` on the global scope, whatever the access policies say. They write an install's first access policies and recover access if those break. - They come from configuration, not the API. `ate-api-server` reads them at startup and exits if the list is empty. Restarting it with a new list takes effect immediately. - Any principal `ate-api-server` can authenticate can be one. - `ate-setup` warns whenever `ate-client` is one. - Keep the list short. Grant day-to-day admins `owner` in the global access policy instead, where it can be revoked through the API. For example, with a JWT provider whose tokens carry the user's email as `sub`: ```sh ATE_API_AUTHZ_BOOTSTRAP_OWNERS=alice@example.com,bob@example.com go run ./cmd/ate-setup deploy ate-system ``` To make your Google account a bootstrap owner, use its numeric `sub`: ```sh gcloud auth print-identity-token | cut -d. -f2 | base64 -d 2>/dev/null | jq -r .sub # your sub ATE_API_AUTHZ_BOOTSTRAP_OWNERS=,system:serviceaccount:ate-system:ate-client \ go run ./cmd/ate-setup deploy ate-system # keep ate-client for demos and benchmarks ``` Then add the Google provider to the `ate-api-authentication` ConfigMap and restart `ate-api-server` ([authentication.md](https://ate.dev/docs/guides/authentication/index.md#configuring-jwt-providers)), and act as yourself: ```sh gcloud auth print-identity-token > me.token kubectl ate --token-file me.token get atespaces ``` ## `kubectl ate` and `ate-setup` identities `kubectl ate` sends the token from `--token-file`. Without one, it mints a token for `ate-client` (or `ATE_CLIENT_SERVICE_ACCOUNT`); see [authentication.md](https://ate.dev/docs/guides/authentication/index.md#people-running-kubectl-ate). `ate-setup` calls the API only to deploy and delete demos and benchmarks, as `ate-client`. If you leave `ate-client` out of the bootstrap owners, grant it access in a policy or those steps are denied. ## Roles Roles are granted on the **global** scope, which covers every atespace, or on one **atespace**. | Permission | Global owner | Global viewer | Atespace owner | Atespace editor | Atespace viewer | |---|:-:|:-:|:-:|:-:|:-:| | Create atespaces; list all atespaces | ✅ | list only | | | | | Get the atespace | ✅ | ✅ | ✅ | ✅ | ✅ | | Delete the atespace | ✅ | | ✅ | | | | Get access policies | ✅ | ✅ | ✅ (its own) | ✅ (its own) | ✅ (its own) | | Create, update, delete access policies | ✅ | | ✅ (its own) | | | | Get and list actors, actor templates, tags, egress policies | ✅ | ✅ | ✅ | ✅ | ✅ | | Create, update, delete actors and actor templates | ✅ | | ✅ | ✅ | | | Resume, suspend, revert actors | ✅ | | ✅ | ✅ | | | Create, update, delete egress policies | ✅ | | ✅ | ✅ | | | Create, publish, delete tags | ✅ | | ✅ | ✅ | | | Create an actor from a tag (`sourceTag`) | ✅ | | ✅ | ✅ | | | Workers: get, list, list assignments | ✅ | ✅ | | | | | Workers: create, update, delete, drain | ✅ | | | | | ### System roles Substrate's components hold roles that access policies cannot grant, and only through their pod certificates. `ate-api-server`'s `--controller-spiffe-id`, `--egress-gateway-spiffe-id`, `--ingress-router-spiffe-id` and `--api-server-sidecar-spiffe-id` flags bind them, defaulting to the install's; `atelet`'s comes from `--atelet-service-account`. | Role | Held by | Allows | |---|---|---| | `controller` | `ate-controller` | Create, get, list, update, delete and drain workers | | `egress_gateway` | `atenet-egress`, and any other SPIFFE IDs passed with `--egress-gateway-spiffe-id`, which ate-setup sets from `ATE_EGRESS_GATEWAY_EXTRA_SPIFFE_IDS` | Get actors and their egress policies; list actors across all atespaces; mint actor JWTs | | `ingress_router` | `atenet-router` | Resume actors; list actors across all atespaces | | `atelet` | `atelet` | Get actors' egress policies, for the worker that enforces them | | `api_server_sidecar` | the `ate-api-server` pod identity, which its sidecars share, today the objectstore control plugin | Get actors, to derive the snapshot prefixes a per-actor storage credential may cover | Only `egress_gateway` can mint actor credentials: not even a global owner can, since a credential lets its holder act as the actor. ## Checks per RPC A gRPC interceptor makes every check before the handler runs ([`registry.go`](https://github.com/agent-substrate/substrate/blob/main/cmd/ateapi/internal/authz/registry.go), [`model.fga`](https://github.com/agent-substrate/substrate/blob/main/cmd/ateapi/internal/authz/model.fga)). When an RPC lists several checks, all must pass. Roles inherit: an owner is also an editor and a viewer, an editor is also a viewer, and a global role holds the same role in every atespace. So "atespace editor" below also means atespace owner, global owner, and so on. ### Atespaces and access policies | RPC | Permission, on | Allowed for | |---|---|---| | `CreateAtespace` | `can_create_atespace`, global | Global owner | | `ListAtespaces` | `can_list_atespaces`, global | Global viewer | | `GetAtespace` | `can_get`, the atespace | Atespace viewer | | `DeleteAtespace` | `can_delete`, the atespace | Atespace owner | | `GetGlobalAccessPolicy` | `can_get_access_policy`, global | Global viewer | | `CreateGlobalAccessPolicy`, `UpdateGlobalAccessPolicy` | `can_create_access_policy` / `can_update_access_policy`, global | Global owner | | `GetAtespaceAccessPolicy` | `can_get_access_policy`, the atespace | Atespace viewer | | `CreateAtespaceAccessPolicy`, `UpdateAtespaceAccessPolicy`, `DeleteAtespaceAccessPolicy` | `can_create_access_policy` / `can_update_access_policy` / `can_delete_access_policy`, the atespace | Atespace owner | ### Actor templates and actors | RPC | Permission, on | Allowed for | |---|---|---| | `CreateActorTemplate` | `can_create_actor_template`, `metadata.atespace` | Atespace editor | | `GetActorTemplate` | `can_get`, the template | Viewer of its atespace | | `DeleteActorTemplate` | `can_delete`, the template | Editor of its atespace | | `ListActorTemplates` | `can_list_actor_templates`, the atespace, or global if none is given | Atespace viewer; global viewer for all atespaces | | `CreateActor` | 1. `can_create_actor`, the actor's atespace
2. `can_use_template`, the actor template
3. `can_use_tag`, the `sourceTag`, if one is given | 1. Atespace editor
2. Viewer of the template's atespace, which may be another one
3. Editor of the tag's atespace. When `sourceTag` is omitted, the handler resolves the template's golden tag without checking `can_use_tag`. | | `UpdateActor` | 1. `can_update`, the actor
2. `can_use_template`, the template in the request | 1. Editor of its atespace
2. Viewer of the template's atespace, so an actor can't be switched to a template its editor can't use | | `GetActor` | `can_get`, the actor | Viewer of its atespace; `atenet-egress`; the objectstore control plugin | | `DeleteActor` | `can_delete`, the actor | Editor of its atespace | | `ListActors` | `can_list_actors`, the atespace, or global if none is given | Atespace viewer; for all atespaces, global viewer, `atenet-router` or `atenet-egress` | | `SuspendActor`, `RevertActor` | `can_suspend` / `can_revert`, the actor | Editor of its atespace | | `ResumeActor` | `can_resume`, the actor | Editor of its atespace; `atenet-router` | | `GetActorEgressPolicy` | `can_get_egress_policy`, the actor | Viewer of its atespace; `atenet-egress`; `atelet` | | `CreateActorEgressPolicy`, `UpdateActorEgressPolicy`, `DeleteActorEgressPolicy` | `can_create_egress_policy` / `can_update_egress_policy` / `can_delete_egress_policy`, the actor | Editor of its atespace | | `MintActorJwt` | `can_mint_actor_jwt`, the actor | `atenet-egress` only | ### Tags | RPC | Permission, on | Allowed for | |---|---|---| | `CreateTag` | `can_create_tag`, the source actor's atespace, where the tag is stored | Atespace editor | | `GetTag` | `can_get`, the tag | Viewer of its atespace | | `UpdateTag` (including publishing) | `can_update`, the tag | Editor of its atespace | | `DeleteTag` | `can_delete`, the tag | Editor of its atespace | | `ListTags` | `can_list_tags`, the atespace, or global if none is given | Atespace viewer; global viewer for all atespaces | ### Workers Workers have no atespace; their permissions come from the global scope. | RPC | Permission, on | Allowed for | |---|---|---| | `CreateWorker` | `can_create_worker`, global | Global owner; `ate-controller` | | `ListWorkers` | `can_list_workers`, global | Global viewer; `ate-controller` | | `GetWorker` | `can_get`, the worker | Global viewer; `ate-controller` | | `UpdateWorker`, `MarkWorkerAvailable`, `MarkWorkerUnavailable` | `can_update`, the worker | Global owner; `ate-controller` | | `DeleteWorker` | `can_delete`, the worker | Global owner; `ate-controller` | | `DrainWorker` | `can_drain`, the worker | Global owner; `ate-controller` | | `ListWorkerActorAssignments` | `can_list_actor_assignments`, the worker | Global viewer | ### WorkerService The interceptor skips these; each handler requires `atelet`'s pod certificate. | RPC | The caller must be | |---|---| | `AcquireWorkerEpoch` | `atelet` on the worker's node | | `RegisterWorker` | `atelet` on the worker's node | | `SetWorkerSandboxRuntimes` | `atelet` on the worker's node | | `ReportActorCrashed` | `atelet` on the node of the worker hosting the actor | | `RequestActorSuspend` | `atelet` on the node of the worker hosting the actor | | `MintWorkerActorJwt` | `atelet` on the node hosting the actor | | `MintWorkerActorCertificate` | `atelet` on the node of the worker the actor is assigned to | | `ReportActorSnapshotState` | `atelet` on the node the actor is placed on | ### Every RPC - An RPC with no rule is denied, and a test fails the build if a served RPC has none. - gRPC reflection is open to any authenticated caller, like Kubernetes API discovery, since it returns only the API schema, which is public in `api/ateapi/v1/ateapi.proto`. Any other streaming RPC is denied. - An unauthenticated caller gets `Unauthenticated` before its request is read. - A missing or invalid resource name returns `InvalidArgument`; no check is skipped. ## Managing access Each scope has one access policy, named `default`, managed with `kubectl ate` using `--global` or `-a `: ```yaml # policy.yaml metadata: name: default bindings: - role: editor members: - user:system:serviceaccount:ate-users:alice - role: viewer members: - user:system:serviceaccount:ate-users:bob ``` ```sh kubectl ate --token-file me.token create access-policy -a team-a -f policy.yaml kubectl ate --token-file me.token get access-policy -a team-a -o yaml > current.yaml kubectl ate --token-file me.token update access-policy -a team-a -f current.yaml kubectl ate --token-file me.token delete access-policy -a team-a ``` - Global policies accept `owner` and `viewer`; atespace policies also accept `editor`. - `update` replaces the whole policy and needs the `uid` and `version` from `get -o yaml`. It fails if the policy changed, or was deleted and recreated, since. - The global policy can't be deleted; update it with no bindings instead. - Deleting an atespace deletes its policy, so a recreated atespace starts with none. A request authorized against the old atespace and still in flight across the delete and recreate can apply to the new one. See [Known limitations](#known-limitations). - Changes take effect on the next call. ## Recovering access If no principal anyone can authenticate as holds `owner` (for example, a Google account listed by email instead of its numeric `sub`), anyone who can redeploy `ate-api-server` can recover: redeploy with a bootstrap owner they can use, fix the policies, then redeploy with the intended owners. ```sh ATE_API_AUTHZ_BOOTSTRAP_OWNERS=alice@example.com go run ./cmd/ate-setup deploy ate-system ``` ## Production readiness For a shared cluster, beyond replacing the default bootstrap owner: - **Give people their own identities** in access policies, and have them pass `--token-file` with a token from a trusted JWT provider or their own ServiceAccount ([authentication.md](https://ate.dev/docs/guides/authentication/index.md#people-running-kubectl-ate)). - **Limit who can act as a ServiceAccount bootstrap owner.** Anyone who can create `serviceaccounts/token` or pods in its namespace, or change RBAC there, can get its token. Check who can, and alert on the audit events: ```sh kubectl auth can-i create serviceaccounts/ate-client --subresource=token -n ate-system --as alice@example.com ``` ### Not yet supported - **Groups.** Policy members are individual principals. - **Signing in.** See [authentication.md](https://ate.dev/docs/guides/authentication/index.md#not-yet-supported). ## Known limitations - **Atespace names must not be reused.** Grants are keyed by atespace name, and calls are authorized before the write runs. If an atespace is deleted and recreated under the same name, a request authorized against the old atespace and still in flight across the delete and recreate can land on the new one, including granting access to it or deleting it. Nothing enforces unique names. Once an atespace is deleted, don't create another one under that name; use a fresh name instead. - **Tags can copy an actor's data.** A tag is a durable copy of an actor's memory and disk, including secrets, and outlives the actor. Any atespace editor can tag, publish and clone any actor in the atespace. - **Principals are not qualified by issuer.** If `ate-api-server` trusts more than one JWT provider, any of them that can issue a token whose `sub` is another principal's ID, a bootstrap owner's included, acts as that principal. System roles are unaffected: they need the component's pod certificate. - **The atespace is the only boundary.** There are no per-actor or per-template grants. - **Any atelet can read any actor's egress policy.** Every node's atelet shares one identity, so the `atelet` role isn't limited to actors on its node ([#1564](https://github.com/agent-substrate/substrate/issues/1564)). ================================================================================ SECTION: HOW-TO GUIDES PAGE: CSI Volumes URL: https://ate.dev/docs/guides/csi-volumes/ MARKDOWN: https://ate.dev/docs/guides/csi-volumes/index.md SOURCE: docs/guides/csi-volumes.md ================================================================================ # CSI Volumes > Attach CSI volumes to actors. > **Status: Preview.** External CSI volumes are disabled by default behind the `ExternalVolumes` preview gate, and APIs or behavior may change. Substrate integrates with the **Container Storage Interface (CSI)** to provide dynamically provisioned, per-actor external volumes that seamlessly attach and detach as actors transition through their lifecycle. > [!IMPORTANT] > External volumes are a preview feature. The ate API server accepts them only > when it runs with `--preview=ExternalVolumes` (ate-setup's `preview` setting) > and the client opts in to preview features: `kubectl ate --preview`, or the > `Ate-Preview-Allowed: true` gRPC metadata header. The opt-in is needed to > create an actor template with external volumes, or an actor from such a > template. Later operations on that actor, such as resume, suspend and delete, > do not need it. --- ## 1. CSI in Substrate vs. Standard Kubernetes In Kubernetes, volumes are reconciled asynchronously via standard Kubernetes objects (e.g. `PersistentVolumeClaim`, `PersistentVolume`). Agent Substrate takes a different approach tailored for actor lifecycle operations: * **No PV or PVC Objects:** External volumes are declaratively defined in the [`ActorTemplate`](https://ate.dev/docs/reference/api-guide/index.md#2-actortemplate-the-workload-blueprint) via `external.template` and provisioned dynamically for each actor instance. Volume operations are coupled directly with the actor lifecycle. * **Direct Network-Based CSI Controller:** The Substrate control plane (`ateapi`) communicates directly with the CSI Controller gRPC service over the network (via TCP or DNS endpoints, optionally secured with TLS/mTLS). --- ## 2. Dynamic CSI Driver Discovery (`CSIDriverConfig`) To discover and communicate with CSI drivers, Substrate uses dynamic discovery driven by the cluster-scoped **`CSIDriverConfig`** Custom Resource Definition (CRD). ### The `CSIDriverConfig` Resource `CSIDriverConfig` defines the gRPC connection parameters for a specific CSI driver. It bridges the Kubernetes `StorageClass` (referenced in the `ActorTemplate`) to the network endpoint of the CSI Controller service and the local socket path of the CSI Node plugin. ```yaml apiVersion: ate.dev/v1 kind: CSIDriverConfig metadata: name: nfs.csi.k8s.io spec: driverName: nfs.csi.k8s.io controllerEndpoint: tcp://csi-nfs-controller.kube-system.svc.cluster.local:50052 nodeSocketOverride: unix:///var/lib/kubelet/plugins/csi-nfsplugin/csi.sock tls: enabled: true usePodIdentity: true serverName: csi-nfs-controller.kube-system.svc ``` ### Specification (`CSIDriverConfigSpec`) | Field | Type | Description | | :--- | :--- | :--- | | `driverName` | `string` | **Required.** The standard CSI driver name (e.g. `nfs.csi.k8s.io`, `hostpath.csi.k8s.io`, `pd.csi.storage.gke.io`). Matches the `provisioner` field on the referenced Kubernetes `StorageClass`. | | `controllerEndpoint` | `string` | **Required.** The gRPC endpoint for the CSI Controller service. Must be a valid URI starting with `tcp://`, `dns:///`, or `unix://` (e.g., `tcp://csi-controller.kube-system.svc:50051` or `dns:///csi-svc.default.svc:9000`). | | `nodeSocketOverride` | `string` | **Optional.** Override for the CSI Node service Unix domain socket on worker nodes. Must begin with `unix://`. If omitted, Substrate defaults to `unix:///var/lib/kubelet/plugins//csi.sock`. | | `tls` | `*CSIDriverTLSConfig` | **Optional.** Configures TLS or mTLS for the gRPC connection to the `controllerEndpoint`. | #### TLS / mTLS Configuration (`spec.tls`) | Field | Type | Description | | :--- | :--- | :--- | | `enabled` | `bool` | **Required.** Enables TLS/mTLS for the gRPC connection. | | `usePodIdentity` | `bool` | **Optional.** When `true`, reuses Substrate's SPIFFE Pod Identity certificates for mutual TLS (mTLS) with dynamic CA trust bundle verification and rotation. Must be `true` when `enabled` is `true`. | | `serverName` | `string` | **Required.** Expected server name (SAN) for TLS certificate verification when TLS is enabled. | > [!NOTE] > For details on exposing CSI controller endpoints over the network and configuring CSI node DaemonSets with required mount propagations, see the [CSI Driver Deployment Guide](https://ate.dev/docs/guides/csi-deployment/index.md). --- ## 3. ActorTemplate: Configuring CSI Volumes External volumes are declared on the `ActorTemplate` resource. For complete details on actor templates, see the [ActorTemplate: The Workload Blueprint](https://ate.dev/docs/reference/api-guide/index.md#2-actortemplate-the-workload-blueprint) section in the Substrate API Guide. ### Volume Configuration Fields To attach a CSI volume to an actor: 1. Define the volume under `volumes` with an `external.template`. 2. Mount the volume inside one or more containers under `containers[].volumeMounts`. #### `volumes[]` ```yaml volumes: - name: my-data-volume external: template: capacityBytes: 10737418240 # 10Gi storageClassName: standard-rwx ``` * `name`: Unique DNS-label-compliant volume name. * `external.template.capacityBytes`: The requested volume size in bytes (e.g. `1073741824` for 1Gi). * `external.template.storageClassName`: Name of a Kubernetes `StorageClass` present in the cluster whose `provisioner` matches a registered `CSIDriverConfig`. #### `containers[].volumeMounts[]` ```yaml volumeMounts: - name: my-data-volume mountPath: /var/data ``` * `name`: Must match the declared `volumes[].name`. * `mountPath`: Unix path inside the container sandbox where the volume will be mounted. > [!NOTE] > All declared volumes in `volumes` must be mounted by at least one container. --- ## 4. End-to-End Example The following example demonstrates setting up an NFS CSI driver with Substrate and deploying an `ActorTemplate` that mounts an external NFS volume. ### Step 1: Create the StorageClass ```yaml apiVersion: storage.k8s.io/v1 kind: StorageClass metadata: name: csi-nfs-sc provisioner: nfs.csi.k8s.io parameters: server: nfs-server.default.svc.cluster.local share: / reclaimPolicy: Delete volumeBindingMode: Immediate mountOptions: - nfsvers=4.1 ``` ### Step 2: Register the CSIDriverConfig ```yaml apiVersion: ate.dev/v1 kind: CSIDriverConfig metadata: name: nfs.csi.k8s.io spec: driverName: nfs.csi.k8s.io controllerEndpoint: tcp://csi-nfs-controller.kube-system.svc.cluster.local:50052 nodeSocketOverride: unix:///var/lib/kubelet/plugins/csi-nfsplugin/csi.sock tls: enabled: true usePodIdentity: true serverName: csi-nfs-controller.kube-system.svc ``` ### Step 3: Define WorkerPool and ActorTemplate Refer to [ActorTemplate: The Workload Blueprint](https://ate.dev/docs/reference/api-guide/index.md#2-actortemplate-the-workload-blueprint) for general template options. The `WorkerPool` is a Kubernetes resource, applied with `kubectl apply`: ```yaml apiVersion: ate.dev/v1 kind: WorkerPool metadata: name: agent-pool namespace: ate-demo labels: workload: stateful-agent spec: replicas: 5 workerImage: ko://go.ate.dev/substrate/cmd/ateworker-gvisor sandboxClasses: - name: gvisor ``` The `ActorTemplate` is a protojson-shaped `ateapi.ActorTemplate`, created through the ate API with `kubectl ate --preview create actor-template -f -` (the `ate-demo` atespace must exist): ```yaml metadata: atespace: ate-demo name: stateful-agent-template workerSelector: matchLabels: workload: stateful-agent containers: - name: agent image: gcr.io/my-project/agent-app@sha256:7f28ab0... volumeMounts: - name: shared-storage mountPath: /mnt/shared wakeupProbe: httpGet: path: /readyz port: 8080 resources: limits: cpuMillicores: 1000 memoryBytes: 1073741824 # 1Gi volumes: - name: shared-storage external: template: capacityBytes: 5368709120 # 5Gi storageClassName: csi-nfs-sc ``` ================================================================================ SECTION: HOW-TO GUIDES PAGE: Deploying CSI Drivers URL: https://ate.dev/docs/guides/csi-deployment/ MARKDOWN: https://ate.dev/docs/guides/csi-deployment/index.md SOURCE: docs/guides/csi-deployment.md ================================================================================ # Deploying CSI Drivers > Deploy CSI drivers for use with Agent Substrate. > **Status: Preview.** External CSI volumes are disabled by default behind the `ExternalVolumes` preview gate, and APIs or behavior may change. This guide explains how to deploy and configure Container Storage Interface (CSI) drivers to work with Agent Substrate. > [!IMPORTANT] > External CSI volumes are a preview feature. Enabling them requires running Substrate with `--preview=ExternalVolumes` and opting in on the client (`kubectl ate --preview`, or the `Ate-Preview-Allowed: true` gRPC metadata header). See [CSI Volumes](https://ate.dev/docs/guides/csi-volumes/index.md) for usage details. In standard Kubernetes, CSI drivers are deployed to interact with asynchronous sidecars and kubelet. Substrate directly orchestrates storage operations from its control plane (`ateapi`) and node daemon (`atelet`), introducing specific deployment and networking requirements for both the **CSI Controller** and the **CSI Node DaemonSet**. --- ## 1. Overview of Deployment Differences | Component | Standard Kubernetes CSI Deployment | Agent Substrate CSI Deployment | | :--- | :--- | :--- | | **CSI Controller Communication** | Controller listens strictly on an in-pod Unix domain socket (`/csi/csi.sock`). Sidecars (`csi-provisioner`, `csi-attacher`) watch `etcd` and talk to the driver locally. | The Substrate control plane (`ateapi`) communicates directly with the CSI Controller via gRPC over the cluster network. The controller must be exposed over TCP via a Kubernetes `Service`. | | **Controller Authentication** | None (in-pod socket communication only). | Optional but recommended: Mutual TLS (mTLS) authenticated via Substrate's SPIFFE Pod Identity certificates and dynamic CA rotation. | | **CSI Node Communication** | `kubelet` communicates with the node plugin Unix domain socket. | Substrate's node daemon (`atelet`) connects directly to the CSI node plugin Unix socket (discovered via `CSIDriverConfig`). | | **Node Mount Propagation** | `kubelet` manages mounts under `/var/lib/kubelet/pods`. | The CSI Node plugin must mount Substrate target directories on the host (e.g. `/var/lib/ate`) with `mountPropagation: Bidirectional` so that `atelet` can bind-mount them into actor sandboxes. | --- ## 2. CSI Controller Deployment Requirements Because Substrate's control plane (`ateapi`) invokes CSI Controller methods directly (`CreateVolume`, `ControllerPublishVolume`, `ControllerUnpublishVolume`, `DeleteVolume`), the CSI Controller's gRPC endpoint must be reachable across the cluster network. ### Exposing the Controller Endpoint Standard CSI controller deployments can be made network-accessible and secured in one of two ways: 1. **TLS Proxy Sidecar with Envoy (Recommended):** Add an Envoy reverse proxy sidecar container to the CSI Controller Deployment/StatefulSet. Envoy terminates incoming TLS/mTLS gRPC connections from `ateapi` and forwards the requests over HTTP/2 to the local Unix domain socket (`/csi/csi.sock`). 2. **Native TCP Endpoint:** If the CSI driver binary natively supports listening on a network socket (e.g. `--endpoint=tcp://0.0.0.0:`), configure the driver container to bind directly to a network port. ### Exposing via a Kubernetes `Service` Create a Kubernetes `Service` targeting the CSI Controller pods. This provides a stable DNS name and load balancing across controller replicas. Create the `Service` before the controller pods when the proxy's serving certificate comes from Substrate's `servicedns.podcert.ate.dev/identity` signer: it issues the certificate for the `..svc` names of the `Service`s selecting the pod, and refuses to issue one for a pod no `Service` selects yet. The `Service` name is also what `spec.tls.serverName` in the `CSIDriverConfig` must be. ### Securing with mTLS and Pod Identity When exposing the CSI Controller over the network, communication should be secured with mutual TLS (mTLS). In the [`CSIDriverConfig`](https://ate.dev/docs/guides/csi-volumes/index.md#2-dynamic-csi-driver-discovery-csidriverconfig) resource, configure the following fields under `spec.tls`: * `enabled: true`: Enables TLS/mTLS for gRPC communication between `ateapi` and the CSI Controller service. If `false` (or omitted), communication falls back to unencrypted plaintext gRPC. * `usePodIdentity: true`: Instructs `ateapi` to authenticate using Substrate's SPIFFE Pod Identity client certificate (`/run/podidentity.podcert.ate.dev/credential-bundle.pem`) and verify the controller's server certificate using Substrate's dynamic Service DNS CA trust bundle (`/run/servicedns.podcert.ate.dev/trust-bundle.pem`). * `serverName: `: Specifies the expected Subject Alternative Name (SAN) of the controller for TLS certificate verification. Required when `enabled` is `true`. > [!IMPORTANT] > **What happens if `usePodIdentity` is `false`?** > Currently, Substrate requires `usePodIdentity: true` whenever `spec.tls.enabled` is `true`. If `usePodIdentity` is set to `false` (or omitted when TLS is enabled), the `CSIDriverConfig` resource will be rejected by CRD validation (`tls.usePodIdentity must be true when tls.enabled is true; manual certificates are not yet supported`), and `ateapi` will return an error at runtime. ### Restricting Controller Communication to `ateapi` Because the CSI Controller executes privileged storage operations (such as creating and deleting volumes), network access should be restricted to the Substrate control plane (`ateapi`). This can be enforced via the Envoy proxy sidecar. In the Envoy reverse proxy sidecar, enable client certificate validation (`require_client_certificate: true`) and configure the validation context with the Substrate Pod Identity CA (`signerName: podidentity.podcert.ate.dev/identity`). In `match_typed_subject_alt_names`, specify `ateapi`'s exact SPIFFE SAN: ```yaml match_typed_subject_alt_names: - san_type: URI matcher: exact: "spiffe://cluster.local/ns/ate-system/sa/ate-api-server" ``` Envoy will reject any connection from clients that do not present a valid certificate issued to `ateapi`. ### Example: `csi-nfs` Controller Deployment with Envoy TLS Proxy The following example demonstrates configuring the `csi-nfs-controller` Deployment with an **Envoy sidecar proxy** to terminate mTLS, authenticate with Service DNS certificates, validate `ateapi`'s SPIFFE identity via Pod Identity CA, and forward gRPC traffic to `/csi/csi.sock`: #### 1. Envoy Proxy ConfigMap ```yaml apiVersion: v1 kind: ConfigMap metadata: name: csi-nfs-envoy-config namespace: kube-system data: envoy.yaml: | # SDS (used for the TLS certs below) requires a node id and cluster. node: id: csi-nfs-envoy cluster: csi-nfs-envoy static_resources: listeners: - name: grpc_listener address: socket_address: address: 0.0.0.0 port_value: 10000 filter_chains: - transport_socket: name: envoy.transport_sockets.tls typed_config: "@type": type.googleapis.com/envoy.extensions.transport_sockets.tls.v3.DownstreamTlsContext require_client_certificate: true common_tls_context: alpn_protocols: - h2 # Serving cert and trust bundle via filesystem SDS (the # sds-*.yaml keys below): watched_directory only works on # SDS-delivered secrets. tls_certificate_sds_secret_configs: - name: servicedns_serving_cert sds_config: resource_api_version: V3 path_config_source: path: /etc/envoy/sds-servicedns-cert.yaml combined_validation_context: default_validation_context: match_typed_subject_alt_names: - san_type: URI matcher: exact: "spiffe://cluster.local/ns/ate-system/sa/ate-api-server" validation_context_sds_secret_config: name: podidentity_validation_context sds_config: resource_api_version: V3 path_config_source: path: /etc/envoy/sds-podidentity-validation.yaml filters: - name: envoy.filters.network.http_connection_manager typed_config: "@type": type.googleapis.com/envoy.extensions.filters.network.http_connection_manager.v3.HttpConnectionManager stat_prefix: grpc_csi http2_protocol_options: {} route_config: name: local_route virtual_hosts: - name: csi_controller domains: ["*"] routes: - match: prefix: "/" route: cluster: csi_unix_socket http_filters: - name: envoy.filters.http.router typed_config: "@type": type.googleapis.com/envoy.extensions.filters.http.router.v3.Router clusters: - name: csi_unix_socket connect_timeout: 0.25s type: STATIC typed_extension_protocol_options: envoy.extensions.upstreams.http.v3.HttpProtocolOptions: "@type": type.googleapis.com/envoy.extensions.upstreams.http.v3.HttpProtocolOptions explicit_http2_config: {} load_assignment: cluster_name: csi_unix_socket endpoints: - lb_endpoints: - endpoint: address: pipe: path: /csi/csi.sock # SDS resources referenced by the listener above. sds-servicedns-cert.yaml: | resources: - "@type": type.googleapis.com/envoy.extensions.transport_sockets.tls.v3.Secret name: servicedns_serving_cert tls_certificate: certificate_chain: filename: /run/servicedns/credential-bundle.pem private_key: filename: /run/servicedns/credential-bundle.pem watched_directory: path: /run/servicedns sds-podidentity-validation.yaml: | resources: - "@type": type.googleapis.com/envoy.extensions.transport_sockets.tls.v3.Secret name: podidentity_validation_context validation_context: trusted_ca: filename: /run/podidentity-ca/trust-bundle.pem watched_directory: path: /run/podidentity-ca ``` > **Note:** `watched_directory` takes effect only on SDS-delivered secrets. Attaching it to an inline `tls_certificates` or `validation_context` entry does nothing: Envoy reads those files once at config load, so a rotated certificate is never picked up and the proxy eventually serves (or trusts) expired material. #### 2. Envoy Sidecar in `csi-nfs-controller` Deployment ```yaml apiVersion: apps/v1 kind: Deployment metadata: name: csi-nfs-controller namespace: kube-system spec: replicas: 1 selector: matchLabels: app: csi-nfs-controller template: metadata: labels: app: csi-nfs-controller spec: containers: # Upstream CSI NFS driver container listening on local Unix socket - name: nfs image: registry.k8s.io/sig-storage/nfsplugin:v4.13.4 args: - "--nodeid=$(NODE_ID)" - "--endpoint=unix:///csi/csi.sock" volumeMounts: - name: socket-dir mountPath: /csi # Envoy proxy sidecar terminating mTLS and forwarding to /csi/csi.sock - name: envoy-proxy image: envoyproxy/envoy:v1.39.5@sha256:8fc594a5d85eb76dca24155cfb9208882406873932b20cf33649a0a611194c43 command: - "/usr/local/bin/envoy" args: - "-c" - "/etc/envoy/envoy.yaml" # The proxy needs no privilege, only to be able to connect to the # driver's socket, which the nfs container creates root-only. securityContext: runAsUser: 0 allowPrivilegeEscalation: false readOnlyRootFilesystem: true capabilities: drop: ["ALL"] volumeMounts: - name: socket-dir mountPath: /csi - name: envoy-config mountPath: /etc/envoy - name: servicedns-certs mountPath: /run/servicedns readOnly: true - name: podidentity-ca mountPath: /run/podidentity-ca readOnly: true volumes: - name: socket-dir emptyDir: {} - name: envoy-config configMap: name: csi-nfs-envoy-config - name: servicedns-certs projected: sources: - podCertificate: signerName: servicedns.podcert.ate.dev/identity keyType: ECDSAP256 credentialBundlePath: credential-bundle.pem - name: podidentity-ca projected: sources: - clusterTrustBundle: signerName: podidentity.podcert.ate.dev/identity labelSelector: matchLabels: podcert.ate.dev/canarying: live path: trust-bundle.pem ``` #### 3. Kubernetes `Service` for the Controller ```yaml apiVersion: v1 kind: Service metadata: name: csi-nfs-controller namespace: kube-system spec: selector: app: csi-nfs-controller ports: - name: grpc port: 50052 targetPort: 10000 ``` #### 4. Corresponding `CSIDriverConfig` ```yaml apiVersion: ate.dev/v1 kind: CSIDriverConfig metadata: name: nfs.csi.k8s.io spec: driverName: nfs.csi.k8s.io controllerEndpoint: tcp://csi-nfs-controller.kube-system.svc.cluster.local:50052 nodeSocketOverride: unix:///var/lib/kubelet/plugins/csi-nfsplugin/csi.sock tls: enabled: true usePodIdentity: true # The one DNS name servicednssigner puts in the proxy's certificate is # ..svc, so that, not the cluster.local form the # endpoint dials, is the server name to verify. serverName: csi-nfs-controller.kube-system.svc ``` --- ## 3. CSI Node DaemonSet Mount Requirements The CSI Node plugin runs as a DaemonSet on each worker node and handles `NodeStageVolume` and `NodePublishVolume` operations when an actor is resumed. ### Mount Propagation Requirements When the CSI Node plugin mounts an external volume (such as an NFS share or a formatted block device), the filesystem mount is created inside the plugin container. For Substrate's node supervisor (`atelet`) and worker sandboxes (`ateworker-gvisor`, `ateworker-microvm`) on the host to see this filesystem mount, the following requirements must be met: 1. **Bidirectional Mount Propagation (`mountPropagation: Bidirectional`):** The volume mount on the host Substrate target directory (e.g. `/var/lib/ate`) in the CSI Node plugin container must have `mountPropagation: Bidirectional`. This ensures that any mounts made by the CSI plugin inside the container propagate back to the host filesystem. 2. **Unix Domain Socket Accessibility:** The CSI Node plugin must place its Unix domain socket under `/var/lib/kubelet/plugins//` (or the path defined in `CSIDriverConfig.spec.nodeSocketOverride`), which is shared with `atelet`. ### Example: `csi-nfs-node` DaemonSet Configuration In the `csi-nfs` driver deployment (see [`hack/third_party/csi-driver-nfs/deploy/csi-nfs-node.yaml`](https://github.com/agent-substrate/substrate/blob/main/hack/third_party/csi-driver-nfs/deploy/csi-nfs-node.yaml) and [`hack/setup-csi-nfs-kind.sh`](https://github.com/agent-substrate/substrate/blob/main/hack/setup-csi-nfs-kind.sh)), the DaemonSet is configured with bidirectional mount propagation to `/var/lib/ate`: ```yaml apiVersion: apps/v1 kind: DaemonSet metadata: name: csi-nfs-node namespace: kube-system spec: selector: matchLabels: app: csi-nfs-node template: metadata: labels: app: csi-nfs-node spec: hostNetwork: true dnsPolicy: ClusterFirstWithHostNet serviceAccountName: csi-nfs-node-sa nodeSelector: kubernetes.io/os: linux containers: - name: nfs image: registry.k8s.io/sig-storage/nfsplugin:v4.13.4 securityContext: privileged: true capabilities: add: ["SYS_ADMIN"] allowPrivilegeEscalation: true args: - "--nodeid=$(NODE_ID)" - "--endpoint=unix:///csi/csi.sock" env: - name: NODE_ID valueFrom: fieldRef: fieldPath: spec.nodeName volumeMounts: # CSI communication socket shared with host/atelet - name: socket-dir mountPath: /csi # Target directory with bidirectional mount propagation - name: ate-base mountPath: /var/lib/ate mountPropagation: Bidirectional volumes: - name: socket-dir hostPath: path: /var/lib/kubelet/plugins/csi-nfsplugin type: DirectoryOrCreate - name: ate-base hostPath: path: /var/lib/ate type: DirectoryOrCreate ``` ================================================================================ SECTION: HOW-TO GUIDES PAGE: Egress Traffic URL: https://ate.dev/docs/guides/egress-traffic/ MARKDOWN: https://ate.dev/docs/guides/egress-traffic/index.md SOURCE: docs/guides/egress-traffic.md ================================================================================ # Egress Traffic > Supported outbound actor protocols, DNS relay filtering, and what actors see when egress is refused. This document specifies the supported egress traffic for [GA](https://github.com/agent-substrate/substrate/milestone/3). Actor TCP egress (except DNS to the sandbox's gateway) is redirected to atunnel, which opens a CONNECT tunnel to the egress gateway. Egress gateway applies policy. DNS over UDP and TCP goes to the sandbox's DNS relay, which forwards a query to the worker pod's resolvers only when the queried hostname matches a host or host wildcard in the actor's egress policy. Queries from an actor without a policy, or for a hostname not allowed by any rule, are answered `NXDOMAIN`. A policy created, updated, or deleted while the actor runs applies to its DNS within about 5 seconds. > [!NOTE] > The actor sees a denied name as an ordinary resolver failure, such as > `wget: bad address 'api.github.com'` or `curl: (6) Could not resolve host`. > Nothing logs the denied name. If an external name does not resolve, check that > the actor's egress policy has a rule for it with > `kubectl ate get egress-policy -a -o yaml`. Other UDP and other traffic is rejected by nftables and never reaches the egress gateway. This applies to an actor's own telemetry too: an OpenTelemetry SDK inside the actor cannot reach the collector until the actor has an `http` egress rule for it. See [Actor telemetry needs an egress policy](https://ate.dev/docs/guides/observability/index.md#actor-telemetry-needs-an-egress-policy). ## TCP | Port | Traffic | Behavior | Path | What the actor sees when refused | | :---- | :---- | :---- | :---- | :---- | | any | HTTP(S) 1.1 / 2 | Supported with policy controls | atunnel -> egress gateway -> origin | `403 Forbidden`. For HTTPS whose SNI matches no `https` or `tls_passthrough` rule, the gateway closes the connection during the TLS handshake, without an HTTP response. | | any | TLS whose SNI matches a `tls_passthrough` rule | Relayed to the SNI hostname without inspection, so WebSocket, CONNECT, and other TCP inside it get through | atunnel -> egress gateway -> origin | Closed during the TLS handshake. | | any | WebSocket | Blocked | atunnel -> egress gateway | `403 Forbidden` | | any | Standard HTTP(S) CONNECT (forward-proxy tunnel) | Blocked | atunnel -> egress gateway | `403 Forbidden` | | 53 | DNS to the sandbox's gateway | Allowed for hostnames matching the actor's egress policy | sandbox DNS relay -> worker pod's resolvers | `NXDOMAIN` | | any | Any other TCP | Blocked | atunnel -> egress gateway | A client that sends first gets `400 Bad Request` before the connection closes. If the client waits for the server to speak first, the connection closes after about 1 second with no data. Only the gateway logs the refusal. | See the [Network Egress Contract](https://ate.dev/docs/guides/network-egress/index.md#refusal-responses-and-policy-changes) for refusal status codes, policy-refresh semantics, and the distinction between DNS refusals and HTTP policy denials. ## UDP | Port | Traffic | Behavior | Path | What the actor sees when refused | | :---- | :---- | :---- | :---- | :---- | | 53 | DNS to the sandbox's gateway | Allowed for hostnames matching the actor's egress policy | sandbox DNS relay -> worker pod's resolvers | `NXDOMAIN` | | any other | Any other UDP | Blocked | n/a | Packets are dropped, not rejected: no ICMP port-unreachable is sent, so the client hangs until its own timeout. | ## Other protocols Everything that is neither TCP nor UDP is rejected with ICMP network unreachable. ## Requesting support If you would like Substrate to support egress traffic that is blocked above, please [file an issue](https://github.com/agent-substrate/substrate/issues/new) describing your use case. ================================================================================ SECTION: HOW-TO GUIDES PAGE: Network Egress URL: https://ate.dev/docs/guides/network-egress/ MARKDOWN: https://ate.dev/docs/guides/network-egress/index.md SOURCE: docs/guides/network-egress.md ================================================================================ # Network Egress > How atunnel and the egress gateway tunnel, authenticate, and apply policy to outbound actor traffic. Last updated: 10/20/2026 ## Overview Agent Substrate enforces that all outbound traffic from an actor, except traffic sent to TCP or UDP destination port 53, MUST go through an egress policy enforcement point (PEP). That PEP can (optionally) inspect, filter, or redirect the traffic according to substrate-native egress policies. This document describes the contract between substrate and egress PEPs. ## Transport At present, actor TCP traffic exits the sandbox through [`atunnel`](https://ate.dev/docs/reference/glossary/index.md#components) and is forwarded on to the PEP. The exception is DNS: TCP and UDP traffic to port 53 on the sandbox's gateway bypasses the PEP and is answered by the sandbox's DNS relay, which forwards a query to the worker pod's resolvers only when the queried hostname matches a host or host wildcard in the actor's egress policy; queries from an actor without a policy, or for a hostname not matched by any rule, are answered `NXDOMAIN`. All other protocols are blocked. `atunnel` is configured with the PEP's network address and port at startup and uses that address to open an mTLS HTTP/1.1 or HTTP/2 CONNECT tunnel to the PEP, as negotiated by ALPN. The CONNECT request target and HTTP/1.1 `Host` header (HTTP/2 `:authority`) are set to the original destination IP and port that `atunnel` derived from the actor's TCP connection. Actor networking is currently IPv4-only. Before traffic reaches the egress PEP, ateworker rejects packets destined for the IPv4 link-local range `169.254.0.0/16` or the IPv6 link-local range `fe80::/10`. The internal actor network `169.254.17.0/30` is exempt because it carries traffic for the actor gateway, DNS relay, ingress, and wakeup probes. Localhost traffic remains inside the actor sandbox and does not traverse the PEP. Access to non-link-local destinations remains controlled by substrate egress policy and, at the `WorkerPool` boundary, Kubernetes NetworkPolicy. ## Refusal responses and policy changes The [supported traffic table](https://ate.dev/docs/guides/egress-traffic/index.md) defines the permitted protocols. For cleartext HTTP requests and HTTP requests on an authorized intercepted TLS connection, a request denied by policy MUST receive HTTP 403. WebSocket upgrades, other HTTP upgrades, and actor-originated CONNECT requests inside the tunnel MUST also receive HTTP 403, regardless of whether the destination is otherwise allowed. This applies to cleartext HTTP and intercepted HTTPS, including mixed-case HTTP upgrade tokens, and does not affect atunnel's outer CONNECT. The PEP MUST NOT open an upstream connection or apply credential effects for these refused requests. Response bodies are implementation-specific. A TLS passthrough authorization lasts for that connection. A policy update MUST apply to new connections after cached policy decisions expire; it need not terminate an already authorized connection. If a new ClientHello no longer matches a passthrough rule, the PEP MUST NOT pass the TLS connection through. If an HTTPS rule matches the SNI and original destination port, the PEP MAY intercept the connection and authorize each HTTP request under that rule. Otherwise, the PEP MUST close the connection at ClientHello, without completing the TLS handshake, returning an HTTP response, or opening an upstream connection. An HTTP rule alone MUST NOT authorize TLS. Missing or invalid SNI MUST also close the connection without completing TLS. Unsupported protocols MAY close the connection without an HTTP response. DNS authorization happens before the PEP and is independent of the protocol and port being dialed. Removing every rule for a hostname can prevent resolution before the connection reaches the PEP; that refusal does not promise an HTTP status or a particular resolver error string. ## Trust Boundary In substrate, we consider everything originating from the actor to be untrusted. The egress PEP MUST NOT trust any decision or attestation coming from the actor itself; it MUST instead trust only information carried through substrate-controlled channels (e.g. the actor certificate and the CONNECT request produced by atunnel). For example, the PEP MUST NOT treat an actor-provided HTTP hostname or TLS SNI as proof that the original destination has that name. The PEP MAY use the hostname as a policy input, but when authorization is based on that hostname, it MUST route the traffic to the authorized hostname rather than use the hostname to authorize an arbitrary IP address from the CONNECT request. ## Server-side TLS The egress PEP MUST serve TLS using a certificate that is valid for the configured PEP hostname and chains to a CA trusted by `atunnel`. `atunnel` uses the configured PEP hostname as the TLS server name and MUST reject the connection if it cannot verify the PEP's certificate. ## Client Authorization `atunnel` MUST present an actor-specific client certificate when connecting to the PEP. Before the actor's workload starts, `atunnel` generates a private key and requests a short-lived certificate from substrate through the node-local atelet, and egress is active from then on: a workload may need the network to become ready, so egress does not wait for the wakeup probe the way ingress does. The private key remains in `atunnel`. The certificate identifies the actor by its atespace, name, and UID and is scoped to the `atunnel` purpose. The PEP MUST require and verify the client certificate before accepting a CONNECT request. It MUST verify the certificate chain and lifetime against the Actor Identity CA, require the client authentication usage, and require exactly one valid `ActorIdentity` extension whose atespace, actor name, actor UID, and purpose are present. The current extension OID is `1.3.6.1.4.1.11129.2.12.2`; it is allocated under Google's enterprise number and will change after the CNCF donation is complete. The purpose MUST be `atunnel`, and the certificate's actor URI SAN MUST identify the same actor as the extension. The PEP MUST then confirm with ate-api-server that the actor still exists, that its current UID matches the certificate, and that it is placed on a worker: `RESUMING` (booting or restoring there) or `RUNNING`. The PEP MUST reject the CONNECT request if any of these checks fail. ```text CURRENT CONNECT EGRESS PATH (one tunnel per actor TCP connection except destination port 53) Actor sandbox Worker pod / ateworker Egress PEP Authorized upstream +------------------+ +--------------------------+ +-------------------------+ +------------------+ | Actor process | | nftables | | mTLS listener | | Target selected | | (untrusted) | | TCP REDIRECT | | + CONNECT terminator | | under policy | +--------+---------+ +------------+-------------+ +------------+------------+ +--------+---------+ | connect(IP:port) | | | +-------------------------------->| | | | SO_ORIGINAL_DST | | v | | +------------+-------------+ | | | atunnel | | | | (trusted tunnel client) | | | +------------+-------------+ | | | | | | TLS 1.2+ mutual authentication | | | actor-specific client certificate | | +---------------------------------->| verify certificate and | | | authorize current actor | | | | | HTTP/1.1 CONNECT IP:port | | | Host: IP:port | | +---------------------------------->| validate destination; | | | evaluate egress policy | | | | | HTTP/1.1 2xx | | |<----------------------------------+ | | | CIDR/all policy: | | | dial now; | | | hostname policy: | | | inspect inner request | | | before dialing | | +------------------------->| | | | |<================ raw bidirectional bytes inside CONNECT ===========>|<========================>| | (PEP may pass through, inspect, redirect, or MITM according to policy) | ``` ================================================================================ SECTION: HOW-TO GUIDES PAGE: MITM Egress Interception URL: https://ate.dev/docs/guides/egress-trust-bundle/ MARKDOWN: https://ate.dev/docs/guides/egress-trust-bundle/index.md SOURCE: docs/guides/egress-trust-bundle.md ================================================================================ # MITM Egress Interception > Configure actors to trust the egress gateway's MITM CA for HTTPS header and credential injection. ## Overview Substrate has special handling for traffic leaving actors --- it is routed through the egress gateway. Connections that use TLS (likely the vast majority of your traffic to external destinations, like requests to google.com) require special configuration inside your actor in order to successfully establish the TLS connection. The exact needed configuration is slightly different depending on whether you are using `tls_passthrough` (no interception) rules, or `https` (interception) rules. This doc gives you a recipe for configuring your actors that will work for both scenarios. Connections allowed via a `tls_passthrough` rule are just forwarded through the egress gateway, with no modifications. The TLS connections are established directly between the actor and the external destination, so the egress gateway has no visibility into the connection (beyond metadata such as the destination). Connections allowed via an `https` rule have TLS terminated at the egress gateway, so that the gateway can inspect the requests and inject credentials. The gateway then open a new TLS connection to the real destination, and sends the modified requests. The egress gateway does the TLS termination using server certificates it mints on-the-fly using a built-in CA. In order for the interception to work, the code you run inside the actor must be explicitly configured to trust this built-in CA. Substrate can automatically inject the root certificates for this CA into the actor's filesystem using a SystemInfo volume with a TrustBundle data source. Note: DNS traffic from the actor (both TCP and UDP) is not forwarded to the gateway. Instead, the sandbox's DNS relay forwards a query to the worker pod's resolvers, which are the host cluster's configured DNS servers, only when the queried hostname matches a host or host wildcard in the actor's egress policy: queries from an actor without a policy, or for a hostname not matched by any rule, are answered `NXDOMAIN`. ## Configuring a workload to trust public and man-in-the-middle CAs Add a `systemInfo` volume with a `trustBundle` data source to the actor template (the manifest you pass to `kubectl ate create actor-template -f`), and mount it in each container that makes TLS connections: ```yaml metadata: atespace: my-atespace name: my-actor containers: - name: app image: registry.example.com/app@sha256: volumeMounts: - name: system-info mountPath: /run/ate/egress-trust-bundle volumes: - name: system-info systemInfo: dataSources: # This file will contain trust anchors for both the MITM CA and standard # public WebPKI CAs, meaning it will work for actors that use `https` rules, # `tls_passthrough` rules, or a mix. - trustBundle: names: - egress-mitm.ate.dev - system-roots.ate.dev path: trust-bundle.pem resources: limits: cpuMillicores: 1000 memoryBytes: 536870912 ``` Substrate only offers a few built-in trust bundles with well-known names: * `egress-mitm.ate.dev` — the egress gateway CA. * `system-roots.ate.dev` — A selection of public CA root certificates provided by Substrate. In official Substrate images, this is the Mozilla Root Store as shipped on Debian (consumed via the distroless-static base image). You then need to configure your application to use the trust-bundle file. In general, this is application specific, but many runtimes respect the SSL_CERT_FILE environment variables. Here are some common scenarios and the configuration they support: | Runtime/Library | Setting | Notes | |---|---|---| | Go | SSL_CERT_FILE=/run/ate/egress-trust-bundle/trust-bundle.pem | | | Python (most things) | SSL_CERT_FILE=/run/ate/egress-trust-bundle/trust-bundle.pem | Anything using OpenSSL's default paths: `ssl`, `urllib`, `aiohttp`, `psql` ... | | Python (requests) | REQUESTS_CA_BUNDLE=/run/ate/egress-trust-bundle/trust-bundle.pem | | | pip (uses requests) | REQUESTS_CA_BUNDLE=/run/ate/egress-trust-bundle/trust-bundle.pem | | | Node.js | NODE_EXTRA_CA_CERTS=/run/ate/egress-trust-bundle/trust-bundle.pem | | | Deno | DENO_CERT=/run/ate/egress-trust-bundle/trust-bundle.pem | | | curl | SSL_CERT_FILE=/run/ate/egress-trust-bundle/trust-bundle.pem || | git (HTTPS remotes) | GIT_SSL_CAINFO=/run/ate/egress-trust-bundle/trust-bundle.pem || Note: If your application relies on an additional private CA built into the system trust store of your base container image (for example, if your enterprise setup has an additional layer of TLS man-in-the-middle for traffic leaving your network), then you will find that whether or not it is respected depends on the software you are running inside your actor. For example, Deno and Python requests treat their environment variable overrides as *completely replacing* the system trust store, where as most things that support SSL_CERT_FILE treat it as *additive* to the system trust store. If you need to support this scenario, you will need to use an entrypoint wrapper to build a file containing the substrate-provided roots along with your custom roots (or directly configure this setup in your application's startup logic). See the [example](#example-entrypoint-wrapper-for-custom-roots) below. ## Example entrypoint wrapper for custom roots ```sh #!/bin/sh set -eu sys=/etc/ssl/certs/ca-certificates.crt # RHEL family: /etc/pki/tls/certs/ca-bundle.crt ca=/run/ate/egress-trust-bundle/trust-bundle.pem out=/tmp/ca-bundle.pem { cat "$sys"; echo; cat "$ca"; } > "$out" export SSL_CERT_FILE="$out" REQUESTS_CA_BUNDLE="$out" CURL_CA_BUNDLE="$out" GIT_SSL_CAINFO="$out" PIP_CERT="$out" export NODE_EXTRA_CA_CERTS="$ca" DENO_CERT="$ca" exec "$@" ``` Note: Do not bake the bundle into the image at build time. It will tie the image to one cluster's CA and breaks on rotation. ## Verify `demos/egress/egress-template.yaml.tmpl` is a complete working template that does exactly this. Deploy it: ```bash go run ./cmd/ate-setup deploy demo egress ``` Then drive an actor's egress at an HTTPS URL an `https` rule allows and confirm it returns a response rather than a certificate error. The minted leaf chains to no public root, so a `200` is positive evidence that the projected bundle did the validating. ## Operational notes * Problems with trust bundles will block actor startup. Error messages: - `... unknown trust bundle "foo"`: You tried to use an unknown trust bundle name. - `... trust bundle "egress-mitm.ate.dev": ClusterTrustBundle "egress-mitm.ate.dev:mitm:primary-bundle" not found`: Your substrate installation is misconfigured. * Certificate errors are likely problems with the actor, not the gateway: If you see an error message like `... certificate signed by unknown authority ...`, the most likely cause is that your actor is not configured to trust the MITM CA correctly. * This CA configuration does not help the actor authenticate to the destination. For that, refer to our documentation on egress credential injection. ## See also * [API Configuration Guide](https://ate.dev/docs/reference/api-guide/index.md) — the full `systemInfo` volume reference. * `demos/egress/README.md` — how tunneled egress and actor-identity authentication fit together. * `cmd/atenet/internal/router/README.md` — the gateway side of the MITM leg. ================================================================================ SECTION: HOW-TO GUIDES PAGE: Egress Credential Injection URL: https://ate.dev/docs/guides/egress-credential-injection/ MARKDOWN: https://ate.dev/docs/guides/egress-credential-injection/index.md SOURCE: docs/guides/egress-credential-injection.md ================================================================================ # Egress Credential Injection > Put secrets into an actor's outbound HTTPS requests without the actor ever holding them. Egress credential injection puts a secret into an actor's outbound HTTP and HTTPS requests without the actor ever holding it. It works on requests the egress gateway can read: cleartext HTTP that an `http` rule in the actor's `EgressPolicy` allows, and HTTPS that an `https` rule allows. An `https` rule has the gateway terminate the actor's TLS for the names it lists and re-originate it (see [egress-trust-bundle.md](https://ate.dev/docs/guides/egress-trust-bundle/index.md)). When the rule carries a `replaceHeaders` effect and the actor's request includes a header it names, the gateway resolves the referenced credential from a **credential provider** and replaces the header's value with it (for example `Authorization: Bearer `) before the request leaves the cluster. The actor never holds the secret. It sends the header with a placeholder value, which the gateway discards, so it cannot read, snapshot, or exfiltrate the secret, nor choose the value that leaves the cluster. The gateway can also inject an **actor JWT** in place of a stored secret: a token that ateapi mints for the actor making the request, which an upstream that trusts Substrate's issuer can verify. See [Actor JWTs](#actor-jwts). ## When you need this Actors call APIs that need bearer tokens or API keys, and those secrets must stay out of the actor's filesystem, environment, and snapshots. Other APIs need to know which actor is calling and accept a signed token that says so; for those, inject an actor JWT. Injection is supported on HTTP and HTTPS requests allowed by an `http` or `https` rule with credential replacement effects. For HTTPS, the actor must trust the gateway CA — see [egress-trust-bundle.md](https://ate.dev/docs/guides/egress-trust-bundle/index.md). ## How it works ```mermaid flowchart LR actor["Actor"] gateway["Egress gateway
applies the EgressPolicy"] provider["Credential provider"] store[("Secret store")] ateapi["ateapi"] upstream["Upstream API"] actor -->|"HTTPS request,
placeholder header"| gateway gateway -->|"credentialUri:
fetch credential
for this actor"| provider provider -->|"authorize, then read"| store gateway -->|"actorJwt:
mint a JWT
for this actor"| ateapi gateway -->|"request with
the credential"| upstream ``` 1. The actor makes an HTTPS request, sending the header the policy names with a placeholder value. For a name an `https` rule lists, the gateway terminates the actor's TLS — the actor trusts the gateway CA through a projected trust bundle — and decides the decrypted request against the actor's `EgressPolicy`. 2. If the deciding rule has `replaceHeaders` and the request carries one of those headers, the gateway asks the credential provider for the credential that header names. 3. The provider decides whether that actor may have that credential, reads it from its secret store, and returns it. The provider is the **only** component in the path with access to secret storage — the gateway never reads secrets at rest. 4. The gateway replaces the placeholder with the credential and sends the request on to the origin over a new TLS connection. For an actor JWT, steps 2 and 3 go to ateapi instead of a credential provider: the gateway asks ateapi to mint a token for the actor, or reuses one it minted earlier. The reference k8s secret provider, `cmd/credential-provider/kubernetes-secrets`, resolves Kubernetes Secrets and is the one [For cluster admins](#for-cluster-admins) deploys. ## The policy Injection is declared in the `effects` of an `http` or `https` rule. This policy allows HTTPS to `api.example.com` on port 443, the default, and replaces the actor's `Authorization` header with a credential: ```yaml rules: - https: hostnames: ["api.example.com"] effects: replaceHeaders: - header: Authorization prefix: "Bearer " credentialUri: ate-secret://k8s.io/default/ns1/example-api/token ``` The actor then sends the header with any placeholder value: ```bash curl -H 'Authorization: placeholder' https://api.example.com/v1/items ``` This URI names the sample Secret deployed under [Enable it](#enable-it). The install's namespace policy grants nothing, so the actor gets a 403 until a cluster admin grants its atespace access to the `ns1` namespace. * `header` names the request header to replace. * `prefix` is prepended verbatim to the credential — include the separator, e.g. `"Bearer "` with the trailing space. * `credentialUri` names the credential; see [Credential URIs](#credential-uris). ### Credential URIs A credential URI has the form `ate-secret:///`. The host names the provider that resolves it, and the provider interprets the path. The API requires the host to be a DNS-1123 subdomain and the path to have at least two non-empty segments. It rejects a port, userinfo, query, fragment, percent-encoding, or trailing slash. A URI the provider refuses denies the request with 403. * **Kubernetes Secrets** (provider class `k8s.io`) — see [The reference provider](#the-reference-provider). * **Google Cloud Secret Manager** (provider class `secretmanager.googleapis.com`) — see its [README](https://github.com/agent-substrate-plugins/credprovider-gcp-secret-manager#credential-uris). ### Actor JWTs An entry can set `actorJwt` in place of `credentialUri`. The gateway then puts the prefix and a JWT that ateapi minted for the actor into the header. No credential provider is involved, so this works on a gateway installed with `--credential-provider='{"enabled":false}'`. ```yaml rules: - https: hostnames: ["api.example.com"] effects: replaceHeaders: - header: Authorization prefix: "Bearer " actorJwt: audiences: ["https://api.example.com"] expirationSeconds: 900 ``` * `audiences` become the token's `aud` claim: 1 to 16 distinct values, each up to 512 characters. Use the value the upstream checks the audience against. * `expirationSeconds` is the token's lifetime, from 300 to 3600 seconds, with no default. * An entry sets exactly one of `credentialUri` and `actorJwt`. The token's `sub` is `actor//`, and its `iss` is the issuer the install gave ateapi (`ACTOR_JWT_ISSUER`). It also names the actor's atespace, name, and UID. ateapi signs it with the install's actor JWT signing key, which `kubectl ate admin` rotates (see the [kubectl-ate README](https://ate.dev/docs/reference/kubectl-ate/index.md)). The gateway reuses a token for the same actor, audiences, and lifetime until a third of its lifetime is left, so the upstream always gets a token with at least that much time remaining: 300 seconds for the 900-second lifetime above. Concurrent requests that need the same token share one mint. ## What the gateway does | Situation | Outcome | |---|---| | Request decided by an `http` or `https` rule, carries the header, provider configured, credential resolves | Header value replaced with the credential; request re-originated upstream | | Request decided by an `http` or `https` rule, does not carry the header | Forwarded unchanged; no credential is fetched | | Request decided by an `http` or `https` rule, carries the header, no provider configured (`--credential-provider='{"enabled":false}'`) | **500** (Envoy) or **403** (agentgateway), fail closed | | Secret missing, or namespace not authorized for the atespace | **403**, fail closed | | Provider unreachable or timed out | **503**, fail closed but retryable | | Provider returns an empty credential, or one containing control characters | **503**, fail closed | | URI names a provider class this gateway does not serve; unusable header name; unparseable URI | **500** (Envoy) or **403** (agentgateway), fail closed | | Request decided by an `http` or `https` rule, carries the header of an `actorJwt` entry, mint succeeds | Header value replaced with the prefix and the actor's JWT, with or without a provider | | `actorJwt` entry, the actor no longer exists | **403**, fail closed | | `actorJwt` entry, ateapi unreachable or timed out | **503**, fail closed but retryable | | `actorJwt` entry, any other mint failure | **500** (Envoy) or **403** (agentgateway), fail closed | A request that does not carry the header skips injection. On an HTTP or HTTPS request that carries it, any failure to produce the credential, including having no provider, denies the request rather than letting it out without the credential. ## For cluster admins ### Enable it **1. The gateway.** The provider is selected when the egress gateway is deployed, with `deploy ate-system` or `deploy atenet`, for either the Envoy or agentgateway dataplane. The selection is required: an install that wants no injection says so with `{"enabled":false}`. ```bash go run ./cmd/ate-setup deploy atenet --credential-provider='{"name":"k8s.io"}' ``` `--credential-provider` (or `ATE_CREDENTIAL_PROVIDER` when the flag is absent) is a JSON object with these keys, or a JSON array of such objects to select several providers; any other key is an error: | Key | Value | |---|---| | `enabled` | Defaults to `true`. `false` turns injection off, and then no other key is allowed. An array entry can't be `false` | | `name` | Required unless `enabled` is `false`. A DNS name: `k8s.io` for the bundled Kubernetes Secrets provider, or the name of a provider you deploy, which is the host of the `ate-secret://` URIs it serves, e.g. `vault.example.com` for `ate-secret://vault.example.com/...`. Names in an array must differ. A policy URI naming no configured provider fails closed | | `address` | The `host:port` the gateway dials the provider at. Required for a provider you deploy; optional with `k8s.io`, where it defaults to the bundled provider's Service, `k8s-credential-provider.ate-system.svc:50051` | For example: ```bash --credential-provider='{"name":"k8s.io"}' --credential-provider='{"enabled":false}' --credential-provider='{"name":"vault.example.com","address":"vault-provider.ate-system.svc:50051"}' --credential-provider='[{"name":"k8s.io"},{"name":"vault.example.com","address":"vault-provider.ate-system.svc:50051"}]' ``` **2. The provider.** With `k8s.io` among the providers, the install deploys the Kubernetes Secrets provider itself, from `manifests/egress-credential-injection/`, and waits for it alongside the gateway. The install applies: * the provider's ServiceAccount, its Deployment and its Service in `ate-system`; * a ClusterRole and ClusterRoleBinding that let it read Secrets in every namespace; * the `k8s-credential-provider-namespace-policy` ConfigMap, only when it is absent (see below); and * a NetworkPolicy that admits only the egress gateway's pods (`app: atenet-egress`), on the provider's gRPC port. Every other connection to the provider pods is dropped. The NetworkPolicy takes effect only on a cluster whose network plugin enforces NetworkPolicy. Elsewhere the provider's check of the gateway's client certificate is the only restriction. A selection without `k8s.io`, including `{"enabled":false}`, removes a bundled provider an earlier install left behind, NetworkPolicy included. A provider of your own is yours to deploy, and until something serves the configured address every matching injection rule fails closed with 503. The provider enforces an atespace→namespace authorization policy, the `k8s-credential-provider-namespace-policy` ConfigMap in `ate-system`. The install creates it empty, granting nothing, and only when it is absent, so a redeploy keeps edits. Grant each atespace the namespaces whose Secrets it may resolve: ```bash kubectl -n ate-system edit configmap k8s-credential-provider-namespace-policy ``` ```yaml namespace-policy.yaml: | policies: - atespace: team-a allowedNamespaces: - ns1 ``` The provider loads the policy **once at startup** and does not yet reload it, so after editing, restart the provider: ```bash kubectl -n ate-system rollout restart deployment/k8s-credential-provider ``` For a first smoke test, grant your actor's atespace `ns1` as above and apply a sample secret that the URI in the sample policy resolves to: ```bash kubectl apply -f manifests/egress-credential-injection/sample-secret.yaml ``` **3. The actors.** Actors can now add `replaceHeaders` effects to their `http` or `https` rules, as described under [The policy](#the-policy). An actor with `https` rules also needs the projected egress trust bundle to do TLS through the gateway at all (see [egress-trust-bundle.md](https://ate.dev/docs/guides/egress-trust-bundle/index.md)). Actor JWT entries need no provider; ateapi signs them with a key that every install creates. ### Verify Confirm the provider is ready and its NetworkPolicy is in place: ```bash kubectl -n ate-system rollout status deployment/k8s-credential-provider kubectl -n ate-system get networkpolicy k8s-credential-provider ``` Then give an actor an `https` rule for `httpbin.org` that replaces a header, have it request `https://httpbin.org/headers` with that header set to a placeholder, and look for the credential in the echoed response. On the Envoy dataplane, the gateway's `ext-proc` container logs why a request was denied: ```bash kubectl -n ate-system logs deployment/atenet-egress -c ext-proc | grep 'egress denied' ``` ### Operational notes **Transient 503s right after (re)deploying the provider.** The gateway keeps a long-lived gRPC channel to the provider. If the provider's Service was deleted and recreated, the channel can sit in connect backoff for up to a couple of minutes before re-resolving; injection fails closed with 503 (deliberately retryable) until it reconnects. **Actor JWTs follow the actor's name.** The gateway caches tokens by actor name. An actor that is deleted and recreated under the same name can get its predecessor's token until that token's refresh point, up to two thirds of its lifetime, and that token's UID claim names the old actor. **Scope of the reference provider's RBAC.** The sample deployment grants read on Secrets cluster-wide so the policy may name any namespace; a production deployment should scope this to the namespaces the provider is allowed to serve (see the note in `k8s-credential-provider.yaml`). ## For credential provider developers ### The plugin API The provider is a plugin: any gRPC service that implements `CredentialProvider.FetchSecret` (`api/credprovider/v1/credprovider.proto`) can back injection. To bring your own — reading HashiCorp Vault, Google Secret Manager, or any other secret store — implement the API under your own provider class (the URI host, e.g. `ate-secret://vault.example.com/...`), then have a cluster admin point the gateway at it with `--credential-provider='{"name":"vault.example.com","address":":"}'`. A gateway can front several providers, one per class. It sends each URI to the provider its host names, and a policy URI naming no configured class fails closed rather than being sent to the wrong provider. The gateway reads only the URI host, to pick the provider the URI targets. The provider interprets the path, within the rules under [Credential URIs](#credential-uris). [credprovider-gcp-secret-manager](https://github.com/agent-substrate-plugins/credprovider-gcp-secret-manager) is such a provider for Google Cloud Secret Manager. **Trust model.** The gateway dials the provider over mTLS with its own pod identity, `spiffe://cluster.local/ns/ate-system/sa/atenet-egress`, and every `FetchSecretRequest` carries the SPIFFE ID of the actor the gateway verified. A provider should: * accept only callers presenting the gateway's identity — the reference provider requires a client certificate chaining to the pod-identity trust bundle and pins that URI SAN on every connection; and * enforce an actor-scoped authorization on the asserted actor SPIFFE ID — the gateway attests *which* actor is asking, but what that actor may resolve is the provider's decision. **Status codes.** The gateway maps a `FetchSecret` failure onto the actor's request: `NotFound` and `PermissionDenied` deny with 403 (retrying cannot succeed); `Unavailable` and `DeadlineExceeded` fail closed with a retryable 503; any other code denies with 403. ### The reference provider `cmd/credential-provider/kubernetes-secrets` is a working example to start from. It resolves URIs of the `k8s.io` class, `ate-secret://k8s.io/default///`, to Kubernetes Secret values, and enforces a **default-deny atespace→namespace policy**, read from the `k8s-credential-provider-namespace-policy` ConfigMap: an actor's atespace may only resolve Secrets in namespaces explicitly granted to it. ## See also * [egress-trust-bundle.md](https://ate.dev/docs/guides/egress-trust-bundle/index.md) — the actor-side trust projection needed for HTTPS injection. * `demos/egress/README.md` — how tunneled egress, actor identity, and policy authorization fit together. * `api/credprovider/v1/credprovider.proto` — the provider plugin API and its trust model. * `cmd/credential-provider/kubernetes-secrets` — the reference provider. * `internal/e2e/suites/egresscredinject` — the e2e suite, run against a cluster installed with `--credential-provider='{"name":"k8s.io"}'`. ================================================================================ SECTION: HOW-TO GUIDES PAGE: Observability URL: https://ate.dev/docs/guides/observability/ MARKDOWN: https://ate.dev/docs/guides/observability/index.md SOURCE: docs/guides/observability.md ================================================================================ # Observability > Logs, metrics, and traces for actors and the Agent Substrate components. Agent Substrate manages actors as virtually long-lived entities that can be suspended when idle and resumed on different Kubernetes worker pods over time. This guide explains how Agent Substrate achieves observability across these suspend/resume cycles, allowing you to monitor logs, metrics, and traces as if an actor has been continuously running on a single dedicated machine. ## The Observability Model To make underlying infrastructure transitions transparent, Agent Substrate establishes a standardized metadata model to identify actors across worker pods. These are the same `ate.*` keys the spans and metrics below use, defined once in [`internal/ateattr`](https://github.com/agent-substrate/substrate/tree/main/internal/ateattr): * `ate.actor.name`: The name of the actor (e.g., `my-counter-1` or `test`). * `ate.atespace`: The atespace the actor lives in (e.g., `ate-demo-counter`). * `ate.actor.uid`: Server-assigned UID of the actor, unique to the lifetime of an actor. * `ate.template.name`: The name of the actor's ActorTemplate (e.g., `counter`). * `ate.template.atespace`: The atespace of the actor's ActorTemplate (e.g., `ate-demo-counter`). * `ate.actor.container.name`: The name of the container within the actor that produced the log line (e.g., `counter`), so a multi-container actor's logs can be demultiplexed by container. Absent on the synthetic lifecycle records (`Actor starting`, `Actor restored`, …): those are about the actor, so no container produced them. The `runsc create`, `start`, and `restore` CLI output for the sandbox is wrapped under `_sandbox`. This does not capture sentry or gofer output, which runsc directs to `/dev/null` when debug logging is disabled. Currently, Agent Substrate automatically wraps container output and injects these metadata labels into **container logs**. For metrics and distributed tracing, Agent Substrate provides foundational system telemetry and head-sampled request tracing (10% by default, 1% at the router), with roadmap plans to fully integrate actor-level correlation. `ate.*` is reserved for Substrate. An actor's own log lines pass through untouched except for keys in that namespace, which are dropped before the record is written, so nothing a workload emits can be read as platform-issued attribution. --- ## 1. Logging Agent Substrate captures container standard output/error, wraps them into structured JSON log entries, and injects the `ate.*` metadata labels. ### Active Actor Inspection via CLI For quick, on-demand debugging of an active actor, use the Agent Substrate CLI: ```bash kubectl ate logs actor --atespace [--follow / -f] ``` `--atespace` (short form `-a`) is required: actor names are only unique within an atespace, so an actor is always addressed by `(atespace, name)`. > **Note:** By default, `kubectl ate logs` queries the Kubernetes API of the worker pod where the actor is *currently* running. It is designed for immediate inspection of active actors. To view historical logs across past worker pods and suspension cycles, use a centralized logging backend. #### Example 1: Actor Not Currently Running If an actor is suspended or not assigned to a worker pod, the CLI informs you immediately: ```bash $ kubectl ate logs actor test -a demo Error: actor demo/test is not currently running on any worker pod ``` #### Example 2: Default Clean JSON Lines Output When an active actor is assigned to a worker pod, the CLI outputs clean, uniform JSON lines stripped of Substrate metadata, perfectly matching standard `kubectl logs` behavior: ```bash $ kubectl ate logs actor test -a demo {"time":"2026-05-22T21:49:15.23700774Z","message":"Actor started"} {"time":"2026-05-22T21:49:15.23700774Z","level":"INFO","msg":"Starting counter server on port 80"} {"time":"2026-05-22T21:49:15.255765354Z","count":0,"fshash":"mCY7G4S318ztOUojPTF2NA/W+ZSmWyr+T5K3udFuP50","level":"INFO","msg":"Count"} {"time":"2026-05-22T21:49:25.263744806Z","count":1,"fshash":"mCY7G4S318ztOUojPTF2NA/W+ZSmWyr+T5K3udFuP50","level":"INFO","msg":"Count"} ``` #### Example 3: Streaming/Live Logs (`--follow` or `-f`) To stream actor logs in real-time, append the `--follow` (or `-f`) flag. The CLI is fully actor-aware, automatically resuming the stream if the actor is suspended or migrates to a different worker pod: ```bash $ kubectl ate logs actor test -a demo -f Actor is currently running on pod ate-demo-counter/counter-d8f99-m7d96 {"time":"2026-05-22T21:49:15.255765354Z","count":0,"fshash":"mCY7...","level":"INFO","msg":"Count"} {"time":"2026-05-22T21:49:25.263744806Z","count":1,"fshash":"mCY7...","level":"INFO","msg":"Count"} Actor is currently running on pod ate-demo-counter/counter-ab123-x4y5z {"time":"2026-05-22T21:50:02.123456789Z","count":2,"fshash":"mCY7...","level":"INFO","msg":"Count"} ``` #### Example 4: Filtering by Container or Source An actor can run several containers. By default application container logs and synthetic lifecycle events (`Actor starting`, `Actor started`, `Actor checkpointing`, `Actor checkpointed`, `Actor restoring`, `Actor restored`, `Actor terminated`) are shown. The `_sandbox` logs are excluded so runsc startup diagnostics do not mix with application output. `--container` (short form `-c`) restricts the output to the named container's logs: ```bash kubectl ate logs actor -a -c ``` To view the sandbox's runsc diagnostics explicitly: ```bash kubectl ate logs actor -a -c _sandbox ``` `--source` selects a class of lines instead of a container. `containers` shows application container output with the lifecycle events removed; `_sandbox` remains excluded unless selected with `-c _sandbox`. `lifecycle` shows only the lifecycle events, which is the quickest way to see how often an actor has been suspended and restored on its current worker (a migration moves the actor to another pod, whose log holds none of the earlier workers' events): ```bash $ kubectl ate logs actor test -a demo --source=lifecycle {"time":"2026-10-07T21:31:48.648599369Z","message":"Actor checkpointing","span_id":"c475...","trace_flags":"01","trace_id":"e617..."} {"time":"2026-10-07T21:31:48.791435032Z","message":"Actor checkpointed","span_id":"c475...","trace_flags":"01","trace_id":"e617..."} {"time":"2026-10-07T21:31:59.772566601Z","message":"Actor restoring","span_id":"a68c...","trace_flags":"01","trace_id":"b387..."} {"time":"2026-10-07T21:31:59.96129419Z","message":"Actor restored","span_id":"a68c...","trace_flags":"01","trace_id":"b387..."} ``` `--container` already implies container output, so it can be combined with `--source=containers` but not with `--source=lifecycle`: lifecycle events are not emitted by a container, and the CLI rejects that combination. --- ### Centralized Logging Backends (Multi-Dimensional Aggregation) To view the continuous log history of actors across past and present worker pods, you can integrate Agent Substrate with any centralized logging backend (such as Grafana Loki or Elasticsearch) that supports structured JSON indexing. Because the logging pipeline indexes the core metadata labels, you can query your logs across multiple dimensions using your logging platform's query language: #### 1. Actor-Centric View To track the unified, continuous lifecycle of a single actor regardless of how many times it migrated across worker pods or was suspended/resumed: ```text labels."ate.actor.name"="test" ``` #### 2. Atespace-Centric View To monitor or debug all actor instances in a specific atespace (e.g., analyzing the collective behavior or error rates of all actors belonging to one tenant): ```text labels."ate.atespace"="ate-demo-counter" ``` #### 3. Template-Centric View To monitor or debug all actor instances created from a specific ActorTemplate (e.g., analyzing the collective behavior or error rates of all counter actors). One atespace can run actors from many templates, so this is a distinct dimension from the atespace view above: ```text labels."ate.template.name"="counter" ``` #### 4. Pod-Centric View To inspect the physical worker pod's aggregate stream and see all co-located actors multiplexed together (useful for investigating pod-level resource exhaustion or noisy neighbor issues): ```text resource.labels.pod_name="counter-c995fdf4c-m7d96" ``` --- ### Joining Logs to Traces Substrate-emitted records carry top-level `trace_id`, `span_id`, and `trace_flags` in lowercase hex, the names the [OpenTelemetry spec](https://opentelemetry.io/docs/specs/otel/compatibility/logging_trace_context/) fixes for non-OTLP log formats. That covers component logs (every `slog.*Context` call, via [`internal/contextlogging`](https://github.com/agent-substrate/substrate/tree/main/internal/contextlogging)) and the synthetic actor lifecycle records, so a suspend or resume log line joins the RPC that drove it. An actor's **own** lines carry trace context only if the actor emits these fields itself, in which case they pass through unchanged. Substrate cannot supply them: one forwarder goroutine covers a container's whole output stream and cannot tell which request produced a given line. Per-line correlation for actor logs arrives with the actor telemetry relay ([#853](https://github.com/agent-substrate/substrate/issues/853)), where the actor's own SDK carries the context. --- ### Actor-Attributed Component Logs A component's own `slog` output can also be about a specific actor. Those records take the identity keys from [`internal/ateattr`](https://github.com/agent-substrate/substrate/tree/main/internal/ateattr) too, flat at the top level rather than inside a label group: a component writes no envelope, so a collector lifts the keys straight onto the log record's attributes. `ateattr.ActorLogAttrs` and `ateattr.ActorLogLabels` return the same five keys for this reason, and a test holds them together. Filtering on `ate.actor.uid` therefore finds a component record and an actor's own output alike. atelet's `Restore timing breakdown` and `Checkpoint timing breakdown` are the unsampled per-actor latency records Substrate produces. Each is emitted once per operation, whether it succeeded or failed; a failed one also carries `error.type` (the gRPC code, `DeadlineExceeded` and `Canceled` for context errors), so a reader can leave it out of a latency distribution: ```json {"time":"…","level":"INFO","msg":"Restore timing breakdown", "ate.atespace":"ate-demo-counter","ate.actor.name":"counter-1","ate.actor.uid":"8f2a…", "ate.template.atespace":"ate-demo-counter","ate.template.name":"counter", "ate.snapshot.fidelity":"memory","ate.snapshot.kind":"latest","ate.sandbox.class":"gvisor", "ate.actor.restore.duration.download":0.310, "ate.actor.restore.duration.oci_unpack":0.050, "ate.actor.restore.duration.worker_restore":0.060, "ate.actor.restore.duration.total":0.420, "trace_id":"4bf92f…","span_id":"00f067…","trace_flags":"01"} ``` The duration keys are the [`ate.actor.restore.duration`](#the-metric-registry) instrument's name with an `ate.snapshot.phase` value appended, and they hold **seconds**, matching that instrument's declared unit. The same rules apply as on the histogram: a phase that never ran is absent rather than zero, and phases overlap and do not sum to the total. There is no `ate.snapshot.phase` key on the record — on a datapoint it names the one step timed, and this record carries them all. This is the record to use for a per-actor wake-up distribution. The histogram cannot answer that question at all, because actor identity is barred from metric labels; traces can, but the data plane is head-sampled at 1%. Both ateworkers write records with the same two messages from inside the `worker_restore` and `worker_checkpoint` phases, under `ateworker.actor.restore.duration.` and `ateworker.actor.checkpoint.duration.`. The phases are each runtime's own and are listed in `cmd/ateworker-gvisor/phaselog.go` and `cmd/ateworker-microvm/phaselog.go`; `internal/ateworker/phaselog` renders the record. They are not instruments. The same identity keys make the two layers' records joinable per actor, and a failed operation's record carries `error.type`. ateapi's `Actor state changed` is written once per committed actor state transition. ateapi owns the state machine, so this is where an actor's state and the time it reached it come from: ```json {"time":"…","level":"INFO","msg":"Actor state changed", "ate.atespace":"ate-demo-counter","ate.actor.name":"counter-1","ate.actor.uid":"8f2a…", "ate.template.atespace":"ate-demo-counter","ate.template.name":"counter", "ate.actor.operation.name":"suspend","ate.actor.state":"suspended", "trace_id":"4bf92f…","span_id":"00f067…","trace_flags":"01"} ``` `ate.actor.state` takes the `ateapi.ActorState` values lowercased, so the log vocabulary and the state machine cannot fork. The last record for an actor's uid is the state it is in now, and its timestamp is when that state began. Query it per actor, not in aggregate: neither key is a metric label, because both only ever appear beside actor identity, which [the cardinality rules](#the-metric-registry) keep off metrics entirely. `ate.actor.operation.name` says which operation drove the transition, which the state alone does not: an actor reaches `suspended` both when it is created and when it is suspended. SuspendActor in either durability mode is one operation, labeled `suspend`. The record goes out after the store commit, never before, and the state is read straight off the committed record rather than named by the caller. Every state commit has a version check too, so if two writers race, the one that lost writes nothing. You will never see a state here that the store did not actually hold. Creating an actor counts as a change. A new actor is born suspended, so it gets a record saying so, with `ate.actor.operation.name` set to `create`. Otherwise an actor that is created and never resumed would have no record at all, no matter how long you keep your logs. `deleted` is the only state with no `ateapi.ActorState` behind it. It is written once the actor row is gone, so there is nothing left to read back. It is also the last record an actor ever gets. Without it, `deleting` would be the end of the story, and a delete that finished would look just like one that got stuck. **What this stream won't tell you.** It only writes when something changes. So an actor that has been sitting in the same state since before your logs roll over has no record, and no state. Ask the control plane what state something is in right now. Use this stream to see how it got there and when. Records can also go missing, like any other log, and a gap looks the same as an actor that just sat still. If you want to count activations, use the router's access log instead. `Actor crashed` is the exception, and carries the same two keys with `ate.actor.state="crashed"`. It is written once per committed transition into `ACTOR_STATE_CRASHED`, beside the [`ate.actor.crashes`](#the-metric-registry) increment and under the same already-crashed guard, so the two can never disagree about how many crashes happened. A consumer deriving state therefore selects on `ate.actor.state` under ateapi's resource (`service.name=ateapi`), not on the message. The resource is what makes the record authoritative. [The ateworker relay](#the-ateworker-otlp-relay) admits only ateworker resources, so once worker pods have no direct path to the collector, a record carrying these keys under any other resource cannot come from a worker pod. Until then this is a rule the consumer applies, not one the transport enforces. ```json {"time":"…","level":"ERROR","msg":"Actor crashed", "ate.atespace":"ate-demo-counter","ate.actor.name":"counter-1","ate.actor.uid":"8f2a…", "ate.template.atespace":"ate-demo-counter","ate.template.name":"counter", "ate.actor.operation.name":"resume","ate.actor.state":"crashed", "trace_id":"4bf92f…","span_id":"00f067…","trace_flags":"01"} ``` The counter carries no actor identity, so this record is the only way to attribute a crash to one agent. The decision-point line that precedes it (`Setting Actor to crashed due to Atelet error`) carries only `ate.atespace` and `ate.actor.name`: it is written before the Actor is loaded, so no uid exists yet. ateworker-gvisor's `Sandbox has no processes left while the actor is hosted` is a WARN written when a running actor's gVisor sandbox has died: the sentry crashed, was killed, or was OOM-killed. The ateworker's usage sweep finds it, so it shows up within one sweep (`--usage-sample-interval`, a minute by default), once per activation. `ate.sandbox.oom_kills` is the cgroup's `oom_kill` count, so a non-zero value points at the kernel OOM killer: ```json {"time":"…","level":"WARN","msg":"Sandbox has no processes left while the actor is hosted", "ate.atespace":"ate-demo-counter","ate.actor.name":"counter-1","ate.actor.uid":"8f2a…", "ate.template.atespace":"ate-demo-counter","ate.template.name":"counter", "ate.sandbox.oom_kills":3} ``` The actor still reports `RUNNING` after this record; nothing moves it to `CRASHED` yet ([#2211](https://github.com/agent-substrate/substrate/issues/2211)). It does not fire when an application container exits while the sentry is still up, or during a checkpoint or terminate. The count covers the life of the cgroup leaf, so a leaf left behind by an earlier activation of the same actor carries its count forward. #### The same records over OTLP With `OTEL_LOGS_EXPORTER=otlp` these records go out as OTLP log events instead of stdout, so a collector reads them without knowing substrate's stdout envelope. Unset or `none`, which is what every environment but kind uses today, keeps them on stdout. `otlp,console` writes both, as kind does for debugging. `console` here is the component's own JSON log line, not the OTel SDK's console exporter format. The value is a comma-separated list: names are trimmed, lowercased, and counted once, and an unknown or empty item logs a warning and is skipped while the rest apply. `none` with an exporter, or a value that names no known exporter, logs a warning and keeps the default, `none`. ateapi emits the lifecycle events and the ateworkers the usage samples, each on its own switch; the ateworkers export through [the ateworker relay](#the-ateworker-otlp-relay), which carries logs, traces, and metrics. atecontroller sets each worker pod's `OTEL_LOGS_EXPORTER` from `ATE_WORKER_OTEL_LOGS_EXPORTER` (or its `--otel-logs-exporter` flag), not from its own value, so the kind ConfigMap sets both to `otlp,console`. Three `event.name` values, which is the OTLP LogRecord's own field rather than an attribute: | `event.name` | Body | Severity | Attributes | |---|---|---|---| | `ate.actor.state_changed` | `Actor state changed` | 9 | the five identity keys, `ate.actor.operation.name`, `ate.actor.state` | | `ate.actor.crashed` | `Actor crashed` | 17 | the same keys | | `ate.actor.usage_sampled` | `Actor usage sampled` | 9 | the five identity keys, `ate.workerpool.*`, `ate.sandbox.class`, `ate.stats.*`, `ate.actor.epoch` | `ate.actor.usage_sampled` is the ateworkers' record: one per actor per sampling interval, plus an `initial` record per activation and a `final` one when the ateworker tears it down, told apart by `ate.stats.kind`. Its timestamp is when the measurement was read. Its measurements are named after the `ate.actor.stats.*` instruments and share their units, so `ate.stats.cpu.time` is seconds. They are absent, not zero, while the actor is not measurable, which the record says with `ate.stats.source` unspecified. `ate.stats.cpu.time` restarts at zero with each `ate.actor.epoch`, the unix-nano time the activation began, so a lifetime figure is the sum over epochs of each epoch's highest value; `ate.stats.memory.usage` and `ate.stats.memory.working_set` are absolute, and `ate.stats.memory.peak` is as the source reports it. The same measurements ride `WorkloadStatsSample` on the stats RPCs, with the epoch beside them. A crash is its own name because an event name promises a set of attributes and a crash has a different severity and shape. There is no name per state: `ate.actor.state` already says which transition happened, so a consumer still selects on that one attribute and needs no map from a name to a state. All three names are in [`docs/metrics/registry/events.yaml`](https://github.com/agent-substrate/substrate/blob/main/docs/metrics/registry/events.yaml), which `hack/verify-all.sh` checks. The attributes are the same flat `ate.*` keys as the stdout record, so they arrive as real log attributes with no transform in front of them. Trace context is not among them: it goes on the record's own `TraceId` and `SpanId` fields, where the stdout record's top-level `trace_id`/`span_id` would be mapped to anyway. The instrumentation scope is `go.ate.dev/substrate/internal/actorevent` for every actor event. Select or drop a stream by `event.name`: the lifecycle events must never be sampled, the usage samples may be. **Never sample or filter the lifecycle stream.** A consumer takes the last event for an actor's uid, so one dropped record reports a stale state with no sign that anything is missing. This is the one stream where a sampling policy is a correctness bug rather than a cost trade. **Each record goes to one place unless asked otherwise.** With `otlp` it goes to OTLP, and otherwise to stdout, so a backend that also collects pod stdout, as GKE's logging agent does, does not store it twice. With `otlp` the records are not in `kubectl logs`. `otlp,console` writes both on purpose; on a cluster that also collects pod stdout, that stores each record twice, and the collector should drop one copy by `event.name`. **With `otlp` alone, a lifecycle record can be lost.** The OTLP processor batches, and it drops records with no signal when its queue is full, when the collector stays unreachable after retries, or when the process exits ungracefully. Records written after the component's logger provider shuts down go to stdout. That shutdown runs last, after the server drains, so records written during the drain are still queued and are lost if the final export fails. Where a lost lifecycle record is not acceptable, set `otlp,console` to keep the stdout record as well. ### Per-Actor Usage Events Each ateworker samples its actors on its own timer and writes [`ate.actor.usage_sampled`](#the-same-records-over-otlp) for each one, in this order: an `initial` record when an activation becomes measurable, a `periodic` record per interval, and a `final` record when the ateworker tears the activation down: a checkpoint, a terminate, a graceful shutdown, or a re-host on a microVM. The sampler reads an actor only after its initial reading and until its final record; until then the discovery read reports it as pending. A failed first reading writes no `initial` record, and periodic records start anyway, so an unreachable guest is still reported. These records and the [`ate.actor.stats.*` metrics](#the-metric-registry) are the halves of one split: the metrics aggregate to the bounded template and pool label set a TSDB can hold, and everything carrying actor identity travels here, on the log stream, where cardinality is free. The stdout record carries the same flat `ate.*` keys as the OTLP event, dated when the measurement was read: ```json {"time":"…","level":"INFO","msg":"Actor usage sampled", "ate.atespace":"ate-demo-counter","ate.actor.name":"counter-1","ate.actor.uid":"8f2a…", "ate.template.atespace":"ate-demo-counter","ate.template.name":"counter", "ate.workerpool.namespace":"ate-demo-counter","ate.workerpool.name":"counter-pool", "ate.sandbox.class":"gvisor","ate.stats.source":"cgroup","ate.stats.kind":"periodic", "ate.actor.epoch":1788903185000000000, "ate.stats.memory.usage":39845888,"ate.stats.memory.working_set":31457280, "ate.stats.cpu.time":0.196776,"ate.stats.memory.peak":52428800} ``` * **Consumers filter on** the message and the keys, which sit at the top level rather than in the label group: ```text jsonPayload.msg="Actor usage sampled" AND jsonPayload."ate.actor.uid"="8f2a…" ``` * **The measurements** follow [the rules above](#the-same-records-over-otlp): absent while the actor is pending, CPU in seconds since the activation began, memory usage and working set absolute. An activation that ends any other way, as by a crash, has no `final` record, so take the highest CPU per epoch rather than wait for one. The `final` reading is taken just before the teardown, bounded on a microVM to 10ms. When it fails or times out, the `final` record repeats the newest measured sample, dated when that sample was read, or is pending when none was measured. * **Cadence**: `--usage-sample-interval` on the ateworker sets the timer (default `1m`, at least `50s`). Each sweep writes one periodic record per actor, and `GetActiveWorkloadStats` serves the latest sample rather than reading again. * **Trace context**: `initial` records come from the Run or Restore call (on a microVM, from a goroutine the call starts, so the record can land just before or after the call returns) and `final` records from the call that tears the activation down, a Checkpoint, a Terminate, or, on a microVM, the Run or Restore that re-hosts it; both carry that call's span, like any record in [Joining Logs to Traces](#joining-logs-to-traces). `periodic` records come from the sampler, and a shutdown's `final` records from the drain, which serve no request, so they carry none. * **Delivery is best-effort**: the stdout record ignores `--log-level` and sits behind a bounded queue, so a stalled log consumer costs records, with a warning, never the sampler. **Do not put actor identity on a log-based metric.** Aggregating these events in the log store is what they are for, but a log-based metric built over them must label only by the bounded set (template, sandbox class, source, pool) — promoting `ate.actor.uid` or `ate.actor.name` into a metric label reintroduces exactly the per-actor cardinality this split keeps out of the TSDB. ### Secrets in Component Logs Control plane and node component logs mask proto fields marked `debug_redact`, such as actor JWTs and env var values: a request or response body in a log line shows `[REDACTED]` in their place. Not every process masks them; the developer guide below lists the ones that do not. An actor's own container output is not touched. > **Developer Guide:** For how to log protos that hold secrets, see [Logging Best Practices](https://github.com/agent-substrate/substrate/blob/main/docs/dev/best-practices/logging.md). --- ## 2. Metrics Agent Substrate emits foundational OpenTelemetry system and server metrics to monitor the overall health and performance of the control plane services. Every metric below is emitted by a service binary over OTLP and is **independent of the deployment** — a Kind dev cluster gets the same instruments as production; only the backend differs (see [Where Telemetry Goes](#4-where-telemetry-goes)). > [`docs/metrics/registry/metrics.yaml`](https://github.com/agent-substrate/substrate/blob/main/docs/metrics/registry/metrics.yaml) defines each instrument. Read it when you need all the labels, the bucket limits, or the permitted values of a label. The table below does not have each instrument. The request-parking instruments and the actor resource-usage instruments (`ate.actor.stats.*`) are in the registry only. Refer to [The metric registry](#the-metric-registry); per-actor usage detail is on the [usage records](#per-actor-usage-events). | Metric | Emitted by | Type | Measures | |--------|------------|------|----------| | `rpc.server.call.duration` | ateapi, atelet, atenet-router, atenet-egress, ateworker-gvisor, ateworker-microvm & credprovider (gRPC servers, via `otelgrpc`) | histogram | per-method gRPC latency, request rate, and errors (labels `rpc.method`, `rpc.response.status_code`) | | `ate.actor.crashes` | ateapi | counter | Number of times actors transitioned to `ACTOR_STATE_CRASHED` (labels `ate.actor.operation.name`, `ate.template.atespace`, `ate.template.name`, `ate.workerpool.namespace`, `ate.workerpool.name`, `ate.sandbox.class`) | | `atenet.router.route.duration` | atenet-router, atenet-egress | histogram | Substrate E2E — Envoy receiving a request to Envoy forwarding it to the resolved worker, excluding actor compute and the response (labels `ate.template.atespace`, `ate.template.name`, `ate.router.outcome`, `ate.router.resume`) | | `atelet.snapshot.size` | atelet | histogram | uncompressed allocated size in bytes of each snapshot image written during checkpoint (`st_blocks * 512`, excluding sparse holes in microVM `memory-ranges`) (labels `file.name`, `ate.template.atespace`, `ate.template.name`) | | `ate.workerpool.desired_workers` | atecontroller | up/down counter | number of worker pods requested for a WorkerPool, from `spec.replicas` (labels `ate.workerpool.namespace`, `ate.workerpool.name`) | | `ate.workerpool.ready_workers` | atecontroller | up/down counter | number of worker pods currently ready for a WorkerPool, from `status.readyReplicas` (labels `ate.workerpool.namespace`, `ate.workerpool.name`) | | `ate.workerpool.workers` | ateapi | up/down counter | live worker count per pool, split by state (`idle`/`partial`/`at_capacity`/`unschedulable`) and sandbox class to provide fleet capacity and saturation at a glance. Every ateapi replica reports the whole fleet: take the `avg` by pool namespace, pool, state, and sandbox class first, then `sum` (example query below the table) | | `ate.actor.lifecycle.operation.duration` | ateapi | histogram | how long each actor operation (create/resume/suspend/delete/revert) takes and whether it failed (`error.type` present = failure, absent = success); labeled by operation, template, pool (`ate.workerpool.namespace` + `ate.workerpool.name`), sandbox class, snapshot kind on resume, and snapshot fidelity on resume and suspend; already-running resume no-ops are not recorded so the histogram tracks actual activations, not router traffic | | `ate.scheduler.assignment.duration` | ateapi | histogram | time of one attempt to assign a worker to an actor, with one record for each resume (ateapi retries an attempt that loses a race for a worker and records only the last attempt), with the outcome (`assigned` / `no_capacity` / `error`), the assigned pool (`ate.workerpool.namespace` + `ate.workerpool.name`) and sandbox class to catch scheduling latency and capacity starvation problems | | `ate.actor.restore.duration` | atelet | histogram | how long each phase of a restore takes on the worker node, which is where cold-start latency actually goes once ateapi hands off (labels `ate.snapshot.phase`, `ate.snapshot.kind`, `ate.snapshot.fidelity`, `ate.template.atespace`, `ate.template.name`, `ate.sandbox.class`) | | `ate.actor.checkpoint.duration` | atelet | histogram | the same phase breakdown for writing a snapshot, so a slow suspend can be attributed to ateworker or to the upload (same labels as the restore histogram) | | `atelet.snapshot.upload.duration` | atelet | histogram | how long a background upload of a local snapshot to object storage takes and whether it failed (`error.type` present = failure, absent = success; label `ate.snapshot.fidelity`) | | `atelet.snapshot.detach.duration` | atelet | histogram | how long a background detach takes to delete an actor's local snapshots and node files, excluding any snapshot upload, and whether it failed (`error.type` present = failure, absent = success; label `ate.snapshot.fidelity`) | | `atelet.snapshot.upload.overdue.duration` | atelet | histogram | how long after a background snapshot upload came due a worker started it: near zero while an upload worker is free, growing when the node's upload workers fall behind (no labels) | | `atelet.snapshot.detach.overdue.duration` | atelet | histogram | how long after a background detach came due a worker started it: near zero while a detach worker is free, growing when the node's detach workers fall behind (no labels) | | `atelet.snapshot.upload.overdue.actors` | atelet | up/down counter | actors on the node with an undurable local snapshot past `--snapshot-upload-delay` waiting for an upload to finish, bucketed by time overdue (label `ate.snapshot.upload_overdue_band`: `under_1m`, `1m_to_5m`, `5m_to_15m`, `over_15m`) | | `atelet.snapshot.undurable.age.max` | atelet | gauge | age of the oldest local snapshot on the node that is not durable yet, counted from when the actor's state stopped being durable, or 0 when every local snapshot is durable; a healthy node stays below `--snapshot-upload-delay` plus one upload | | `ate.imagecache.requests` | atelet | counter | image lookups in the node-local image cache, by outcome (`ate.imagecache.outcome`), with `error.type` on the `error` outcome. A miss pays for the pull and the unpack, so the hit ratio per node is a leading indicator of resume latency | | `ate.filecache.size` | atelet | up/down counter | bytes of disk a node-local file cache occupies, by `ate.filecache.name`, read on each collection; compare with `--sandbox-asset-cache-max-bytes` | | `ate.filecache.evictions` | atelet | counter | entries the eviction passes removed from a node-local file cache, by `ate.filecache.name`; a steady rate at the budget means the cache thrashes and actors pay for downloads again | | `ate.filecache.gc.passes` | atelet | counter | eviction passes over a node-local file cache, by `ate.filecache.name`, that had work to do, by outcome (`ate.filecache.gc.outcome`: `complete`, `shortfall`, `skipped`); a pass within budget is not counted | The table lists the OpenTelemetry instrument names. How a name appears in a query depends on the backend (Cloud Monitoring (GMP) / Kind collector). For `ate.workerpool.desired_workers` and `ate.workerpool.ready_workers`: * **Supply-Side Saturation Golden Signal**: Measures whether commanded capacity was delivered. Dedicated instruments match Kubernetes semantic conventions (`k8s.deployment.desired_pods` / `k8s.deployment.available_pods`) because desired + ready is not a disjoint sum. * **Autoscaling Control Loop & Anti-Windup**: `desired - ready > 0` sustained beyond a few minutes indicates undelivered capacity due to node pool exhaustion, quota limits, or stuck worker pods, serving as anti-windup input for demand-reactive capacity scaling. For `ate.workerpool.workers`: * Every ateapi replica reports the whole fleet, so a plain `sum` counts each worker once per replica ([#3055](https://github.com/agent-substrate/substrate/issues/3055)). The replicas read separate worker caches and can disagree for a few seconds: a worker that changed state sits under its old state on one replica and its new state on the other ([#3162](https://github.com/agent-substrate/substrate/issues/3162)). Take the `avg` across replicas first, keeping the pool namespace, pool, state, and sandbox class, then `sum`. A `max` would keep that worker under both states and the summed pool outgrows its real size. The average keeps the sum equal to the pool size; the counts can be fractional while the replicas disagree. Workers per pool: ```promql sum by ("ate.workerpool.namespace", "ate.workerpool.name") ( avg by ("ate.workerpool.namespace", "ate.workerpool.name", "ate.worker.state", "ate.sandbox.class") ( {"ate.workerpool.workers"} ) ) ``` When one backend receives several clusters, add `cluster` to both groupings, or same-named pools in different clusters merge. For `atenet.router.route.duration`: * `ate.router.outcome` categorizes the route attempt result: `ok`, `cancelled`, `timeout`, `no_capacity`, `failed_precondition`, `lock_conflict`, `not_found`, `unavailable`, `rate_limited`, `denied`, or `resume_error`. `denied` is a request the router refused by policy (a 403 or `PermissionDenied`), such as an egress request outside the actor's egress policy; `resume_error` is left for a resume that failed for any other reason, a router defect. * The router and the scheduler both report `no_capacity` for one condition, but they count different events. The scheduler records one sample on `ate.scheduler.assignment.duration` for each resume. The router records one sample for each request. Many requests can join one resume, and one parked request can cause more than one resume. Thus do not compare the two counts directly. * `ate.router.resume` indicates the singleflight execution state of actor resumption: `none` (the resume found the actor already running), `triggered` (this request completed a cold activation), `joined` (this request waited on another request's resume, which activated the actor), or `unknown` (the resume did not complete, so whether an activation ran is unknown). `ate.template.atespace` and `ate.template.name` hold `unknown` when the router has no template to name. For `ate.imagecache.requests`: * `ate.imagecache.outcome` is `hit` when the node holds a complete image record — every layer directory the record names is present — and `miss` when the lookup must pull. A failed lookup is neither: `error` is a failed lookup whatever the cause, and `cancelled` or `timeout` is the caller giving up, as on `ate.router.outcome`. So the hit ratio is `hit / (hit + miss)`, with failures and abandoned lookups out of the denominator. * `error.type` is present only on the `error` outcome, and carries the registry's own HTTP status for its rejection, from a fixed set: `401`, `403`, `404`, `429`, `500`, `502`, `503`, `504`. The set is an allow-list because the registry client reports whatever the remote returned. Each other status, and each failure that carries no status, reports `_OTHER`. For the `ate.filecache.*` instruments, one series per cache under `ate.filecache.name`: `sandbox_asset` for the sandbox asset cache and `snapshot` for the snapshot cache (see the [Sandbox Asset Cache](https://github.com/agent-substrate/substrate/blob/main/docs/concepts/sandbox-asset-cache.md) page): * The cache evicts only what nothing links: an actor's staged assets and each worker's default runtime are hard links into the cache, and an entry with a link stays whatever the pressure. `ate.filecache.gc.outcome` is `shortfall` when a pass could not reach the budget for that reason; the size then measures what the actors and the defaults on the node need, not waste. `skipped` means atelet could not read the cache directory and did nothing. `ate.workerpool.namespace` and `ate.workerpool.name` identify a pool together, on every instrument that names one. A WorkerPool is a namespaced resource, so the name on its own merges same-named pools from different namespaces into one series. The pair means that capacity (`ate.workerpool.workers`, `ate.workerpool.desired_workers`, `ate.workerpool.ready_workers`) joins to demand (`ate.scheduler.assignment.duration`, `ate.actor.lifecycle.operation.duration`, `ate.actor.crashes`) by pool. **No keys** means the operation has no pool. The actor-centric instruments omit the pair, so a crash before the actor reached a worker, or the `no_capacity` outcome, names no pool. No instrument sends the pair with both keys empty. The three snapshot labels are orthogonal and mean the same thing on every histogram that carries them: * `ate.snapshot.kind`: which snapshot the operation reads or writes. `local` (the copy on the node: an asynchronous suspend writes it, and a resume reads it while the node still has it, whether or not its upload to durable storage has finished. A synchronous suspend is `latest`, but it also keeps a copy on the node, so the resume after it can be `local`), `latest` (a durable snapshot, including a template's golden snapshot), or `boot` (from scratch, so it never appears on the atelet histograms). * `ate.snapshot.fidelity`: what state it holds. `volumes`, `rootfs`, or `memory`. * `ate.snapshot.phase`: which step was timed. `volume_mount`, `manifest_fetch`, `sandbox_assets`, `download`, `oci_unpack`, `worker_restore` on restore; `sandbox_assets`, `worker_checkpoint`, `persist`, `cleanup` on checkpoint; `total` on both. Golden snapshots use `latest` on both control-plane and atelet histograms, so the resulting series also includes other durable restores and does not isolate golden first restores. To investigate an individual activation, correlate the Actor's recorded snapshots with its logs and traces instead of adding Actor identity to metric labels. **Phases overlap and do not sum to `total`.** The download runs concurrently with the asset fetch and OCI unpack, so each is an independent observation; use `total` as the denominator. A phase that never started is absent rather than zero. The `ate.*` control-plane metric labels are either fixed value sets (operation, outcome, state, class, kind, scope, phase) or scoped to the deployment catalog (template and pool names are operator-created, never derived from request payloads), and the label set varies per operation: resume carries the most dimensions, delete only the operation and error type. `ate.sandbox.class` is derived from the runtime on the actor's worker assignment or snapshot (each worker pool has exactly one class), so it adds no extra series next to the pool label; it exists so dashboards can aggregate by class without enumerating pool names. High-cardinality actor identity (name/uid/atespace) stays off metrics entirely and lives on logs and traces instead — for resource usage, on the [per-actor usage events](#per-actor-usage-events). ### The metric registry [`docs/metrics/registry/metrics.yaml`](https://github.com/agent-substrate/substrate/blob/main/docs/metrics/registry/metrics.yaml) defines each instrument that the ate system components send, every attribute any signal uses, and the permitted values of each. [`events.yaml`](https://github.com/agent-substrate/substrate/blob/main/docs/metrics/registry/events.yaml) beside it defines the [log events](#the-same-records-over-otlp). Use them as the only source of this data. A few attributes are defined there but barred from metric labels — actor identity above all. The `note` on each says so, and the cardinality rules below are where the bar lives. It is an [OpenTelemetry Weaver](https://github.com/open-telemetry/weaver) registry. Weaver reads the whole directory, resolves each `ref`, and refuses a group or an attribute that is not correct: ```sh hack/verify/metrics.sh # the command that CI uses weaver registry check -r docs/metrics/registry # the same command, direct ``` `hack/verify-all.sh` runs the script. It uses a local `weaver` binary if there is one, and the official image if there is none. **To add or change an instrument:** follow [Metrics Best Practices](https://github.com/agent-substrate/substrate/blob/main/docs/dev/best-practices/metrics.md), which covers the instrument choice, the labels, the Go, the test, and the registry entry; then change `metrics.yaml` and run the script. Four files, and not one: | File | Content | Who reads it | |---|---|---| | `docs/metrics/registry/manifest.yaml` | The name and the schema URL of the registry. | Weaver | | `docs/metrics/registry/metrics.yaml` | The `groups`: each metric, each attribute. | Weaver | | `docs/metrics/registry/events.yaml` | The [actor events](#the-same-records-over-otlp). | Weaver, `TestEventsMatchTheRegistry` | | `docs/metrics/substrate.yaml` | The rules of Substrate that Weaver cannot hold. | A person, an agent | Weaver permits only `groups` and `imports` at the top level of a registry file, and it refuses a file that has any other top-level key. Thus `substrate.yaml` is beside the registry directory and not in it. It records: * **`upstream_semconv_version`** — the version of the upstream semantic conventions that Substrate borrows `error.type`, `file.name` and the `rpc.*` attributes from. * **`bridged_metric_families`** — the metrics that atecontroller exports but Substrate does not define, as prefixes. `controller_runtime_version` records the version of controller-runtime that the list comes from. * **`cardinality_rules`** — the rules that keep the number of series small. No rule is enforced at this time. Each rule says what could enforce it. * **`lint_exceptions`** — the known debt. * **`blind_spots`** — the subsystems with no metrics. Read this list before you give a cause to a fault. The store has no instruments, but each lifecycle operation uses it. **What the check does not do.** Weaver reads the registry, and not the Go code. Thus the build stays correct if a person adds an instrument to the code and not to the registry, or renames one in the code only. `weaver registry generate` could make `internal/ateattr` from the registry, which would remove that difference for the attributes. `weaver registry live-check` could compare the registry with the telemetry of the end-to-end tests. Both are future work. ### Bridged controller-runtime metrics (atecontroller) atecontroller bridges controller-runtime's private Prometheus registry, which the manager serves on `:8080` only while `OTEL_METRICS_EXPORTER` leaves it on (see [Scraping instead of pushing](#scraping-instead-of-pushing)), onto its OTLP reader. So `controller_runtime_*`, `workqueue_*`, `certwatcher_*`, `rest_client_*`, `leader_election_*`, `go_*`, and `process_*` reach the collector too, keeping their Prometheus names because they are upstream instruments and renaming them would break existing controller-runtime dashboards. These can be used to answer whether the controller is keeping up, e.g. rising `workqueue_depth` or `workqueue_queue_duration_seconds` means reconciles are falling behind, and `controller_runtime_reconcile_errors_total` says which controller. `docs/metrics/substrate.yaml` records these as prefixes under `bridged_metric_families`, and not one metric at a time. The upstream library owns the names, the labels and the buckets, and a version bump can add a family. A copy in the registry becomes wrong with no signal. `controller_runtime_version` dates the list, thus a bump has an obvious place to check. Note that controller-runtime enables native histograms on `controller_runtime_reconcile_time_seconds`, `workqueue_queue_duration_seconds`, and `workqueue_work_duration_seconds`, so those three arrive as OTLP exponential histograms rather than fixed-bucket ones. A queue that has never processed an item bridges as an exponential histogram with no positive buckets, which the Telemetry API (the Cloud Monitoring OTLP endpoint) rejects on ingest. atecontroller gives each such data point one positive bucket with a count of 0 before the OTLP push (`cmd/atecontroller/metrics.go`), so an idle queue no longer causes an error every tick. The padding carries no observations, so other backends are unaffected. ### Scraping instead of pushing ateapi, atelet, atenet-router and the credential provider also serve every instrument on their Prometheus `/metrics` endpoint. A cluster that scrapes those endpoints sets `OTEL_METRICS_EXPORTER=prometheus` on the components, so each series reaches the backend once. With `prometheus` the components install no OTLP metric reader and keep the Prometheus one; traces and logs are unaffected. `none` stops the push too. For now `/metrics` stays on for every value, including `otlp` and `none`. The health probes have moved to their own server on `:9091`, so `:9090` serves only `/metrics`; turning it off for `otlp` and `none` is tracked in [#2067](https://github.com/agent-substrate/substrate/issues/2067). The variable takes a comma-separated list, with the same rules as `OTEL_LOGS_EXPORTER`. Unset is `otlp,prometheus`, or `otlp` alone on ateworker. `otlp` keeps the push, and `prometheus` or `none` stops it. ateworker serves no `/metrics`, so there `prometheus` is skipped as unknown with a warning, and the push stays on unless the variable is `none`. `ate-controller` does not propagate its own value to the worker pods: ateworker serves no `/metrics`, and `none` would leave it exporting no metrics at all. To set the variable on ateworker, set `ATE_WORKER_OTEL_METRICS_EXPORTER` in `ate-otel-config` (or the controller's `--otel-metrics-exporter` flag). atecontroller's `:8080` listener follows the variable. When the variable is unset or `otlp,prometheus`, atecontroller pushes every metric over OTLP, and `:8080` serves the controller-runtime families and the `ate.workerpool.*` instruments. With `otlp`, atecontroller pushes every metric and closes `:8080`. With `prometheus`, atecontroller does not push, and `:8080` serves the controller-runtime families and the `ate.workerpool.*` instruments. With `none`, atecontroller does not push and closes `:8080`. ### Local Metrics with Prometheus (Kind Cluster) For local development inside a `kind` cluster, Agent Substrate automatically provisions a Prometheus server in the `otel-system` namespace. To explore metrics locally: 1. **Expose the Prometheus UI** via port forwarding: ```bash kubectl port-forward -n otel-system svc/prometheus 9090:9090 ``` 2. **Open the Prometheus UI** in your web browser: [http://localhost:9090](http://localhost:9090) 3. **Query metrics**: Run `up` to confirm each scraped target is up (one series per target, value `1`), then explore the `rpc_*` series via the expression browser's autocomplete. **Status > Targets** lists the discovered pods. > **Note:** Storage is ephemeral (`emptyDir`), so metrics are lost when the Prometheus pod restarts. > **Note:** This Prometheus does not see every instrument. It scrapes the `/metrics` endpoint (`:9090`) of the pods annotated `prometheus.io/scrape`, which are `ate-api-server`, `atelet`, and `atenet-router`. From the collector it keeps only the `substrate_*` counts of the count connector and the collector's own `otelcol_*` counters ([`manifests/ate-install/kind/prometheus.yaml`](https://github.com/agent-substrate/substrate/blob/main/manifests/ate-install/kind/prometheus.yaml)), so that the scraped services are not counted twice. The metrics that ateworker (including its `rpc.server.call.duration`) and atecontroller (`ate.workerpool.desired_workers`, `ate.workerpool.ready_workers`, and the bridged controller-runtime families) push over OTLP reach the collector but do not appear in this UI. The credential provider's metrics appear in neither: its pod carries no scrape annotation and does not read `ate-otel-config`, so its OTLP push goes to the SDK default `localhost:4317` and fails. > **Roadmap Note (Actor-Level Metrics):** The control plane latency, snapshot performance, and fleet utilization instruments above are in place. What remains is actor-level correlation: telemetry that an actor emits itself carries no Substrate-injected identity yet ([#853](https://github.com/agent-substrate/substrate/issues/853), [#761](https://github.com/agent-substrate/substrate/issues/761)). ### SLOs and alerting Please see the [SLOs and alerting](https://ate.dev/docs/guides/slos-and-alerts/index.md) page for suggestions about how to set up SLOs, SLO-based alerts, and other alerts to ensure your deployment is working well. --- ## 3. Tracing Distributed tracing tracks the end-to-end flow of requests as they pass through the Agent Substrate gateway, router, worker pods, and external services. Agent Substrate samples traces by default. Each component roots parentless requests at a per-component ratio (10% on the control plane components, 1% at the atenet router), overridable per component through the standard `OTEL_TRACES_SAMPLER` / `OTEL_TRACES_SAMPLER_ARG` environment variables. Every component uses a parent based sampler, so a client can also force a request to be traced end to end (e.g. via the `--trace` flag). Agent Substrate leverages OpenTelemetry (OTel) for context propagation across the call stack. Each traced request generates a unique trace hash/ID, which you can use to inspect the detailed request lifecycle and span hierarchy inside Jaeger or your OpenTelemetry tracing backend. See the per-component defaults table in [Tracing Best Practices](https://github.com/agent-substrate/substrate/blob/main/docs/dev/best-practices/tracing.md). A component with `OTEL_TRACES_EXPORTER=none` exports no spans. It still samples and propagates trace context, so the traces of the components around it stay whole. The variable takes a comma-separated list, with the same rules as `OTEL_LOGS_EXPORTER`, and knows `otlp` (the default) and `none`. It does not cover the worker pods: set `ATE_WORKER_OTEL_TRACES_EXPORTER` in `ate-otel-config` (or `ate-controller`'s `--otel-traces-exporter` flag) for them. The atelet relay forwards ateworker's spans as they are and does not apply atelet's own value. ### Local Tracing with Jaeger (Kind Cluster) For local development inside a `kind` cluster, Agent Substrate automatically provisions a local OpenTelemetry Collector and Jaeger instance. To visualize traces locally: 1. **Expose the Jaeger query UI** via port forwarding: ```bash kubectl port-forward -n otel-system svc/jaeger 16686:16686 ``` 2. **Open the Jaeger UI** in your web browser: [http://localhost:16686](http://localhost:16686) 3. **Generate Traces**: Run a CLI command or API call with the `--trace` flag, e.g.: ```bash kubectl ate get actor -A --trace # or kubectl ate suspend actor -a --trace ``` The kind overlay pins `ateapi` to `parentbased_always_on`, so API calls show up even without `--trace`; the flag additionally prints the trace ID and forces sampling on every hop. 4. **Search and Inspect**: Copy the printed Trace ID from the CLI output and paste it into the Jaeger search box (top right), or select `ateapi`, `atelet`, or `ateworker-gvisor` under the **Service** dropdown and click **Find Traces** to inspect detailed call stacks, DB transactions, state updates, and worker pod handoffs. > ateworker carries no manual spans — its only instrumentation is the `otelgrpc` interceptor on the gRPC surface `atelet` calls. So it produces a span for an actor lifecycle operation (`suspend`, `resume`) and nothing at all for a read like `kubectl ate get actor`. Its sampler is parent based, so a lifecycle command is traced end to end into ateworker whenever `ateapi` roots a sampled trace, which the kind overlay makes unconditional; the per-component ratio never enters into it. To check whether ateworker exported its spans through the [OTLP relay](#the-ateworker-otlp-relay) rather than falling back to direct network egress, inspect the span's resource attributes: `ate.otlp.relay` will be set to `"relay"` (instead of `"direct"`). > `ateapi` traces its PostgreSQL store. Every statement issued while serving a sampled request becomes a client span under the RPC span, named by its query summary (`SELECT actors`, `UPDATE workers`), with the parameterized statement in `db.query.text` and never the argument values. Transaction control appears as well, as `begin` and `commit` spans, so commit latency is visible. Store spans only join a trace that is already sampled: they follow `ateapi`'s sampler and have no switch of their own, and background work such as outbox polling and lease upkeep produces none. > **Developer Guide:** For detailed instructions on configuring OpenTelemetry tracer providers, middleware, and exporters in your servers or clients, please refer to the [Tracing Best Practices](https://github.com/agent-substrate/substrate/blob/main/docs/dev/best-practices/tracing.md) guide. --- ## 4. Where Telemetry Goes Telemetry is emitted over OTLP to your configured OpenTelemetry Collector. > In Kind, `ateapi`, `atelet`, `ate-controller`, `atenet-router`, and `atenet-egress` are pointed at the in-cluster `opentelemetry-collector` in `otel-system`, and the controller propagates the endpoint to the worker pods it creates, so the telemetry of these components and of the workers lands locally (Prometheus exporter on `:8889`, Jaeger UI for traces, and the collector's `debug` exporter for OTLP logs). The Kubernetes credential provider is the exception: it does not read the endpoint (see the [note on the Kind Prometheus](#2-metrics)). > > Each of these components reads that endpoint from the shared `ate-otel-config` ConfigMap ([`manifests/ate-install/ate-otel-config.yaml`](https://github.com/agent-substrate/substrate/blob/main/manifests/ate-install/ate-otel-config.yaml), with a Kind replacement of the same name under [`manifests/ate-install/kind/`](https://github.com/agent-substrate/substrate/blob/main/manifests/ate-install/kind/ate-otel-config.yaml)). Editing it does not restart the pods that consume it — follow a change with `kubectl rollout restart`. > > Workers don't read the ConfigMap at all. They normally export through the [`atelet` relay](#the-ateworker-otlp-relay), which forwards to `atelet`'s own endpoint, so restarting `atelet` moves the relayed workers to a new endpoint. `ate-controller` also copies the value into each worker pod at creation, for the direct fallback. That copy changes only once the controller itself restarts, and that restart then rolls every WorkerPool Deployment, replacing the running workers along with the actors on them. ```yaml apiVersion: v1 kind: ConfigMap metadata: name: ate-otel-config namespace: ate-system data: OTEL_EXPORTER_OTLP_ENDPOINT: http://opentelemetry-collector.otel-system.svc:4317 OTEL_METRIC_EXPORT_INTERVAL: "10000" OTEL_METRIC_EXPORT_TIMEOUT: "10000" OTEL_LOGS_EXPORTER: otlp ``` ### The ateworker OTLP relay ateworker is the one component that does not talk to the collector directly. It exports logs, traces, and metrics over a unix socket at `/var/lib/ate/atelet-otlp.sock`, which `atelet` serves and forwards to the collector on the node's network ([`internal/otlprelay`](https://github.com/agent-substrate/substrate/tree/main/internal/otlprelay)): ``` ateworker ──OTLP/gRPC over unix socket──► atelet relay ──OTLP/gRPC──► collector ``` The socket sits in the `BasePath` hostPath already mounted into both, so nothing new is mounted. `atelet` is a DaemonSet, so every ateworker on a node shares one relay, and the many per-pod collector connections collapse into one per node. Four things motivate it: the worker pod runs untrusted agent code and will not need egress to the collector once direct fallback is phased out; the connection count drops; ateworker's own telemetry stays clear of the transparent egress redirect it installs for the actor; and `atelet` outlives the worker pod, so spans still queued at teardown are not lost with it. The relay is best-effort. If the socket is absent when ateworker starts — `atelet` not up yet, `--otlp-relay-socket=""`, or no collector configured for the relay to forward to — ateworker logs it and exports directly to `OTEL_EXPORTER_OTLP_ENDPOINT` (or the per-signal `OTEL_EXPORTER_OTLP_{TRACES,METRICS,LOGS}_ENDPOINT`). `atelet` logs `Failed to create the OTLP relay` and starts no relay when its endpoint uses `https` (the relay forwards over plaintext gRPC and refuses to downgrade), when its per-signal endpoints or compression settings disagree (the relay carries every signal over one connection), or when its `OTEL_EXPORTER_OTLP_*_HEADERS` cannot be parsed. ateworker checks only that the socket file exists, so a socket left on the node by an earlier `atelet` still sends it to the relay: nothing serves the socket, and every export fails with `connection refused` instead of falling back. Without any endpoint either, ateworker exports no traces, metrics, or logs, rather than dialing the SDK default `localhost:4317`, where no collector runs, and logs a warning per signal with `reason="no OTLP relay or endpoint"`. That fallback is decided once at startup, not per export, and is stamped on telemetry as the `ate.otlp.relay` resource attribute (`relay` vs `direct`). On an install that sets no endpoint for `ate-controller` to propagate, an ateworker that starts before its node's relay socket exists therefore exports nothing until the worker pod restarts. > **Note on Network Egress Lockdown:** Complete network policy lockdown of worker pod egress to the collector is planned as a Phase 2 milestone once the relay path is fully proven and direct fallback is deprecated. While the fallback path remains active, worker pods retain network egress to the collector and `ate-controller` continues to inject `OTEL_EXPORTER_OTLP_ENDPOINT` when it has one. For verified ateworker sources, the relay forwards each request verbatim rather than decoding and re-exporting, which is what keeps every ateworker its own service in Jaeger or your tracing backend instead of being absorbed into `atelet`'s. `ate-controller` injects `k8s.pod.name`, `k8s.namespace.name`, `k8s.pod.uid`, `k8s.node.name`, and `service.instance.id` directly into `OTEL_RESOURCE_ATTRIBUTES` via the Kubernetes Downward API, whether or not it has an endpoint to propagate; because the relay preserves resources verbatim, Kubernetes attributes remain intact even though the TCP connection to the collector originates from `atelet` rather than the worker pod IP (bypassing reliance on collector-side IP-based `k8sattributes` enrichment). Verbatim forwarding is restricted to known ateworker sources and refuses anything else with `PermissionDenied`. Actor telemetry is what that excludes: actors share a hostname (`actor`) and an interior IP, so their series merge unless identity is injected from outside the actor ([#761](https://github.com/agent-substrate/substrate/issues/761)) — a rewrite, which will be implemented as an explicit rewriting path alongside this forwarder. ### Actor telemetry needs an egress policy An actor's own OpenTelemetry SDK does not use the relay. It dials the collector over the network like any other outbound connection, and that connection is subject to the actor egress lockdown described in [Network Egress](https://ate.dev/docs/guides/network-egress/index.md): every actor TCP connection is redirected into `atunnel` and carried to the egress gateway, which refuses anything the actor's `EgressPolicy` does not allow. An actor with no policy, or a policy with no matching rule, cannot reach the collector at all. With no policy, or a policy whose rules do not include the collector's hostname, the actor cannot even resolve the collector's name: the sandbox's DNS relay answers its queries `NXDOMAIN`, so each export fails before it connects. If a rule allows the hostname on another protocol or port (for example, `*` on port `443`), DNS resolves the name, the SDK sees `Unavailable: connection reset by peer` on each export, and the gateway logs why it denied the connection. To let an actor export, give it an `http` rule naming the collector's Service and the OTLP ports. It must be an `http` rule, not `https`: the OTLP/gRPC exporter is gRPC over cleartext HTTP/2 (h2c) on `4317`, which an `http` rule admits, and the OTLP/HTTP exporter is cleartext HTTP/1.1 on `4318`. On Kind the collector is `opentelemetry-collector.otel-system.svc`. Whatever the name, it has to match the host the actor's `OTEL_EXPORTER_OTLP_ENDPOINT` names, because the rule is matched on the request authority. ```bash kubectl ate create egress-policy -a -f - <<'EOF' rules: - http: hostnames: ["opentelemetry-collector.otel-system.svc"] ports: numbers: [4317, 4318] EOF ``` An actor that already has a policy for other destinations keeps those rules and gains this one: `kubectl ate update egress-policy` replaces the whole rule list, so include the existing rules in the manifest. `kubectl ate get egress-policy -a -o yaml` prints the current one. Three things to know when applying this: * **It is per actor.** The policy is a resource under the actor, and nothing applies one by default today. Every actor that exports telemetry needs its own, created after the actor and before, or shortly after, it is resumed. A template-level default policy is requested in [#1430](https://github.com/agent-substrate/substrate/issues/1430). * **A running actor recovers slowly.** If the policy is created while the actor is already exporting and failing, the SDK's gRPC client is in reconnect backoff, the sandbox's DNS relay takes up to 5s to notice the policy, and the gateway caches the missing policy for `--egress-policy-cache-ttl` (10s by default). Expect the first successful export 45s to two minutes after the policy appears. An actor whose policy exists before it starts has no such delay. * **The collector sees the gateway, not the actor.** The TCP connection to the collector comes from the egress gateway pod, so a collector that enriches by source IP (the `k8sattributes` processor) labels the actor's spans and metrics with the gateway's `k8s.pod.name`, `k8s.deployment.name=atenet-egress`, and `k8s.namespace.name`. The actor's `service.name` and anything it puts in `OTEL_RESOURCE_ATTRIBUTES` survive. Set the actor's identity there yourself if you need it on the backend; the relay-based path that will do this for actors is tracked in [#853](https://github.com/agent-substrate/substrate/issues/853) and [#761](https://github.com/agent-substrate/substrate/issues/761). **Logs written to stdout and stderr need none of this.** ateworker reads a container's output stream inside the worker pod and writes it to the pod's own stdout, which is where [`kubectl ate logs`](#active-actor-inspection-via-cli) and a node log agent pick it up. That path never leaves the sandbox over the network, so it is untouched by the egress lockdown and works for an actor with no policy at all. The policy is only for telemetry an SDK inside the actor pushes itself: traces, metrics, and OTLP logs if `OTEL_LOGS_EXPORTER=otlp` is set in the actor. Substrate puts no OTLP configuration into an actor container. Set `OTEL_EXPORTER_OTLP_ENDPOINT` in the ActorTemplate's container `env`; without it the SDK defaults to `localhost:4317`, where nothing listens inside the sandbox, and every export is dropped with no error at the collector. ================================================================================ SECTION: HOW-TO GUIDES PAGE: SLOs and Alerting URL: https://ate.dev/docs/guides/slos-and-alerts/ MARKDOWN: https://ate.dev/docs/guides/slos-and-alerts/index.md SOURCE: docs/guides/slos-and-alerts.md ================================================================================ # SLOs and Alerting > Define SLIs, SLOs, and Prometheus alerting rules to monitor Agent Substrate in production. This page explains how operators can [defend](#appendix-monitoring-ate-and-defending-slos) SLOs and monitor ATE in production. It defines SLIs, suggests SLOs, and suggests other alerts operators can use to ensure their ATE deployments are working correctly. # Background ## Goals Operators need to monitor their ATE deployments for two reasons: * **Incident management:** Incidents include cases such as bad ATE releases, problematic clients, or other misconfigurations of their ATE deployment that lead to bad behavior for their users. * **Optimizing tradeoffs:** Operators can set a "reasonable" rate of failed actor resumptions to maintain a high density, or vice versa. There are other cases where operators may want to trade off factors such as availability and performance for cost. All the scenarios below are tagged with "ICT" if they could represent incidents, or "OPT" if they help optimize the deployment. Of course, some scenarios can represent both. ## Surfaces Each primary surface of ATE needs alerting to meet the goals above. These surfaces are: * **ateapi (API server):** Start and manage actors, snapshots, etc. * **K8s APIs and operations:** Create the ATE installation itself and ensure it's healthy; create and manage worker pools. * **Actor execution:** As actors execute, they must be able to use local \*nix mechanisms like process exits and signals (e.g. OOMs), exposing probers, and producing stdout/stderr. We include mounting access to storage under this category. * **Actor network requests:** Actors can accept incoming network requests, even if the actor is currently suspended, and can make outbound network requests through the ATE-managed gateway. The following surfaces are *not* considered to be primary, at least for now, because their behavior is easily observable through a different (primary) surface. Observability of these surfaces is important for diagnosing problems, but is not required for alerting: * **Snapshot storage:** The time spent storing or retrieving an actor's snapshot is visible via the API server, but see also section B.4 below for some subtleties in how the times should be interpreted. * **PostgreSQL or other backends:** Similarly, any problems here should be visible in the API server or other surfaces. Finally, the following are out of scope: * **Sandbox operation:** ATE is not responsible for the correct operation of the underlying sandbox technology (e.g. gVisor, microVMs); that is, ATE could work correctly even though the sandboxes are running slowly. These components are pulled directly from upstream; see the documentation of those projects for observability options and recommended monitoring. # High-level approach This document defines three types of signals and associated alerting. ## SLIs, SLOs, and SLO-based alerts These are traditional alerts that cover good and bad events, and have error budgets. SLIs and SLOs are expressed using PromQL rules, following the [SRE workbook standard](https://sre.google/workbook/alerting-on-slos/). This document uses OTel-style metric names (e.g. UTF-8 metrics). Operators are encouraged to review and customize these alerts, including converting to traditional Prometheus syntax if needed. While ATE developers cannot *defend* SLOs, we will *demonstrate* that the P0 and P1 SLOs can be met under reasonable conditions. (**details tbd**). ## Non-SLI metrics and custom alerts These are conditions that can be described as metrics, but not good-bad events and don't have an error budget. Examples include K8s-style "not-at-intent" problems, such as the intended and actual number of workers not converging after a period of time. It's not an error condition for the two to be out of sync (e.g. if the number of intended workers is constantly changing), but it is an error if they are not constantly converging. As with SLOs, recommended alerts are expressed as Prometheus configs. ## Error logs Many other errors are not worth having dedicated metrics; these often include failures that will *eventually* cause another recommended SLO or alert to fail. While we *could* introduce a one-off metric for every possible failure condition, operators would likely not update their alerting configs and it seems like unnecessary work to keep them in sync. The problem with using logging for alerting (as opposed to debugging or observability) is that it is hard to enforce that error messages are only produced when there are genuine, non-retryable, actionable conditions that an operator needs to investigate. While we will attempt to reduce the number of unactionable errors that ATE components emit, it's unlikely we'll get to (or stay at) zero, especially considering errors that could come from libraries. This may seem resolvable if we introduce a new general-purpose "failure" metric that can be set by many parts of the code, but unfortunately the same problem would eventually apply: some authors might overuse this metric, and others might ignore it. This is a fundamental problem associated with providing a complex piece of infrastructure with a huge number of points of failures. As a result, operators will have two options: * They could configure alerts whenever the error rate breaches some threshold, and accept that occasionally this will lead to false alerts. * They could manually monitor error messages at high-risk times, e.g. while upgrading Substrate. Finally, in as many cases as possible in the scenarios below, we've described what happens if error messages are missed. In many cases, this will cause another SLO or signal to move, but potentially much later when the situation is already affecting users and/or is harder to resolve. There are a few cases where this isn't true (e.g. losing stdout from actors), but these are generally lower-severity problems. # Initial set of monitoring scenarios In this section, we discuss the general approach for each surface listed above, as well as specific CUJs within that surface and how those CUJs will be monitored and alerted. Ultimately, the owners of each ATE surface will be responsible for implementing SLOs, non-SLO alerts, and error logs as described above, and this document will become incomplete and/or out of date. The purpose of this section is to list the current (as of late 2026), most critical alerting requirements that we'll implement up-front, both to kickstart the observability, and also to illustrate how the approach described above will actually work in practice. ## A: API server All metrics related to `ate-api-server` are reported per-RPC, including pass/fail results and latency. ### \[ICT, OPT\] A.1: Actors cannot be executed at the expected success rate There are legitimate reasons that actors might not be able to be run; most importantly, the operator may not choose to size the worker pool for absolute peak capacity. Therefore, the failure rate of `ResumeActor` due to exhaustion is a critical SLI, with one alert for P1 (e.g. pool is not large enough) and a lower threshold for P0 (e.g. most or all workers are broken). (For failures due to causes other than exhaustion, see section A.2). Note that this configuration uses `ate.actor.lifecycle.operation`, not `rpc.server.call`, since the former correctly (in this case) excludes "resume" operations where the actor was already running, while the latter does not. One current weakness of this alert is that it greatly overcounts failures that occur during request parking, since request parking can retry about 20 times. (100ms initial delay, 1.1x multiplier, 5s budget). Right now, there's no way to tell the difference between "retryable" failures and "final" failures. **Example:** In the [sample rules](https://github.com/agent-substrate/substrate/blob/main/docs/examples/gmp-alerting-rules.yaml), see `ate_resume_actor_capacity_slo_recording_rules` and `ate_resume_actor_capacity_slo_alerts`. ### \[ICT\] A.2: The API server or its dependencies are broken or degraded Most operations should be insensitive to the state of the system, and we should maintain very high (e.g. \>99.9%) success rate. These are fairly straightforward. These SLIs/SLOs only track the equivalent of 5xx codes. Notably, resource exhaustion is excluded; see section A.1 for that. **Example:** In the [sample rules](https://github.com/agent-substrate/substrate/blob/main/docs/examples/gmp-alerting-rules.yaml), see `ateapi_rpc_availability_slo_recording_rules` and `ateapi_rpc_availability_slo_alerts`. Note that these may be too sensitive for low-traffic methods (e.g. `CreateActorTemplate`); see the file for possible approaches to resolve this. _NB: the sample rules only cover availability, not latency, but the availability SLOs can easily be adapted into multiple tiers of latency SLOs as needed._ ## B: K8s APIs and asynchronous operations Substrate introduces the following CRDs: * `WorkerPool`: declares and manages workers. Inherently K8s-centric since it creates and manages pods, and can be autoscaled via HPA. * `SandboxConfig`: declares sandbox types. Might be moved to the API server now that fine-grained auth is supported through OpenFGA. Read directly by atelets. * `CSIDriverConfig`: gives ATE direct access to CSI drivers without using PV/PVCs. Read directly by atelets. In addition, ATE uses Secrets to project trust bundles into Actors, `StorageClass` to set up access to storage, and `PodCertificateRequest` to provision SPIFFE mTLS identities to workers. Finally, the atelet is deployed as a daemonset on all nodes. While failure conditions may be displayed on CRD status fields, these are hard to monitor and do not cover all the cases listed above. As a result, the primary signal for failures with K8s resources and their controllers will be error logs, unless otherwise listed below. ### \[ICT, OPT\] B.1: Workers cannot be created K8s may not be able to schedule the requested pods for workers, which could be a tradeoff issue (e.g. the cluster is not large enough). But there are many other reasons the number of workers may not be at intent: for example, the `WorkerPool` reconciler could fail to create the associated `Deployment`, or the pods could be created but not noticed by the rest of the system due to a faulty network policy. These are always incidents. We already track the two necessary metrics for this scenario: * `ate.workerpool.desired_workers` is emitted as soon as the data is read from the WorkerPool CRD, so will be accurate as long as the controller is running at all. * `ate.workerpool.workers` is emitted once workers are actually registered with ateapi. Since the intended worker pool size can change over time, the following alert verifies that the current size is \>= the smallest requested size over the last five minutes, and fires an alert after five minutes if not. While this will not catch failures to *release* workers, this seems like a far less likely problem. **Example:** In the [sample rules](https://github.com/agent-substrate/substrate/blob/main/docs/examples/gmp-alerting-rules.yaml), see `ate_workerpool_alerts`. ### \[ICT\] B.2: Existing workers become unreachable If ateapi stops being able to reach the atelet *after* creation \- for example, because the atelet has crashed, or because a new NetworkPolicy breaks communication \- there is currently no metric that will move (as of Oct 2026), until a client attempts to contact it directly or indirectly via `ate-api-server` (at which point the client will receive `UNAVAILABLE`, which is caught by A.2). Worse, because ateapi assigns an actor to a worker before attempting to contact the atelet, and because it treats `Unavailable` as a transient error, any actor assigned to an unreachable worker will fail permanently. The worker doesn't become a traditional black hole \- it will only "accept" as many actors as it originally had capacity for \- but once the actor is removed, the zombie worker will trap *future* actors. **TODO:** consider adding a heartbeat between ateapi and atelets; after a period of time, declare the worker lost, decrease ate.workerpool.workers (so the alert from B.1 fires) and stop assigning new actors to it. We could also wait for external RPCs instead of heartbeats combined with a timeout or failure count. ### \[ICT\] B.3: The atelet (or other components) experiences local failures The atelet can fail in any number of ways which are not necessarily exposed to the API server RPCs. For example: * It could fail to receive the capacity or status of the worker or register this information back to the API server. This could silently prevent the API server from attempting to use the worker, without causing any RPC to fail. * The atelet could fail to clean up after an actor eviction or to prune local snapshots. In addition, components should emit errors if there is an *impending* failure. For example, a certificate used for mTLS that is expiring in fewer than M days very likely indicates that the certificate distribution system has broken down, and should be exposed as an error (not warning) message. **Signal and alerting**: error logs. If ignored, these will eventually reduce capacity, leading to violations of A.1. ### \[ICT, OPT\] B.4: The atelet experiences performance degradations The atelet is responsible for storing and retrieving OCI images, actor snapshots, and other assets. If these operations fail completely, the failures will be detected via other surfaces (e.g. A.1 and B.3 at a minimum), but if they're merely slower than expected, we would need dedicated signals to detect the problem. Unfortunately, pure durations (e.g. `{"ate.actor.checkpoint.duration", "ate.snapshot.phase"="persist"}`) are not an adequate metric, since assets can vary in size by orders of magnitude. So we need a measure of throughput that's resistant to very small assets. One possible approach is: * Batch together all assets so we're not measuring very small files * Only calculate throughput for batches larger than the expected [bandwidth-delay product](https://en.wikipedia.org/wiki/Bandwidth-delay_product). For example, for a minimum delay of 25ms and an expected throughput of 200MiB/S, we would only include batches larger than \~5MiB. **Signals and alerting:** Add throughput metrics and set up the associated SLOs and SLO-based alerting. This is low priority at this time. ## C: Actor execution For the most part, ATE is not responsible for ensuring that actors can execute correctly; this is primarily the responsibility of the owners of the actors (e.g. the authors of the code and whomever sets resource requests) and the sandboxes themselves. However, there are some possible failures from ATE itself, plus ATE operators must be able to monitor the operation of actors. ### \[OPT\] C.1: Actors do not have enough resources (e.g. CPU or RAM) **Signal:** metrics showing CPU usage, RAM usage, and counters showing the number of OOMs. Many of these are available in the [metrics](https://github.com/agent-substrate/substrate/blob/main/docs/metrics/registry/metrics.yaml) and [events](https://github.com/agent-substrate/substrate/blob/main/docs/metrics/registry/events.yaml) registries, such as `ate.actor.stats` and `ate.actor.usage_sampled`. Also see the [observability](https://ate.dev/docs/guides/observability/index.md) docs for information on log attributes such as `ate.sandbox.oom_kills`. **Alerting:** n/a. Operators should monitor these signals for overall system health, and *may* set SLOs on the percentage of successful executions if they also control the workloads themselves, but resource exhaustion does not indicate a problem with ATE. ### \[ICT\] C.2: Ateworker cannot access the probers **Signal and alerting:** Error logs. If ignored, see C.4. ### \[ICT\] C.3: Ateworker cannot receive and store stdout/stderr **Signal and alerting:** Error logs. No backup if ignored. ### \[ICT\] C.4: Actors are failing for other reasons. E.g. atelets could fail to mount/unmount drives, or any other kind of local failure. **Signal:** See C.1, and specifically the `ate.actor.crashes` metric and `ate.actor.crashed` OTLP log event. **Alerting:** n/a ## D: Actor network requests ### \[ICT\] D.1: Inbound network requests cannot be routed End-to-end ingress routing success can be measured by Envoy using `envoy_http_downstream_rq_total` and `envoy_http_downstream_rq_xx`. Unfortunately, if the actor returns a 5xx status code, Envoy can't distinguish between an error from ATE and from the Actor, so these are coarse signals that shouldn't be used for SLOs. By contrast, `atenet.router.route.duration` contains all the outcomes (see [observability](https://ate.dev/docs/guides/observability/index.md)) which is a more reliable SLI, but may be missing some failure cases. ### \[ICT\] D.2: Outbound network requests cannot be routed TBD # Appendix: monitoring ATE and defending SLOs "Defending" an SLO on ATE means: 1. Monitoring its SLIs and [alerting on potential violations](https://sre.google/workbook/alerting-on-slos/); 2. Taking actions to resolve the violations. In a self-managed deployment such as ATE, it is obviously not possible for Google (or any other cloud provider) to take actions to resolve violations on user-owned systems. For example, Google cannot: * Increase the number of workers (or underlying nodes) to increase capacity * Roll back a problematic change * Ban or rate-limit problematic clients There may be a certain class of problems that cloud providers *could* resolve, but it's hard to say what those might be with any confidence, and it's highly likely that these problems would not be specific to ATE anyway. In addition, even centralized monitoring is not possible. Google generally only has access to metrics from Google-owned components (aka "system metrics"), but since an ATE deployment is not a Google-owned component, its metrics are not accessible to Google, [except under exceptional conditions](https://cloud.google.com/security/products/access-transparency). There could be a theoretical path by which we ask customers to provide permission for Google to view these metrics, but they could always say no. These restrictions presumably apply to other cloud providers as well. For this reason, the best we can do is to: * Give users useful SLIs so they can defend their own SLOs; * Test ATE in controlled environments to maximize the likelihood that users will be able to successfully defend their own SLOs, e.g. through performance optimization. ================================================================================ SECTION: HOW-TO GUIDES PAGE: Rolling Upgrades URL: https://ate.dev/docs/guides/upgrade/ MARKDOWN: https://ate.dev/docs/guides/upgrade/index.md SOURCE: docs/guides/upgrade.md ================================================================================ # Rolling Upgrades > Runbook for a rolling upgrade of Agent Substrate. This runbook upgrades a running Agent Substrate install to a newer release. | From → to | Supported | |---|---| | A later patch release of the same minor, for example 1.2.1 → 1.2.4 | Yes. Patch releases can be skipped | | The next minor release, for example 1.2.x → 1.3.y | Yes | | Skipping a minor release, for example 1.2.x → 1.4.y | No. Upgrade to 1.3 first | | A new major release, for example 1.x → 2.x | No. Reinstall | | A minor release before 1.0, for example 0.3.x → 0.4.y | No. Reinstall. Before 1.0, breakage still exists for minor versions | If an upgrade goes wrong, you can roll back to the release you upgraded from, and no further. These rules and the step order below come from the [version skew policy](https://ate.dev/docs/reference/version-skew-policy/index.md). | Step | What changes | What running actors see | How long | |---|---|---|---| | 1 | podcertificate-controller | Nothing | A minute | | 2 | atelet, one node at a time | Nothing | Seconds to about 6 minutes per node | | 3 | workers | `SIGTERM`, then a 30-minute window to be suspended before `SIGKILL` | Seconds to hours per pool | | 4 | ate-api-server and ate-idp-server, the CRDs and ate-controller, atenet, SandboxConfig, then the rest of the system bundle | API calls, long requests and outbound connections can be cut once. See step 4 | A few minutes | Rollback is the same list in reverse, from the old release. An actor that crashes along the way goes back to its last snapshot with `kubectl ate revert actor -a `. The system upgrade does not change the database engine, the CSI drivers `--setup-csi` installs, the Kubernetes version or the node OS version. ## Where am I? Come back here after a break or a handoff. Each time a Deployment moves to a new image it gets a new ReplicaSet, and the atelet DaemonSet a new ControllerRevision, so their `AGE` tells what has moved. A component has moved when the `AGE` of its ReplicaSet with pods (`DESIRED` above 0), or of atelet's ControllerRevision with the highest `REVISION`, is shorter than the time since the upgrade began. Two ReplicaSets with pods mean a roll is under way. `podcertificate-controller` moves in step 1, `atelet` in step 2, the rest in step 4: ```bash kubectl -n podcertificate-controller-system get rs kubectl -n ate-system get rs,controllerrevision ``` The worker pools move in step 3, and their ReplicaSets read the same way. A pool is done rolling when `UP-TO-DATE` and `READY` equal `WANTED`: ```bash kubectl get rs -A -l ate.dev/worker-pool kubectl get deploy -A -l ate.dev/worker-pool -o custom-columns='NAMESPACE:.metadata.namespace,NAME:.metadata.name,UP-TO-DATE:.status.updatedReplicas,READY:.status.readyReplicas,WANTED:.spec.replicas,PAUSED:.spec.paused' ``` ## Before you start You need git, go and kubectl, and crane for prebuilt images. Agree on a time with actor owners: their running actors get `SIGTERM` in step 3. Every `ate-setup deploy` run records the settings it was given in `installs/` under the record directory: `ATE_RECORD_DIR` if set, by default `~/.cache/ate-setup` on Linux and `~/Library/Caches/ate-setup` on macOS. It prints the path as `recorded this configuration in `. The record's `cluster.substrateVersion` is the installed release. If this machine has no record of the cluster, write one from the settings you installed with ([operator-install.md](https://ate.dev/docs/guides/operator-install/index.md) lists them), and set `OLD_RELEASE` to the installed release before you run the block below. If `~/ate-upgrade` is left from an earlier upgrade, delete it first. Fill in the first three lines, then run the rest. `ate-setup` runs from the new checkout. The old checkout is kept for rollback. ```bash NEW_RELEASE=v0.4.2 # change me RECORD=~/.cache/ate-setup/installs/my-cluster.yaml # change me: this cluster's record kubectl config use-context my-cluster # change me: keep it until the upgrade is done # export ATE_API_POSTGRES_READ_WRITE_CONNECTION_STRING=... # external database only OLD_RELEASE=${OLD_RELEASE:-$(sed -n 's/^ *substrateVersion: //p' $RECORD)} echo "Upgrading '$OLD_RELEASE' -> '$NEW_RELEASE'" # stop if either is empty or not a release tag mkdir ~/ate-upgrade && cd ~/ate-upgrade cp $RECORD old.yaml && cp $RECORD new.yaml sed -i.bak "s/^ tag: .*/ tag: $NEW_RELEASE/" new.yaml # prebuilt images: the new tag grep 'tag:' old.yaml new.yaml # prebuilt images: new.yaml has the new tag git clone --branch $OLD_RELEASE https://github.com/agent-substrate/substrate.git old git clone --branch $NEW_RELEASE https://github.com/agent-substrate/substrate.git new (cd old && go install ./cmd/kubectl-ate) cd new ``` Run every step from this shell. A record holds no database connection strings: with an external database, uncomment the `export` line, and export `ATE_API_POSTGRES_OWNER_CONNECTION_STRING` too if you set it. Keep `old.yaml` as it is, for rollback. With prebuilt images, `new.yaml` now names the new tag. From source, the checkout you run from decides the release. > [!CAUTION] > Do not run `ate-setup setup csi` or `ate-setup delete` during the upgrade, or `ate-setup deploy ate-system` before step 4e. They change components out of upgrade order. ## Upgrade ### Step 1. Upgrade podcertificate-controller ```bash go run ./cmd/ate-setup deploy podcertificate-controller --config ~/ate-upgrade/new.yaml ``` This deploys the new podcertificate-controller and waits until it publishes its trust bundles. Rerunning is safe. - **Done:** the command exits 0. - **Stuck:** the command fails. [Roll back step 1](#roll-back-step-1). ### Step 2. Upgrade atelet ```bash go run ./cmd/ate-setup deploy atelet --config ~/ate-upgrade/new.yaml --rollout-timeout 1h ``` `ate-setup` prints nothing until the roll is done. To watch it, open a second terminal and run: ```bash kubectl -n ate-system rollout status ds/atelet ``` - **Progress:** `Waiting for daemon set "atelet" rollout to finish: 2 out of 5 new pods have been updated...`, then `... 4 of 5 updated pods are available...`. - **Done:** `daemon set "atelet" successfully rolled out`. - **Stuck:** the `N out of M` count does not move for more than about 6 minutes. A new atelet pod that cannot start stops the roll and leaves its node without atelet; if you cannot fix it quickly, [roll back step 2](#roll-back-step-2). A pod stuck in ContainerCreating usually means podcertificate-controller is not running. If `deploy atelet` times out, the roll still goes on; keep watching. - **Actors:** running actors keep running. A node's atelet is down for a few seconds while it restarts. ### Step 3. Move each WorkerPool to the new worker image > [!IMPORTANT] > Do not start step 3 until step 2 is done: new workers need the new atelet. Change the WorkerPool, not its Deployment: `ate-controller` overwrites edits to the Deployment. If your WorkerPools live in Git or are applied by CI, change `workerImage` there as well, or the next sync puts the old image back and rolls every worker again. Before you patch any pool, list the pools and save the list for rollback. Each pool needs the new image for its sandbox CLASS: ```bash kubectl get workerpools -A -o custom-columns='NAMESPACE:.metadata.namespace,NAME:.metadata.name,CLASS:.spec.sandboxClasses[0].name,IMAGE:.spec.workerImage' | tee ~/ate-upgrade/before-workerpools.txt ``` If you build from source, build and push both worker images once. This touches nothing in the cluster and ends by printing `ateworker-gvisor: ` and `ateworker-microvm: `: ```bash go run ./cmd/ate-setup publish worker-images --config ~/ate-upgrade/new.yaml | tee ~/ate-upgrade/worker-images.txt ``` Then move the pools, starting with a small pool of each CLASS that you can afford to disrupt. For each pool: 1. Set the pool: ```bash NS=my-namespace # change me WORKERPOOL=my-pool # change me CLASS=$(kubectl -n $NS get workerpool $WORKERPOOL -o jsonpath='{.spec.sandboxClasses[0].name}') ``` 2. Set its new image. From source, take the ref after `ateworker-$CLASS: ` in `~/ate-upgrade/worker-images.txt`: ```bash NEW_IMAGE=$(sed -n "s/^ateworker-$CLASS: //p" ~/ate-upgrade/worker-images.txt) ``` With prebuilt images, take `ateworker-$CLASS` from your image repo at the new tag, pinned by digest: ```bash NEW_IMAGE=$(sed -n 's/^ repo: //p' ~/ate-upgrade/new.yaml)/ateworker-$CLASS:$NEW_RELEASE NEW_IMAGE=$NEW_IMAGE@$(crane digest $NEW_IMAGE) ``` 3. Check that `echo "$NS/$WORKERPOOL $CLASS $NEW_IMAGE"` shows `ateworker-` plus the pool's CLASS and a full `@sha256:` digest. A bad image still removes the pool's first old worker. 4. Patch the worker pool and wait until ate-controller moves its Deployment to the new image: ```bash kubectl -n $NS patch workerpool $WORKERPOOL --type merge -p "{\"spec\":{\"workerImage\":\"$NEW_IMAGE\"}}" kubectl -n $NS wait deployment/$WORKERPOOL --for=jsonpath='{.spec.template.spec.containers[0].image}'="$NEW_IMAGE" --timeout=2m ``` 5. Once the first pool of a CLASS has one new worker running, patch the other pools of that CLASS without waiting for the roll to finish. A new worker is running when the newest ReplicaSet of the pool shows READY 1 or more: ```bash kubectl -n $NS get rs -l ate.dev/worker-pool=$WORKERPOOL ``` Then repeat on the other CLASS. What to expect: - **Progress:** `kubectl -n $NS rollout status deployment/$WORKERPOOL` prints how many new workers are updated and available. Ctrl-C stops only the watch. - **Done:** Step 3 is done when every pool is done rolling with `kubectl get deploy -A -l ate.dev/worker-pool`. Go to Step 4 once done. - **Pace:** a WorkerPool replaces 10% of its workers per batch, rounded down, at least one. With spare capacity, every batch has its `SIGTERM` within minutes and the old workers drain in parallel. Without it, each batch waits up to 30 minutes for old workers to exit. - **Stuck:** rollout status fails with `exceeded its progress deadline` (the roll goes on), or a worker pod stays Terminating for more than 60 minutes, its grace period (a worker normally exits within about 30). Check `kubectl -n $NS get pods -l ate.dev/worker-pool=$WORKERPOOL`. A bad image stops the roll after the first batch: [roll back that pool](#roll-back-step-3). - **Actors:** each gets `SIGTERM` and 30 minutes to be suspended. Substrate does not suspend them. An actor that is not suspended by then becomes `CRASHED` and loses its work since its last snapshot. [Eviction](https://ate.dev/docs/reference/api-guide/index.md#eviction) in the API guide says what an actor needs to keep its state. ### Step 4. Upgrade the Substrate control plane #### 4a. ate-api-server and ate-idp-server The first new API server pod migrates the database schema, and rolling back does not undo that. Back up the database first: with `pg_dump` for the bundled one, or an on-demand backup for Cloud SQL. With an external database, check that `ATE_API_POSTGRES_READ_WRITE_CONNECTION_STRING` is set in this shell: without it, `deploy apiserver` points the API server at the bundled database. ```bash go run ./cmd/ate-setup deploy apiserver --config ~/ate-upgrade/new.yaml kubectl -n ate-system rollout status deploy/ate-api-server kubectl -n ate-system rollout status deploy/ate-idp-server kubectl ate get actors -A ``` - **Done:** both print `successfully rolled out` and `kubectl ate get actors -A` lists your actors. - **Stuck:** rollout status does not finish, or a new pod keeps restarting. [Roll back step 4](#roll-back-step-4). - **API clients:** an API call can fail once with a retriable error while an API server pod restarts. Retry it. #### 4b. CRDs and ate-controller ```bash go run ./cmd/ate-setup deploy ate-controller --config ~/ate-upgrade/new.yaml ``` This applies the new CRDs and ate-controller's RBAC, then deploys the new `ate-controller`. Rerunning is safe. Within a major the worker pod template stays the same, so no pool rolls. - **Done:** the command exits 0, `kubectl -n ate-system get pods -l app=ate-controller` shows one pod, Running, with RESTARTS 0, and a minute later `kubectl get pods -A -l ate.dev/worker-pool` shows no worker pod younger than this step. - **Stuck:** the new ate-controller pod keeps restarting. [Roll back step 4](#roll-back-step-4). #### 4c. atenet ```bash go run ./cmd/ate-setup deploy atenet --config ~/ate-upgrade/new.yaml kubectl -n ate-system rollout status deploy/atenet-router kubectl -n ate-system rollout status deploy/atenet-egress ``` - **Done:** both print `successfully rolled out`. - **Stuck:** rollout status does not finish, or a new pod keeps restarting. [Roll back step 4](#roll-back-step-4). - **Actors:** a request to an actor, or an outbound connection, still open about 15 seconds after its `atenet` pod starts draining can be cut once. #### 4d. SandboxConfig ```bash go run ./cmd/ate-setup deploy sandboxconfig --config ~/ate-upgrade/new.yaml ``` This applies the SandboxConfig admission policy and the default SandboxConfig. It does not roll workers. If the release changes a sandbox version, new actors start on the new version, and actors suspended on the previous one still restore on it. - **Done:** the command exits 0. - **Stuck:** the command fails. [Roll back step 4](#roll-back-step-4). - **Actors:** a suspended actor whose sandbox version the release drops resumes without its memory, keeping its files. #### 4e. The rest of the system bundle ```bash go run ./cmd/ate-setup deploy ate-system --config ~/ate-upgrade/new.yaml --setup-csi=none ``` This applies what the steps above do not, such as the ate-system NetworkPolicies and the bundled PostgreSQL. The components above are already on the new release and do not roll again. `--setup-csi=none` leaves the CSI drivers as they are. Rerunning is safe. - **Done:** the command exits 0. - **Stuck:** the command fails. [Roll back step 4](#roll-back-step-4). - **API clients:** if the release changes the bundled PostgreSQL, it restarts once, and API calls fail until it is back. Then install the new `kubectl ate` with `go install ./cmd/kubectl-ate`. Actor owners can now use API fields and commands new in this release, and you can set WorkerPool, SandboxConfig and CSIDriverConfig fields new in this release. The upgrade is done when step 4 is done and [Where am I?](#where-am-i) shows every component and pool moved, every pool done rolling and none paused. Keep `~/ate-upgrade` while you might still roll back, then delete it. ## Something's wrong, or I need to roll back Stopping between steps is safe, and so is stopping partway through step 3. A pool you have patched keeps rolling unless you pause it. ### Pause a pool's roll If a pool's new workers look wrong but you are not ready to roll it back, or you want to stop interrupting actors for a while, pause its Deployment. ate-controller leaves this one Deployment change in place. ```bash kubectl -n $NS rollout pause deployment/$WORKERPOOL kubectl -n $NS rollout resume deployment/$WORKERPOOL ``` Workers already removed still get their 30 minutes. Do not leave a pool paused, and do not start step 4 while one is: a paused pool ignores every later change to its workers. ### Roll back To go back to the old release, roll back every step you began, finished or not, starting with the last. To fix one pool, roll back only that pool. Rollback runs from the old checkout, with `old.yaml`: ```bash cd ~/ate-upgrade/old ``` #### Roll back step 4 Tell actor owners to stop using API fields new in this release, because the old API server rejects them. Then: ```bash go run ./cmd/ate-setup deploy sandboxconfig --config ~/ate-upgrade/old.yaml go run ./cmd/ate-setup deploy atenet --config ~/ate-upgrade/old.yaml go run ./cmd/ate-setup deploy ate-controller --config ~/ate-upgrade/old.yaml go run ./cmd/ate-setup deploy apiserver --config ~/ate-upgrade/old.yaml go install ./cmd/kubectl-ate ``` `deploy sandboxconfig` removes a sandbox version new in this release, so actors suspended on it resume without their memory. `deploy ate-controller` restores the old CRDs, RBAC and `ate-controller`. No pool rolls. Fields that the new release added to WorkerPools, SandboxConfigs or CSIDriverConfigs are dropped from those objects. Remove them from your YAML files too, or `kubectl apply` rejects the files. The database schema stays new; the old API server drops fields it does not know when it rewrites a record. Step 4e is not undone either: the NetworkPolicies and the bundled PostgreSQL stay as the new release left them. #### Roll back step 3 Set each patched pool back to its image in `~/ate-upgrade/before-workerpools.txt`: ```bash NS=my-namespace # change me WORKERPOOL=my-pool # change me OLD_IMAGE=$(awk -v ns=$NS -v name=$WORKERPOOL '$1 == ns && $2 == name {print $4}' ~/ate-upgrade/before-workerpools.txt) kubectl -n $NS patch workerpool $WORKERPOOL --type merge -p "{\"spec\":{\"workerImage\":\"$OLD_IMAGE\"}}" ``` If you paused a WorkerPool, resume it now with `kubectl -n $NS rollout resume deployment/$WORKERPOOL`. Then watch: ```bash kubectl -n $NS rollout status deployment/$WORKERPOOL ``` Only workers already on the new image are replaced, and their actors get `SIGTERM` again. Wait as in [step 3](#step-3-move-each-workerpool-to-the-new-worker-image). #### Roll back step 2 > [!WARNING] > Do not roll back step 2 before every pool is back on its old image and `kubectl get pods -A -l ate.dev/worker-pool` shows no Terminating pod. A new worker with an old atelet is not supported. ```bash go run ./cmd/ate-setup deploy atelet --config ~/ate-upgrade/old.yaml --rollout-timeout 1h ``` #### Roll back step 1 ```bash go run ./cmd/ate-setup deploy podcertificate-controller --config ~/ate-upgrade/old.yaml ``` ================================================================================ SECTION: HOW-TO GUIDES PAGE: Actor JWTs URL: https://ate.dev/docs/guides/actor-jwts/ MARKDOWN: https://ate.dev/docs/guides/actor-jwts/index.md SOURCE: docs/guides/actor-jwts.md ================================================================================ # Actor JWTs > What an actor JWT contains and how a relying party verifies it. ate-api-server mints short-lived JWTs that identify an actor, and ate-idp-server publishes the keys that verify them. A relying party, such as a cloud provider's token exchange or a service the actor calls, trusts a token by checking it against the issuer's OpenID Connect discovery document. This page describes what a token contains and how a relying party verifies it. Tokens come from the `Control.MintActorJwt` RPC. The egress gateway calls it to attach an actor JWT to an actor's outbound requests. ## Verifying a token A relying party is configured with the issuer URL it trusts and the audience it expects. To verify a token: 1. Check that the `iss` claim equals the trusted issuer. 2. Fetch `/.well-known/openid-configuration` over TLS, verifying the server's certificate, and check that its `issuer` equals the same value. See [Reaching the issuer](#reaching-the-issuer) for how to trust the certificate. 3. Fetch the document's `jwks_uri` over TLS with the same certificate check, and take the key whose `kid` matches the token header. Check the signature with that key and its `alg`. 4. Check that `aud` contains the expected audience, and check `exp` and `nbf` against the current time. 5. Check that `sub` starts with `actor/`. The same keys sign `worker-for-actor` tokens, which a worker or another Substrate component uses when acting for an actor. Those have `sub` `worker-for-actor//` and their claims under `worker-for-actor.ate.dev`, and a relying party that wants the actor itself must reject them. When a token names a `kid` the relying party has not seen, it should refetch the key set: a new key is published before it signs anything. Rate-limit key set fetches instead of fetching on every unrecognized `kid`, so invalid tokens cannot overload ate-idp-server. ## The token ### Header | Field | Value | |-------|-------| | `typ` | `JWT` | | `alg` | The signing key's algorithm: `ES256`, or `RS256` | | `kid` | The signing key's ID. Treat it as an opaque string that matches one key in the key set | ### Claims | Claim | Value | |-------|-------| | `iss` | The issuer URL. See [The issuer](#the-issuer) | | `sub` | `actor//` | | `aud` | The audiences the caller requested, always a JSON array | | `exp` | Expiry, in Unix seconds | | `nbf` | Five minutes before `iat`, to allow for clock skew | | `iat` | Issue time, in Unix seconds | | `jti` | A random token ID | | `actor.ate.dev` | An object with the actor's `atespace`, `actorName`, and `actorUID` | The caller chooses the lifetime with `expiration_seconds`, which must be between 300 and 3600. The RPC also returns `expire_time`, equal to `exp`. A request names between 1 and 16 audiences of up to 512 characters each. Substrate adds no audience of its own. The subject names the actor, not one incarnation of it: an actor deleted and recreated under the same name gets the same `sub`. A relying party that has to tell the two apart can bind to the `actorUID` in `actor.ate.dev`, which differs. Atespace and actor names are each up to 63 characters, so `sub` can be up to 133 bytes long. ### Example A decoded token, with an example key ID and UID: ```json { "typ": "JWT", "alg": "ES256", "kid": "9XGqai5pBy8t4HsHIxcu68X4_e4fOg7MYyKYMHc7WPA" } ``` ```json { "iss": "https://idp.ate-system.svc", "sub": "actor/team-a/agent-1", "aud": ["https://api.example.com"], "exp": 1760401800, "nbf": 1760399700, "iat": 1760400000, "jti": "MZ4RHTOXLJ6GSVNQ2YAKE7WB3C", "actor.ate.dev": { "atespace": "team-a", "actorName": "agent-1", "actorUID": "4f9c3a1e-8d2b-4c77-9e0a-6b5f1d2c3e4a" } } ``` ## The issuer The issuer is `https://` followed by a host, with no path, query, or fragment, for example `https://idp.ate-system.svc` or `https://cluster-a.idp.example.com`. A port is allowed. ate-api-server puts it in `iss`, and ate-idp-server serves the discovery document and key set at fixed paths under it. The installer sets it through the `actorJWT.issuer` setting (`ACTOR_JWT_ISSUER`, or `--actor-jwt-issuer`; see [operator-install.md](https://ate.dev/docs/guides/operator-install/index.md#settings)). It defaults to `https://idp..svc`, the in-cluster URL of ate-idp-server. The installer gives the same value to ate-api-server, through the `ate-api-server-envvars` ConfigMap, and to ate-idp-server, as its `--issuer` flag, so the token and the discovery document cannot name different issuers. A changed issuer reaches both on the next `ate-setup deploy ate-system`. Tokens signed before a change keep the old `iss` until they expire, at most an hour later. A relying party that trusts only the new issuer rejects them for that time. ## Discovery document and key set ate-idp-server answers `GET` and `HEAD` on two paths, both as `application/json`. The discovery document, at `/.well-known/openid-configuration`: ```json { "issuer": "https://idp.ate-system.svc", "jwks_uri": "https://idp.ate-system.svc/openid/v1/jwks", "response_types_supported": ["id_token"], "subject_types_supported": ["public"], "id_token_signing_alg_values_supported": ["ES256"], "claims_supported": ["iss", "sub", "aud", "exp", "nbf", "iat", "jti", "actor.ate.dev", "worker-for-actor.ate.dev"] } ``` `id_token_signing_alg_values_supported` lists each algorithm in the signing pool. There is no `authorization_endpoint`, because the issuer has no login flow. The key set, at `/openid/v1/jwks`, is `{"keys": [...]}` with one entry per key in the pool, sorted by `kid`. Each entry has `kty`, `kid`, `use` (`sig`), `alg`, and the public key: `crv` (`P-256`), `x`, and `y` for an ES256 key, or `n` and `e` for an RS256 key. The set includes keys that are not signing, so tokens signed by an older key keep verifying during a rotation. Both documents are built from the signing pool on every request. ate-idp-server rereads the pool at most once a minute. ## Reaching the issuer ate-idp-server runs in the system namespace behind the `idp` Service on port 443. It serves HTTPS with a certificate from the `servicedns.podcert.ate.dev/identity` signer, which names `idp..svc`. A verifier inside the cluster trusts that certificate through the signer's ClusterTrustBundle, projected the way [ate-controller.yaml](https://github.com/agent-substrate/substrate/blob/main/manifests/ate-install/ate-controller.yaml) mounts it. A relying party outside the cluster cannot resolve a Service name. To serve one, put an Ingress or load balancer with a publicly trusted WebPKI certificate on a host you own, route `/.well-known/openid-configuration` and `/openid/v1/jwks` to the `idp` Service over HTTPS, and set the issuer to `https://`. The installer does not create the Ingress, the DNS record, or the certificate. [actor-jwt-federation.md](https://ate.dev/docs/guides/actor-jwt-federation/index.md) walks through it, and through setting up a relying party to trust the issuer. ## Signing keys and rotation The signing keys live in the `actor-id-jwt-pool` Secret. One key signs at a time, and the key set publishes all of them. The installer creates the pool with one ES256 key. Set `actorJWT.algorithm` (`ACTOR_JWT_ALGORITHM`) to `RS256` for a 4096-bit RSA key instead, for relying parties that don't support ES256; the setting only affects a pool the installer creates. A key's ID defaults to the base64url SHA-256 of its public key's PKIX encoding, without padding. Keys created with an explicit ID keep it. Rotate with the `kubectl ate admin` commands, against the pool Secret (`--name actor-id-jwt-pool --secret-namespace ate-system`): 1. `add-jwt-key` adds a key that does not sign yet. ate-idp-server publishes it once the updated Secret reaches its pods and it rereads the pool. 2. `activate-jwt-key --key-id ` makes it the signing key, once relying parties have refetched the key set. 3. `remove-jwt-key --key-id ` removes the old key, once every token it signed has expired. `list-jwt-keys` shows each key's ID and algorithm and marks the one that signs. `add-jwt-key` takes `--alg` and `--key-id` like `make-jwt-pool`. [actor-jwt-key-rotation.md](https://ate.dev/docs/guides/actor-jwt-key-rotation/index.md) gives the timings for each step and what to do about a compromised key. ## Known limitations - **No revocation.** A token stays valid until `exp`, up to an hour after it was minted, even if its actor is deleted. Removing a key from the pool stops every token that key signed, for every actor, once relying parties refetch the key set. - **Tokens are bearer tokens.** Whoever receives one can present it to any relying party that accepts its audience until it expires. Atespace editors write the egress policy, so they choose both the audience and the host that receives the token. An upstream that echoes request headers hands the token back to the actor. A relying party should check `sub` or `actor.ate.dev` as well as `aud`. - **No cloud token exchange.** Cloud token exchange endpoints take the JWT as a request parameter, and the gateway only replaces headers, so Substrate can't yet trade an actor JWT for cloud credentials ([#1661](https://github.com/agent-substrate/substrate/issues/1661)). - **A recreated actor can get a stale UID.** The gateway caches tokens by actor name. For up to two thirds of a token's lifetime after an actor is recreated, the gateway can inject a token whose `actorUID` names the old actor ([#2208](https://github.com/agent-substrate/substrate/issues/2208)). - **Rotation is manual.** Keys don't expire, and nothing rotates them on a schedule. An operator runs the steps under [Signing keys and rotation](#signing-keys-and-rotation). ================================================================================ SECTION: HOW-TO GUIDES PAGE: Federating Actor JWTs URL: https://ate.dev/docs/guides/actor-jwt-federation/ MARKDOWN: https://ate.dev/docs/guides/actor-jwt-federation/index.md SOURCE: docs/guides/actor-jwt-federation.md ================================================================================ # Federating Actor JWTs > Serve the actor JWT issuer publicly and set up a relying party outside the cluster to trust it. A relying party outside the cluster, such as a cloud provider's workload identity federation, can trust actor JWTs by fetching the issuer's discovery document and key set. This page serves the issuer where it can reach them and lists what to configure on the relying party. [actor-jwts.md](https://ate.dev/docs/guides/actor-jwts/index.md) describes the token and how a relying party verifies it. ## Serve the issuer publicly A relying party outside the cluster fetches the discovery document over the internet, so the default in-cluster issuer, `https://idp..svc`, does not work. 1. Pick a host you own, for example `idp.example.com`, and put an Ingress, Gateway, or load balancer on it with a publicly trusted WebPKI certificate. 2. Route `/.well-known/openid-configuration` and `/openid/v1/jwks` to the `idp` Service in the system namespace, port 443, over HTTPS. The pods present a certificate from the cluster's `servicedns.podcert.ate.dev/identity` signer, not a public CA, so have the load balancer trust that signer's ClusterTrustBundle, or skip backend verification if it can't. 3. Point the load balancer's health check at `/healthz` over HTTPS. The other paths return 404. 4. Redeploy with the issuer set. Pass the record of your last install so the run keeps its other settings; the flag overrides the record: ```sh ate-setup deploy ate-system \ --config ~/.cache/ate-setup/installs/.yaml \ --actor-jwt-issuer=https://idp.example.com ``` The record doesn't include secrets, such as database connection strings, so you need to pass secrets again. See [Install runs record what they used](https://ate.dev/docs/guides/operator-install/index.md#install-runs-record-what-they-used) for where records are kept. 5. Check from outside the cluster that the document names the same issuer: ```sh curl -s https://idp.example.com/.well-known/openid-configuration | jq -r .issuer ``` Tokens signed before the change keep the old `iss` for up to an hour. ## Set up the relying party Configure the relying party with: - The issuer, exactly as the discovery document names it. - The audience it expects. The egress policy's `actorJwt.audiences` has to include it; see [Actor JWTs](https://ate.dev/docs/guides/egress-credential-injection/index.md#actor-jwts) in the egress guide. - Authorization rules for the subjects that should have access. An actor's `sub` is `actor//`. If the relying party doesn't accept ES256, [switch the pool to RS256](https://ate.dev/docs/guides/actor-jwt-key-rotation/index.md#changing-the-algorithm). To check the setup, have an egress policy inject a token for the relying party's audience into a request to a server you control, and present that token to the relying party. Substrate can't yet present an actor JWT to a cloud token exchange endpoint itself; see [Known limitations](https://ate.dev/docs/guides/actor-jwts/index.md#known-limitations). ================================================================================ SECTION: HOW-TO GUIDES PAGE: Rotating the Actor JWT Signing Key URL: https://ate.dev/docs/guides/actor-jwt-key-rotation/ MARKDOWN: https://ate.dev/docs/guides/actor-jwt-key-rotation/index.md SOURCE: docs/guides/actor-jwt-key-rotation.md ================================================================================ # Rotating the Actor JWT Signing Key > The rotation steps, how long to wait between them, and replacing a compromised key. ate-api-server signs actor JWTs with the active key in the `actor-id-jwt-pool` Secret, and ate-idp-server publishes every key in the pool. A rotation adds a key, waits until relying parties have it, makes it the signing key, and removes the old key once nothing it signed can still be presented. [actor-jwts.md](https://ate.dev/docs/guides/actor-jwts/index.md#signing-keys-and-rotation) describes the pool. The commands below use the default Secret. Set `ISSUER` to your `actorJWT.issuer`: ```sh ISSUER=https://idp.example.com POOL="--name actor-id-jwt-pool --secret-namespace ate-system" ``` ## How long each step takes - **Reaching the pods.** A change to the Secret reaches the ate-api-server and ate-idp-server pods when the kubelet next syncs the mounted Secret, and each process rereads the pool within a minute after that. Allow a few minutes. - **Reaching relying parties.** Each relying party caches the key set for its own period. Unless every relying party documents a shorter one, wait a day. - **Tokens in flight.** A token is valid for up to an hour. The egress gateway reuses a token until a third of its lifetime is left, so a token signed just before the new key took over can still be presented, and has to verify, for up to an hour after that. ## Routine rotation 1. Note the active key: ```sh kubectl ate admin list-jwt-keys $POOL ``` 2. Add the new key. It is published but signs nothing yet. Pass `--alg RS256` to change the algorithm; see [Changing the algorithm](#changing-the-algorithm). ```sh kubectl ate admin add-jwt-key $POOL ``` 3. Wait until the key set lists the new key ID: ```sh curl -s "$ISSUER/openid/v1/jwks" | jq -r '.keys[].kid' ``` Then wait out the longest key set cache among your relying parties. 4. Make the new key the signing key: ```sh kubectl ate admin activate-jwt-key $POOL --key-id ``` 5. Wait at least an hour, plus a few minutes for the change to reach every ate-api-server pod. 6. Remove the old key: ```sh kubectl ate admin remove-jwt-key $POOL --key-id ``` ## Changing the algorithm Follow the same steps and pass `--alg` to `add-jwt-key`, for example to move to RS256 for a relying party that doesn't accept ES256. Every relying party has to accept the new algorithm before step 4. During the overlap the discovery document's `id_token_signing_alg_values_supported` lists both. The `actorJWT.algorithm` setting only affects a pool the installer creates, so rotating is the way to change an existing install. ## Replacing a compromised key Skip the waits: 1. Add a new key and activate it right away. Relying parties that haven't fetched the new key reject tokens it signs until they refetch the key set. 2. Once the activation has reached every ate-api-server pod, remove the compromised key. 3. Restart the egress gateway so it drops the tokens it cached: ```sh kubectl rollout restart deployment/atenet-egress -n ate-system ``` A relying party keeps accepting the compromised key until it refetches the key set, so make each one refetch if it offers a way to. ================================================================================ SECTION: REFERENCE PAGE: API Guide URL: https://ate.dev/docs/reference/api-guide/ MARKDOWN: https://ate.dev/docs/reference/api-guide/index.md SOURCE: docs/reference/api-guide.md ================================================================================ # API Guide > Define WorkerPools, ActorTemplates, and SandboxConfigs, and call the Control Plane gRPC API. This guide explains how to configure Substrate resources to deploy high-density, stateful agents. ## 1. WorkerPool: The Physical Capacity The `WorkerPool` defines the pool of physical "warm" compute capacity. It manages a fleet of standby worker pods that are ready to receive and execute actor states. ### Specification (`WorkerPoolSpec`) | Field | Type | Description | | :--- | :--- | :--- | | `replicas` | `int32` | **Required.** Number of physical standby pods to maintain in the cluster. | | `workerImage` | `string` | **Required.** The container image for the `ateworker` process (e.g. `ko://go.ate.dev/substrate/cmd/ateworker-gvisor`). | | `sandboxClasses` | `[]WorkerPoolSandboxClass` | **Required.** The sandbox runtime families the pool runs; exactly one entry today. Each entry has `name` (**required**, `gvisor` or `microvm`, immutable), which drives the worker pod shape (e.g. KVM device mounts, node placement), and `configRef.name` (optional), which names a cluster-scoped [`SandboxConfig`](#3-sandboxconfig-the-sandbox-itself) of the same class. Without `configRef`, the pool uses the class default (see [Class default](#class-default)). The controller records the result in `status.sandboxClasses`. The sandbox binaries an actor boots with come from the `SandboxConfig` the pool of its assigned worker resolved. | | `template` | `WorkerPoolPodTemplate` | **Optional.** Metadata, scheduling, and resource settings for worker workloads. | #### `WorkerPoolPodTemplate` (`spec.template`) | Field | Type | Workload mapping | | :--- | :--- | :--- | | `labels` | `map[string]string` | Generated Deployment and `spec.template.metadata.labels` (max 64) | | `annotations` | `map[string]string` | Generated Deployment and `spec.template.metadata.annotations` (max 64) | | `nodeSelector` | `map[string]string` | `spec.nodeSelector` | | `tolerations` | `[]Toleration` | `spec.tolerations` (max 16) | | `priorityClassName` | `string` | `spec.priorityClassName` | | `nodeAffinity` | `NodeAffinity` | `spec.affinity.nodeAffinity` | | `resources` | `ResourceRequirements` | `spec.containers[].resources` | | `serviceAccountName` | `string` | `spec.serviceAccountName`; defaults to `ateworker`, which the controller creates | Keys in `ate.dev/` and its subdomains (for example, `policy.ate.dev/`) are reserved for controllers and cannot be set in `template.labels` or `template.annotations`. Metadata keys and label values must follow Kubernetes syntax. `template.labels` and `template.annotations` only configure Kubernetes workload metadata; they do not affect actor scheduling. Actor selectors match `WorkerPool.metadata.labels`, not `WorkerPool.spec.template.labels`. #### Worker security Substrate sets a fixed capability set for each sandbox runtime. Every worker drops `ALL` container-runtime default capabilities and adds back only the set below: 9 capabilities for gVisor and 11 for microVM. These permissions are not configurable per pool. Actor containers are configured separately through `ActorTemplate.containers[].securityContext`. Workers run as UID/GID 0 with `privileged: false` and `allowPrivilegeEscalation: false`. They retain `SYS_ADMIN` and `NET_ADMIN` for mounts and networking, so they still require trusted nodes and the deployment permissions described in [Security](https://ate.dev/docs/security/security/index.md). | Capability | `gvisor` | `microvm` | | --- | --- | --- | | `SYS_CHROOT`, `SYS_PTRACE` | Not granted; runsc's sandbox user namespace supplies its own | Not granted | | `SETPCAP` | Not granted | Retained so virtiofsd's serving child can drop its bounding set (`PR_CAPBSET_DROP` needs `CAP_SETPCAP`) | | `CHOWN`, `FOWNER` | Retained for worker filesystem ownership and mode operations | Same; also used by virtiofsd for guest filesystem operations | | `MKNOD` | Not granted; since Linux 5.8, 0:0 overlay whiteouts do not require it | Retained for guest device-node creation | | `DAC_READ_SEARCH` | Not granted | Not granted; virtiofsd runs with file handles off (`--inode-file-handles=never`) and needs it only to open them | | `FSETID` | Not granted | Retained for virtiofsd's setgid-file behavior | | `NET_RAW` | Needed to open the sandbox's raw socket | Not granted | | `SETFCAP`, `SETUID` | Needed to start runsc's gofer | Retained in virtiofsd's child set; `SETUID` is needed for non-root guest file creation | | `SETGID` | Needed to start runsc's gofer | Needed by virtiofsd to drop supplementary groups | | `DAC_OVERRIDE` | Needed to mount the rootfs overlay | Same | | `NET_ADMIN` | Needed to set up actor networking | Same | | `SYS_ADMIN` | Needed to set up the worker's mounts | Same | MicroVM workers retain virtiofsd's eight default child capabilities to preserve guest filesystem operations. virtiofsd tracks guest inodes with `O_PATH` file descriptors instead of file handles, which is also what it falls back to by default on the worker's overlayfs rootfs. virtiofsd explicitly uses namespace isolation (`pivot_root`, not `chroot`) and its own seccomp filter. The outer worker's seccomp and AppArmor profiles remain `Unconfined` so the sandbox runtimes can set up their mounts; worker seccomp hardening is separate from these capability defaults. #### Worker Capacity (`spec.template.resources`) Setting `resources.limits` (CPU and Memory) on a `WorkerPool` establishes each worker pod's **capacity** — the envelope its actor sandboxes share, taken from the `ateworker` container's limits. The scheduler only places an actor on a worker whose remaining capacity is `>=` the actor's declared resource limits (see [Sandbox Right-Sizing](#sandbox-right-sizing-resources) on the `ActorTemplate`). - Worker capacity is a shared budget: each actor placed on a worker subtracts its declared limits from what is left. Size a pool's `limits` for the actors it should host together. - A worker also has an actor limit, set by the ateworker's `--max-actors` flag (default 1000). Placement stops at whichever runs out first. - A worker keeps some of its memory limit for itself and reports the rest as capacity. Each actor can also use memory on its worker outside its own limits, e.g. for the microVM's VMM. The worker reports this overhead with each runtime, placement counts it against capacity, and it shows in the worker's allocated resources. - Capacity is advisory for placement only: a worker that declares no CPU/memory limit reports zero capacity for that dimension, which the scheduler treats as **unconstrained** (placement is never blocked by missing data). The actual sandbox size still comes from the `ActorTemplate`. ### Example ```yaml apiVersion: ate.dev/v1 kind: WorkerPool metadata: name: agent-pool namespace: ate-demo labels: workload: secret-agent spec: replicas: 10 workerImage: ko://go.ate.dev/substrate/cmd/ateworker-gvisor sandboxClasses: - name: gvisor configRef: name: gvisor-default template: labels: project: agent-platform annotations: policy.example.com/exemption: sandbox-host ``` ### Devices (GPUs) — temporarily unsupported Substrate does not pass devices through to actors. A GPU worker pod's devices were injected into every actor container; that was removed while the device API is designed, since a resource name and a count cannot express which container gets a device, sharing one, or resuming onto compatible hardware. A `nvidia.com/gpu` pool limit is now an ordinary extended resource: it places the worker pod on a GPU node and reserves the device, and nothing else reads it. ### Status (`WorkerPoolStatus`) | Field | Type | Description | | :--- | :--- | :--- | | `replicas` | `int32` | Total number of worker pods, mirrored from the managed Deployment. | | `selector` | `string` | Label selector for the worker pods. | | `sandboxClasses` | `[]WorkerPoolSandboxClassStatus` | One entry per sandbox class, with `name`, `configRef.name` (the `SandboxConfig` in use), `pendingConfigRef.name` (transient, the config the class is switching to; see [Deletion protection](#deletion-protection)), and `conditions`. The `Resolved` condition is `True` (reason `Resolved`) when the class's spec resolves. Otherwise it is `False` with reason `SandboxConfigNotFound`, `SandboxConfigDeleting`, `SandboxClassMismatch`, or `DefaultConfigNotFound`, the controller emits a Warning event, and `configRef` keeps the last resolved config. `configRef` is omitted if the class has never resolved. Each time `configRef` changes, the controller emits a Normal `SandboxConfigSwitched` event naming the old and new config, so `kubectl events --for workerpool/NAME` shows the configs the pool used recently. Node-side failures to sync a resolved config's runtimes (for example, an unreachable asset URL or SHA256 mismatch) are emitted by atelet as Warning `SandboxRuntimeSyncFailed` events on the worker pod (`kubectl events --for pod/POD_NAME`). The API server keeps an event for one hour by default, so the full history is only in a logging backend that stores events. The controller creates the worker Deployment only once every class has a config, and keeps updating an existing Deployment while a class does not resolve. | --- ## 2. ActorTemplate: The Workload Blueprint The `ActorTemplate` defines the code, environment, and state-management policies for a specific type of agent. By default, it is used to generate a "Golden Snapshot" that new actors can start from. ### Specification (`ActorTemplate`) | Field | Type | Description | | :--- | :--- | :--- | | `containers` | `[]Container` | **Required.** The workload definition — see [Container Fields](#container-fields) below. Each container may also declare an optional `wakeupProbe` HTTP probe — see [Container Wakeup Probe](#container-wakeup-probe-wakeupprobe). | | `workerSelector` | `*Selector` | Optional. Gates which `WorkerPool`s actors from this template may use, by matching against each pool's labels (`matchLabels`). If unset, all pools are eligible (subject to the actor's own `worker_selector`). The pool sets the sandbox class, so this is also how a template pins one: if your cluster has pools of more than one sandbox class and the workload needs a particular class, make sure the selector points only at pools of that class. An actor booting with no snapshot and no selector may run on any class, and keeps it from its first checkpoint on. An actor created from the golden snapshot or a `sourceTag` runs on that snapshot's class. See [Sandbox Classes](https://ate.dev/docs/reference/sandbox-classes/index.md) for what differs between classes. The pools a template selects must also share a CPU architecture; see `sandboxRuntime` under [Snapshot Storage Layout](#snapshot-storage-layout). | | `snapshotConfig` | `SnapshotConfig` | Optional; omitted fields take their defaults. The base object-storage location snapshots are written under (`storageLocation`; **Preview**, `PerTemplateSnapshotLocation` gate), the snapshot fidelity (`preferredFidelity`) every Suspend captures, and automated golden snapshot creation (`goldenSnapshotConfig.mode`). See [Snapshot Storage Layout](#snapshot-storage-layout). | | `volumes` | `[]Volume` | Optional. Volumes the containers may mount, each a `durableDir`, an `image` read-only OCI image volume (`image.reference`, pinned by digest), an `external` CSI volume source (**Preview**, `ExternalVolumes` gate; see [CSI Volumes Guide](https://ate.dev/docs/guides/csi-volumes/index.md)), or a `systemInfo` volume (see [SystemInfo Volumes](#systeminfo-volumes)). Every declared volume must be mounted by at least one container. A `durableDir` is created `0777`, as Kubernetes creates an emptyDir, so a container running as any user can write it. A snapshot restores it with the mode it had, so the container's own chmod survives suspend and resume. | | `resources` | `*ResourceRequirements` | Required. Declares each actor's compute size via `limits`, which must set both `cpuMillicores` and `memoryBytes` — see [Sandbox Right-Sizing](#sandbox-right-sizing-resources). Immutable, like the rest of the template. | | `goldenEgressPolicy` | `EgressPolicyTemplate` | **Preview** (`GoldenEgressPolicy` gate). Optional egress policy for the golden actor during initialization. Requires at least one rule, using the same rule defaults and validation as `EgressPolicy`. See [Golden Egress Policy](#golden-egress-policy-goldenegresspolicy). | The sandbox runtime and its binaries (e.g. the gVisor `runsc` binary, or a microVM kernel/firmware/config) are configured on the [`WorkerPool`](#1-workerpool-the-physical-capacity) via its `sandboxClasses` and the cluster-scoped [`SandboxConfig`](#3-sandboxconfig-the-sandbox-itself) it resolves, rather than on the `ActorTemplate`. An `ActorTemplate` selects its `WorkerPool`s via `workerSelector` (AND'd with the actor's `worker_selector`). When an actor runs on a worker, the runtime it runs under is stamped onto the actor's assignment and snapshots (`sandboxRuntime`: the sandbox class and the runtime's compatibility version) so subsequent resumes prefer runtime-compatible workers. ### Golden Egress Policy (`goldenEgressPolicy`) > [!IMPORTANT] > `goldenEgressPolicy` is a **preview** feature. Enable > `--preview=GoldenEgressPolicy` on the server (or through > ate-setup's `preview` setting), and opt in on the client with `kubectl ate --preview` > or the `Ate-Preview-Allowed: true` gRPC metadata header. See [Preview gates](https://github.com/agent-substrate/substrate/blob/main/docs/dev/preview-gates.md). Set `goldenEgressPolicy` on an ActorTemplate to give its golden actor network access during initialization. For example, this template field allows HTTPS to `api.example.com` on the default port, 443: ```yaml goldenEgressPolicy: rules: - https: hostnames: [api.example.com] ``` The template controller creates the golden actor suspended, then creates its `EgressPolicy` through `CreateActorEgressPolicy` before calling `ResumeActor`. The policy uses the actor's atespace and the name `default`, and receives its own UID, version, and timestamps. The template field contains only rules. If policy creation fails, the controller retries without resuming the actor. On a retry, an existing policy must match the template's rules. A mismatch stops golden initialization and records a `GoldenEgressPolicyConflict` error in the template's status. Actor and policy creation are separate operations: callers must leave golden startup to the controller, since a direct `ResumeActor` call can start it before its policy exists. Other actors created from the template do not inherit this policy, including actors restored from its golden snapshot. Configure their network access using `CreateActorEgressPolicy` before resuming them. The controller checks the policy before each resume it attempts during golden initialization; it does not continuously reconcile the policy after the actor is running. Updating or deleting that policy does not change the immutable template. Omitting `goldenEgressPolicy` creates no golden policy. When set, it must contain at least one rule. It cannot be set when `snapshotConfig.goldenSnapshotConfig.mode` is `GOLDEN_SNAPSHOT_MODE_DISABLED`. Turning the gate off prevents creating new templates with `goldenEgressPolicy`. Existing templates retain their configuration, including when their golden is created after the gate is turned off. Existing policies and resource cleanup continue to work. ### Sandbox Right-Sizing (`resources`) Unlike a Pod, an actor is sized by its **`limits`** (CPU and Memory): the size is a property of the template, baked into snapshots, so it lives on the immutable `ActorTemplate`. Both are required. Declared limits do two things: 1. **Size the sandbox.** The limits are supplied to the sandbox over the actor RPCs (control plane → atelet → ateworker): - **gVisor (`ateworker-gvisor`):** The limits go into the container OCI spec. `limits.cpuMillicores` sets the cgroup v2 CPU quota (`cpu.max`) and the CPU count the Sentry reports (`--cpu-num-from-quota`, at least 2; see [CPU Count and Throttling](https://ate.dev/docs/reference/sandbox-classes/index.md#cpu-count-and-throttling)). `limits.memoryBytes` caps the sandbox's memory and bounds the virtual total memory it reports (so JVM/Go do not over-allocate from host RAM). When the node's kernel has cgroup memory pressure (PSI) enabled, an actor that stays over its limit is killed and reported crashed, and other actors on its worker keep running. - **microVM (`ateworker-microvm`):** `limits.cpuMillicores` sets Cloud Hypervisor `BootVcpus` / `MaxVcpus` (rounded up to whole vCPUs, minimum 1), and the host caps the VMM and `virtiofsd` at the same limit with cgroup v2 `cpu.max`. `limits.memoryBytes` sets guest RAM, all of it. The microVM worker refuses to boot an actor with less than 128 MiB. What the actor costs outside the guest comes out of the worker's capacity instead (see [Worker Capacity](#worker-capacity-spectemplateresources)). When the node's kernel has cgroup memory pressure (PSI) enabled, an actor whose VMM and virtiofsd stay over what it books is killed and reported crashed, and other actors on its worker keep running. 2. **Gate scheduling.** An actor is only placed on a worker whose remaining [capacity](#worker-capacity-spectemplateresources) is `>=` these limits, plus its runtime's overhead on that worker. There is no `requests` field. To change an actor's limits, point the suspended actor at a new template with `UpdateActor`. Its next resume restores its `VOLUMES` data and cold-boots. Container environment variables support literal `value` entries only. Values are not interpolated (`$(VAR)` references are not expanded), and Kubernetes `envFrom`/`valueFrom` sources are not supported. ### Workload Connectivity A higher-order system reaches an actor through the **Substrate Router** by setting `ate-target-actor` to `/`. This value selects the Actor; `Host` and HTTP/2 `:authority` remain application metadata. Normal HTTP requirements still apply: clients must send a valid `Host` or `:authority`, usually derived automatically from the request URL, and reverse proxies should preserve it when the application depends on the original authority. Clients that construct HTTP requests should add the routing header directly. For example, curl uses `-H`, Go uses `request.Header.Set`, and Python clients use their request `headers` mapping. WebSocket clients add the same header to the opening HTTP upgrade request. gRPC clients send it as outgoing metadata using the lowercase name `ate-target-actor`. For an HTTP `CONNECT` tunnel to a non-default Actor port, put the routing header on the outer `CONNECT` request and keep the target port in its authority. With curl and the projected volumes from the router's [common setups](https://github.com/agent-substrate/substrate/blob/main/cmd/atenet/internal/router/README.md#common-setups), use `--proxy-header` instead of `-H`: ```bash curl --proxytunnel \ --proxy https://atenet-router.ate-system.svc:8444 \ --proxy-cacert /run/router-client/router-ca.pem \ --proxy-cert /run/router-client/credential-bundle.pem \ --proxy-key /run/router-client/credential-bundle.pem \ --proxy-header "ate-target-actor: my-atespace/my-actor" \ http://actor-upstream:9090/ ``` From outside the cluster, `kubectl ate proxy actor` opens this mTLS `CONNECT` tunnel automatically: ```bash kubectl ate proxy actor my-actor 9090:9090 -a my-atespace curl http://localhost:9090/ ``` Browser navigation cannot add custom request headers or present the router's client certificate. For local development and demos (such as the [Jupyter demo](https://ate.dev/docs/tutorials/jupyter/index.md)), `kubectl ate proxy actor` serves a local port that forwards browser traffic and WebSockets through `atenet-router`. In production, browser-based and other fixed clients connect through a user-controlled reverse proxy or policy enforcement point that authenticates to `atenet-router` and sets `ate-target-actor` from trusted configuration or authenticated request context rather than forwarding values supplied by an untrusted caller. ### SystemInfo Volumes To deliver identity information, including credentials, to a running actor, you can use a SystemInfo volume. Define it in `volumes`, and mount it into each container that needs it. Available information sources: #### actorMetadata The actorMetadata data source projects the actor's identity fields to files, one per item, analogous to the [Kubernetes downwardAPI volume](https://kubernetes.io/docs/concepts/storage/volumes/#downwardapi). Each item selects a `field` — `ACTOR_METADATA_FIELD_NAME` (unique within an atespace), `ACTOR_METADATA_FIELD_ATESPACE` (together with the name, the actor's full identity), or `ACTOR_METADATA_FIELD_UID` (server-generated, distinguishes incarnations of the same name) — and the `path` the value is written to, raw with no trailing newline. `path` is a clean relative path from the root of the volume (no leading `/`, no `.` or `..` segments, at most 16 segments) and must not repeat another path projected into the same volume. ```yaml volumes: - name: system-info systemInfo: dataSources: - actorMetadata: items: - field: ACTOR_METADATA_FIELD_NAME path: actor-name - field: ACTOR_METADATA_FIELD_ATESPACE path: atespace - field: ACTOR_METADATA_FIELD_UID path: actor-uid containers: - name: main # ... volumeMounts: - name: system-info mountPath: /run/ate # the actor reads e.g. /run/ate/actor-name ``` The values are delivered as files on a read-only per-actor bind mount, not environment variables, precisely so they carry the correct values after a resume from a shared snapshot — an env var (or a file baked into the image) would be frozen at the snapshot-source actor's values, since it lives in the checkpointed process memory, and would therefore be identical for every actor restored from that snapshot. The metadata fields themselves are fixed for the actor's lifetime, so workloads may cache them; future data sources that rotate (identity tokens and certificates) must be re-read at time of use. #### trustBundle The trustBundle data source projects the union of the trust anchors of one or more named trust bundles (at most 8) to a single PEM file — inspired by the [Kubernetes clusterTrustBundle projected volume source](https://kubernetes.io/docs/concepts/storage/projected-volumes/#clustertrustbundle), but source-neutral: the name selects a bundle substrate knows how to fetch, and where it is fetched from is a deployment concern, not part of the API. Supported names are allowlisted: * `egress-mitm.ate.dev` — the egress gateway man-in-the-middle CA bundle. * `system-roots.ate.dev` — A selection of public CA root certificates provided by Substrate. In official Substrate images, this is the Mozilla Root Store as shipped on Debian (consumed via the distroless-static base image). ```yaml volumes: - name: trust systemInfo: dataSources: - trustBundle: names: - egress-mitm.ate.dev path: ca.pem containers: - name: main # ... volumeMounts: - name: trust mountPath: /run/substrate/certs # the actor reads /run/substrate/certs/ca.pem ``` atelet resolves the bundle on the node when the actor starts, reading the backing object through a cluster-wide watch (the same informer that drives live refresh) and sanitizing it the way kubelet does for projections: only `CERTIFICATE` PEM blocks are kept, deduplicated across all the named bundles, with block headers stripped and the anchors deliberately shuffled, so consumers must not depend on their order. The actor itself never talks to any bundle backend. Starting the actor fails, with an error naming the bundle, if any name is not on the allowlist, the bundle's backend is unavailable in this deployment, or the resolved bundle is missing, empty, or contains no certificates. Bundle contents are re-resolved on every Run/Restore and refreshed while the actor runs: when any backing bundle changes, atelet rewrites the projected file atomically at the same path. As with the Kubernetes clusterTrustBundle projection, the application must re-read the file to pick up a rotation; a runtime that loads trust anchors once at startup sees the change at its next start or resume. If a refresh fails because the backing object was deleted or is unusable, the file keeps its last good contents. Bundle publishers should rotate with overlap: add the new CA before minting leaves under it, and keep the old CA until its last leaf expires. ### Container Fields Each entry in `containers` describes one process to run in the actor's sandbox. | Field | Type | Description | | :--- | :--- | :--- | | `name` | `string` | **Required.** DNS-label-safe container name. | | `image` | `string` | **Required.** Container image name; must include a digest (`name@sha256:...`). | | `command` | `[]string` | Optional. Entrypoint array. If unset, the image's `ENTRYPOINT` is used. If set, it replaces **both** the image's `ENTRYPOINT` and `CMD`. | | `args` | `[]string` | Optional. Arguments to the entrypoint. If unset, the image's `CMD` is used (unless `command` is set, which discards the image's `CMD`). If set, it replaces the image's `CMD`. | | `env` | `[]EnvVar` | Optional. Literal `value` entries. | | `wakeupProbe` | `WakeupProbe` | Optional. HTTP wakeup probe — see [Container Wakeup Probe](#container-wakeup-probe-wakeupprobe). | | `volumeMounts` | `[]VolumeMount` | Optional. Mounts a `volumes` entry (e.g. `durableDir`) into this container. | | `securityContext` | `SecurityContext` | Optional. Security settings for the container process — see [Container Capabilities](#container-capabilities-securitycontextcapabilities). | | `resources` | `ResourceRequirements` | Optional. Compute limits for this container (`limits.cpuMillicores` and `limits.memoryBytes`), enforced inside the actor's sandbox. See [Per-container limits](#per-container-limits). | `command` and `args` resolve against the container image's `ENTRYPOINT`/`CMD` the same way [Kubernetes Pod `command`/`args`](https://kubernetes.io/docs/tasks/inject-data-application/define-command-argument-container/) resolve against `ENTRYPOINT`/`CMD`. If the resolved argv is empty — the image sets neither `ENTRYPOINT` nor `CMD`, and the container sets neither `command` nor `args` — `Run`/`Restore` fails. The process runs as the image's `USER`, resolved as Docker does against the image's own `/etc/passwd` and `/etc/group`: a numeric `uid:gid` as is; a bare uid with its login group, or gid 0 when it has no entry; a name must have an entry or `Run`/`Restore` fails. Group memberships become supplementary groups. Ids are at most 2147483647. No `USER` means root. There is no `runAsUser` field. The process starts in the image's `WORKDIR`, or in `/` when the image sets none. A `WORKDIR` the image lacks is created in the actor's writable layer. There is no `workingDir` field. ### Container Capabilities (`securityContext.capabilities`) Each container runs with a default set of Linux capabilities — `AUDIT_WRITE`, `KILL` and `NET_BIND_SERVICE`. `securityContext.capabilities` adjusts that set, mirroring `securityContext.capabilities` on a Kubernetes Pod container. A container running as root has that set as its effective capabilities. A container whose image `USER` is not root starts with no effective capabilities, as under Kubernetes, because ambient capabilities are not supported. Its set then only limits what a setuid-root program can gain. | Field | Type | Description | | :--- | :--- | :--- | | `securityContext.capabilities.add` | `[]string` | Optional. Capabilities to grant on top of the default set. Only the allowed capabilities listed below are accepted. | | `securityContext.capabilities.drop` | `[]string` | Optional. Capabilities to remove from the default set. `ALL` drops the whole set. | - **Naming.** Capabilities are named **without** the `CAP_` prefix, as in Kubernetes — `NET_BIND_SERVICE`, not `CAP_NET_BIND_SERVICE`. The prefixed spelling is rejected at admission rather than silently granting nothing. Unknown names are rejected too, in `drop` as well as `add`, so a typo in `drop` can't leave the capability granted. - **Order.** `drop` is applied first, then `add`. A capability named in both is therefore **granted**. - **Exact sets.** Because `drop: ["ALL"]` clears the default set, combining it with `add` expresses an exact capability set rather than a relative one: ```yaml securityContext: capabilities: drop: ["ALL"] add: ["NET_BIND_SERVICE"] ``` - **Allowed capabilities.** `add` accepts only `AUDIT_WRITE`, `CHOWN`, `DAC_OVERRIDE`, `FOWNER`, `FSETID`, `KILL`, `NET_BIND_SERVICE`, `SETFCAP`, `SETGID`, `SETPCAP`, `SETUID` and `SYS_CHROOT`. Any other name, including `ALL`, `DAC_READ_SEARCH`, `MKNOD`, `NET_RAW`, `SYS_ADMIN` and `SYS_PTRACE`, is rejected at admission. These capabilities act on the microVM guest kernel and its devices, not just on the container. `drop` accepts any capability name. - **Ambient capabilities are not supported** ([gvisor#3166](https://github.com/google/gvisor/issues/3166)). - **Low ports.** A container may bind a port below 1024 without `NET_BIND_SERVICE` on both runtimes: gVisor's network stack has no privileged-port limit, and the microVM guest sets `net.ipv4.ip_unprivileged_port_start=0` to match. - **`ping` without `NET_RAW`.** `NET_RAW` can't be added. Both runtimes let any user open ICMP echo sockets, which is what `ping` uses: gVisor's network stack allows them, and the microVM guest sets `net.ipv4.ping_group_range=0 2147483647` to match. - **Setuid binaries work.** Neither runtime sets `no_new_privileges`, as for a Kubernetes container with `allowPrivilegeEscalation: true`; gVisor runs with `--allow-suid`. - **Image files owned by other users.** Image files keep the owner their image's layers record, as under containerd. A container running as its image's `USER` can use what the image gave that user, for example after `RUN chown -R app:app /app` and `USER app`. A container running as root, because its image sets no `USER` or `USER root`, has no `DAC_OVERRIDE` by default. On a file or directory another user owns, it gets only what the permission bits grant to others. Add `DAC_OVERRIDE` if it reads or writes such files. The sandbox — gVisor or microVM — remains the isolation boundary; capabilities constrain the workload *inside* it. ### Per-container limits A container may cap its own CPU and memory so it cannot starve or kill its siblings in the same actor (on a template whose `workerSelector` targets a microVM pool): ```yaml containers: - name: trainer resources: limits: {memoryBytes: 1572864000} # 1500Mi - name: sidecar resources: limits: {memoryBytes: 268435456, cpuMillicores: 200} # 256Mi, 0.2 CPU ``` A container that exceeds its memory limit is OOM-killed on its own; the actor's other containers are unaffected. A `cpuMillicores` limit below 10 is raised to 10, because the kernel rejects a CFS quota under 1ms. Per-container limits are microVM only today. gVisor applies cgroup limits at the sandbox level: one sentry backs every container in the actor, so a per-container cgroup is created and then stays empty ([google/gvisor#190](https://github.com/google/gvisor/issues/190)). On a gVisor worker the per-container limits are accepted but not enforced; only the sandbox-level size applies. These limits subdivide the sandbox that [`resources`](#sandbox-right-sizing-resources) already sized; a container that declares none is bounded by the guest as a whole, not by a copy of the actor's total. A microVM guest's memory is `resources.limits.memoryBytes`, and its vCPU count is `resources.limits.cpuMillicores` rounded up to whole cores. A container limit above either ceiling can never bind, so the actor fails to start with an error naming both the limit and the ceiling. Each limit is validated on its own at apply, but the sum across the actor's containers is only checked when the actor first runs, against the real guest size. A template whose limits do not fit is accepted by the API server and fails on its first actor. ### Container Wakeup Probe (`wakeupProbe`) Each entry in `containers` may declare an optional **HTTP wakeup probe** so `ResumeActor` only returns once the workload is actually serving traffic. Unlike a Kubernetes `readinessProbe`, which reruns for the life of the pod to gate traffic to it, this is a one-shot gate on each wakeup: `ResumeActor` blocks until the endpoint returns 200, and the probe does not run again while the actor stays `RUNNING`. | Field | Type | Description | | :--- | :--- | :--- | | `wakeupProbe.httpGet.path` | `string` | Optional. URL path to GET. Defaults to `/`. Must begin with `/` and contain only RFC 3986 path characters (no query string `?` or fragment `#`). | | `wakeupProbe.httpGet.port` | `int32` | **Required.** TCP port on the container to probe (`1..65535`). | | `wakeupProbe.timeoutSeconds` | `int32` | Optional. How long to keep polling before the wakeup fails (`1..3600`). Defaults to `30`. | How it behaves: - **When it runs.** On every `ResumeActor` that actually wakes the actor. A `ResumeActor` on an actor that is already `RUNNING` is a no-op and does not probe. - **Block-until-ready semantics.** `ResumeActor` returns successfully only after every container with a `wakeupProbe` has returned HTTP 200. If any container has not done so within `timeoutSeconds` (30s by default), `ResumeActor` fails and the actor moves to `ACTOR_STATE_CRASHED`, as it does for any other error while waking. A crashed actor cannot be resumed directly: `ResumeActor` rejects it with `FAILED_PRECONDITION`. Call `RevertActor` first, which returns it to `ACTOR_STATE_SUSPENDED` with its last snapshot, then resume it again. - **Aggressive polling.** The poll loop is tuned for single-millisecond detection latency: a keep-alive HTTP client with a 1ms interval and 250ms per-request timeout. While the workload is still booting, kernel `RST`s return in microseconds, so the loop spends almost no time blocked; once the listener is up, the next attempt completes on veth-local latency. - **Egress while waking.** The actor's egress is active before its containers start, so a workload can download what it needs to become ready (a model, a Git repository) before it answers the probe. A microVM restored from a memory snapshot is already past its boot, and gets egress just before the probe starts. Its EgressPolicy applies as it does once the actor is `RUNNING`; inbound traffic still waits for the probe. A wake that fails or is cut short after egress is active can be retried from the same snapshot, so outbound requests made while waking can be sent more than once. - **Golden snapshot warm-up shortcut.** When **every** container in a template declares `wakeupProbe`, the actor template controller skips its default ~20s "give the workload time to settle" delay before taking the golden snapshot — `ResumeActor` already blocked until the workload reported 200, so the workload is known to be initialized. Templates that omit `wakeupProbe` on any container keep the 20s warm-up as a safety net. - **Snapshot/restore interaction.** The TCP listener is part of the checkpointed RAM, so on resume `wakeupProbe` typically returns 200 on the first attempt, with no observable latency penalty. If `wakeupProbe` is omitted from a container, `ResumeActor` returns as soon as the sandbox has started that container, which can be before the workload is listening. ### Example A protojson-shaped `ateapi.ActorTemplate`, created through the ate API with `kubectl ate create actor-template -f secret-agent.yaml` (the `ate-demo` atespace must exist): ```yaml metadata: atespace: ate-demo name: secret-agent containers: - name: agent image: gcr.io/my-project/my-agent@sha256:7f28ab0e... # Optional: gate ResumeActor on the agent's HTTP wakeup probe endpoint. # See "Container Wakeup Probe (wakeupProbe)" above. wakeupProbe: httpGet: path: /readyz port: 80 workerSelector: matchLabels: workload: secret-agent # The actor's size, booked against worker capacity. resources: limits: cpuMillicores: 1000 memoryBytes: 1073741824 # 1Gi ``` ### Snapshot Storage Layout `snapshotConfig.storageLocation` is a **base prefix**, not the address of any one snapshot. Every external snapshot has exactly one owner: the actor that took it, or the tag that copied it. And the owner is part of the path, so an object's name says who it belongs to: ``` /atespaces//actors//snapshots/ /atespaces//tags/ ``` > [!IMPORTANT] > Snapshots are stored under ate-api-server's cluster-wide `--snapshot-location`. `snapshotConfig.storageLocation` is an optional preview field: with the `PerTemplateSnapshotLocation` preview gate (`--preview=PerTemplateSnapshotLocation`), a template that sets it stores its snapshots there instead. `` above is whichever of the two applies. > > Only a client that opts in to preview features (`kubectl ate --preview`, or the `Ate-Preview-Allowed: true` header) may set `storageLocation`, and only with the gate enabled. Otherwise the request fails with `InvalidArgument`. A template stored before the API enforced this keeps the field, and ate-api-server ignores it while the gate is off. `CreateActorTemplate` fails with `FailedPrecondition` when the template leaves it empty and ate-api-server has no `--snapshot-location`. > > Changing the flag or toggling the gate moves `` for every template it affects. Snapshots already taken stay where they are and can still be resumed from. An actor's next suspend writes under the new location and releases its previous snapshot from the old one. Until then, the actor cannot change its `actor_template` (see `UpdateActor` below), and deleting it collects its prefix under the old location. If that release fails, the previous snapshot's objects stay under the old location, and deleting the actor later does not collect them. The objects of a snapshot (its manifest, memory image, durable-data tar) are named below it. So with `--snapshot-location=gs://my-bucket/`, a snapshot of an actor in atespace `team-a` is stored at `gs://my-bucket/atespaces/team-a/actors/3f8b…/snapshots/f47ac10b-…`, and a golden snapshot under `gs://my-bucket/atespaces/team-a/tags/`. An actor takes a series of snapshots over its life, so it gets a prefix of its own and each snapshot sits below it. A tag holds exactly one, so the tag's prefix *is* its snapshot's. Both owners are keyed on their UID, so recreating an actor or tag under the same name never inherits its predecessor's objects. A tag records its snapshot's URI in `status.snapshot` right after it is reserved, before anything is copied, so a tag whose creation failed still names any partial copy to collect. Every owner's prefix starts with `/atespaces/`. So where a location is set, through `--snapshot-location` or a template's `storageLocation`, each path segment may contain only ASCII letters, digits, `_` and `-`, and no segment may be `atespaces` in any letter case. The bucket is not restricted. A location such as `gs://my-bucket/atespaces/team-a/actors/` would otherwise put one owner's objects under another owner's or atespace's prefix, inside its grants and its cleanup. ate-api-server refuses to start with a `--snapshot-location` that breaks these rules, `ate-setup deploy` rejects it, and creating a template rejects such a `storageLocation`. Creating a tag checks the location the tag is stored under and fails with `FAILED_PRECONDITION` on one that breaks them. An owner is collected by deleting everything under its prefix, and it can delete nothing else. That is what makes a borrowed snapshot safe: an actor created from a tag points at a URI under `tags/`, which its own prefix does not cover. See [Snapshot lifetime](#snapshot-lifetime). An `Actor` reports its snapshots in the server-managed `status.snapshots` (and `status.lastAssignedGeneration`) and a `Tag` in `status.snapshot`. Each entry is a `Snapshot` with the following fields: - `uuid`: The snapshot's server-generated name. It names the node-local checkpoint and forms the last segment of the snapshot's URI in object storage (so a tag's snapshot is named by the tag's UID). - `generation`: The monotonically increasing snapshot generation on the owner. - `owner`: The resource kind that owns the snapshot (`SNAPSHOT_OWNER_ACTOR` or `SNAPSHOT_OWNER_TAG`). - `fidelity`: The state layers the snapshot holds (`SNAPSHOT_FIDELITY_VOLUMES`, `SNAPSHOT_FIDELITY_ROOTFS`, or `SNAPSHOT_FIDELITY_MEMORY`). - `actorTemplateUid`: The UID of the `ActorTemplate` used to capture this snapshot. An actor can change templates while `SUSPENDED`. A resume checks the snapshot it restores. A snapshot captured under a replaced template restores volume data and boots fresh. - `locality`: The node holding the snapshot's local files. For attachment rules, see `ActorStatus.assigned_node` in the [API schema](https://github.com/agent-substrate/substrate/blob/main/api/ateapi/v1/ateapi.proto). - `durableSnapshot`: The copy in object storage, with its URI in `object.snapshotUri` and a `state` that is `PENDING` while the copy is being written and `COMPLETED` once it can be restored from. A synchronous suspend uploads the copy before returning (`COMPLETED`); an asynchronous suspend records the copy as `PENDING`, and atelet uploads it in the background. A tag's snapshot always has one, and the tag can seed actors only once it is `COMPLETED`. - `sandboxRuntime`: The sandbox runtime and hardware compatibility attributes the guest state was captured under, copied from `status.workerAssignment.sandboxRuntime` when the checkpoint is taken. A resume that restores memory state (`SNAPSHOT_FIDELITY_MEMORY`) first schedules onto an eligible worker whose `status.sandboxClasses` entry for the snapshot's `sandboxClass` has a `defaultRuntime` or `restorableRuntimes` entry whose `version` is compatible with `sandboxRuntime.version` (same schema version, and every attribute recorded on the snapshot present on the worker with the same value, except `cpu_features`, whose hexadecimal bitmask on the snapshot must be a bitwise subset of the worker's; attributes only the worker reports are ignored). When no compatible worker has capacity: - A durable snapshot — the actor's own, a `sourceTag` clone, or the template's golden snapshot a new actor starts from — falls back to any eligible worker and downgrades the restore to `SNAPSHOT_FIDELITY_ROOTFS`, preserving the snapshot's rootfs and volume data while booting the guest fresh under the `defaultRuntime` of that class on the new worker. - A local checkpoint first tries compatible workers on its attached node. If that fails, `ResumeActor` checks capacity without the node and runtime restrictions. When that capacity exists, it detaches the checkpoint and retries placement. Detach uploads it if needed, deletes the local files, and releases the binding. The retry prefers compatible workers before using the durable fallback above. A detach error leaves the actor attached, unless its node was deleted. See [`ResumeActor`](#resumeactor) for that case. A snapshot without `sandboxRuntime` or with `SNAPSHOT_FIDELITY_ROOTFS` or `SNAPSHOT_FIDELITY_VOLUMES` imposes no runtime-version placement constraint. On gVisor, such a restore runs under the worker's default `runsc`, which may not be the version that wrote its files (see the [Version Skew Policy](https://ate.dev/docs/reference/version-skew-policy/index.md#sandboxconfig)). A snapshot that records a sandbox class always restores on a worker of that class. Because a `ROOTFS` restore reuses the container's rootfs upper layer over the template's image, pools selected by an `ActorTemplate` must share a CPU architecture. An `ActorTemplate` references its golden tag with the `ObjectRef` in `status.goldenSnapshotStatus.goldenTag`. These status fields are server-owned and ignored on input. Parse a URI only against the scheme above. An `ActorTemplate` belongs to one atespace, but one `storageLocation` still holds snapshots for many atespaces: the golden actor and tag live in the template's atespace, and a `PUBLISHED` snapshot may be cloned from other atespaces. The `` level exists so that access can be granted per tenant: an object-storage policy can only condition on an **object-name prefix**, and cannot read the identity recorded inside a snapshot's manifest. Binding a per-atespace grant on GCS looks like: ```yaml # Read-only on team-a's snapshots for this template, and nothing else. - members: ["serviceAccount:node-runtime@my-project.iam.gserviceaccount.com"] role: roles/storage.objectViewer condition: title: team-a-snapshots expression: > resource.name.startsWith( "projects/_/buckets/my-bucket/objects/secret-agent/atespaces/team-a/") ``` One consequence worth planning for: **a published snapshot is read from the atespace that took it.** Cloning across atespaces via a `PUBLISHED` tag reads the source atespace's prefix, so the reader needs a grant covering it — the target atespace's grant is not enough. --- ## 3. SandboxConfig: The Sandbox Itself `SandboxConfig` is a **cluster-scoped** resource that decouples the sandbox binaries (the gVisor `runsc` binary, or a microVM kernel/firmware/config) from the workload definition in the `ActorTemplate`. A `WorkerPool`'s `sandboxClasses` entry resolves its `SandboxConfig` (either by explicit `configRef` or via the class default), and `atelet` provisions each worker pod with the enabled versions of that config, using `defaultVersion` for cold boots and keeping other enabled versions in `restorableRuntimes` so existing snapshots can still restore across upgrades. ### Specification (`SandboxConfigSpec`) | Field | Type | Description | | :--- | :--- | :--- | | `sandboxClass` | `string` | Optional. Runtime family this config applies to: `gvisor` (default) or `microvm`. A `WorkerPool` only uses `SandboxConfig`s whose `sandboxClass` matches its entry's `name`. | | `defaultVersion` | `string` | **Required.** Name of the entry in `versions` that actors boot with. It must name an existing version that is not `Disabled`. | | `versions` | `[]SandboxVersionConfig` | **Required.** 1–16 sandbox versions, unique by `name`. | Each entry of `versions` (`SandboxVersionConfig`): | Field | Type | Description | | :--- | :--- | :--- | | `name` | `string` | **Required.** Identifies the version within the config. A DNS label (lower-case alphanumerics and `-`, at most 63 characters). | | `state` | `string` | Optional. `Enabled` (default) or `Disabled`. The version named by `defaultVersion` cannot be `Disabled`. | | `assets` | `map[arch]map[name]AssetFile` | Content-addressed files atelet fetches, keyed by architecture (`amd64`, `arm64`) then asset name. gVisor expects a `gvisor` asset (the release's `gvisor.tar.zstd`), which atelet auto-extracts. A microVM backend expects several. Each `AssetFile` is a `{ url, sha256 }` pair; `url` is at most 2048 bytes. A `ValidatingAdmissionPolicy` enforces each class's required assets on every version. atelet holds no storage credentials: it fetches every URL through the node's object-store plugin (`AssetProvider`), with the plugin's credentials. Which schemes work depends on the plugin; the in-tree one serves only its storage backend's scheme, `gs://` on GCS and `s3://` on S3, as `/`, and rejects other schemes without reading them. A `gs://` URL the plugin cannot serve (it has no such object, rejects the URL, does not implement `AssetProvider`, is unreachable, or fails with an error it does not recognize, such as GCS refusing its credentials for a public bucket) is then read anonymously, which serves public buckets such as `gs://gvisor`; any other refusal by the plugin is final. The plugin returns an object only if its content matches `sha256`, and refuses any URL under a snapshot location (a snapshot location followed by `/atespaces/`), so assets must not be stored there. On GKE, substrate-gke's setup-gcp lets the plugin read only the asset bucket, `-assets`, so keep assets there. atelet verifies the sha256 again before using an asset. Its logs and errors show an asset URL as `scheme://host/path`. | A cluster-wide gVisor `SandboxConfig` (`gvisor-default`, annotated as the class default) is installed with the platform, so gVisor `WorkerPool`s use it automatically without any extra setup. ### Class default A `SandboxConfig` annotated `sandboxconfig.ate.dev/is-class-default: "true"` is a default for its `sandboxClass`. A `WorkerPool` entry without `configRef` uses the newest such config of its class that is not being deleted. Among configs created in the same second, the name that sorts first wins. More than one config of a class may carry the annotation, so a new default can be rolled out by creating it, and pools move to it without restarting their workers. While no default is available, the class reports `Resolved=False` with reason `DefaultConfigNotFound`, or `SandboxConfigDeleting` if the only defaults are being deleted. A pool that already uses a config keeps it meanwhile, even if that config is being deleted or has lost the annotation. ### Deletion protection atecontroller keeps a `SandboxConfig` from being deleted while a `WorkerPool` uses it. - When a `WorkerPool` resolves a `SandboxConfig`, atecontroller puts the finalizer `sandboxconfig.ate.dev/workerpool-protection` on the config before it records the config in the pool's `status.sandboxClasses`. - A pool that switches to another config first records the new config in `pendingConfigRef` and keeps its current config in `configRef`. It then reads the new config from the API server, and moves `configRef` only if the config is not being deleted. Otherwise it keeps its current config. - The finalizer stays until the config is deleted. Deleting the config marks it for deletion. atecontroller removes the finalizer once no `WorkerPool`'s `status.sandboxClasses` names the config. It confirms that against the API server, not only its cache. - A config that pools stopped using is gone within a reconcile. A config that pools still use stays `Terminating` until they move or are deleted. A config no pool has ever used carries no finalizer and is deleted at once. - A config that is being deleted takes no new pools. A `configRef` to it reports `SandboxConfigDeleting`, and it stops being a class default. A pool already on it keeps it and reports `Resolved=False`. A pinned pool does so until its `configRef` changes, an unpinned pool until another default exists. So a `Terminating` config only loses users. - `WorkerPool`s carry no finalizer and delete without waiting. A delete cannot be undone, and an object that is being deleted cannot be recreated under the same name. Applying its manifest again only updates the terminating object. To recover from deleting a class default that pools still use: 1. Create a replacement under a different name with the `is-class-default` annotation. Pools move to it on their own, the finalizer is released, and the old config disappears. 2. Once the old config is gone, the next `ate-setup` upgrade recreates the shipped config under its original name. Pools move back to it, because it is then the newest default. atecontroller alone adds and releases the finalizer. While it is down, `kubectl delete sandboxconfig` of a config that carries the finalizer hangs until atecontroller is back. `ate-setup` strips the finalizer from every config when it deletes the control plane. ### Example ```yaml apiVersion: ate.dev/v1 kind: SandboxConfig metadata: name: gvisor-default annotations: sandboxconfig.ate.dev/is-class-default: "true" spec: sandboxClass: gvisor defaultVersion: v7 versions: - name: v6 state: Enabled assets: amd64: gvisor: url: "gs://gvisor/releases/release/20261026.0/x86_64/gvisor.tar.zstd" sha256: "708afc62f2ee82d19930c703e85f9dab4f27a78c0aec40c0f55eb1040107f1a7" arm64: gvisor: url: "gs://gvisor/releases/release/20261026.0/aarch64/gvisor.tar.zstd" sha256: "6257e49955f13e6caceb2c4181c5dc5406c460e805940c944e67ad65553f5b02" - name: v7 state: Enabled assets: amd64: gvisor: url: "gs://gvisor/releases/release/20261102.0/x86_64/gvisor.tar.zstd" sha256: "a1439ffbba7d212e0fde32ea79fa9bb51cbc4c5f6c16af1f047260290b861d42" arm64: gvisor: url: "gs://gvisor/releases/release/20261102.0/aarch64/gvisor.tar.zstd" sha256: "f702ae92ad7a0c4134be34ca23f52966dbb696db0c0f5b8ec439210ab2cd7c6a" ``` ### MicroVM SandboxConfig A `microvm` `SandboxConfig` supplies the [Kata Containers](https://katacontainers.io/) + [Cloud Hypervisor](https://www.cloudhypervisor.org/) toolchain instead of `runsc`. Each architecture of each version must define the full asset set — `cloud-hypervisor`, `virtiofsd`, `kata-kernel`, and `kata-image` — which a `ValidatingAdmissionPolicy` enforces at apply time. Worker pods for a microVM pool require `/dev/kvm` and nested-virtualization-capable nodes. The controller requests those devices on the pod automatically, and atelet advertises them only where they exist, so placement follows the hardware rather than a node label. Clusters that reserve nested-virt nodes with an `ate.dev/sandboxClass=microvm` taint are still tolerated: advertising a device attracts these pods to capable nodes but repels nothing else from them. The same convention applies to every class: worker pods of a pool tolerate `ate.dev/sandboxClass=:NoSchedule`, so a cluster can reserve a node pool per sandbox class with that taint, and the atelet DaemonSet tolerates the key for any value. On GKE, a pool can also select a ComputeClass with `template.nodeSelector` (`cloud.google.com/compute-class: `): GKE taints the nodes it creates for the class `cloud.google.com/compute-class=:NoSchedule` and adds the matching toleration to the worker pods, and the atelet DaemonSet tolerates that key as well. `go run ./cmd/ate-setup deploy ate-system` (and `deploy sandboxconfig`) assembles and stages these assets automatically; see [`docs/dev/microvm-local.md`](https://github.com/agent-substrate/substrate/blob/main/docs/dev/microvm-local.md) for local bring-up and a worked counter demo (`demos/counter/counter-microvm.yaml.tmpl`) that suspends and resumes an in-RAM counter across worker pods. --- ## 4. Operational Workflow ### The Golden Snapshot Golden snapshot creation is enabled by default. To disable it, set `snapshotConfig.goldenSnapshotConfig.mode: GOLDEN_SNAPSHOT_MODE_DISABLED` on the template. Omitting the config or leaving `mode` unspecified defaults to `GOLDEN_SNAPSHOT_MODE_ENABLED`. A disabled template creates no golden actor or tag. Actors created without a `sourceTag` cold-boot and can still suspend and resume using their own snapshots. `kubectl ate get actor-templates` shows `DISABLED` in the `GOLDEN TAG` column for these templates. When an `ActorTemplate` is created with golden snapshots enabled: 1. Substrate creates and resumes a temporary golden actor in the template's atespace, using the same atespace-scoped policies and storage grants as ordinary actors. 2. It waits for readiness plus a 20-second warmup, then suspends the actor at the template's `preferredFidelity`. The extra interval is zero when every container has a wakeup probe, since Resume already waits for readiness. 3. It creates a published tag named after the template UID, copying the snapshot into tag-owned storage in the same atespace. The golden actor uses that name too. 4. It deletes the golden actor and records the tag reference in the template status. Golden actors follow ordinary lifecycle and ingress policies and appear in Actor lists. Leave these resources to the controller: modifying or deleting them can interrupt Golden creation, and deleting a published Golden tag invalidates the template's recorded reference. `DeleteActorTemplate` collects its Golden tag and any unfinished Golden actor. Golden capture follows `preferredFidelity`. MEMORY preserves processes and memory; VOLUMES preserves startup data in mounted volumes and starts new processes on restore. ROOTFS preserves the sandbox's root filesystem writes and also starts new processes. `CreateActor` uses an explicit `sourceTag` when supplied; otherwise it resolves the template's golden tag and records that snapshot on the new actor. If the golden tag is not ready yet, the actor starts without a snapshot and cold-boots even if the tag becomes ready before its first resume. The default does not populate the caller-owned `sourceTag` field. ### Resumption Lifecycle CreateActor fixes the Actor's initial snapshot source at creation. Resume places it on an eligible worker. A MEMORY snapshot restores saved processes; VOLUMES and ROOTFS snapshots restore their persisted data and start new processes. An Actor created without a snapshot boots from its template's images. ### Eviction A worker pod can go away while it hosts running actors: when its `WorkerPool` rolls to a new worker image or pod template, as in an [upgrade](https://ate.dev/docs/guides/upgrade/index.md), or scales down, when its node is drained, or when its Spot VM is reclaimed. When the pod gets `SIGTERM`, its worker stops accepting actors and sends `SIGTERM` to the main process of each container of every actor it hosts. Each actor then has 30 minutes, counted from the pod's `SIGTERM`, to be suspended. The 30 minutes is the worker's shutdown budget, fixed in ateworker and kept shorter than the pod's one-hour termination grace period so that the worker, not the kubelet, kills what is left. An actor suspended in that window keeps its state and resumes on another worker like any suspended actor. Substrate does not suspend the actor for you. Whatever drives the actor, usually its harness, has to call `SuspendActor` with `DURABILITY_MODE_SYNCHRONOUS` (`kubectl ate suspend actor --mode=sync`) before those 30 minutes are up. An asynchronous suspend can lose the snapshot if the node goes away: - It returns once the snapshot is on the node. - atelet schedules the upload to object storage about `--snapshot-upload-delay` (5 minutes by default) after the snapshot is taken. Neither when the upload starts nor when it finishes is promised. - Until atelet detaches the actor, `--snapshot-detach-delay` (30 minutes by default, unrelated to the worker's 30 minutes), the actor stays pinned to the node and resumes there. - If the node goes away before the upload completes, the snapshot goes with it. Only the actor gets the `SIGTERM`, so it has to pass that on, for example on an endpoint the harness polls. Every actor that must keep its state through an eviction needs a `SIGTERM` handler that: - **does not exit.** A process that has exited cannot be suspended. - **runs in the container's main process.** Substrate signals only to the main process. - **gets the actor suspended within the 30 minutes.** An actor still running when the 30 minutes are up is killed. Once the pod is gone, the actor moves to `ACTOR_STATE_CRASHED`, and everything since its last snapshot is lost. `RevertActor` (`kubectl ate revert`) returns a crashed actor to `ACTOR_STATE_SUSPENDED` at its last external snapshot. The node can cut the 30 minutes short. GKE node upgrades and cluster autoscaler scale-downs wait up to an hour for a pod, so they leave the full 30 minutes. A reclaimed GKE [Spot VM](https://docs.cloud.google.com/kubernetes-engine/docs/concepts/spot-vms) gives pods 15 seconds by default and at most 2 minutes, so an actor there has to be suspended in that time or it crashes. --- ## 5. Best Practices * **Startup Logic:** Place expensive initialization (loading large models, establishing baseline connections) in your application's entry point. When golden snapshots are enabled, this initialization can be captured in the Golden Snapshot. Actors that cold-boot run initialization again. * **Placement:** The `WorkerPool` sets the sandbox class (`sandboxClasses[].name`). Use the template's `workerSelector` to target the pools whose class your workload needs (see [Sandbox Classes](https://ate.dev/docs/reference/sandbox-classes/index.md)). Pool selection matches pool labels only. A pool's namespace and RBAC do not limit which templates can select it. Without a selector an actor may run on any pool. * **Version Management:** When updating code, create a new `ActorTemplate` (e.g. `v2`). Substrate treats each template as an immutable state root. Suspended actors can be repointed to the new template via `UpdateActor` (`kubectl ate update actor --template`). * **Eviction:** Give every actor a `SIGTERM` handler that does not exit, and have the actor suspended within 30 minutes of it. See [Eviction](#eviction). --- ## 6. Control Plane gRPC API The Substrate Control Plane (`ate-api-server`) exposes a gRPC interface for managing actors and workers. This is the primary API used by the `kubectl-ate` CLI and higher-level frameworks. ### Service: `substrate.ateapi.v1.Control` #### `CreateActor` Registers a new logical actor in the system. * **Request:** `CreateActorRequest` * `actor`: `Actor` — the actor to create. Its `metadata` carries the atespace and name (name must be a DNS-1123 label); the `actor_template` ref (atespace + name) selects the `ActorTemplate`. * `actor.source_tag`: (Optional) `ObjectRef` of a `Tag` to seed the actor from. The tag must be taken under the same `ActorTemplate`, must not be marked for deletion, and must be either in the actor's own atespace or `PUBLISHED`. Nothing is copied: the new actor's `status.snapshots` points at the tag's snapshot, under the tag's prefix, until its own first suspend. * `actor.worker_selector`: (Optional) Per-actor `WorkerSelector` merged with the template's `worker_selector` at resume time (actor keys override template keys). * **Response:** the initialized `Actor`. #### `UpdateActor` Replaces the mutable fields of an existing actor with the ones in the request (`kubectl ate update actor`). * **Request:** `UpdateActorRequest` * `actor`: `Actor` — the complete replacement actor. `metadata.atespace` and `metadata.name` identify the resource; `metadata.uid` and `metadata.version` are **required** preconditions. `metadata` and `status` are server-owned and whatever the request carries in them is ignored. `source_tag` is immutable. * `actor_template` may change only while the actor is `SUSPENDED`. The new template must declare the same volumes and volume mounts. Once the actor owns an external snapshot, the new template must also store snapshots under the same location: its `snapshotConfig.storageLocation` with the `PerTemplateSnapshotLocation` preview gate, otherwise ate-api-server's `--snapshot-location` (see [Snapshot Storage Layout](#snapshot-storage-layout)). * `worker_selector` may be updated or cleared (`nil`) in any state, but not while another operation is changing the actor. The updated selector takes effect on the next `ResumeActor`. * A suspended actor with a local snapshot is initially pinned to its node (see `locality` in [Snapshot Storage Layout](#snapshot-storage-layout)). `ResumeActor` first looks for a worker there matching the template and actor selectors. If none is available but a worker satisfies the remaining constraints without node or runtime restrictions, resume detaches the checkpoint and retries placement. If no eligible worker has room, it returns `RESOURCE_EXHAUSTED`. * **Response:** the updated `Actor`. * **Errors:** `INVALID_ARGUMENT` if `uid` or `version` is unset, or if the request changes an immutable field (including by leaving one unset). `FAILED_PRECONDITION` if the actor is marked for deletion or if a change to `actor_template` breaks one of the rules above. `ABORTED` if either guard no longer matches the stored resource, or while another operation is changing the actor (retry the same request). Because the guards are required and only a read supplies them, an update is always a read-modify-write. To Update an `Actor`, you must first `GetActor`/`CreateActor`, instead of building a new one — see [§7.2 of the API style guide](https://github.com/agent-substrate/substrate/blob/main/docs/api-style-guide.md#72-using-version-and-uid-to-guard-writes) for why reconstructing the message can silently drop data. #### `ResumeActor` Activates a suspended actor by restoring it onto a physical worker. * **Request:** `ResumeActorRequest` * `actor`: `ObjectRef` of the actor to resume. * **Response:** `ResumeActorResponse` containing the updated `Actor` object (including the physical worker placement in `status.worker_assignment`). * **Errors:** `RESOURCE_EXHAUSTED` if no eligible worker has room. `FAILED_PRECONDITION` if the actor is `ACTOR_STATE_CRASHED`. When a suspended actor's node is deleted, resume restores its local snapshot on another node from the snapshot's upload, even if atelet never reported the upload complete. Resume does this once another node has room. If atelet finds no upload, resume moves the actor to `ACTOR_STATE_CRASHED` and returns `FAILED_PRECONDITION`. `RevertActor` then returns it to its last completed external snapshot. Revert deletes an upload that was never reported complete, so a snapshot is lost if its restore fails for another reason, or if the actor is reverted before it resumes. #### `SuspendActor` Hibernate a running actor, capturing its current RAM and disk state into a snapshot. * **Request:** `SuspendActorRequest` * `actor`: `ObjectRef` of the actor to suspend. * `options.mode`: (Optional) `DURABILITY_MODE_SYNCHRONOUS` returns once the snapshot is `COMPLETED` in object storage. `DURABILITY_MODE_ASYNCHRONOUS` returns once the snapshot is captured on the worker's node, and atelet uploads it in the background. Unset runs an asynchronous suspend. * **Response:** `SuspendActorResponse` containing the `Actor` object in `ACTOR_STATE_SUSPENDED`, with its snapshot in `status.snapshots`. * Called on an actor that is already `ACTOR_STATE_SUSPENDED`, it returns the actor as it is, in either mode. After an asynchronous suspend, atelet uploads the snapshot in the background, so `status.snapshots` may still show a `PENDING` `durableSnapshot`. * A snapshot generation conflict returns `ABORTED` without crashing the actor. The actor can remain `ACTOR_STATE_SUSPENDING`, with its worker assignment and previous completed snapshot kept for recovery. Read the actor's latest state before retrying; repeating a stale request cannot resolve a persistent disagreement with atelet. * A successful suspend releases the actor's previous external snapshot: an actor keeps one, and only tags outlive it. To keep the snapshot a suspend just wrote, tag it with `CreateTag` while the actor is still suspended. After an asynchronous suspend, `CreateTag` first has atelet upload the snapshot if it has not already, so the call can take as long as that upload. If the upload fails, so does `CreateTag`, and the actor is left as it was. #### Snapshot lifetime Every external snapshot has exactly one owner, and the control plane deletes it when that owner lets go: | Owner | Released when | | :--- | :--- | | The actor that took it (`status.snapshots`) | The actor's next successful suspend replaces it, or the actor is deleted. | | The tag that copied it (`status.snapshot`) | The tag is deleted. | An actor created from a tag borrows the tag's copy instead of taking one of its own. The borrowed URI sits under the tag's prefix, which the actor's own prefix does not cover, so neither suspending nor deleting the actor can reach it; its first own suspend writes a snapshot under the actor's prefix, and it owns its snapshots from then on. Deletion always runs before the database reference is dropped, and a failure fails the whole RPC. Clients are expected to retry: destinations are deterministic and every phase tolerates a partly-completed predecessor, so a retry resumes rather than duplicating work. The cost of that ordering is that a crash between the two can leave an external snapshot no row names; the reverse order would instead lose the handle needed to ever delete it. > **Do not delete a tag while actors created from it exist.** A clone borrows the tag's snapshot rather than copying it, and only stops borrowing at its own first suspend (its `status.snapshots` entry still names the tag's prefix while it is). Deleting the tag leaves such a clone unable to resume. This is not prevented today. #### `RevertActor` Discards an actor's live or crashed execution and transitions it to `ACTOR_STATE_SUSPENDED` at its last completed external snapshot (`status.snapshots`). * **Request:** `RevertActorRequest` * `actor`: `ObjectRef` of the actor to revert. Accepted from `ACTOR_STATE_RUNNING` and `ACTOR_STATE_CRASHED` (plus `ACTOR_STATE_REVERTING` for idempotent retries), and from `ACTOR_STATE_SUSPENDED` when the actor's latest snapshot is node-local. Calling `RevertActor` on an `ACTOR_STATE_SUSPENDED` actor with no node-local snapshot returns `FAILED_PRECONDITION`. * **Response:** `RevertActorResponse` containing the reverted `Actor` in `ACTOR_STATE_SUSPENDED`. * Reverting terminates any bound worker sandbox, clears node-local checkpoints, and garbage-collects any partial external snapshot left by an interrupted suspend while preserving the last committed durable snapshot in `status.snapshots`. * External volumes are not reverted. Their contents are never part of a snapshot, so a reverted actor comes back with its memory and root filesystem rewound but its volumes exactly as the discarded execution left them. #### `DeleteActor` Removes an actor in any state from the registry and cleans up associated resources, terminating active workloads, detaching volumes, and releasing worker allocations. * **Request:** `DeleteActorRequest` * `actor`: `ObjectRef` of the actor to delete. * `options`: (Optional) `DeleteOptions`. `uid` and `version` are preconditions checked against the actor as the caller last read it: a mismatch returns `ABORTED`, an omitted guard is skipped. They are checked before the workflow starts, so a guarded delete that fails part-way leaves the actor `ACTOR_STATE_DELETING` at a higher version. Retry it with the `uid` guard alone, or re-read first. * **Response:** the deleted `Actor`, as it was immediately before removal. * Deleting an actor also deletes the external snapshot it owns, along with one an interrupted suspend left behind. Snapshots it only borrows from a tag are left alone, and its tags are unaffected — they hold their own copies. #### `GetActor` / `ListActors` Query the state of logical actors. * **GetActor:** Retrieves a single actor by ID. * **ListActors:** Lists all actors currently tracked in the database. #### `GetWorker` / `ListWorkers` / `ListWorkerActorAssignments` Query the physical resource pool. * **GetWorker / ListWorkers:** Retrieve one or more `Worker` records, including each worker's `status.state`, `status.capacity`, `status.allocated`, and `status.sandbox_classes`. * **ListWorkerActorAssignments:** Lists the `ActorAssignment` subresources for the actors currently hosted on a worker. --- ## 7. Advanced: Actor Identity Credentials ate-api-server mints credentials that identify an actor to services outside Substrate. They name the logical actor, so they stay the same as the actor moves between workers. This is distinct from the `actorMetadata` data source described under [SystemInfo Volumes](#systeminfo-volumes), which only tells an actor its own identity fields (name, atespace, uid). `MintActorJwt` is on `substrate.ateapi.v1.Control`. Only the egress gateway may call it; see [authorization.md](https://ate.dev/docs/guides/authorization/index.md). It calls this RPC when an egress policy injects an actor JWT into a request. ### `MintActorJwt` * **Request:** `actor`, the actor's atespace and name; `audiences`, one to 16 values for the token's `aud`; and `expiration_seconds`, the lifetime, from 300 to 3600. The actor must exist, but it doesn't need to be running. * **Response:** `actor_jwt`, the signed token, and `expire_time`, which equals its `exp` claim. The token's header carries `typ: JWT`, the algorithm of the pool's active signing key (ES256 or RS256), and that key's ID as `kid`. Its claims are: * `iss`: ate-api-server's `--actor-jwt-issuer`. ate-setup sets it from the `actorJWT.issuer` setting, which defaults to `https://idp..svc`. * `sub`: `actor//`. * `aud`: the requested audiences, always a JSON array. * `exp`, `nbf`, `iat`: Unix seconds. `nbf` is five minutes before `iat`, to allow for clock skew. * `jti`: a random token ID. * `actor.ate.dev`: an object with the actor's `atespace`, `actorName`, and `actorUID`. Relying parties find the signing keys through OpenID Connect Discovery at `/.well-known/openid-configuration`, which ate-idp-server serves in every install. --- ## 8. Framework & Ecosystem Integration Agent Substrate is designed to be the foundational execution layer for any agentic framework. ### Agent Development Kit (ADK) An ADK agent running as an actor can have the egress gateway attach an actor JWT ([section 7](#7-advanced-actor-identity-credentials)) to its outbound requests, so the tools and services it calls can tell which actor is calling. ### LangChain Substrate is an ideal runtime for stateful LangChain agents. By defining a LangChain agent as an `ActorTemplate`, you can preserve the agent's internal "thought process" and conversation history in memory across hibernations, while sandboxing its tool execution for security. ### Claude Code & CodeX For developer-focused agents, Substrate enables massive multiplexing of coding environments. Each developer can have a dedicated, persistent terminal session (Actor) that preserves filesystem deltas, while the cluster only runs physical pods for active users. ================================================================================ SECTION: REFERENCE PAGE: API Reference URL: https://ate.dev/docs/reference/api/ MARKDOWN: https://ate.dev/docs/reference/api/index.md SOURCE: docs/reference/api.md ================================================================================ # API Reference > Reference for the Substrate ate-api-server gRPC API and Kubernetes CRD types. The website generates the API reference from the ate-api-server gRPC API and the Kubernetes CRD types. To change the reference, edit the doc comments in [`ateapi.proto`](https://github.com/agent-substrate/substrate/blob/main/api/ateapi/v1/ateapi.proto) or in [`api/infra/k8s/ate/v1`](https://github.com/agent-substrate/substrate/tree/main/api/infra/k8s/ate/v1), following [Documentation Comments](https://github.com/agent-substrate/substrate/blob/main/docs/api-style-guide.md#13-documentation-comments). ================================================================================ SECTION: REFERENCE PAGE: Resource Model URL: https://ate.dev/docs/reference/resource-model/ MARKDOWN: https://ate.dev/docs/reference/resource-model/index.md SOURCE: docs/reference/resource-model.md ================================================================================ # Resource Model > The resources Agent Substrate exposes through its gRPC and Kubernetes APIs, and how they relate. Substrate exposes resources through two APIs: - a **gRPC API** (`api/ateapi/v1/ateapi.proto`), served by the Substrate API server. Its resources follow the [API Style Guide](https://github.com/agent-substrate/substrate/blob/main/docs/api-style-guide.md). - The **Kubernetes API**, through CRDs in `api/infra/k8s/ate/v1`, for managing compute infrastructure. Its resources follow the [Kubernetes API conventions](https://github.com/kubernetes/community/blob/main/contributors/devel/sig-architecture/api-conventions.md) ## Scope and parent Every resource belongs either to the whole Substrate installation or to a single atespace. A **global** resource, such as an `Atespace` or a `Worker`, belongs to the whole Substrate installation. An **atespace-scoped** resource, such as an `Actor` or a `Tag`, belongs to one atespace, and atespaces keep their resources apart from each other. Some resources exist only as part of another resource. These are **subresources**, and the resource they belong to is their **parent**. For example, an `EgressPolicy` is part of an `Actor`. A subresource is always reached through its parent and goes away when its parent does. A subresource's parent can also be **Global**, which stands for the whole Substrate installation. Global is not a resource. It is the root of the hierarchy below. ## Hierarchy ``` Global ├── AccessPolicy "default" subresource (singleton) ├── Atespace │ ├── AccessPolicy "default" subresource (singleton) │ ├── ActorTemplate │ ├── Actor │ │ └── EgressPolicy "default" subresource (singleton) │ └── Tag └── Worker └── ActorAssignment subresource (one per hosted Actor, managed by Substrate) ``` ## Top-level resources | Resource | Scope | Cardinality | Written by | | :--- | :--- | :--- | :--- | | `Atespace` | Global | 0..N | Clients | | `ActorTemplate` | Atespace | 0..N | Clients | | `Actor` | Atespace | 0..N | Clients | | `Tag` | Atespace | 0..N | Clients | | `Worker` | Global | 0..N | Substrate | ## Subresources | Subresource | Parent | Cardinality | Written by | | :--- | :--- | :--- | :--- | | `EgressPolicy` | `Actor` | 0..1, named `default` | Clients | | `AccessPolicy` | Global | 0..1, named `default` | Clients | | `AccessPolicy` | `Atespace` | 0..1, named `default` | Clients | | `ActorAssignment` | `Worker` | 0..N | Substrate | ## Kubernetes resources | CRD | Scope | | :--- | :--- | | `WorkerPool` | Namespaced | | `SandboxConfig` | Cluster | | `CSIDriverConfig` | Cluster | ## Relationships ```mermaid classDiagram namespace kube-apiserver { class WorkerPool { <> } class SandboxConfig { <> } class WorkerPod } namespace substrate-api-server { class Atespace class AtespaceAccessPolicy class ActorTemplate class Actor class EgressPolicy class Tag class Worker class ActorAssignment } Atespace "1" *-- "0..1" AtespaceAccessPolicy Atespace "1" o-- "*" ActorTemplate Atespace "1" o-- "*" Actor Atespace "1" o-- "*" Tag Actor "1" *-- "0..1" EgressPolicy Worker "1" *-- "*" ActorAssignment Actor "*" --> "1" ActorTemplate : actor_template Actor "*" --> "0..1" Tag : source_tag Tag "*" --> "1" Actor : source_actor ActorAssignment "0..1" --> "1" Actor : actor ActorTemplate ..> WorkerPool : worker_selector WorkerPool ..> SandboxConfig : sandboxClasses WorkerPool "1" *-- "*" WorkerPod Worker "1" --> "1" WorkerPod ``` ================================================================================ SECTION: REFERENCE PAGE: kubectl-ate CLI Reference URL: https://ate.dev/docs/reference/kubectl-ate/ MARKDOWN: https://ate.dev/docs/reference/kubectl-ate/index.md SOURCE: docs/reference/kubectl-ate.md ================================================================================ # kubectl-ate CLI Reference > Manage atespaces, actor templates, actors, snapshots, egress policies, and logs with the kubectl-ate plugin. A Kubernetes-native CLI plugin for managing Substrate Actor and Worker lifecycles. ## Running the CLI There are two ways to run the tool, depending on whether you are developing locally or installing it permanently. ### 1. Install as a native `kubectl` Plugin You can use `go install` to compile the tool and place the binary directly into your Go bin directory (which should be in your `$PATH`). Because the source folder is named `kubectl-ate`, Kubernetes will automatically recognize the resulting binary! ```bash go install ./cmd/kubectl-ate ``` You can now run it seamlessly anywhere as a native Kubernetes command: `kubectl ate `. ### 2. Run directly from source (Development) If you are testing changes to the codebase, you can bypass compilation and run the CLI directly from the source tree: ```bash go run ./cmd/kubectl-ate ``` ## Connection & Auto Port-Forwarding By default, `kubectl-ate` will automatically read your `~/.kube/config`, discover the `ate-api-server` pods in your cluster, and establish a temporary background port-forward tunnel to execute gRPC calls securely. If you prefer to route traffic directly (e.g., from a pod in a namespace labeled `ate.dev/api-access=true`), simply provide the `--endpoint` flag to bypass the tunnel. Either way, `kubectl-ate` reads the CA that verifies `ate-api-server` from the cluster, so it needs a kubeconfig or in-cluster config. For which credential to pass with `--token-file`, see the [Authentication Guide](https://ate.dev/docs/guides/authentication/index.md#people-running-kubectl-ate). ## Tracing The CLI supports on-demand tracing using the `--trace` flag. When enabled, the CLI will generate a trace ID and signal to the server that it wants the request to be traced. **Local (kind):** The kind overlay installed by `ate-setup --kind deploy ate-system` already provisions an in-cluster OpenTelemetry Collector and Jaeger in the `otel-system` namespace. No additional setup is required. Port-forward the Jaeger UI and invoke any command with `--trace`: ```bash kubectl port-forward -n otel-system svc/jaeger 16686:16686 & kubectl ate get actor my-counter-1 --trace # open http://localhost:16686 and search for the most recent trace ``` ## Global Flags These flags can be appended to any command: | Flag | Short | Description | Default | |---|---|---|---| | `--kubeconfig` | | Path to your kubeconfig file | `~/.kube/config` | | `--context` | | Name of the kubeconfig context to use | current context | | `--endpoint` | | Manual gRPC endpoint override (e.g., `localhost:8080`) | | | `--token-file` | | Path to a bearer token for ate-api authentication, or `-` for stdin | a token for the shared `ate-client` ServiceAccount | | `--output` | `-o` | Output format (`table`, `json`, `yaml`) | `table` | | `--preview` | | Allow the use of preview features enabled on the server | `false` | | `--trace` | | Enable on-demand tracing for the request | `false` | | `--verbose` | `-v` | Increase output verbosity; may be repeated or stacked (e.g., `-v`, `-vv`, `-vvvvv`) | `0` | Pass `--version` (`kubectl ate --version`) to print the CLI version and exit. --- ## Command Reference & Examples ### Getting Resources List and inspect the state of actors and workers across the cluster. ```bash # List actors in one atespace; -a is shorthand for --atespace kubectl ate get actors --atespace kubectl ate get actors -a # List actors across all atespaces kubectl ate get actors -A # Get a specific actor by name and output as raw YAML kubectl ate get actor --atespace -o yaml # List all physical workers or get specific workers by name kubectl ate get workers kubectl ate get worker # Filter workers by Kubernetes namespace, assigned-actor atespace, # worker pool labels, or sandbox class (same flags as `top workers`) kubectl ate get workers -n kubectl ate get workers -a kubectl ate get workers -l kubectl ate get workers --sandbox-class # Show live CPU and memory usage of worker pods kubectl ate top workers ``` > **Note:** `get actors` requires either `--atespace ` / `-a ` (one atespace) or `-A`/`--all-atespaces` (all atespaces) — there is no default atespace. Getting a single actor always requires `--atespace`/`-a`, since an actor is addressed by `(atespace, name)`. `-a` (lower-case) scopes to one atespace; `-A` (upper-case) spans all. > **Note:** Actors, workers, and actor templates are not Kubernetes CRDs — they live in the Substrate control plane's PostgreSQL database, not `etcd`. `kubectl get actor`, `kubectl get worker`, and `kubectl get actortemplate` will not return anything; only `kubectl ate get …` queries the control plane (see [Actor Templates](#actor-templates)). `kubectl get workerpool` *does* work, because pools are CRDs. #### `kubectl ate get actor` output columns | Column | Meaning | |---|---| | `ATESPACE` | The atespace the actor belongs to. Part of the actor's identity; folded into the storage key as `actor::`. | | `NAME` | The actor's name. User-provided for application actors; UUID for the golden actor that each template materialises while building its golden tag. | | `TEMPLATE` | The `ActorTemplate` the actor references, displayed as `/`. | | `STATE` | Current lifecycle state (`ACTOR_STATE_RESUMING`, `ACTOR_STATE_RUNNING`, `ACTOR_STATE_SUSPENDING`, `ACTOR_STATE_SUSPENDED`, `ACTOR_STATE_CRASHED`, `ACTOR_STATE_DELETING`, `ACTOR_STATE_REVERTING`). | | `WORKER POD` | The worker pod (`/`) currently hosting the actor, or `` while suspended. | | `WORKER IPS` | Comma-separated pod IPs of that worker. Empty while suspended. | | `VERSION` | Monotonic integer that increments on every state transition (resume / suspend / checkpoint). Useful for distinguishing snapshots. | | `AGE` | Time elapsed since the actor was created. | #### `kubectl ate get worker` output columns | Column | Meaning | |---|---| | `NAME` | The worker's name. The control plane assigns it. Treat it as opaque. | | `POOL` | The `WorkerPool` name. | | `STATE` | Current worker state. `WORKER_STATE_ACTIVE` when schedulable. `WORKER_STATE_UNAVAILABLE` while the worker is not ready or is restarting. `WORKER_STATE_DRAINING` once the pod is terminating. | | `ACTORS` | Actors assigned to the worker vs. the worker's actor limit (`/`). | | `CPU(MILLICORES)` | Allocated vs. capacity CPU in millicores (`/`). Allocated includes each runtime's per-actor overhead. `-` if the worker reports no CPU capacity. | | `MEMORY` | Allocated vs. capacity memory with binary suffixes, such as `0/960Mi`. Allocated includes each runtime's per-actor overhead. `-` if the worker reports no memory capacity. | | `POD` | The worker pod (`/`). | | `AGE` | Time elapsed since `ate-controller` created the Worker for its pod. | ### Atespaces An **atespace** is the isolation boundary an actor belongs to. It must exist before you can create actors in it. ```bash # Create an atespace kubectl ate create atespace # List all atespaces kubectl ate get atespaces # Get an atespace kubectl ate get atespace # Delete an atespace (must be empty: fails if any actors, templates, or tags remain) kubectl ate delete atespace ``` > **Note:** `create actor … -a ` requires the atespace to already exist, otherwise it fails with `FailedPrecondition`. `delete atespace` only removes an **empty** atespace. Delete its actors, actor templates, and tags first (cascade delete is not yet supported). #### `kubectl ate get atespace` output columns | Column | Meaning | |---|---| | `NAME` | The atespace name. Globally unique — atespaces are global-scoped. | | `AGE` | Time elapsed since the atespace was created. | ### Actor Templates An **actor template** describes what an actor runs: containers, volumes, snapshot policy, and worker selection. Templates live in an atespace and are immutable — there is no update; delete and recreate to change one (or create a new template and repoint suspended actors to it with `kubectl ate update actor --template`). ```bash # Create a template from a manifest (protojson-shaped ateapi.ActorTemplate, # a single YAML/JSON document; use -f - for stdin). The metadata's atespace # must already exist. kubectl ate create actor-template -f template.yaml # List templates, or get one (also: -o yaml prints the re-applyable manifest). kubectl ate get actor-templates -a kubectl ate get actor-template -a -o yaml # Delete a template. This also deletes its golden actor and golden snapshot. kubectl ate delete actor-template -a ``` See [`demos/counter/counter-template.yaml.tmpl`](https://github.com/agent-substrate/substrate/blob/main/demos/counter/counter-template.yaml.tmpl) for a complete manifest example. #### `kubectl ate get actor-templates` output columns | Column | Meaning | |---|---| | `ATESPACE` | The atespace the template belongs to. | | `NAME` | The template's name. | | `GOLDEN TAG` | The golden tag's name once it exists; `DISABLED` when the template opts out of golden snapshot creation; empty while the golden build is still running. | | `ERROR` | `ERROR` when the golden build failed; `-o yaml` shows the full message. | | `AGE` | Time elapsed since the template was created. | ### Actor Lifecycle Manage the execution state of your workloads. *(Note: Actors are identified by a user-provided name, which must be a valid DNS-1123 label)* ```bash # Create a new actor from an ActorTemplate. kubectl ate create actor my-actor --template= -a # Update an actor's template (requires the actor to be SUSPENDED) or worker selector. kubectl ate update actor my-actor --template= -a kubectl ate update actor my-actor --worker-selector=tier=paid,env=prod -a kubectl ate update actor my-actor --worker-selector="" -a # Resume an actor (assigns it to a free worker and restores its state) kubectl ate resume actor my-actor -a # Suspend an actor (snapshots its state and frees the worker). By default the # snapshot stays on the worker's node and is uploaded to storage in the background. kubectl ate suspend actor my-actor -a # Suspend an actor and wait for its snapshot to reach storage. kubectl ate suspend actor my-actor -a --mode=sync # Revert an actor to its last external snapshot (discards live or crashed state, or suspended state with a node-local snapshot, and returns to SUSPENDED) # A suspended actor without a node-local snapshot is already at its external snapshot, so it cannot be reverted kubectl ate revert actor my-actor -a # Delete an actor, terminating workloads and detaching volumes. kubectl ate delete actor my-actor -a ``` ### Sending Requests to an Actor `proxy actor` forwards one or more local TCP ports to an actor. It presents the client certificate for `spiffe://cluster.local/ns/ate-system/sa/atenet-client`, so the install must list that ID in `--ingress-allowed-spiffe-ids`. ```bash # Forward 127.0.0.1:8000 to actor port 80 (default when no ports are given). kubectl ate proxy actor my-actor -a curl http://127.0.0.1:8000/ # Forward specific [local-port:]remote-port mappings. kubectl ate proxy actor my-actor 8000:80 9090:9090 -a ``` ### Actor Snapshots Suspending an actor creates a durable snapshot. Tags give snapshots stable, Atespace-owned names; published tags may be used from other Atespaces. A tag is created from a suspended actor and gets its own copy of that actor's snapshot, so suspending or deleting the actor afterwards cannot collect it. ```bash # List an Atespace's tags, all Atespaces' tags, or resolve tags by name. kubectl ate get tags -a kubectl ate get tags -A kubectl ate get tag [ ...] -a # Tag the snapshot a suspended actor holds, then publish or unpublish the tag. # After an asynchronous suspend, create tag uploads the local snapshot first if needed. kubectl ate create tag -a --actor [--scope published] kubectl ate update tag -a --scope published kubectl ate update tag -a --scope atespace # Create an actor from a tag. kubectl ate create actor -a --template --tag kubectl ate delete tag -a ``` ### Egress Policies An actor has at most one egress policy. See [Egress Traffic](https://ate.dev/docs/guides/egress-traffic/index.md), [Network Egress](https://ate.dev/docs/guides/network-egress/index.md), [MITM Trust Bundle](https://ate.dev/docs/guides/egress-trust-bundle/index.md), and [Egress Credential Injection](https://ate.dev/docs/guides/egress-credential-injection/index.md) for how outbound traffic is mediated and how rules are evaluated. Example `policy.yaml`: ```yaml rules: - https: hostnames: ["api.anthropic.com", "api.openai.com"] ports: numbers: [443] ``` ```bash # Get an actor's egress policy. kubectl ate get egress-policy -a kubectl ate get egress-policy -a -o yaml # Create an egress policy. kubectl ate create egress-policy -a -f policy.yaml # Copy the egress policy of another actor. kubectl ate get egress-policy -a -o yaml | \ kubectl ate create egress-policy -a -f - # Replace an actor's egress policy: dump it, edit the rules, send it back. kubectl ate get egress-policy -a -o yaml > policy.yaml $EDITOR policy.yaml kubectl ate update egress-policy -a -f policy.yaml # Delete an actor's egress policy; the actor keeps running with all egress denied. kubectl ate delete egress-policy -a # Delete it only if it is still the policy you read. kubectl ate delete egress-policy -a --uid --version ``` #### Details * `create` can take a manifest with no `metadata`, taking `name` and `atespace` from the command line. * `update` replaces the entire policy and the manifest metadata must match `uid` and `version` for the `EgressPolicy` being updated. * `delete` with the optional `--uid` and `--version` flags removes the policy only if the given values still match what `get -o yaml` reported; the server answers `Aborted` otherwise. #### `kubectl ate get egress-policy` output columns | Column | Meaning | |---|---| | `ATESPACE` | The atespace the actor and its policy belong to. | | `ACTOR` | The actor the policy applies to. | | `RULES` | Number of rules; `-o yaml` shows them in evaluation order. | | `VERSION` | The policy's version, bumped on every update. | | `AGE` | Time elapsed since the policy was created. | ### Logs `kubectl ate logs` requires a resource-type subcommand. The only supported resource type is `actor`: ```bash # Print the logs an actor has produced on its current worker. # -a/--atespace is required, since an actor is addressed by (atespace, name). kubectl ate logs actor my-actor -a # Follow the logs with -f. The stream is aggregated across worker # reassignments, so the same actor stays queryable as it teleports between pods. kubectl ate logs actor my-actor -a -f # Show only one container's logs with -c/--container. kubectl ate logs actor my-actor -a -c my-container # Pick a class of lines with --source: containers (every container's output, # no lifecycle events) or lifecycle (only the actor's lifecycle events). kubectl ate logs actor my-actor -a --source=lifecycle ``` Logs are streamable only while the actor is bound to a worker (i.e., `ACTOR_STATE_RUNNING`). For history across worker migrations, route through a centralized log backend (Loki, Elasticsearch, etc.); see the [Observability Guide](https://ate.dev/docs/guides/observability/index.md). ### Administration & Setup Commands for bootstrapping the Substrate control plane. ```bash # Generate a new Actor ID CA pool and push it directly to a Kubernetes Secret kubectl ate admin make-ca-pool \ --name actor-id-ca-pool \ --secret-namespace ate-system \ --ca-id "1" # Generate a new JWT authority pool and push it to a Kubernetes Secret. The # key is ES256 (--alg RS256, a 4096-bit RSA key, for relying parties that don't # support ES256) and its ID defaults to the base64url SHA-256 of the public # key's PKIX encoding (--key-id to override). The Secret holds the pool under # "pool". kubectl ate admin make-jwt-pool \ --name actor-id-jwt-pool \ --secret-namespace ate-system # Rotate the JWT signing key. The new key is published but signs nothing until # activated; activate it once relying parties have refetched the key set, and # remove the old key once every token it signed has expired. list-jwt-keys # shows each key's ID and algorithm and marks the active one. kubectl ate admin list-jwt-keys --name actor-id-jwt-pool --secret-namespace ate-system kubectl ate admin add-jwt-key --name actor-id-jwt-pool --secret-namespace ate-system kubectl ate admin activate-jwt-key --name actor-id-jwt-pool --secret-namespace ate-system --key-id kubectl ate admin remove-jwt-key --name actor-id-jwt-pool --secret-namespace ate-system --key-id ``` ================================================================================ SECTION: REFERENCE PAGE: Glossary URL: https://ate.dev/docs/reference/glossary/ MARKDOWN: https://ate.dev/docs/reference/glossary/index.md SOURCE: docs/reference/glossary.md ================================================================================ # Glossary > Definitions of core terms, resources, and components across Agent Substrate. This document defines the core terms used across Agent Substrate. For how the pieces fit together, see the [Architecture](https://ate.dev/docs/concepts/architecture/index.md) and [API Guide](https://ate.dev/docs/reference/api-guide/index.md). ## Naming & Prefix - **ATE vs. Agent Substrate**: **Agent Substrate** is the name of the overall platform and runtime, while **`ate`** (originally *Agent Task / Execution Engine*) is the internal prefix used across Kubernetes namespaces (`ate-system`), control-plane binaries (`ate-api-server`, `atecontroller`), node and pod daemons (`atelet`, `ateworker`), networking components (`atenet`, `atunnel`), HTTP routing headers (`ate-target-actor`), and the `kubectl-ate` CLI plugin. ## Resources (declarative) - **ActorTemplate** (ate API resource): the definition of an actor "class": the container image(s) and snapshot configuration. Creating an `ActorTemplate` triggers creation of a [Golden Snapshot](#snapshots) by default. Set `snapshotConfig.goldenSnapshotConfig.mode` to `GOLDEN_SNAPSHOT_MODE_DISABLED` to skip it. The template is treated as immutable: you create a new template for a new version rather than editing an existing one. It is analogous to a Pod template, but for a checkpointable workload. ActorTemplates are created and managed through the substrate gRPC API (e.g. `kubectl ate create actor-template`) and stored in the control-plane database; they are not Kubernetes objects. - **WorkerPool** (Kubernetes CRD): declares warm compute capacity, a fleet of pre-started worker pods. It is reconciled into a Kubernetes `Deployment` by the [atecontroller](#components). - **SandboxConfig** (Kubernetes CRD): a cluster-scoped resource holding the sandbox binaries for one runtime family (the gVisor `runsc` binary, or a microVM kernel/firmware/config). Each `WorkerPool` resolves a `SandboxConfig` for its sandbox class, either by explicit `configRef` or via the class default. A new or cold-booted actor gets the config's default version. A memory restore gets the version its snapshot was taken with. ## Records (dynamic state, in the control-plane store) These live in the control-plane database because they change too frequently for etcd. - **Atespace**: the isolation boundary an Actor belongs to, and the first half of its identity: an Actor is addressed by `(atespace, name)`, so the same name can exist in two atespaces. Atespaces are global-scoped, not Kubernetes namespaces. `ActorTemplate`s and `Tag`s also live in an atespace, while `WorkerPool`s live in Kubernetes namespaces. One must exist before any resource can be created in it, and it can only be deleted once its actors, actor templates, and tags are gone. - **Actor**: a single instance derived from an `ActorTemplate`, identified by a DNS-1123 name. It is the unit that is suspended and resumed, and it moves between workers over its lifetime. An Actor record tracks its lifecycle status and snapshot references. - **Tag**: a durable, atespace-scoped snapshot record created from a suspended Actor (or by `ate-api-server` for a template's golden snapshot). Each Tag holds its own copy of the snapshot in object storage and can be scoped to its own atespace or published across atespaces. - **Worker**: a record representing one worker pod in a `WorkerPool`. A Worker hosts several Actors at once, each in its own sandbox, up to its actor limit and its compute capacity. Many more are multiplexed across a pool over time. ## Components - **ate-api-server** (binary `ateapi`): the control plane. It owns the Actor lifecycle, schedules Actors onto Workers, and coordinates their snapshots, all backed by the state store. The `kubectl-ate` CLI talks to it. - **atecontroller**: the Kubernetes controller that reconciles the CRDs (for example, it turns a `WorkerPool` into a `Deployment`). - **atelet**: the node-level supervisor, run as a DaemonSet. It pulls images, assembles OCI bundles, drives the sandbox lifecycle on the node via ateworker, and streams snapshots to and from snapshot storage. - **ateworker**: the coordinator that runs inside each worker pod and drives the sandbox runtime on behalf of atelet. This decouples the physical pod lifecycle from the sandboxed agent process. It embeds a networking service called `atunnel` that handles network traffic for the sandboxed Actor. ateworker is Substrate's worker implementation, one binary per sandbox class (`ateworker-gvisor`, `ateworker-microvm`). - **atenet**: the networking stack. Its router resumes suspended Actors on demand and routes traffic to the right worker pod. - **podcertcontroller**: issues short-lived pod certificates that components use as their TLS identity to authenticate connections to one another (mutual TLS). - **kubectl-ate**: a `kubectl` plugin CLI for managing the Actor lifecycle and listing Workers. ## Lifecycle - **Suspend**: hibernate a running Actor into a snapshot. The Actor is checkpointed on its Worker, which is then freed. Snapshot files remain on the node VM so the following Resume can restore from them locally (until `atelet` or a Resume detaches them), and are also uploaded to external storage: a synchronous suspend uploads the snapshot before it returns, while an asynchronous suspend (the default) returns once the local snapshot is captured and `atelet` uploads it in the background. Suspending an Actor that is already suspended returns it as it is. - **Resume**: activate a suspended Actor by restoring it onto a Worker. The common path restores from a snapshot rather than cold-booting. ## Volumes - **DurableDir volume**: a directory mounted into one or more containers whose contents are preserved by the [`VOLUMES` snapshot fidelity](#snapshots) and therefore survive across Suspend/Resume independently of process memory or other rootfs writes. A volume may be mounted into multiple containers, potentially at different paths. This is the per-Actor application-data surface. An `ActorTemplate` may declare several, on either sandbox class. ## Snapshots - **Snapshot fidelity**: the state layers a snapshot holds, ordered lowest to highest; each level includes the ones below it. - **`VOLUMES`**: only the contents of attached volumes that support snapshots — currently `DurableDir` volumes. Process memory and rootfs changes are discarded. Used to persist application data cheaply without the cost of a memory image. On Resume the containers start afresh from the OCI image with the `DurableDir` contents restored. - **`ROOTFS`**: volumes plus the root filesystem changes made since boot. Process memory is discarded; on Resume the containers cold-boot from the OCI image with their rootfs writes back in place. - **`MEMORY`**: volumes, rootfs changes, and process memory. Used to capture everything needed to resume hot. Fidelity describes only what a snapshot *holds*. A template sets one level, `preferredFidelity`, the ceiling for every snapshot of its actors: both the node-local checkpoint a [Suspend](#lifecycle) keeps on the node and the snapshot uploaded to snapshot storage. It is best effort: the system never packages more than that level, and may package less when a layer cannot be captured or is too expensive to capture at the time. A process resumed in place keeps its memory regardless. - **Golden Snapshot**: the initial checkpoint normally captured once, when an `ActorTemplate` is created, from a temporary "golden" boot of the workload. Templates with golden snapshot creation disabled do not create one. By default an Actor of that template is first restored from this shared snapshot when it is available, and otherwise cold-boots. It is captured at the template's `preferredFidelity` (`MEMORY` by default). - **Last Snapshot**: the most recent per-Actor snapshot, written on Suspend and used to restore that specific Actor on the next Resume. - **Snapshot storage**: the object store (GCS or S3) where snapshots are persisted so Actor state is durable and portable across the cluster. ## Networking - **Actor routing header**: a higher-order system sends traffic to the Substrate router with `ate-target-actor: /`. The router uses this header to locate and resume the Actor. ================================================================================ SECTION: REFERENCE PAGE: Version Skew Policy URL: https://ate.dev/docs/reference/version-skew-policy/ MARKDOWN: https://ate.dev/docs/reference/version-skew-policy/index.md SOURCE: docs/reference/version-skew-policy.md ================================================================================ # Version Skew Policy > Which component releases may run together, and the order to upgrade them in. The [compatibility policy](https://docs.google.com/document/d/1obSIvfcafLNniLYTQCcT2eCgxHqa2AQ3Ga7YTsju49s/edit?usp=sharing&resourcekey=0-5VqJ13Vj1QctB7SnnfmtkQ) says what may change between two releases. This document says which releases of the substrate components may run together, and the order in which they are upgraded so that only supported combinations occur. The [upgrade runbook](https://ate.dev/docs/guides/upgrade/index.md) implements it and may add restrictions of its own. ## Scope Two releases are compatible when every change between them is compatible under the compatibility policy. This document covers moving an installation between two compatible releases, and the mixture of components that exists while that happens. Between incompatible releases, reinstall. This policy applies from 1.0.0. From 1.0.0, the patch and minor releases of one major are compatible. Below, N and N+1 are two consecutive compatible releases. All components of a release are built from one commit and carry one version string. Patch releases add no API fields, public or internal, and no schema migrations. ## Components * `ate-api-server`: serves the public API, calls `atelet`, talks to the database. * `ate-idp-server`: publishes the keys that verify actor JWTs. It shares only the actor JWT key pool with `ate-api-server`, and is upgraded with it. * `ate-controller`: renders WorkerPools into Deployments, registers workers with `ate-api-server`. * `atelet`: one pod per node (DaemonSet), drives the `ateworker` processes on its node. * Workers: the pods of a WorkerPool, running `ateworker` and the actors. * `atenet`: router and egress gateway. * `podcertificate-controller`: signs the certificates components present to each other. * Clients: `kubectl-ate` and anything else using the public API. See more in [architecture.md](https://ate.dev/docs/concepts/architecture/index.md). ## Supported version skew The components are ordered: `podcertificate-controller`, `atelet`, workers, `ate-api-server`, `ate-controller`, `atenet`, clients. * A component may be one release newer than any component after it in this order. * A component must not be newer than any component before it. * While a component rolls, its own replicas, nodes or pods are at N and N+1. * The API schema is not shared. Until every `ate-api-server` replica is at N+1, a request can reach an N replica, which rejects fields new in N+1. So clients must not use fields new in N+1 until `ate-api-server` finishes rolling. * The database schema is shared. The first N+1 `ate-api-server` replica migrates it, so `ate-api-server` replicas at N and N+1 use the N+1 database schema while `ate-api-server` rolls. So during an upgrade from N to N+1, a new `atelet` drives old `ateworker` processes on its node, a new `atelet` calls an old `ate-api-server`, an old `ate-controller` renders Deployments for new workers, and old `ate-controller` and clients talk to a new `ate-api-server`. The reverse pairs are not supported: in particular, a worker must not be newer than the `atelet` on its node, and `ate-api-server` must not be newer than any worker. `ate-api-server` reaches the workers through `atelet`, and a new `ate-api-server` may rely on new worker behavior behind existing fields, so the workers go first. ## Requirement of a release The skew rules above hold only if every release keeps the following. ### Worker pod template `ate-controller` upgrades after the workers. So the workers start the `ateworker` N+1 image with the Deployment `ate-controller` N renders. Within a major, `ate-controller` renders the same worker pod template for the same WorkerPool. New `ateworker` behavior comes from the image or from a new WorkerPool field, not from the template. So `ateworker` accepts the flags and environment the template passes for the whole major, and no WorkerPool rolls at `ate-controller`'s step, in an upgrade or a rollback. ### API fields These rules say what a component does with a protobuf field it does not know. Servers: * `ate-api-server` rejects a request that carries a field it does not know, with `INVALID_ARGUMENT`. * `ate-api-server` drops the fields it does not know when it reads a stored object, and serves the reduced object. Such a field was written by a newer replica, during `ate-api-server`'s roll or before a rollback, and this release cannot validate it. Rejecting it would make the object unreadable. Keeping it would serve a field this release cannot validate. * `atelet` rejects fields it does not know. Its callers, `ate-api-server` and `ateworker`, are never newer than it during an upgrade or a rollback in the supported order, so an unknown field means the upgrade order was not followed. * `ateworker` rejects fields it does not know. Its caller, `atelet`, can be one release newer, so `atelet` sends a new field only from the minor release after the one in which `ateworker` learns it (two-step releases). Clients: * A client that reads an object, changes it and writes it back keeps the unknown fields. An older client must not strip fields that a newer server still understands. * `kubectl-ate` rejects a manifest file that has fields it does not know, as the server does for a create. Wherever a caller can be newer than its server, the server learns a new field in one minor release and the caller sends it only from the next minor release on (two-step releases). ### CRDs The N+1 CRDs are applied with `ate-controller` N+1. Components that upgrade before it run at N+1 against the N CRDs. `atelet`, for one, watches WorkerPool and SandboxConfig. So these components must work without a new CRD field or kind. Users must not set new CRD fields until the upgrade completes. Between compatible releases, a CRD change only adds optional fields. The CRDs serve one version with no conversion. So a rollback from N+1 that applies the N CRDs again removes every field N does not know from every object. ### SandboxConfig Each sandbox class has a default SandboxConfig, such as `gvisor-default`, that lists the sandbox versions a cluster may run. A release that changes a sandbox version adds the new one, points `defaultVersion` at it, and keeps the previous release's version unless that version has a known vulnerability. A kept version's assets must stay fetchable, so a release does not overwrite them. The upgrade applies these lists. That does not change the pod template, so workers do not roll. An actor that starts without memory state, new or cold booting, gets the default version. An actor restoring memory state gets the version its snapshot was taken with, and stays on it until it next starts without memory state. If that version is gone, it cold boots, keeping its files and losing its memory. Rollback applies the N lists again, which removes the N+1 versions. On gVisor, every restore without memory state runs under the worker's default `runsc`. After an upgrade or a rollback, that `runsc` may read `fscheckpoint` bundles that another `runsc` version wrote. Substrate does not check that it can, and a restore that fails crashes the actor. ### Database schema The schema is updated with `ate-api-server`, and migrations follow [postgresql-schema-evolution.md](https://github.com/agent-substrate/substrate/blob/main/docs/dev/postgresql-schema-evolution.md): every migration keeps working for the previous `ate-api-server` binary. Migrations are never rolled back. ### Data on nodes and in snapshots This covers Substrate's own data only. The sandbox runtime's checkpoint files follow the runtime's own rules: a restore across runtime versions keeps only the file systems. On gVisor, `runsc` writes those file systems as `fscheckpoint` bundles (see [SandboxConfig](#sandboxconfig)). On microVM, `ateworker-microvm` writes them as tar files, which are Substrate's own data and follow the snapshot rules below. This section is about data one release writes and another release reads, whether that is a different component or the same component after an upgrade or a rollback. Substrate's data comes in two kinds, by how long it lives. * Data on the node: the directory layout and the files `atelet` keeps there. * What N writes, N+1 can read. What N+1 writes, N can read. * So a layout change takes two minor releases: the first reads the old and the new layout but still writes the old one, and the next writes the new one. This keeps a rollback possible. Neither step can go in a patch release: an upgrade can skip patch releases, and a rollback can land on any of them. * `atelet` passes the per-actor directories to `ateworker` in the RPC. `ateworker` also reads some files `atelet` writes under the paths in `internal/nodepath`, such as `sandbox-runtimes.json`, and rejects fields it does not know in them. So a new field in those files follows the two-step rule of the RPC: `ateworker` learns it in one minor release, and `atelet` writes it from the next. The sockets `atelet` and `ateworker` use to reach each other do not change within a major. * Data in snapshots: `manifest.json`, which `atelet` writes and reads. * Within a major, a release reads the snapshot data any earlier release of that major wrote. * A new field must be safe to ignore. A release ignores manifest fields it does not know, so a snapshot taken at N+1 still restores after a rollback to N. ### System bundle The upgrade applies the rest of the system bundle last, and a rollback does not undo it. So a component at N+1 must work with these resources at N, and a component at N must work with them at N+1. Changes that only take something away, such as a NetworkPolicy that denies traffic, can go in this step. Anything a new component needs to run, such as a new Secret or a rule that allows its traffic, ships in that component's own step. ## Supported upgrade order To move from N to N+1, in this order, waiting for each step to finish: 1. `podcertificate-controller`. 2. `atelet`, node by node. 3. Every WorkerPool's `spec.workerImage`, which rolls its workers. 4. `ate-api-server` and `ate-idp-server`. The first N+1 `ate-api-server` replica migrates the schema. 5. The CRDs, then `ate-controller`. 6. `atenet`. 7. The SandboxConfig admission policy and the default SandboxConfig. 8. The rest of the substrate system bundle: what no step above deploys. 9. Clients. Rollback from N+1 to N is the same list in reverse, with release N, except that the schema migration and step 8 are not undone. Clients stop sending N+1 fields before `ate-api-server` goes back, and `atelet` goes back only after every worker is back at N. Rolling back further than N is not supported. ### Skipping releases Patch releases can be skipped: `x.y.1` can upgrade straight to `x.y.5`. Minor releases cannot: to go from `x.1` to `x.3`, upgrade to `x.2` first. The two-release steps and the database schema evolution both assume each upgrade moves one minor release. ## Outside this policy The Kubernetes version, the database engine, the node OS, the CSI drivers and the actor images are not changed by the Agent Substrate upgrade. ## Testing Before a release is cut, an installation at the previous compatible release is upgraded to it with actors running and rolled back, in the order above. ## Exceptions As in the compatibility policy. ================================================================================ SECTION: REFERENCE PAGE: Sandbox Classes URL: https://ate.dev/docs/reference/sandbox-classes/ MARKDOWN: https://ate.dev/docs/reference/sandbox-classes/index.md SOURCE: docs/reference/sandbox-classes.md ================================================================================ # Sandbox Classes > How the gvisor and microvm sandbox classes differ in sizing, capabilities, and snapshots. Each [`WorkerPool`](https://ate.dev/docs/reference/api-guide/index.md#1-workerpool-the-physical-capacity) runs one sandbox class, set in `spec.sandboxClasses[].name`: `gvisor` (`ateworker-gvisor`) or `microvm` (`ateworker-microvm`). The pool resolves a cluster-scoped [`SandboxConfig`](https://ate.dev/docs/reference/api-guide/index.md#3-sandboxconfig-the-sandbox-itself) for that class. An [`ActorTemplate`](https://ate.dev/docs/reference/api-guide/index.md#2-actortemplate-the-workload-blueprint) selects eligible pools through `workerSelector`, combined with the actor's own `worker_selector`. An actor without a snapshot may start on either class if its selectors match pools of both. Its first snapshot records the class in `sandboxRuntime`. Every later restore, at any fidelity, runs on a worker of that class. A new actor created from the template's golden snapshot or from a `sourceTag` starts with that snapshot, so it runs on the class the snapshot was taken on. ## Summary of Differences | Area | `gvisor` (`ateworker-gvisor`) | `microvm` (`ateworker-microvm`) | | :--- | :--- | :--- | | Isolation | User-space kernel (`runsc`) | KVM guest ([Kata Containers](https://katacontainers.io/) on [Cloud Hypervisor](https://www.cloudhypervisor.org/)) | | Node requirements | Standard Linux node | Linux node with `/dev/kvm`, requested as the `ate.dev/kvm` device resource | | Reported CPU count | `max(2, ceil(cpuMillicores / 1000))`, at most the CPUs the sandbox can use | `max(1, ceil(cpuMillicores / 1000))` | | CPU limit enforcement | Host cgroup v2 `cpu.max` on the sandbox | Guest vCPU count, plus host cgroup v2 `cpu.max` on the VMM and `virtiofsd` | | Minimum `limits.memoryBytes` | None enforced by worker | `128 MiB` (`134217728` bytes) | | Per-container limits (`containers[].resources.limits`) | Accepted in the API, ignored at runtime | Enforced via guest cgroups (`cpuMillicores` floored at 10) | | `VOLUMES` and `ROOTFS` snapshot format | `runsc` filesystem checkpoint bundles (`fs/`, `rootfs/`) | Tar archives of the host directories | ## CPU Count and Throttling An actor's CPU size comes from `resources.limits.cpuMillicores` on its `ActorTemplate`: - **gVisor (`gvisor`):** `ateworker-gvisor` sets the sandbox's cgroup v2 `cpu.max` quota from `cpuMillicores` and passes `--cpu-num-from-quota` to `runsc`. `runsc` rounds the quota up to whole CPUs, with a minimum of 2, and never reports more CPUs than the sandbox can use. An actor with `cpuMillicores: 500` or `1000` sees 2 CPUs, and the host kernel throttles it to its quota. - **MicroVM (`microvm`):** `ateworker-microvm` sets Cloud Hypervisor `BootVcpus` and `MaxVcpus` to `ceil(cpuMillicores / 1000)` with a minimum of 1. It also caps the VMM and `virtiofsd` at `cpuMillicores` with a host cgroup v2 `cpu.max`. An actor with `cpuMillicores: 500` boots with 1 vCPU and gets half a CPU of host time. ## Memory Sizing An actor's memory size comes from `resources.limits.memoryBytes` on its `ActorTemplate`: - **gVisor (`gvisor`):** `memoryBytes` sets the sandbox cgroup memory limit and the total memory reported by the Sentry. - **MicroVM (`microvm`):** `memoryBytes` sets the guest RAM size and must be at least `128 MiB`. The VMM and `virtiofsd` overhead counts against the worker's capacity on top of `memoryBytes`. A `MEMORY` restore reuses the guest RAM and vCPU count recorded in the snapshot. ## Per-Container Resource Limits Containers may declare optional `containers[].resources.limits` (`cpuMillicores` and `memoryBytes`): - **gVisor (`gvisor`):** One Sentry process runs every container in the actor ([google/gvisor#190](https://github.com/google/gvisor/issues/190)). The API accepts per-container limits, and gVisor enforces only the actor's `resources.limits`. - **MicroVM (`microvm`):** The guest enforces each container's limits in its own cgroup. Substrate raises a `cpuMillicores` limit below 10 to 10. A container that exceeds its memory limit is OOM-killed on its own while sibling containers in the actor keep running. See [Per-container limits](https://ate.dev/docs/reference/api-guide/index.md#per-container-limits) in the API Guide. ## System Mounts (`/dev/shm` and `/tmp`) Each container gets its own `tmpfs` at `/dev/shm` (`64 MiB`, mode `1777`, `noexec`) and `/tmp` (mode `1777`). `/tmp` holds at most half of `limits.memoryBytes` on `gvisor`, and half of usable guest RAM (`MemTotal`) on `microvm`. The `/tmp` mount hides any files the image has in `/tmp`. A volume at either path replaces the `tmpfs`. Files in these `tmpfs` mounts count against the actor's memory limit. They live outside the container's rootfs overlay, so `MEMORY` snapshots keep them and `ROOTFS` snapshots start with them empty. Neither sandbox class mounts anything at `/run`. As with containerd, `/run` is part of the container's rootfs overlay, so the image's files there stay visible and `ROOTFS` snapshots keep them. Volumes cannot mount at `/dev`, or at or under `/dev/pts`, `/proc`, or `/sys`. ## Snapshots (`VOLUMES`, `ROOTFS`, and `MEMORY`) On both classes, a `MEMORY` restore prefers a worker that offers a runtime compatible with the snapshot's `sandboxRuntime`. If none of those workers has room, a durable snapshot restores its `ROOTFS` state on any eligible worker of the same class and cold-boots under that worker's default runtime. See `sandboxRuntime` in [Snapshot Storage Layout](https://ate.dev/docs/reference/api-guide/index.md#snapshot-storage-layout). How `VOLUMES` and `ROOTFS` state is stored differs by class: - **gVisor (`gvisor`):** `durableDir` volumes and container rootfs overlays live inside the Sentry's internal filesystems. `runsc` saves them as filesystem checkpoint bundles in the snapshot's `fs/` and `rootfs/` directories. A `VOLUMES` or `ROOTFS` restore uses the worker's default `runsc`, which may be a different version from the one that wrote the bundles. See the [Version Skew Policy](https://ate.dev/docs/reference/version-skew-policy/index.md#sandboxconfig). - **MicroVM (`microvm`):** Container rootfs upper layers and `durableDir` volumes live on the host filesystem and are shared into the guest over `virtio-fs`. `ateworker-microvm` archives them as tar files, one per volume and one per container. The format does not depend on the guest kernel or Cloud Hypervisor version. ================================================================================ SECTION: SECURITY PAGE: Security Overview URL: https://ate.dev/docs/security/security/ MARKDOWN: https://ate.dev/docs/security/security/index.md SOURCE: docs/security/security.md ================================================================================ # Security Overview > The security scope, extensibility, and core principles of Agent Substrate. Substrate delivers zero trust for agents. Every agent has an identity, but no credentials, no open network, and no escape from the sandbox. Agent Substrate executes untrusted actors at scale in a multi-tenant environment where workloads are dynamic and memory states migrate continuously across physical infrastructure. All security controls operate within Substrate's cold-start and resume latency budget. This document defines the high-level security principles. For identity and credential configuration, see the [Authentication Guide](https://ate.dev/docs/guides/authentication/index.md). Detailed threat scenarios and mitigating invariants are documented [in the threat model](https://ate.dev/docs/security/threat-model/index.md). ## Security Boundaries ![Substrate security boundaries](https://ate.dev/docs/assets/security-boundaries.svg) The numbered tags mark the boundaries Substrate holds: 1. **Sandbox.** Each actor runs in its own gVisor or microVM sandbox, with its own network namespace and storage. Actor code cannot reach the node or other actors; it can only talk through its gateway path and DNS relay, where policy applies. 2. **Gateways.** Traffic reaches actors only through the ingress gateway, from authenticated callers. Outbound traffic leaves only through the egress gateway and DNS relay. The gateway enforces destination policies and injects credentials, so sandboxes hold none. 3. **Control plane.** Actors have no direct network path to the Substrate API, its database, or the Kubernetes API; their egress policy decides what external destinations or cluster services they can reach. Worker pods cannot reach the Substrate API or bundled database. API callers are authorized per atespace. 4. **Storage.** Sandboxes never touch snapshot storage. On GCS, the node supervisor moves an actor's snapshots using credentials scoped to that one actor (S3 per-actor scoping is planned), and only the control plane copies or deletes them. Your application is outside these boundaries: it authenticates end users and decides which actors they can reach. A more detailed diagram is available in the [threat model](https://ate.dev/docs/security/threat-model/index.md#architecture-of-substrate). ## Security Scope Substrate provides the foundational base for agent security: - Sandboxing: built-in compute, storage, and networking separation. - Identity: identity issuance is built-in and designed to federate with other identity systems. See the [Substrate Identity Critical User Journeys](https://docs.google.com/document/d/1seJNYQmsQYLCrYYzB9AmuxpSGSPCC3-u3tPNFsK2edc/edit?resourcekey=0-D6q8_0vqF6LPEk0BxWMTaA&tab=t.0#heading=h.6qjmtncwtihi). - Gateway: built-in ingress and egress filtering that can do destination filtering and credential injection. We expect the application layer creating actors using substrate to be responsible for: - Handling human-to-actor authentication and authorization. - Registering and maintaining information about agents in an agent registry. - Maintaining agent hierarchy/state to be able to reconstruct agent lineage: who made this sub-sub-sub agent. ## Security Extensibility Substrate provides extension points for operators and vendors to augment or replace the built-in ingress and egress gateways. See [Extending Substrate](https://ate.dev/docs/security/extensibility/index.md). ## Core Security Principles **Status Legend:** - Met: Fully implemented in the current codebase. - In Progress: Partially implemented or under active development. - Future: Planned architecture and roadmap capabilities not yet implemented. - Out of Scope: Not substrate’s job, handled by a layer outside substrate. ### Defensible Multi-Tenancy Substrate is designed to safely run mutually hostile, untrusted actors from disparate tenants side-by-side on the same physical host to maximize resource utilization and bin-packing efficiency. | Principle | Description | Status | | :--- | :--- | :--- | | **Always Sandboxed** | Every actor is required to run inside a strong compute sandbox. A linux container is not sufficient or supported. Our security design relies on strong containment. | Met (gVisor, CHV microVM) | | **Per-Actor Storage Segregation** | Storage and persistent volume state must be strictly segregated per actor. Actors cannot traverse, enumerate, or access persistence belonging to other actors. Restoring a snapshot must re-validate the actor's current tenant authorization before execution resumes. | Met (Segregation)
Future (Per-Actor Encryption, Restore Authorization) | | **Per-Actor Network Isolation** | Guest network stacks are segregated per actor. Lateral peer-to-peer network communication between co-located actors on the same host or across the cluster is blocked by default; inter-actor communication must route through authenticated, policy-enforced gateways. | Met (Private netns, atunnel) | | **Delete Guarantee** | Reset, scrub, and sanitize local ephemeral volumes, cached state, and worker runtime environments between actor scheduling cycles to prevent cross-tenant data leakage. | In Progress (atelet resetActorDirs) | | **Worker Capability Reduction** | Worker pods use `privileged: false`, `allowPrivilegeEscalation: false` and fixed runtime-specific capability sets. They still run as root with `SYS_ADMIN` and `NET_ADMIN`; see [Worker security](https://ate.dev/docs/reference/api-guide/index.md#worker-security). | In Progress (root with `SYS_ADMIN`; no worker user namespace yet) | | **Node Blast Radius Containment** | A compromise of an individual sandbox guest or node supervisor restricts impact to the actors and identities actively co-located on that node. | In Progress (Node-attested PodIdentity, checkActorOnNode) | | **Atespace Separation** | The Substrate equivalent of a Kubernetes namespace that is used to group resources and partition object storage snapshot paths. OpenFGA relationship-based access control (ReBAC) is integrated for API authorization, but it does not yet enforce inter-atespace network policies or resource quotas. | In-Progress (OpenFGA ReBAC; network policies and quotas pending) | Other forms of tenancy: - **[Possible, but not currently supported] Multiple Substrate installations inside a single Kubernetes cluster**, with nodes separated by taints/tolerations. This gets you Substrate API separation but gives up some node bin-packing efficiency. Care needs to be taken at the Kubernetes layer to keep everything separate. - **[May be possible, but not currently supported] Attaching multiple Kubernetes clusters to a single Substrate installation**. This may be possible with some work today, but is not supported. We’re expecting a single large K8s cluster to be capable of running 100s of millions of mostly-idle agents. - **[Out of Scope] Multi-tenancy inside an actor**. There is no meaningful security boundary inside an actor. Actors are designed to be cheap and efficient enough to run many, including one per human user, so that it’s possible to run them without handling content for multiple users inside the same actor. - **[Not Currently Supported] Per-Actor ingress/egress**. Actor-facing traffic proxies (`atenet-router` for ingress, `atenet-egress` for egress) are shared across all actors in the substrate installation by default. Dedicated proxy-per-actor won't scale to large numbers of actors. Multiple proxies, such as per-atespace, could be used in the future. ### Identity | Principle | Description | Status | | :--- | :--- | :--- | | **Zero Ambient Credentials** | Default actor sandboxes execute completely credential-free. Actors may hold credential identifiers, encrypted credentials, or unprivileged tokens that need to be hydrated or decrypted out-of-band by Substrate infrastructure before accessing external services. | Met (no credentials inside sandbox) | | **Distinct Workload Identity** | Every actor receives a short lifetime (1h) SPIFFE ID X.509 certificate and on-demand actor JWTs for external and egress authentication. It is possible to distinguish an actor that was deleted and then re-created with the same name using their UID. | Met (X.509, on-demand JWTs, OIDC discovery) | | **Universal Component mTLS** | All control plane and data plane components (`ate-api-server`, `atelet`, `ateworker`, routers, tunnels) authenticate and communicate exclusively over mutual TLS using node- and service-attested identities. Unauthenticated traffic is rejected, exempting only local unauthenticated health/liveness probes (`/healthz`, `/readyz`). | Met (Mutual TLS) | | **Sub-Actor Provenance** | Delegated sub-actors inherit cryptographically signed lineage. Every tool invocation, data access, and egress transaction can be deterministically resolved to the root actor and execution chain. This provenance/hierarchy needs to be recorded in the system that calls substrate. | Out of Scope | ### Storage | Principle | Description | Status | | :--- | :--- | :--- | | **Untrusted Storage** | Persistence layers are treated as untrusted environments. Remote storage is protected with encryption. Local snapshot caching on storage attached to the node is multi-tenant and protected from cross-tenant attacks with encryption and strict filesystem permissions. | Met (Plugin isolation, SHA-256 asset verification)
Future (Per-Actor Encryption) | | **Dual-Party Snapshot Authorization** | Reading or writing snapshot storage requires dual-identity verification proving both the actor's identity and the node supervisor (`atelet`) operating on its behalf. Sandboxed guests cannot directly interact with snapshot storage. | Met (Supervisor-only storage access, Per-actor token exchange on GCS)
Future (Per-actor tokens on S3) | ### Network | Principle | Description | Status | | :--- | :--- | :--- | | **No Access by Default** | Sandboxed actors and worker nodes launch with default-deny networking and have zero network or API access to control plane services (`ate-api-server`, Kubernetes API servers, PostgreSQL state store). DNS is filtered against allowed policy hostnames. Every control plane pod in `ate-system` also denies [ingress by default](https://github.com/agent-substrate/substrate/blob/main/manifests/ate-install/ate-system-networkpolicies.yaml), admitting only the flows the control plane needs. | Met (Loopback, private netns, link-local block, DNS relay filtering, ate-system NetworkPolicies) | | **Authenticated and Policy-Enforced Ingress** | Inbound invocations to actors terminate at authenticated edge routers (`atenet-router`), which authenticate callers and verify invocation policies before triggering on-demand actor cold-start or resume. | Met (atenet-router mTLS, atunnel mTLS) | | **Fail-Closed Policy-Enforced Egress Gateway** | Outbound traffic is funneled through the multi-tenant egress proxy (`atenet-egress`), which validates actor identity, enforces destination host allowlists, executes TLS inspection or prompt filtering, and dynamically injects target credentials. If an egress policy enforcement point (PEP) is unconfigured/fails egress is dropped. Unsupported protocols are dropped; see [Egress Traffic](https://ate.dev/docs/guides/egress-traffic/index.md), [Network Egress](https://ate.dev/docs/guides/network-egress/index.md), [MITM Trust Bundle](https://ate.dev/docs/guides/egress-trust-bundle/index.md), and [Egress Credential Injection](https://ate.dev/docs/guides/egress-credential-injection/index.md) for specifics. | Met (atenet-egress fail-closed PEP, DNS filtering, non-TCP drop) | ### Authorization | Principle | Description | Status | | :--- | :--- | :--- | | **Actor to Actor Authorization** | Built-in authorization policy for agent to agent communication will be added in the future. Before that exists users can install an authenticating reverse proxy, or implement authorization at the application level. | Future | | **Atespace and Node-level containment** | Actors are grouped into logical Atespaces for declarative RBAC, resource quotas, and independent metadata/label authorization to prevent privilege escalation. | In Progress (embedded OpenFGA) | | **Comprehensive Quotas & Anti-DoS Throttling** | The platform enforces strict limits across API call rates, memory/CPU allocations, and egress bandwidth to eliminate runaway actors and shared resource exhaustion. | Future | ### Auditability and Response | Principle | Description | Status | | :--- | :--- | :--- | | **Structured Audit Trail** | All lifecycle transitions (start, suspend, resume, terminate), snapshot access, gateway routing events, and policy violations generate structured telemetry linked to cryptographic actor identity and parent lineage. Token log leaks are prevented with sanitization. | Met (OTLP Weaver registry, credential redaction) | | **Dynamic Quarantine & Snapshot Tainting** | The control plane supports immediate revocation and network blackholing of misbehaving/compromised actors. Suspended states from compromised actors receive an immutable cryptographic taint, preventing resumption across the fleet. | Met (Dynamic egress revocation, lifecycle halt)
Future (Snapshot tainting) | ================================================================================ SECTION: SECURITY PAGE: Threat Model URL: https://ate.dev/docs/security/threat-model/ MARKDOWN: https://ate.dev/docs/security/threat-model/index.md SOURCE: docs/security/threat-model.md ================================================================================ # Threat Model > Assets, trust boundaries, and threats considered for Agent Substrate. Last updated: Oct 29, 2026 # Overview Agent Substrate orchestrates stateful AI agents at high scale and density on Kubernetes, multiplexing untrusted, mutually hostile actors across a shared pool of worker Pods while maintaining sub-second cold-start and snapshot resume latency. Because actors execute untrusted code and migrate across physical infrastructure, Substrate enforces VM-grade sandboxing (`gVisor` and `Cloud Hypervisor` microVMs), per-actor cryptographic identity (`SPIFFE` X.509 certificates and JWTs), zero ambient credentials inside sandboxes, default-deny networking with authenticated ingress and egress proxies, and relationship-based access control (`OpenFGA`). This threat model catalogs the security threats inherent to high-density stateful agent orchestration and defines the mitigating invariants and suggested mechanisms required to neutralize each threat. See the [Security Guide](https://ate.dev/docs/security/security/index.md) for the high-level security architecture, principles, and implementation status. # Scope * While shown in diagrams for reference, this threat model does not cover application-layer agent frameworks or harnesses running inside Substrate actors. * This threat model does not cover vulnerabilities contained entirely within the underlying cloud provider, host Linux kernel, Kubernetes control plane, or upstream sandbox runtimes (`gVisor`, `Cloud Hypervisor`). # Trust Boundaries * The sandbox (`gVisor` or microVM) is the primary boundary between an actor and everything else. * The worker Pod is trusted with the actors it hosts. An escape into the worker compromises those actors, and workers are not a boundary between actors, since actors move between workers on resume (T-30). The worker Pod's own confinement is defense in depth for the boundary between the worker and the node, atelet, and other workers. * A threat that matters only after another control has failed, such as a sandbox escape or a storage compromise, is rated High priority at most. # Architecture of Substrate Components that appear in both diagrams are in the same positions. The first prepares an actor template. The second creates an actor from it and serves a request. ## Creating an actor template ![Substrate actor template creation](https://ate.dev/docs/assets/actor-template-diagram.svg) Creating a template runs a golden actor once, snapshots it, and keeps that snapshot. Actors created from the template start from it. | Step | What happens | What's checked | | :---- | :---- | :---- | | P1 | The application creates an actor template. `ate-api-server` stores it, and a background reconciler starts building its golden snapshot. | OpenFGA allows the caller to create templates in the atespace. Container images must be pinned by digest. | | P2 | The reconciler creates a golden actor in the template's atespace, picks a worker, and tells that node's `atelet` to run it. | `atelet` accepts only `ate-api-server`'s identity. | | P3 | `atelet` pulls the container image into the node's layer cache, which all actors on the node share. Nodes also fetch sandbox assets (such as runsc or guest kernel and rootfs) specified by the SandboxConfig. | The image is fetched by digest (using node credential provider config when available). Each sandbox asset must match the sha256 pinned in the SandboxConfig. | | P4 | `atelet` tells `ateworker` to start the sandbox. | The call stays on the node. | | P5 | `atunnel` asks `atelet` for the actor's certificate. | `atelet` accepts only workers on its own node. | | P6 | `atelet` asks `ate-api-server` to mint it. | It is minted only for the worker the actor is assigned to. | | P7 | Once the workload is ready and warmed up, the reconciler suspends the golden actor, and `atelet` checkpoints its memory and disk. | Only `ate-api-server` can order a suspend. | | P8 | `atelet` trades a JWT for this actor for storage credentials. | On GCS, the credentials cover only this actor's snapshot location (S3 uses the plugin's own pod credentials). | | P9 | `atelet` uploads the golden snapshot. | The sandbox never touches storage. | | P10 | The control plane copies the snapshot to the tag, deletes the golden actor, and marks the template ready. | The golden actor's local directories are removed when it is deleted. | The golden actor has no network egress unless the template sets a golden egress policy. ## Creating an actor and serving a request ![Substrate architecture and data flows](https://ate.dev/docs/assets/actor-call-diagram.svg) In **A**, an application creates an actor. In **B**, it sends the actor a request, which resumes the actor on a worker, and the actor then calls an external tool. | Step | What happens | What's checked | | :---- | :---- | :---- | | A1 | The application asks `ate-api-server` to create an actor from a template. | The caller is authenticated, and OpenFGA allows it to create actors in the atespace and use the template. | | A2 | `ate-api-server` saves the actor as suspended. It points at the template's golden snapshot without copying it, and nothing runs until its first request. | For bundled PostgreSQL, NetworkPolicy restricts access to `ate-api-server` with a client certificate; managed databases use provider network and IAM/TLS controls. | | B1 | The application sends a request for `/` to `atenet-router`. | The caller's mTLS identity must be on the router's allowlist. | | B2 | The router asks `ate-api-server` to resume the actor and return its worker. | OpenFGA allows the router (or an authorized user/editor) to resume actors in the atespace. | | B3 | `ate-api-server` tells the `atelet` on the chosen node to restore the actor. | `atelet` accepts only `ate-api-server`'s identity. | | B4 | `atelet` downloads the actor's snapshot. | On GCS, credentials grant read/write access to this actor's snapshot prefix and read-only access to the template's source tag prefix (S3 uses the plugin's own pod credentials). | | B5 | `atelet` tells `ateworker` to start a sandbox from the snapshot. If the node hasn't cached the image, `atelet` pulls it first. | The call stays on the node. | | B6 | The router forwards the request to `atunnel` in the actor's Worker Pod. | `atunnel` accepts only the router's identity, and NetworkPolicy admits only the router. | | B7 | `atunnel` delivers the request to the actor's sandbox. | It goes only to the actor named in the request. | | B8 | The actor calls a tool. nftables outside the sandbox send the connection to `atunnel`. | DNS answers only names in the actor's egress policy. Other protocols are dropped. | | B9 | `atunnel` forwards the call to `atenet-egress` with a certificate for this actor. | The certificate is issued only to the worker the actor is assigned to. | | B10 | `atenet-egress` fetches the credential the egress rule names. | Only `atenet-egress` can reach the credential provider. | | B11 | `atenet-egress` calls the external service and injects the credential. | Destinations not in the actor's egress policy are refused. The credential never enters the sandbox. | # Key Components * **Actor:** The core stateful compute primitive, grouped into atespaces and scheduled onto workers via Run or Resume. * **Actor Network:** Private per-actor network namespace with a point-to-point veth (gVisor) or tap device (microVM). nftables rules outside the sandbox send all of its traffic to `atunnel`. * **Actor Routing:** Ingress requests identify the target actor via `ate-target-actor: /`. * **ate-api-server:** Control plane API that manages actor lifecycle, scheduling, OpenFGA authorization, and actor credential minting. Its `objectstore-plugin` sidecar copies and deletes snapshots and issues per-actor storage credentials. * **ate-controller:** Kubernetes controller that turns WorkerPools into worker Deployments and their NetworkPolicies. * **ate-idp-server:** Serves the public OIDC discovery document and JWKS that relying parties use to verify actor JWTs. * **atelet:** Per-node supervisor (DaemonSet) that coordinates sandbox lifecycle and checkpoint/restore, and brokers credentials for the actors on its node. Its `objectstore-plugin` sidecar moves snapshots. * **atenet-egress:** Multi-tenant Envoy egress forward proxy that enforces declarative egress policies and injects credentials. * **atenet-router:** Envoy ingress proxy with an `ext_proc` processor that authenticates callers over mTLS, resolves worker assignments, and routes to `atunnel`. * **ateworker:** Runtime daemon in each Worker Pod. It runs the sandboxes (`gVisor` or `Cloud Hypervisor` microVMs) and includes `atunnel`. * **atunnel:** Terminates mTLS ingress from `atenet-router`, tunnels each actor's redirected outbound TCP to `atenet-egress`, and relays DNS filtered by egress policy. * **Filesystem Support:** Container rootfs overlays and working directories captured in snapshots and wiped on actor teardown. * **k8s-credential-provider:** Gives `atenet-egress` the Kubernetes Secrets that egress rules reference. * **Kubernetes:** The underlying cluster infrastructure providing node isolation and network policies. * **Object Storage:** A snapshot bucket, partitioned by atespace and actor (scoped per actor on GCS), and storage locations for sandbox assets specified by the SandboxConfig. * **podcertcontroller:** Substrate's Kubernetes controller, in the `podcertificate-controller-system` namespace, that issues short-lived mTLS certificates for pod identity, service DNS, and database access. * **Substrate Database:** PostgreSQL (or a managed PostgreSQL service). Bundled installs restrict network access to `ate-api-server` over TLS with a client certificate; managed installs use provider network and IAM controls. * **Worker Pod:** Pre-provisioned Pod that hosts many actors, each in its own sandbox. # Threats and Mitigations **Table Schema:** * **Priority:** Critical, High, Medium, Low * **Threats:** Expected risks in a naïve implementation of Substrate. * **Mitigating Invariants:** High-level properties which, if true, mitigate the threat. * **Suggested Concrete Mitigations:** Specific options and architectural mechanisms for implementing the mitigating invariants. * **Notes:** Contextual notes, edge cases, and design considerations. Implementation status is intentionally not tracked in this table. High-level principle status is maintained in the [Security Guide](https://ate.dev/docs/security/security/index.md), while threat-by-threat mitigation quality is dynamically evaluated against the codebase by automated security review agents in CI (see `.agents/skills/security-status-report/`). Recording static status claims in this reference catalog would risk biasing automated evaluators away from independently verifying the code. ## Attacks from External Networks Substrate is expected to sit behind the user's service, not on the internet, so these threats assume that outer layer has failed. | Threat ID | Priority | Threat | Mitigating Invariants | Suggested Concrete Mitigations | Notes | | :---- | :---- | :---- | :---- | :---- | :---- | | T-01 | High | External attacker can access actors over the internet | Access to actors over the external internet is blocked by default. | ClusterIP-only Services. Default-deny ingress NetworkPolicies for ate-system and worker pools. Exposing anything outside the cluster is an explicit operator choice. Requires a CNI that enforces NetworkPolicy. | | | T-02 | High | External attacker can access nodes over the internet | Access to nodes over the external internet is blocked by default. | No hostNetwork, hostPort, NodePort, or LoadBalancer in Substrate components. Cluster setup tooling creates private nodes. | Node firewalls are cluster configuration and outside Substrate's scope, but are recommended. | | T-03 | High | External attacker can access Substrate API or backend database over the internet | Access to Substrate API over the external internet is blocked by default. | ClusterIP-only Services. Default-deny ingress NetworkPolicies for ate-system; only ate-api-server can reach the database. Exposing the API outside the cluster is an explicit operator choice. Requires a CNI that enforces NetworkPolicy. | | ## Attacks from Internal Network, API Clients | Threat ID | Priority | Threat | Mitigating Invariants | Suggested Concrete Mitigations | Notes | | :---- | :---- | :---- | :---- | :---- | :---- | | T-04 | Critical | Access to the internal network allows arbitrary actions to be performed on ate-apiserver, atelet, substrate backend database, etc. | All system components must have basic mutual authentication and authorization, and communicate over TLS. All clients (including end users and actors) must be authenticated and authorized. Unauthenticated traffic must be rejected. Where possible, use firewalls to block access from ranges that have no business communicating with Substrate components. | mTLS or a UDS between components, with each caller's identity checked. Per-node identity for atelet. Ingress checks the caller's permission on the target actor before resuming or forwarding. Only ate-api-server can reach the database. Default-deny NetworkPolicies for ate-system. | | | T-05 | High | Privilege escalation via access to sensitive labels. | If Substrate offers its own resource labeling mechanism, it must also offer a way to authorize label updates on a per-label basis. | Restrict who can change labels that drive behavior, such as the WorkerPool labels that `worker_selector` matches. Authorize label updates separately from the resource body. | Kubernetes let the /status subresource update labels. Don't repeat this. | | T-06 | High | Attacker gains control of the Substrate API server or an ingress/egress gateway. | Isolate the control plane from the data plane, and isolate data plane ingress from sandboxes. | Run control-plane components and gateways on separate nodes from sandboxes. Block network access from workers to the control plane. mTLS between all components. | Separation also limits noisy-neighbor and reliability problems. | | T-07 | High | Attacker who can create ActorTemplates specifies malicious runtime. | Ensure available runtime can only be configured by administrators. | Configure runtimes in admin-owned cluster resources, similar to RuntimeClass, not in ActorTemplates. | | | T-08 | High | Attacker who can create ActorTemplates can read or write any storage buckets atelet has access to. | Ensure that bucket access is least-privilege. | Per-actor, downscoped credentials for snapshot transfers. No storage credentials on atelet or nodes. Derive storage locations internally instead of accepting arbitrary URLs. Verify asset digests. | Example: a template points its snapshot or asset URL at an internal bucket to read from it or write to it. | | T-09 | Medium | DoS attack against API, gateway, or available cluster resources. | Minimize exposure and ensure APIs and proxies implement appropriately scoped quotas and rate limiting. | Keep the API and gateways off the internet and unreachable from sandboxes. Run control-plane components on separate nodes from sandboxes. Rate limits and quotas in the API and gateways, including per-user actor quotas. Consider an identity-aware WAF if external access is required. | Bound the API load from gateway assignment lookups during traffic floods. | | T-10 | Medium | Internal network traffic is intercepted or spoofed | Encrypt all traffic by default | mTLS between all components, including gateway to worker. | | ## Misconfiguration Risks | Threat ID | Priority | Threat | Mitigating Invariants | Suggested Concrete Mitigations | Notes | | :---- | :---- | :---- | :---- | :---- | :---- | | T-11 | High | Improper handling of Secrets | Ensure there is an official, secure, recommended way for actors to use secrets, like API access tokens, without the secret entering the sandbox. | Inject credentials at the egress gateway so secrets never enter the sandbox. | Without an official path, users put secrets in plaintext fields. | | T-13 | Medium | Flat namespace of actors encourages broad permission grants or complex graph-oriented policy. | Support a grouping mechanism that can be used in policy controls. | Atespaces as the policy grouping unit, like Kubernetes namespaces. | | | T-14 | Medium | DNS misconfiguration | Access to DNS configuration should be limited to authoritative controllers. Routing should use stable configurations and query the API for the current worker assignment before forwarding requests. | Keep controllers with sensitive access off actor nodes. Limit who can update DNS configuration. Gateways get the worker assignment from the Substrate API, with bounded caching. mTLS from gateway to worker. | Invalidate cached assignments when actors move. | ## Attacks from Actors | Threat ID | Priority | Threat | Mitigating Invariants | Suggested Concrete Mitigations | Notes | | :---- | :---- | :---- | :---- | :---- | :---- | | T-15 | Critical | Malicious actor gains access to the underlying node or other actors via container breakout (Linux local privilege escalation) | Actors are always sandboxed using a hardened sandbox solution like gvisor or microvm. Traditional containers are not a secure sandbox. | gVisor or microVM sandboxes. Defense in depth around the sandbox runtime: user namespace, pivot_root, seccomp, minimal capabilities. Low-privilege sidecars. Control sandbox lifecycle from outside the sandbox. | | | T-16 | Critical | Malicious actor gains access to the underlying node via node-local endpoints exposed on the network | Network policies must prevent the actor from accessing system services on the node (e.g. instance metadata, host network namespace, host network interface). | Per-actor network namespace, no host network access. Block node-local services such as instance metadata. Targeted ingress policies on host services. | | | T-17 | Critical | Malicious actor gains access to other actors via network | Network policies must deny ingress and egress by default, and selectively allow access to specific actors only when necessary. Network policies must be synchronized with actor lifecycle. | Default-deny traffic between actors, enforced by the worker so policy updates are fast. | | | T-18 | Critical | Malicious actor gains access to the underlying node or other actors via filesystem | Filesystem access is limited to the actor's local fs and remote filesystems the actor is directly, explicitly authorized to access. | Isolate actor filesystems with user namespaces or per-actor UIDs. Guard filesystem setup and APIs against symlink, mount, and path traversal. | Vectors: file-read bugs, symlink and path traversal, OCI image unpacking, shared directories on root or privileged workers. | | T-19 | Critical | Malicious actor gains access to the underlying Worker Pod or node via local Substrate services (atelet, ateworker, etc). | Actor access to system services must be denied or explicitly scoped to the actor. Local services must run with the fewest privileges possible to limit the ability for an actor to escalate privilege. | Limit actor access to local services. Local services authenticate the actor and authorize by its identity. | | | T-20 | Critical | Malicious actor gains access to the underlying node or other actors via Substrate APIs (remote ate-apiserver) | Either actors are completely unable to access Substrate APIs, or actor access is authenticated and authorized such that the actor cannot escalate beyond its intended scope. Especially prevent self-modification, which was a common escalation path in K8s. | Don't let actors or workers modify themselves, e.g. read or write their own snapshots or change their own labels. Keep resource-modifying logic in the control plane. | | | T-21 | High | Attacker who has escaped the sandbox escalates to the node, atelet, or other workers via excessive Worker Pod permissions | Worker Pods hold only the permissions sandbox lifecycle needs, and each permission beyond the container defaults has a documented reason. A worker can't modify or execute anything that atelet or other workers trust. | Run worker Pods non-root or in a user namespace, unprivileged, with only required capabilities and devices. A tailored seccomp profile instead of Unconfined. No ServiceAccount token. Mount only the worker's own host paths, read-only where it doesn't write. Verify shared assets before every use. | Rated High priority because it requires a sandbox escape first (T-15). Until workers run in their own user namespace, they may run as root, hold the capabilities the sandbox runtime needs (for example, `SYS_ADMIN` for mounts and `pivot_root`, and `NET_ADMIN` to build the actor network), and run with AppArmor Unconfined, because the default profile denies `mount`. By design, a worker can observe and modify the actors it hosts (such as pause containers that share a cgroup). | | T-22 | Critical | Malicious actor gains access to the underlying node or other actors via Kubernetes APIs | Either actors are completely unable to access the Kubernetes APIs, or actor access is authenticated and authorized such that the actor cannot escalate beyond its intended scope. **Strong preference on blocking actor access to Kubernetes.** | Block network access to the Kubernetes API. Give worker Pods no service account token, or one with no privileges. | | | T-23 | Critical | Malicious actor workload gains access to snapshots of other actors and steals data from them. | Worker Pods and actors must not have direct access to snapshots. | Make snapshot credentials usable only by atelet, via extra claims or binding to atelet's mTLS. Envelope encrypt snapshots, keyed to actor identity. Avoid snapshotting credentials. | | | T-24 | Critical | Malicious actor workload overrides snapshots of other actors with malicious snapshots. | Worker Pods and actors must not have direct access to snapshots. | Make snapshot credentials usable only by atelet, via extra claims or binding to atelet's mTLS. | | | T-25 | Medium | Malicious snapshot is restored | Cryptographically verify each snapshot before restoration | Check snapshots against a trusted digest or signature before restore. Deliver the digest or key over a trusted channel. | Rated Medium priority because it requires control of snapshot storage, or of the path to it, which is a failure of other controls (T-23, T-24, T-36). Those with that access are mostly storage administrators, as in T-39. | | T-26 | Critical | Stale policies propagated out-of-band with actor scheduling result in incorrect access boundaries (including network policies, IAM permissions, etc.) | An actor's network and authorization policy is in effect before it starts running. Enforcement points may cache policy, as long as a policy change takes effect within a short time. An enforcement point that can't get current policy denies rather than serving stale policy indefinitely. | Program authz and network policy as part of actor create and resume, or fetch it on first use and deny until it arrives. Short cache TTLs. Invalidate caches on policy change where it's cheap. | Caching is required for performance, since enforcement points check policy on every connection or query. Operators may tune cache TTLs. Policy applies to new connections; a policy change doesn't cut connections that are already open. Credentials already issued are limited by their own lifetime (T-28), not by policy propagation. | | T-27 | Critical | Worker reuse enables malicious actor to escalate to other actors by persisting a threat across reuse, reading leftover state from a previous actor, or taking advantage of stale policy configuration. | All actor-specific worker state, including process state, filesystem, env vars, mounted config, network policy, and security policy, must be completely reset between actors that subsequently run on the same worker. | Synchronize local policy updates with actor lifecycle. Test cleanup across suspend/resume for each sandbox type, with honeypots on both sides. | | | T-28 | Critical | Malicious actor tricks Substrate identity broker into returning identity credentials for a different actor. | Add defense-in-depth to ensure that credentials cannot be mis-issued, and that mis-issued credentials are unusable in practice. | Check the actor's worker assignment before issuing credentials. Bind credentials to the worker and node, check those claims in authz, and invalidate old credentials on reschedule. | | | T-29 | High | Agent leaks credentials exposed in sandbox, because LLMs are unreliable. Due to prompt injection or just agent silliness. | Credentials are not exposed in sandboxes by default. | No credentials in sandboxes by default. Inject credentials at a proxy. Delegate to drivers (like CSI) rather than sandboxed clients. Credentials that must enter the sandbox are wrapped and only usable through the proxy. | Exposing credentials can be opt-in when the process is not an AI agent or can't read them. | | T-30 | Medium | Malicious actor moves laterally to other nodes via suspend/resume, especially if self-suspend is allowed. | Actor isolation doesn't depend on which node an actor runs on. Admins can confine actors to specific node pools. | Let admins restrict which node pools an actor can resume on. | Actors move between workers by design, so nodes aren't a boundary between actors. This only matters combined with a sandbox, network, or filesystem escape. | | T-31 | High | DoS/resource exhaustion via malicious container image | Enforce limits when pulling and extracting container images | Limit uncompressed layer size and unpack time. | Example: a zip bomb. | | T-32 | Medium | Compromised process inside actor uses actor identity to write its own snapshot, enabling local privilege escalation (still in actor sandbox). | Issue separate credentials for accessing snapshots. | Snapshot access requires an extra claim, audience, or second identity proving atelet is acting for the actor. | Medium because it's still a local exploit inside the sandbox. Example: a nonroot process rewrites the snapshot so it can `su` to root after restore. | | T-33 | Medium | Attacker creates a large number of malicious actors to rapidly spread across the cluster or exhaust resources (via direct create or fork-bomb equivalent). | Limit resource consumption and number of child actors for each actor. | Make actor creation a privileged permission. Rate limits and quotas that count child actors and their resources, attached to atespaces. | | | T-34 | Medium | Attacker discovers Kubernetes-internal network topology from inside sandboxed actor | Don't allow actors to explore the cluster network | Don't expose cluster DNS to actors. Give actors their own resolv.conf, not the worker Pod's. | | | T-35 | Low | Attacker discovers enough information about actor names or namespace names to be consequential, from inside a sandboxed actor. | Only expose actor names for peers the actor needs to interact with. | Don't expose Substrate-internal DNS to actors by default. | What an actor can send traffic to matters more than the names it learns. | ## Attacks from Nodes | Threat ID | Priority | Threat | Mitigating Invariants | Suggested Concrete Mitigations | Notes | | :---- | :---- | :---- | :---- | :---- | :---- | | T-36 | High | A compromised node accesses all snapshots for the cluster. | Node access to snapshots must be scoped to actors actively scheduled to the node. | Per-actor snapshot credentials, issued only to the node the actor is scheduled on and downscoped to that actor's snapshots. No node-wide storage role. Write-once snapshot versions. Wipe previous actors' data from the node on suspend. | Detecting nodes that request snapshots for actors not scheduled on them adds defense in depth. | | T-37 | High | Compromised node accesses filesystem storage for actors on other nodes. | Node access to network filesystems must be scoped to actors actively scheduled to the node. | Per-actor filesystem credentials, issued only to the node the actor is scheduled on. | Depends on what the network filesystem supports. | | T-38 | High | Compromised node can escalate to other nodes or control plane via Substrate or Kubernetes. | Node access to Substrate or Kubernetes APIs must be scoped to operations relevant to actors actively scheduled to the node, and to the node itself. | Scope atelet's API access to actors on its node. Tight NetworkPolicies between data-plane and control-plane components. Limit what node identities can do in Kubernetes, as NodeRestriction does. | | ## Insider Attacks | Threat ID | Priority | Threat | Mitigating Invariants | Suggested Concrete Mitigations | Notes | | :---- | :---- | :---- | :---- | :---- | :---- | | T-39 | Medium | Insider access to snapshots | Substrate's storage models for snapshots must enable users to grant granular access control when needed for system administration, rather than only supporting broad access across all actors. | Encrypt snapshots so storage is untrusted and only the node running the actor can decrypt them (T-23). A storage layout that maps to per-actor identity, so access can be granted per actor. | | | T-40 | Medium | Insider access to disks, e.g. for substrate backend DB | Support envelope encryption of sensitive data, or avoid storing sensitive data. | Full-disk encryption, which most clouds enable by default. Keep secrets out of Substrate's database, and envelope encrypt any that must be stored, with HSM-backed keys. | Credentials are injected at egress, not passed to actors (T-29), so actor specs such as environment variables are not a place for secrets. | ## Detection and Response | Threat ID | Priority | Threat | Mitigating Invariants | Suggested Concrete Mitigations | Notes | | :---- | :---- | :---- | :---- | :---- | :---- | | T-41 | High | Malicious actions are not recorded for forensics. | Enable audit logging for all Substrate components that serve APIs, including node-local components. | Every component that serves an API or accesses snapshot storage logs each request with the caller's identity, so the cluster's log collection captures it. | Data-access logs on busy snapshot buckets are impractical, so rely on logs from the components allowed access. | | T-42 | Medium | Malicious activity proceeds undetected. | Enable integration with threat detections systems. | Pluggable threat detection fed by telemetry from the API, node components, and sandboxes. Optional per-node or per-worker security sidecars. | | | T-43 | Medium | Detected malicious actor cannot be contained. | Quarantine or suspension option. | Dynamic policy updates for fast quarantine. Snapshot and suspend on detection, and taint the snapshot so it can't resume automatically. | | ================================================================================ SECTION: SECURITY PAGE: Extending Substrate URL: https://ate.dev/docs/security/extensibility/ MARKDOWN: https://ate.dev/docs/security/extensibility/index.md SOURCE: docs/security/extensibility.md ================================================================================ # Extending Substrate > Augment or replace the built-in ingress and egress gateways with custom policy enforcement points. Substrate ships with built-in sandboxing, actor identity, and ingress and egress gateways. Security vendors and platform teams can plug their own capabilities into the gateways, either to **augment** the built-in controls or to **replace** them. | Path | Ingress | Egress | | --- | --- | --- | | **Augment:** add your `ext_proc` callout to Substrate's gateway | Planned ([#3259](https://github.com/agent-substrate/substrate/issues/3259)) | Available (experimental) | | **Replace:** bring your own gateway | Planned ([#3258](https://github.com/agent-substrate/substrate/issues/3258)) | Partial | ## Augment: add an ext_proc callout You can add your own [`ext_proc`](https://www.envoyproxy.io/docs/envoy/latest/configuration/http/http_filters/ext_proc_filter) service. Substrate gateways call it on every request. - **Egress:** set `--experimental-additional-egress-extproc-service` (see [Operator install](https://ate.dev/docs/guides/operator-install/index.md#settings)). This may be converted to an API in the future. Your service runs after Substrate has authenticated the actor and applied its `EgressPolicy`. It currently receives the request headers and the actor's identity, and it can deny the request or add headers; this can be changed if necessary. Today it sees headers only, not bodies. - **Ingress:** planned. ## Replace: bring your own gateway - **Egress:** you can point actors at your own policy enforcement point (PEP) instead of `atenet-egress`. Your PEP must meet the [Network Egress Contract](https://ate.dev/docs/guides/network-egress/index.md): it must verify the actor's certificate and reject any traffic it can't attribute to a running actor. Honoring Substrate's `EgressPolicy` is encouraged but not required. The installer doesn't have a bring-your-own mode yet. - **Ingress:** planned. `atenet-router` does more than filter traffic: it wakes suspended actors and routes requests to their workers. A replacement router will need to call Substrate, either through the Substrate ext_proc or via the Substrate API, for those steps. If you replace a gateway, the authentication and authorization it enforced become your responsibility. ## Get help with integrations We want to help you integrate, and we're discussing other ways to extend Substrate beyond gateways. Talk to us in [#substrate-dev](https://cloud-native.slack.com/archives/C0B6M3E2J3D) on CNCF Slack, [open a GitHub issue](https://github.com/agent-substrate/substrate/issues/new), or join the weekly [community meeting](https://github.com/agent-substrate/substrate/blob/main/README.md#community). ================================================================================ SECTION: PROJECT PAGE: Community URL: https://ate.dev/docs/project/community/ MARKDOWN: https://ate.dev/docs/project/community/index.md SOURCE: docs/project/community.md ================================================================================ # Community > Connect with the Agent Substrate community on Slack, the mailing list, and weekly sync meetings. We welcome questions, feedback, and contributions from users and developers building on Agent Substrate. ## Communication Channels | Channel | Purpose | Link | | :--- | :--- | :--- | | **#substrate-users** | Questions, operational support, and discussions on running Agent Substrate | [Join on CNCF Slack](https://cloud-native.slack.com/archives/C0B6RCAJULW) | | **#substrate-dev** | Architecture, design proposals, and contributor development discussions | [Join on CNCF Slack](https://cloud-native.slack.com/archives/C0B6M3E2J3D) | | **ate-dev Mailing List** | Release announcements, RFCs, and asynchronous technical discussions | [Join Google Group](https://groups.google.com/g/ate-dev) | | **YouTube Channel** | Demos, walkthroughs, and recorded technical deep dives | [agent-substrate on YouTube](https://www.youtube.com/channel/UCN9PPqlTtVxlcpbQ-NWpfZQ) | > [!TIP] > Don't have a CNCF Slack account yet? Request a free invite at **[slack.cncf.io](https://slack.cncf.io/)**. ## Weekly Community Meeting We host an open community sync every week to discuss roadmap progress, design proposals, and user questions. * **Schedule:** Every **Thursday from 10:00 AM – 11:00 AM PT** (PST/PDT) * **Video Call:** [Google Meet (`meet.google.com/uhq-cxvn-dhy`)](https://meet.google.com/uhq-cxvn-dhy) * **Dial-in:** `(US) +1 253-289-6971` — PIN: `787 664 574 59#` ([international numbers](https://tel.meet/uhq-cxvn-dhy?pin=9044088223662)) * **Agenda & Notes:** [Weekly Sync Meeting Notes](https://docs.google.com/document/d/1obSIvfcafLNniLYTQCcT2eCgxHqa2AQ3Ga7YTsju49s) * **Archive:** [Meeting Recordings & Transcripts](https://drive.google.com/corp/drive/u/0/folders/1rX1S6vPxPrR8dA1lEBuBEXkGKjHtG-mL) ## Contributing Interested in contributing code, documentation, or bug reports? Review our **[Contributing Guidelines](https://github.com/agent-substrate/substrate/blob/main/CONTRIBUTING.md)** and **[Integration Repositories](https://github.com/agent-substrate/substrate/blob/main/docs/project/integration-repos.md)** guides to get started.