Always-on agents need to continuously watch and respond to events, from many more sources than it can keep track of in its context window. Comma's distributed harness, Salix, gives agents the capability to load tiny eBPF programs into the cluster and have them execute persistently, 24/7, to monitor events, call Jev for fast intelligent decisions, and wake up the main agent loop when something interesting happens.
The inner loop is not enough
The familiar agent loop is simple: send the context to a model, run the tool calls it returns, append the results, repeat. That loop is good at doing work. It is bad at waiting for work.
Ask an agent to “tell me when the contract comes back signed” or “watch this repo and flag anything that touches billing”, and the inner loop has two bad options. It can poll: wake up every few minutes, re-read the inbox, and spend a full model call deciding that nothing happened. Or it can hand the job to a heartbeat or cron schedule, which is the same polling with a longer interval and a worse latency. Either way, the expensive part of the system, a large model reading a large context, runs on every tick, and almost every tick is a quiet one.
What an always-on agent actually needs is an outer loop: something that sits on the event stream, does the cheap filtering, and only escalates to the model when there is something worth thinking about. And because every agent watches different things in different ways, that outer loop can’t be a fixed product feature. The agent has to write it.
In Salix, that outer loop is called a Loop.
A Loop is a small C program
A Loop is a single C file, written by the agent, compiled to eBPF, and run on the cluster next to the agent that owns it. Every Loop has the same shape:
It waits for a timer or an event, checks something, and, rarely, wakes the agent. The agent gets the exact SDK header and a programming guide from a loop.sdk tool, writes the program, compiles it with loop.build, and starts it with loop.create. There is no template and no DSL. If the agent can describe the watch in C, it can run it.
Why C and eBPF, rather than, say, a Python script in a container? Because of the numbers we designed Salix around. Salix runs millions of agents at more than 100 agents per CPU core. An agent can’t have a sandbox that stays on, so its watches can’t either. A Loop has to be cheap enough to keep running forever, for every agent, on a shared, multi-tenant node, while running code that no human reviewed.
eBPF fits that brief unusually well:
It is small. A waiting Loop is tens of kilobytes of resident memory. There is no process, no interpreter, no heap.
It is safe to run untrusted. The target has no syscalls, no function pointers and no unbounded stack. All a program can do is compute over its own memory and call the host functions we hand it.
It is cheap to wait. A Loop that is blocked in
sf_event_nextorsf_sleep_mscosts nothing but its memory.
The price is a restricted dialect, and the SDK guide is blunt about it: integer C only (no floating point, no signed division), 4 KiB stack frames, at most 8 frames deep, no recursion, no sprintf, and at most 128 live handles. Models turn out to be fine with this. They already know C, and the constraints are the kind that a compiler error explains well.
Spinfoam: the runtime under every Loop
Loops run on spinfoam, our eBPF loop runtime. Spinfoam is built on async-ebpf, a userspace eBPF runtime that is async-friendly, fully preemptive, and has formally verified memory safety at its core.
Userspace eBPF, made async
Spinfoam is not kernel eBPF. It is an ordinary Rust process, one per Salix node, that needs no kernel eBPF support and no privilege, and runs the same on Linux and macOS. Salix talks to it over JSON-RPC on stdin and stdout. The split of responsibilities is strict: spinfoam owns local execution, isolation between programs, cancellation and bounded message delivery. Salix owns everything that needs to survive a crash: placement, durable state, restart policy, credentials and the network.
The “async” in async-ebpf is what makes a Loop look like ordinary sequential C. Each loaded program gets one long-lived invocation, running on its own coroutine. When the program calls sf_sleep_ms, sf_event_next or sf_host_call, the helper suspends the coroutine and hands the wait to Tokio. The C stack and the program’s globals stay exactly where they were. When the timer fires, the event arrives or Salix answers the host call, the coroutine resumes on the next line. The agent writes for (;;) { wait; check; wake; } and never sees a callback.
“Fully preemptive” is what makes this safe to share. All guest code runs on a single Tokio thread, and a watcher thread interrupts any program that runs too long without yielding. A Loop that spins in a tight for (;;) {} loses its time slice like any other, and stopping it doesn’t need its cooperation. One buggy Loop can’t stall its neighbours.
Most Loops spend nearly all their time waiting, so this design packs densely. In our qualification run, 10,000 small monitoring Loops, all compiled by the embedded compiler, ran on one execution thread at about 64 KB of resident memory each. Loading and starting all 10,000 took 3.07 seconds. Delivering one event to every Loop, and answering the host call each one made in response, took 1.56 seconds. Control requests stayed under a quarter of a millisecond at p99. The whole process used four OS threads while loading and two when idle.
Memory safety you can check
Running thousands of agent-written programs in one process is only reasonable if none of them can touch memory that isn’t theirs. In async-ebpf, that guarantee starts with the memory layout and ends with machine-checked proofs.
The layout first. Each program’s data sits inside a pointer cage: a reserved region with randomized guard pages around it. The guest stack is laid out as islands of one frame each, separated by inaccessible gaps wider than any eBPF memory instruction can reach, so a function that walks off its own frame faults instead of reading its caller’s. JIT code pages are never writable and executable at the same time.
Then the proofs. async-ebpf compiles each eBPF function to native code the first time it runs. Its x86_64 backend is structured so the part that makes decisions can be proved:
The validator is sound. Along every execution of a program it accepts, the program never reaches an undefined instruction, never jumps into the middle of an instruction or out of the program, and the frame pointer is always the frame base for the current call depth.
Functions are closed. Control never leaves a function except through a local call or a return, which is what lets the JIT compile one function at a time.
The generated code is memory-safe. If the checker accepts a function, every execution of its native code touches only an allowed set of addresses (its frame, its own guest regions, its own stack and literal pool) and returns with the caller’s state intact. Lazily compiled callees compose with the same theorem.
The bytes say what the model says. Every instruction the assembler emits decodes back to the instruction the model reasoned about.
The proofs connect to the hardware in two places. An executable simulator is proved to be a run of the Lean machine model, and a differential test runs twenty thousand random instruction sequences through both the simulator and the real processor. Before each invocation, the runtime checks the concrete memory layout against the theorem’s hypothesis, and refuses to run if it doesn’t hold.
We are explicit about what is not proved. The trusted base includes the x86 instruction semantics (tested against hardware, not proved), the entry trampolines, the fault handler, the memory mappings, the host side of every helper call, and Charon and Aeneas themselves. The theorem is about memory safety, not functional correctness or information leaks. And it covers the x86_64 backend only; the arm64 backend is tested but not proved.
The proofs keep the guest in its own memory. Everything a Loop does in the outside world goes through a host call, and that boundary belongs to Salix: the capability allowlist and authority rules described below.
The compiler runs inside the sandbox too
Agents compile their own Loops, so the compiler is untrusted-input territory too. Spinfoam embeds TinyCC, compiled to eBPF, and runs it on async-ebpf like any other guest. Each build gets a fresh compiler instance with an 8 MiB arena, a memory-only file system holding just the submitted sources and the SDK header, a 15-second deadline, and no network, process or host-file access. The output is treated as untrusted as well. Every object goes through the same validation when it loads, whether it came from the compiler or not.
That is why loop.build needs no toolchain, no container and no privilege on the node. The agent’s C source never touches a native compiler.
Waking the agent is the expensive part
A Loop runs on its own, but it is useless if it can’t tell the agent anything. The one way it does that is agent.notify:
The notification is delivered to the Loop’s target Session as an ordinary message. Only then does the main agent loop run, and only then does it spend model tokens. A quiet hour costs no tokens at all.
The dedup_key matters more than it looks. Loops get restarted, events get redelivered, and a guest may retry a call it isn’t sure succeeded. Salix delivers the wake as loop:<id>:<dedup_key>, so the same mail, issue or alert wakes the agent once.
Because a wake is the expensive part, it is also budgeted. A Loop may wake its agent six times per ten minutes. A Loop that stays at that limit for an hour is paused with reason budget, and the agent is told. A buggy or over-eager Loop degrades into a visible, paused Loop, not into a bill.
Fast decisions without a large model
The hard part of a watch is usually not “did something change” but “does this change matter”. Is this email something the owner has to act on today? Is this PR touching the parts of the system we care about? A Loop written in integer C can’t answer that by itself, and calling a frontier model on every event would put us right back where we started.
So Loops get one more capability: decide. It sends a small, typed question to a fast decision model (Jev) and gets back a structured answer. Here is the question from our email-watch prototype:
decide supports choice for picking one candidate, separate yes/no questions for multiple matches, and score for ordered relevance. It only ever returns a decision. It does not read sources, run tools or grant authority. The Loop supplies the data, asks the question, and applies the threshold in its own code.
Two design choices made this work well:
Uncertainty is a valid answer. An explicit
noneordeferchoice means “I can’t tell”, and that is a successful result, not a transport error. The Loop keeps the event instead of fabricating a quiet decision.It is cheap enough to run on every event. Our pricing catalog lists
typesafe/jev-1.13.0at $0.042 per million input tokens, with free output tokens. That is cheap enough to screen every email, not just the ones a keyword filter lets through.
The result is a two-tier system: a small model screens every event, and the large model only sees the ones that pass.
Events in, durably
Timers are enough for polling, but many sources can push. A Loop has a mailbox, and external systems feed it in three ways:
the Salix API:
POST /v1/agent-groups/:group_id/loops/:loop_id/eventswith a group API key;a secret webhook URL, which the agent enables, rotates or revokes with
loop.webhook;Composio triggers from the user’s connected apps, which arrive with the provider’s event ID and trigger slug.
A 202 from any of these means PostgreSQL has retained the event on its Loop. It does not mean the Loop processed it. The guest calls loop.ack when it has reached a safe point: a quiet decision, or a durable handoff such as a successful agent.notify. Until then, the event stays pending, and the Reconciler replays pending events after the Loop moves or restarts, and once a minute in any case. An event that is not acknowledged within 15 minutes fails the Loop visibly, with the pending events kept for the agent to retry or discard.
One lesson from building this: spinfoam’s mailbox deduplicates event IDs at admission, not at business completion. In an early prototype, we expected a failed model call to be retried by the provider redelivering the event. It wasn’t; the redelivery was correctly dropped as a duplicate. So retries belong to the guest, which keeps the event and retries a bounded number of times, and recovery after a crash belongs to the durable inbox. We now write that rule into the SDK guide.
Acknowledgement, durable Session admission, a visible message and an external effect are four separate facts. We give each one its own owner, require stable deduplication keys downstream, and do not promise exactly-once external effects.
Running forever on a moving cluster
“24/7” is easy to say and hard to do on a cluster where nodes come and go. A Loop runs on the node that holds its agent’s lease, and it follows that lease.
Placement. When an agent’s server process claims its lease, it adopts its active Loops. When the agent is passivated or fenced, it releases them. An active Loop keeps its agent resident, so an idle agent with a Loop renews its lease instead of parking.
Incarnations. Every load of a Loop bumps its incarnation, a fence value on the Loop’s row. A host call from an object that is not the current incarnation is refused, and so is an
ackfrom one. A stale copy of a Loop on a departing node can’t act.Checkpoints.
loop.state.putsaves up to 16 KiB, and the next load gets it back asconfig.state. Agents checkpoint cursors and last-seen IDs, not everything.Faults. A fault reloads the Loop from its checkpoint, at most three times an hour. After that it is
failed, and the agent is told once.Stranding. A periodic sweep finds active Loops with no object attached, or attached on a node that left, and restarts their owner.
Loops also have a source of truth outside the runtime. loop.build writes the compiled ELF to the agent’s file system, and loop.create records its path and SHA-256. Every load rereads the file and checks the hash, so a Loop runs exactly the program it was created with, or not at all.
What a Loop is allowed to do
A Loop is code written by a model, running without supervision, around the clock. Its authority has to be smaller than the agent’s, not equal to it.
Loops call a closed allowlist of host capabilities: agent.notify, the loop.state.*, loop.ack and loop.log calls, Salix tools classified as read-only, environment and device tools, SSH tools, reading a conversation, web.http_request, composio.execute for connected apps, and decide. Spinfoam refuses every other name with SF_DENIED. There is no grant step to get wrong.
Each call goes through the same tool dispatch and information-flow checks as a normal agent turn. A Loop acts as its creator, through a delegated principal schedule|loop:<id>|<creator>, with the creator’s disclosure rules and the Loop’s sealed origin. Event payloads are data. They carry no authority and can’t widen what the Loop may do, which is why the decision prompt above explicitly treats the email as untrusted input.
Finally, quotas keep the system bounded: 20 active Loops per agent, 100 per group, 32 pending events of 16 KiB each per Loop, and rate limits on webhook ingress.
Scripts: the same runtime, for one turn
Once we had a safe, cheap runtime for agent-written C, a second use appeared. script.run compiles and runs an integer-C program once, inside a single tool call, with access to the calling turn’s own tools through salix.call. A script has no row, no incarnation, no checkpoint and no agent.notify. It is a way for an agent to batch many tool calls into one deterministic program instead of many model round-trips. Two of the skills we ship are C programs run this way.
Agents that program their own event system
Put together, a Comma agent that is asked to keep watch does three things that traditional agents can’t: it writes the watch as a program, the cluster runs that program for as long as it is needed at the cost of a few kilobytes, and a small model decides, event by event, whether the large model needs to wake up at all.
The inner loop is still where the agent thinks. The outer loop is where it pays attention. Salix lets the agent build both.