Slot lifecycle reference

The canonical state machine lives in hal0.slots.state and is used by SlotManager, the dashboard SSE stream, and the state.json persistence layer.

States

State	Meaning
OFFLINE	Not running. No systemd unit active.
PULLING	Model files are being downloaded or verified. systemd unit not yet started.
STARTING	systemd unit has been started; waiting for the container to come up.
WARMING	Container is up; the health probe is returning non-ready responses while the model loads into VRAM/GTT.
READY	Passed the full health probe (non-empty /v1/models plus a sentinel completion). Ready to serve.
SERVING	An inference request is actively in-flight on this slot.
IDLE	Reached two ways, both render identically on the wire: warming → idle (process up but /v1/models is empty — launched with --model "" or a missing file; routers must not treat this as ready), or ready → idle (no request for longer than the idle timeout — eviction candidate).
UNLOADING	Graceful shutdown in progress. systemd stop issued; waiting for the container to exit.
ERROR	Failed. Details in state.json and journald.

offline → pulling → starting → warming → ready ←──┐
                                   │      ↑       │
                                   │      ↓       │
                                   └──→ idle ←──serving
                                          │
                                          ↓
                                      unloading → offline
                                          ↑
                                        error

Legal transitions

LEGAL_TRANSITIONS is the authoritative map SlotManager enforces:

From	Legal next states
OFFLINE	PULLING, STARTING
PULLING	STARTING, ERROR, OFFLINE
STARTING	WARMING, ERROR, OFFLINE
WARMING	READY, IDLE, ERROR, OFFLINE
READY	SERVING, IDLE, UNLOADING, ERROR
SERVING	READY, IDLE, ERROR
IDLE	SERVING, UNLOADING, READY
UNLOADING	OFFLINE, ERROR
ERROR	OFFLINE, PULLING, STARTING (re-loadable, subject to the crash-loop breaker)

Dispatchable states

DISPATCHABLE_STATES = {READY, SERVING, IDLE} — the single source of truth used by SlotManager, the GPU arbiter, stack-apply, the Prometheus renderer, and the slot-view aggregator to decide whether a slot can take a request right now.

Selection ranking

When more than one candidate slot binds the same model, _SELECTION_ORDER ranks them:

1. DISPATCHABLE_STATES (ready/serving/idle) — answers now.
2. {WARMING, STARTING, PULLING} — will answer shortly.
3. {OFFLINE, UNLOADING} — needs a cold load.
4. {ERROR} — last resort (already failed once).

Unknown states rank as ERROR.

Typed errors

Each is an HTTP-status-bearing error under the slot.* namespace:

Error	HTTP status	Meaning
SlotNotFound	404	—
IllegalSlotTransition	409	Attempted transition not in LEGAL_TRANSITIONS.
SlotNotReady	503	—
SlotSpawnFailed	500	—
SlotHealthFailed	503	—
SlotCrashLooping	503	Reload blocked during exponential backoff, or after too many consecutive failures. Carries retry_after_s.
SlotTerminateTimeout	504	Stop didn’t return in time — the request unblocks anyway.
SlotConfigError	400	—
NpuExclusivityViolation	409	Only one device=npu, type=llm, enabled=true slot is allowed at a time (AMD XDNA hardware context limit).
SlotPinned	409	A pinned slot needs ?force=true to unload or delete.

Health-probe resolution on timeout: unresolved health checks resolve to WARMING, not ERROR — an ambiguous outcome is treated as “still loading,” not “failed.”

Persistence

SlotStateRecord (name, state, model_id, port, updated_at, message, extra) is written atomically (tempfile + fsync + os.replace) to /var/lib/hal0/slots/<name>/state.json, so SSE readers never observe a torn write.

Self-managed providers

SELF_MANAGED_PROVIDERS = {kokoro, qwen3tts, moonshine, vibevoice} — these providers serve a baked-in model, so no explicit model_id is needed to dispatch to them (provider_requires_model() returns False for this set). This set must stay in sync with the dashboard’s own SELF_MANAGED_PROVIDERS constant.

Activation

There is no enabled field on SlotConfig. A non-empty model.default is the activation signal — it is what makes a slot routable and loadable.

Boot autostart is a separate, explicit setting: autoload. The generated Quadlet unit carries [Install] WantedBy=hal0.target only when autoload = true, and that stanza is the only thing that starts a slot at boot. With autoload = false the unit still exists and is start-able (load/swap use systemctl restart, which never consults [Install]) — nothing just pulls it up on reboot. Binding a model no longer implies a boot start.

Slots written before the field existed (key absent on disk) migrate to autoload = true when they have a bound model, and false otherwise, so an upgrade preserves the old behaviour until you toggle it. Slots created through POST /api/slots always persist an explicit value, defaulting to false.

Editing autoload via PUT /api/slots/{name}/config re-renders the on-disk unit, so the change takes effect from the next boot without a manual reload.

Eviction priority

Every non-pinned slot is an eviction candidate. priority (int, 0–100, default 50) orders the victims for both memory-pressure eviction (SlotReaper.pressure_evict_once) and pre-load eviction (preload_evict): the lowest priority is unloaded first, with least-recently-used as the tie-break inside a priority tier. priority = 100 is not a pin — it is merely evicted last; pinned = true is what exempts a slot entirely, as do the default-pinned anchors (agent and friends). A slot with a request in flight is never a victim, and only resident slots (READY/IDLE) are considered at all.

Caution

The old lru = true eviction opt-in is retired. The key is still accepted in slot TOML but ignored, with a one-time slot.lru_flag_retired warning per slot — remove it and use priority/pinned instead. Slots that never set lru are now evictable where they previously were not.

The v1.0 write-boundary guard

Caution

New in v1.0 (Stream F, fix/write-boundary-enforcement — reviewed PASS, not yet merged to main at time of writing). Before this change, stacks/apply.py and SlotManager.create() could both write slot TOML in-process without ever routing through the HTTP-layer’s model-owned-key guard or the [server].extra_args hardware-flag screen — a real bypass of the model/slot key-ownership split described in Config schema.

guard_slot_write_payload (hal0.slots.config_write) is now the single shared in-process gate every writer of slot TOML must pass through — API routes, stacks-apply, and SlotManager alike. It partitions model-owned vs. slot-owned keys and screens extra_args for hardware flags, so a stack-apply or in-process create can no longer silently write a key the model is supposed to own. PATCH /slots/{name}/defaults correctly reports model-owned keys as such after this change.

For the CLI surface that drives slot lifecycle transitions directly, see CLI reference → hal0 slot.