Hermes Agent Crash Recovery Strategies

Hermes doesn't run as one program that either works or doesn't. It runs as a set of independent profiles, each with its own config, its own memory store, its own gateway process, and its own cron schedule, and that structure means the question after a crash is never "is the agent down." It's which layer broke while the other five kept running.

The real cost of an undetected agent failure

The failures that hurt the most are rarely the ones where Hermes gives a wrong answer. They're the quiet ones: a 429 that never got surfaced, a log that grew past a limit nobody set, an API key that expired at 2 a.m. while the gateway kept accepting requests. None of those look dramatic in the moment. None of those look dramatic in the moment, and that's what makes them the problem.

Run the math on a modest setup. Ten agentic workflows a week, one silent failure that takes an operator 45 minutes to notice, diagnose, and fix, at $150 an hour: that's $112.50 per incident. If it happens weekly, that's $1,125 a month, and that's the optimistic version, because it assumes someone catches the failure the same session it happens.

The SQLite corruption case from August 2026 shows the pessimistic version. A blog agent's memory store suffered page-level corruption, but the gateway stayed up and the agent kept answering messages, because reads still worked even as the underlying file was breaking. Nothing about the outside behavior signaled a problem. The failure was invisible for an extended period, and recovering it took a custom row-level salvage script.

That's the real argument for a layer-aware recovery model. Restarting a process fixes the failures that announce themselves. It does nothing for the ones that don't, and those are the expensive ones. The sections below map which of Hermes's six resilience layers is supposed to catch a given failure, and what an operator can actually do at each one when it doesn't.

This isn't a niche operational concern. HUMAN Security's 2026 Bad Bot Report clocked AI agent traffic growth at 7,851% year over year in 2025, with automated traffic now making up 51% of all internet activity. Agent infrastructure is running in production at a scale that makes "just watch the logs" an unworkable recovery strategy on its own.

How Hermes classifies errors

Every API error that Hermes handles through its built-in pipeline passes through classify_api_error() in agent/error_classifier.py. This is the fork in the road: it decides whether Hermes retries the call, falls back to another provider, compresses the context, or just surfaces the error to the person waiting on a response. Model-provider plugin errors may not follow the same classification path as built-in error types, so a plugin-sourced failure may not behave the same way.

The classifier assigns one of a defined set of FailoverReason values: rate_limit, overloaded, context_overflow, payload_too_large, long_context_tier, auth, billing, content_policy_blocked, ssl_cert_verification, timeout, stream_drop, thinking_signature, model_incompatible, invalid_request, server_error, format_error, reasoning_mandatory, unknown, among others. Each of those carries flags including retryable, should_fallback, and should_compress, among others. Which reason got assigned decides which of the recovery layers below actually engage.

The reasons aren't symmetrical, and that asymmetry is deliberate rather than a gap. content_policy_blocked is marked retryable=False, so Hermes won't hit the same provider again with the same message. The fallback provider chain may still activate even though retry on the original provider is closed off. That's a designed distinction, not an oversight: retrying a blocked request against the same content filter is pointless, but trying a different provider entirely might not be.

Tool handlers get a simpler rule. Any exception a handler throws gets converted into a tool result the model itself can see and react to, rather than being allowed to crash the loop. Every handler returns a JSON string, not a raw Python object, and that is what makes that conversion possible.

Retry behavior and jitter in multi-instance deployments

Hermes doesn't retry everything on the same clock. Backoff policy differs by scenario rather than applying one blanket wait-and-try-again timer everywhere.

Jitter gets added on top of that backoff, and the reason is specific to how Hermes actually gets deployed. Operators frequently run several profiles against a shared pool of API keys. If a provider has an outage and every instance backs off on an identical schedule, they all come back online at the same instant and hit the provider simultaneously, right as it's trying to recover. Jitter breaks that synchronization.

While a retry loop is running, Hermes prints a status block showing the error type, the provider, the model, elapsed time, context size, and a countdown to the next attempt. On a gateway or a cron job running unattended at 3 a.m., there's no one watching an interactive terminal, so that status block becomes the audit trail after the fact. If retries aren't resolving and an operator wants to know why, that output answers the first question that matters: is Hermes actually still retrying, or has it already classified the error as non-retryable and moved on.

Provider fallback's three sub-layers and where each one breaks

Fallback is three mechanisms, and they fail in different ways. It's three, and they fail in different ways.

Credential pools come first. If one API key for a provider hits a billing limit or a rate cap, Hermes rotates to another key for the same provider before doing anything more drastic. This is the cheapest and least disruptive form of fallback, since it changes nothing about the model or the session, only the credential behind the call.

Primary model fallback is the heavier layer. It switches to a different provider-and-model pair mid-session, without losing conversation history, tool calls, or accumulated context. It activates at most once per turn, and resets back to the primary provider on the next user message, so a fallback doesn't quietly become the new permanent default without the operator noticing.

Auxiliary task fallback runs independently of both. Side tasks like vision processing or context compression resolve their own provider chain, separate from whatever the primary conversation is using. That separation is exactly where the current known issue lives: as of GitHub issue #52392 in June 2026, when compression falls back between auxiliary routes, it can land on a model with a smaller context window than the one it left. Everything continues to run. Nothing errors out. But the session's actual continuity has quietly degraded, because the smaller window means more of the conversation gets dropped or summarized than the operator's configuration intended.

The chain that decides where fallback goes, for billing and rate-limit errors specifically, checks configured fallback options in sequence, and if none resolves, skips the task and logs a warning.

For high-concurrency setups hitting Nous Portal, Hermes writes a cross-session rate-limit record when it gets a 429, so other workers sharing that deployment know not to hit the same exhausted bucket. That's specifically a protection for long-running, multi-worker installs, not something a single-profile setup will typically see engage.

And if compression fails outright and no provider is available to catch it, Hermes drops the middle turns of the conversation rather than generating any kind of summary. That's intentional behavior, not a defect, but it means an operator depending on long-session continuity needs to know it can happen silently.

When fallback looks like it failed, the fix depends entirely on which of the three sub-layers actually broke. Credential exhaustion, a model-incompatibility failure in the primary switch, and an auxiliary compression failure each need a different response, and treating them as one problem wastes the diagnostic time that layer-aware recovery is supposed to save.

Context compression as a resilience dependency, not just a cost-saving feature

Compression gets configured through its own config block, with the option to override the summarization model and provider separately under auxiliary.compression. Compression is configured separately from the primary model, which allows it to be managed as its own dependency.

Compression is most likely to be needed exactly when a session is already under stress, when the primary provider is failing and Hermes is mid-fallback and the context is long enough to need trimming. That's the worst possible moment for the compression layer itself to fail, because a compression failure at that point doesn't just cost tokens, it compounds a continuity loss that's already happening.

That's the direct throughline from the fallback bug above. Because a compression fallback can land on a smaller-context-window model, the session may be truncating more than the operator's configuration was meant to allow. The context ends up smaller than it should be.

The fix isn't automatic. Auxiliary compression needs its own provider and model chosen with a context window that actually matches the typical session length it'll be asked to handle, and it needs to be monitored as its own dependency, not assumed to inherit the primary model's capacity by default. There's no single data point that proves this risk; the argument is structural, built into how Layer 3 (fallback) and Layer 4 (compression) interact when both are under pressure at once.

Checkpoints v2 and rollback: what the shadow git store does and does not protect

Checkpoints v2 shipped in the Tenacity Release, v0.13.0, on May 7, 2026, and rewrote how Hermes persists state with actual pruning behavior instead of unbounded accumulation.

The mechanism runs on an internal Checkpoint Manager that keeps a single shared shadow git repository under ~/.hermes/checkpoints/store/. The real project's own .git directory is never touched. Every project shares that one store: git's content-addressable object database deduplicates data across projects and across turns, rather than each project paying the storage cost independently. Before any file mutation, the system snapshots the working directory automatically, and old checkpoints get pruned rather than piling up forever.

Checkpoints are opt-in as of v2. Most users never touch /rollback, and the shadow store isn't free, it grows over time, so the default is off rather than on. Enabling it requires the --checkpoints flag on a per-session basis.

That has one hard consequence. If a destructive operation goes wrong and checkpoints weren't enabled beforehand, /rollback has nothing to roll back to. This is the one layer in the whole stack where the recovery option has to be turned on before the failure, not after. There's no retroactive fix.

Checkpoints also aren't a general-purpose safety net. They don't cover API-layer failures, they don't touch session database corruption, and they have nothing to do with cron job failures, those are the other five layers' job. Checkpoints exist for one thing specifically: file-system state after the agent itself makes a mutation.

Gateway auto-resume and session handoff changes in v0.13.0 and later hardening

Before the Tenacity Release, any mid-session gateway restart, whether from a system update, an OOM kill, or a network drop, meant every in-progress agent session was simply gone. Recovery was manual, every time.

v0.13.0 changed that baseline. The gateway now resumes interrupted sessions automatically after a restart, Checkpoints v2's rewritten state persistence gives Kanban board durability, and in-progress tasks get reclaimed so the agent picks back up close to where it left off. The scenario this fixes most directly is the overnight cron job: a restart at 3 a.m. no longer silently orphans every session that was active when it hit.

That wasn't the end of the work, though. v0.20.3, released August 16, 2026, hardened the remote-gateway connection and session-handoff path. v0.21.3, on September 14, 2026, fixed a specific failure where remote dashboard sessions expired during refresh bursts: both refresh paths, the cookie gate and Desktop's native bearer route, now coalesce concurrent requests that carry the same rotating refresh token, which stops a Desktop wake burst from replaying an already-rotated token and revoking the entire session. That same release also moved refresh work off the event loop, so a slow identity provider no longer freezes /api/status for everyone waiting on it.

v0.21.3 also closed a quieter leak: long-lived processes were accumulating duplicate state.db writer handles. Now the gateway, the dashboard and Desktop backend, and the ACP and CLI readers attach read-only, and in-process writers share a single registry handle instead of each opening their own.

If sessions aren't resuming after a restart, the first thing to check is whether the install is at or above v0.13.0. If it's specifically remote sessions expiring without warning, v0.21.3 from September 14, 2026 is the fix that addresses it.

The state.db corruption class: a distinct failure surface that the six layers do not fully cover

v0.21.2, released September 11, 2026, was billed as "The state.db Patch Release." It existed because v0.21.0 had shipped a large rewrite of the session store's connection handling that, on some installs, made state.db fragile in ways the earlier architecture hadn't been. The release that fixed it was sizable in its own right: 312 merged pull requests, 140 contributors, 947 non-merge commits, 1,869 changed files.

Six distinct failure modes got closed in that patch, and each one is identifiable by its symptom rather than its fix, because recognizing the symptom is what tells an operator which patch actually applies.

Multiple writer handles were cancelling each other's POSIX locks, which is close to a textbook recipe for corrupting SQLite. Profile gateways, the dashboard, and cron's lifecycle guard were each opening their own independent writable handles into the same file. The fix routes hosted rooms through a shared state.db, has the dashboard open read-only first, moves the lifecycle guard through a tracked connection registry, and has doctor --fix refuse to touch a checkpoint it can't prove is safe.

Separately, healthy write-ahead-log databases were getting flagged as corrupt when they weren't. OpenZFS deleted dentries, combined with a close() call racing an append_message write, produced a persistent DeletedWalGenerationError on a store that had nothing actually wrong with it.

Full-text search index damage was being classified as whole-file corruption, which used to force entire conversations to fail closed. It's now scoped to just the fts_index: search degrades gracefully, the index rebuilds on its own later, and the transcript store underneath is never touched.

A single corrupt row, whether the culprit was a malformed TEXT timestamp or an absurd epoch value like 1e30, was enough to crash the sessions list, exports, and insights. A new coerce_epoch() helper now lets a bad row render with a warning attached instead of taking down the whole listing.

Sessions could also bind to, or read from, the wrong profile's database entirely, a race condition in Desktop's launch backend under a HERMES_HOME override could pin a session to another profile's state.db. And state.db connection handling had been rewritten in ways that, on some installs, made the session store fragile for several seconds and then fail outright with "database is locked."

If state.db Problems showed up after an upgrade to v0.21.0, so v0.21.2 from September 11, 2026 is the release built specifically to address them. Operators still on older versions should treat multiple simultaneous writable handles into the same database as the primary corruption risk to audit for.

This closes the loop back to where this piece started. The August 23, 2026 SQLite corruption incident, the one where the gateway stayed up and kept answering messages while the underlying store silently broke, predates all six of these fixes. Recovering it required a manual, custom salvage script that rebuilt the schema and recovered 20,644 messages and 353 of the original 360 sessions. The connection registry design introduced in v0.21.2 exists specifically to prevent the multi-writer scenario that made that kind of silent corruption possible.