Metrics
CPU per core, memory, load, disks, filesystems, network and the top processes, every 15 seconds, under OpenTelemetry names.
The watcher of the Proteus family
It sees every host.
It strikes what breaks.
An agent on every machine. A mothership that swallows a million records a second. An AI you talk to that hunts the fault down and fixes it, the moment a human signs off.
Click the dark. It bites.
01 The agent 200 m · twilight
One binary.
Every host.
Nothing gets past it.
leviathan-agent is a single static binary that tears everything it can off a machine (metrics, logs, events) and ships it home over OTLP. If the mothership goes dark, it holds the line on disk and delivers when it comes back.
Logs
Tails files by glob, keeps multi-line stack traces whole, and reads journald from a saved cursor, so a restart never loses or repeats a line.
Events
Services dying, OOM kills, failed logins, reboots, and config files changing under you, with the file's hash before and after.
Delivery
At least once. Every batch lands in a 512 MiB on-disk queue before the first send, and leaves only when the mothership answers 200.
listener:
url: https://mothership.internal:4318
token: ${LEVIATHAN_TOKEN}
logs:
- name: nginx
type: file
paths: [/var/log/nginx/*.log]
- name: journal
type: journald
events:
service_state: true
oom_kills: true
logins: true
file_watch:
- /etc/nginx/nginx.conf- ~1.5%
- of a vCPU on a light host
- 19–27 MiB
- of memory, bounded by batch size, not rate
- 20k/s
- records per agent before you need a second one
- 512M
- systemd memory cap, so a log storm can't starve the host
Linux today. The Windows agent (Event Log, performance counters, services) is next.
02 The mothership 1,000 m · midnight
It swallows
a million records
a second.
leviathan-listener takes OTLP from every agent, or from any OpenTelemetry SDK or Collector. It writes everything durably to EBS, keeps the last fifteen minutes searchable in memory, and answers questions about it.
mothership1.42M/s
- 1.42M/s
- durable ingest measured on a slow virtual disk, every acknowledged batch on disk
- 26×
- faster than an fsync per request: group commit flushes many batches at once
- 83 B
- per record on disk, stored exactly as the agent sent it and ready to replay anywhere
- 2.2 µs
- of listener CPU per record from busy hosts
- Segment files framed with CRC32C, so a torn write is found, not served.
- Backpressure with
Retry-After: agents slow down instead of the mothership falling over. - Retention by size, because 100k records a second is 717 GB a day.
- A query API (
/api/v1/logs,/metrics,/agents,/stats) behind its own read token.
03 Strike 4,000 m · abyss
Tell it what hurts.
It finds the wound.
Leviathan's MCP endpoint runs on Amazon Bedrock AgentCore Gateway. Claude Code, or Strike, the AI in the Leviathan app, connects to one URL and gets sixteen tools: six that see everything the mothership knows, and ten that diagnose, fix, undo, and write down what they learned.
observe read-only
list_agents get_host_health search_logs query_metrics list_events get_ingest_stats
act audited, human-approved
run_diagnostic list_reachable_hosts get_fix_policy propose_fix apply_fix rollback get_changes get_command_result write_runbook_entry search_runbook
Every fix it learns goes into the Depths, a runbook wiki the AI writes and a human reviews.
04 The leash 6,000 m · hadal
It only bites
what you point it at.
AI gets things wrong, and tokens leak. So every layer assumes the one above it has already failed.
Nothing arbitrary runs.
Three pinned Systems Manager documents, sent only to hosts tagged
leviathan:managed=true. No shell, no Session Manager, no other document.What runs is what you approved.
The plan pins the document's version and SHA-256 and is sealed with a KMS HMAC. Change one byte after approval and it refuses to run.
Only a human can sign.
Approvals are KMS signatures. Every role the AI runs as is explicitly denied
kms:Sign. It can carry a token. It can never mint one.One plan, once, briefly.
A token binds the exact plan, expires within 15 minutes, and spends its nonce in the same transaction that starts the change.
No audit, no action.
Every step lands in an Object Lock bucket before the next one runs, and nothing written there can ever be replaced.
Small bites.
Three hosts per fix at most, one at a time, stopping at the first failure. sshd, the SSM Agent, systemd and Leviathan's own agent are never touched.
Kill switches.
Read-only mode. Zero concurrency on the act function. Untag a host. Remove yourself as an approver. Any one of them stops it cold.
It ships read-only. Fixes stay off until you turn them on.
05 Feed it 8,000 m · trench
How big does
the mothership
need to be?
From measurements, not guesses. Pick a fleet.
Planning figures from the capacity study: measured on a shared desktop, then doubled for cloud vCPUs. Run the benchmark on your instance before you buy.
06 The family 10,935 m · challenger deep
Proteus makes
the monsters.
Leviathan hunts them.
Proteus
Builds a distributed system that doesn't exist, then makes it behave like production: metrics, logs, traces, RUM and real-looking failures, injected into Amazon CloudWatch.
proteus.jeremyfhall.com →Leviathan
Watches real systems, finds what's wrong, and fixes it with an AI you talk to, on a leash a human holds.
you are hereNext: the proving ground. Point Proteus at the mothership and every fault it invents becomes a drill Leviathan has to catch.