Faultline

Faultline injects failures into an agent’s tools: a timeout after the action has already gone through, a rate limit, or a malformed response. Then it checks the service’s own records instead of the agent’s reply, and saves each failure as a test that can be rerun. In the demo a refund goes through but its response times out. A naive agent retries and refunds $258 on a $129 order. An agent that reuses a stable idempotency key refunds $129 once.

There is a React dashboard for running and comparing these. The fault injection is also a small SDK on npm with no dependencies, so it can wrap the tools of an agent you already have.

Making the results trustworthy. Runs go through a job queue in PostgreSQL, where workers claim jobs with SKIP LOCKED and hold a lease. The tricky case is a worker that stalls, loses its lease, then wakes up and tries to save an old result over a newer one. Every write checks the lease first, so it can’t.

The agents it ships with are fixed scripts, not live models. The Claude and OpenAI adapters are written but haven’t been tested with paid model calls yet, so that comes first. After that, running it on AWS instead of only on my machine.