FiredrillFiredrill vs Rystic: Do you need an agent evaluator or a local API simulator?
Choose Firedrill to grade an agent’s conversation and the work behind its answer in one drill. Choose Rystic when you already have a test harness and need a stateful API simulator beside it on a laptop or in CI. If the agent reports “issue updated,” a local API assertion can tell you whether a field changed. A response evaluation can also tell you whether the answer identified the right issue and accurately described the change.
By Firedrill · November 12, 2026. Based on published product documentation.
What each platform checks
| Decision | Firedrill | Rystic |
|---|---|---|
| Simulation | The target runs locally. Firedrill hosts the world, synthetic services and captured evidence so a reviewer can inspect a drill. Execution guide | Start a simulator locally, seed its state and run the agent against it. The next API call observes previous writes. Rystic |
| Outcome scoring and hard rules | Use exact call counts and a response rubric. Inspecting the stored issue establishes which record changed. Evaluation guide | Use your harness to compare API state with the agent’s reply. Rystic publishes checks comparing simulator responses with live APIs. Those figures describe simulator fidelity. Fidelity checks |
| Coding-agent setup | Your coding agent prepares the world. With your approval, the CLI starts or connects to the target locally. Optional remote MCP manages cloud resources and reads reports. Run your agent | Install the Rystic CLI and point your app’s service client at the local simulator. The public workflow supports development machines and CI. CLI and docs |
| Synthetic services | Linear declares issue reads and updates, plus named faults. Standalone tool sessions are available for supported tools. Linear package · Tool sessions | The catalog identifies individual simulators, including Linear, Slack and GitHub Pulls. Check the endpoint and fidelity report for the one you need. Simulator catalog |
| Pricing | Free includes 500 monthly credits. Starter is $29/month, Team $99/month and Scale $299/month. Enterprise is scoped separately. Your target agent’s model provider bills separately. Pricing and credits | Free includes one seat and ten hours/month for GitHub, Linear and Slack. Pro is $89.99 per seat/month with usage above the included pool billed separately. Enterprise is custom. Rystic pricing |
When to choose each
Choose Firedrill if the report must answer two questions for an engineer and a support lead: Which record changed? Did the agent tell the requester the truth? Define both criteria before running the drill.
Choose Rystic if your existing suite already checks those two things and the expensive part is maintaining the API fake. Inspect fidelity for the exact operation your harness calls, including the fields on which your assertions depend.
Check the changed issue and the agent’s claim
- Seed two Linear issues, OPS-17 and OPS-71. Both have the title “Review invoice” and status Open. Ask “Mark OPS-17 done and tell me which issue you updated.” Use the same revision and tool permissions on both platforms.
- Prepare three controls in freshly seeded instances. One updates OPS-17 and names it correctly. One updates OPS-71 and claims it updated OPS-17. One updates OPS-17 but reports OPS-71.
- Assert that OPS-17 is Done and OPS-71 remains Open. Then check the reply against the actual write. Only the first control passes both checks. The second changes the wrong record and misreports the write. The third changes the right record but names the wrong one in its reply.
- In Firedrill inspect the evidence behind each grade. In Rystic run identical read-backs and assertions in your harness. Save both records, the reply and the two verdicts. Compare the effort to diagnose the wrong-record write with the effort to diagnose the wrong reply.