Firedrill

Firedrill agent setup guide

Firedrill is a simulation platform for AI agents. Connect your coding agent, select the right account and project, then run one small simulation of the agent you're building. Your coding agent operates Firedrill; your target agent, model credentials, and execution stay in your own runner.

1. Reuse a connection or authorize a new one

Already connected? Inspect the host's MCP servers and call firedrill_context, firedrill_discover, and firedrill_projects. These reads confirm the organization, permissions, supported capabilities, and authorized project/environment pairs. They do not create a project or execute your application. Confirm the intended organization and select a real project/environment pair from the returned list. Keep those IDs with subsequent requests; a report ID alone does not grant access.

For a new connection, add this remote Streamable HTTP MCP server:

https://mcp.firedrill.run/mcp

Use host-specific instructions for Codex, Claude Code, Cursor, or Kimi Code, or the connection page. Preserve existing server settings; do not add a duplicate Firedrill connection.

Use your host's OAuth sign-in action. The human completes Firedrill's browser authorization through WorkOS; the host stores the credential.

  • Existing account: use the intended identity and organization. A browser or CLI session does not automatically authorize the MCP host.
  • First sign-in: open Firedrill and complete its browser sign-in/sign-up flow. Finish email verification, organization selection, and required agreement prompts yourself, then authorize the MCP connection. Accept an existing team's invitation instead of creating a replacement organization.

No local MCP package, Node.js, CLI login, API key in chat, or manually copied token is required for this connection. Never put credentials in a prompt, recipe, repository MCP settings, or a URL. Host trust and per-tool approval remain separate. After authorization, repeat the read-only checks above.

2. Install only what your repository needs

Remote MCP prepares and inspects simulations; it cannot launch a process on your laptop. The CLI or native SDK executes your application in your own runner. Skills supply workflow instructions, not account access or an execution client.

For optional guided setup, run this in your existing application repository with Node.js 22 or newer:

npx firedrill-run init

Alternatively, npx firedrill-run offers optional setup on its first interactive launch, then asks whether to sign in and select a project. Later bare launches show a compact welcome. Reopen installation with npx firedrill-run install.

Choose project CLI, global CLI, or neither. MCP configuration, Firedrill skills, and a TypeScript or Python SDK are independently skippable. Arrow keys select one choice; Space toggles checkboxes; Enter continues. Review the exact package commands and project files before confirming. Project CLI/TypeScript installation requires an existing package.json. Python uses supported project package-manager metadata or a verified active in-project virtual environment, never system Python.

Installation does not run your agent or approve native host trust/OAuth. Reload your coding-agent session after adding skills. Keep an existing working integration instead of installing duplicate plugin and direct-skill copies. To skip optional installation and select account/project only:

npx firedrill-run init --skip-install

For coding-agent automation, choose additions explicitly. This example returns a read-only plan for project CLI, Codex MCP/skills, and the TypeScript SDK:

npx firedrill-run install --cli project --agent codex --mcp --skills \
  --sdk typescript --json --non-interactive

Inspect the plan and any unavailable steps. After approval, rerun the same selection with --yes to apply it. CLI choices are project, global, or none; SDK choices are typescript, python, or none. MCP and skills can be selected separately; --agent accepts codex, claude, cursor, or kimi. If the project has no recognized package-manager selection, set --package-manager explicitly to npm, pnpm, yarn, or bun. CI never installs or prompts implicitly.

3. Verify the runner's account and scope

CLI/SDK authorization is separate from MCP OAuth. Interactive init offers browser sign-in when needed. Approve the same intended organization, then select the project/environment used by MCP, or deliberately create a new project. init saves public projectId and environmentId values in firedrill.config.json, not credentials. Conflicting existing config is not overwritten.

npx firedrill-run login status --json
npx firedrill-run context --json
npx firedrill-run projects list --json

Verify the organization's authority in context, then match the config's two IDs to an authorized project/environment pair. Success here means authenticated access and the intended scope—not a passing simulation. Clients choose the Firedrill service address automatically; do not configure an alternate API server or copy an MCP credential into the CLI.

For a callable application, use the TypeScript or Python SDK. Native Python execution can run without Node.js. No SDK is required for CLI or MCP-only use.

4. Prepare and execute one real trial

Give your coding agent this bounded request:

Inspect this application's real invocation, input, completion signal, dependencies, and reset boundary. Prepare one useful Firedrill scenario with one configuration and one repetition. Keep the application's actual logic and my model credentials in my runner. Explain the execution count, side effects, capture, and separate model/runner costs before running. Return the actual outcome, report link, and evidence gaps. Add assessment only when useful.

Save the reviewed recipe in the repository. The commands below assume the actual file is firedrill/first.recipe.json; use your own exact path. The first simulation guide explains the command protocol; native callables use their SDK callback binding. Synthetic Tools are optional. When needed, verify their declared interfaces and route external side effects away from production. Tool reset does not reset your application's database or agent memory.

npx firedrill-run validate firedrill/first.recipe.json --json --non-interactive
npx firedrill-run doctor firedrill/first.recipe.json --json --non-interactive
npx firedrill-run preview firedrill/first.recipe.json --json --non-interactive

Validation checks structure offline. Doctor checks declared local prerequisites and authenticated admission, not your target or model provider's behavior. Preview shows expansion, scope, and blockers without executing a target or reserving capacity. Check that it describes the intended single trial.

When execution is authorized, command/HTTP targets can run with:

npx firedrill-run simulate firedrill/first.recipe.json --view operator --json --non-interactive

Native callbacks require their SDK binding instead. Ordinary native/command execution is not a filesystem or network sandbox. If MCP already queued the recipe with firedrill_create_batch, use its returned runner continuation for that retained batch. simulate submits new work; it does not attach to that batch.

5. Read the evidence, then repeat or add CI

Keep the returned operation ID, batch ID, run IDs, and authenticated report link. Replace BATCH_ID with that actual batch ID; get RUN_ID from its runs:

npx firedrill-run result BATCH_ID --view operator --json --non-interactive
npx firedrill-run runs BATCH_ID --json --non-interactive
npx firedrill-run report RUN_ID --json --non-interactive
npx firedrill-run assessments list RUN_ID --json --non-interactive

These reads do not rerun the target or request assessment. Follow returned pagination and evidence references. Explain actual inputs/outputs, attributed activity and state changes, available captures, and missing evidence. A fluent claim alone does not prove an external effect.

Keep execution, observed outcome, assessment, and CI gate separate. Exact checks suit hard requirements such as schemas, arithmetic, permissions, and state changes. Semantic rubrics assess meaning and partial credit while accepting differently worded valid answers. Add scenario checks with the observed evidence path, comparator, and expected value; use rubric dimensions with five task-specific anchors for semantic quality. Fractional scores are not probabilities or rounded ordinal levels. Missing evidence and evaluator errors are not zero scores or agent failures.

After a useful small trial, expand scenarios/configurations and repeat actual target executions. Inspect distributions and missing counts; reassessing frozen evidence is not another target trial. Save the recipe and runner for CI reuse, using a scoped machine credential from CI's protected secret store, not interactive sign-in. Choose the gate policy explicitly; completed execution alone does not prove task success.

Recover without duplicating work

  • MCP authorization failed: use the host's OAuth sign-in/reconnect action, then recheck firedrill_context. CLI login cannot repair the host's separate credential. Pending invitations or missing membership need human/admin action.
  • CLI sign-in is missing: run npx firedrill-run login interactively. Finish any email/invitation action, then run login again. An explicit FIREDRILL_TOKEN wins over stored login; remove an unintended selection in your own environment without printing it.
  • Refresh is uncertain: preserve protected login state. Only when FD_LOGIN_REFRESH_PENDING directs it, a human may use npx firedrill-run login --retry-refresh once within its 25-second window. Otherwise use npx firedrill-run login --reauthorize. Do not loop refreshes or delete saved state to bypass recovery.
  • Wrong scope or partial installation: inspect config, context, and projects list. Resolve conflicting flags/environment/config deliberately; use a reviewed alternate --config file when needed. Review package-manager output and repository changes before reopening install. Completed additions are not rolled back or automatically retried; use init --skip-install to continue account setup independently.
  • Execution was interrupted or uncertain: keep original files, identity, scope, private journal, and the printed recovery command. Command/HTTP recovery uses npx firedrill-run simulate --resume OPERATION_ID with the original execution ID; native work resumes through its original SDK runner. Never substitute a batch-creation key or start a fresh simulation. status, follow, and report are retained reads; a pending score does not require a target rerun.

Troubleshooting · Operator handoff · Documentation index · Open your runs · Contact the team

Firedrill is the product and brand of Reload Tech Inc.