Building test data and environments for AI agents means generating structured, ground-truth-labeled simulations, not just static datasets, so agents can be trained and evaluated against tasks of controllable difficulty. Because the data is generated rather than collected, every task carries a known-correct answer built in — which is what makes rigorous agent training and evaluation possible in the first place. The result is a simulated environment, populated with realistic context like emails, tool calls, and multi-step workflows, that mirrors production conditions without exposing real systems or data.
What "test data and environments" means for AI agents
An agent environment is not a dataset in the way a classifier's training set is a dataset. A fraud-detection model learns from rows of transactions, each labeled fraud or not-fraud, and once it has seen enough rows it can score a new transaction on its own. An agent doesn't operate on isolated rows — it acts inside a context: it reads a message, decides what to do, calls a tool, and that action changes the state it acts on next. Training and evaluating an agent means giving it that context to act inside of, not a spreadsheet to classify.
That context has to include the raw material an agent would actually encounter — emails to read, records to look up, tools to call, a calendar to check — plus a defined task with a correct outcome the agent's actions can be scored against. Without that correct outcome, you can watch an agent act, but you can't say whether it acted well. This is the same requirement that shapes AI training data generally — a model is only as good as the ground truth attached to what it learns from — but agent environments raise the bar, because the "correct answer" has to account for a multi-step process, not a single label.
Tonic Fabricate is built around this distinction: rather than producing a table of rows, Fabricate generates a simulated environment — structured records, unstructured messages, and the tools an agent would call — with the tasks and their correct outcomes generated alongside it. Every layer of that environment, from a single lookup task to a full simulated company, exists so the agent has something real to act on and a correct outcome to be judged against.
Why real-world data and public benchmarks fall short
Two options come before generation, and both run out quickly. The first is production logs: message threads, ticket histories, CRM activity. These are realistic, but they carry no ground truth — nobody recorded the "correct" resolution alongside each support ticket, so there's nothing to score an agent's output against without a separate, expensive annotation pass. Logs are also uneven by nature: the rare, hard cases that actually stress an agent — the ambiguous request, the multi-system escalation — show up too infrequently in raw history to build a reliable evaluation from.
The second option is public agent benchmarks — AgentBench-style suites, GAIA, WebArena, and similar collections. These solve the ground-truth problem: tasks are pre-defined with known-correct answers, which is exactly what real logs lack. But they're built around generic environments — a browser, a shopping site, a set of general tools — not the specific databases, internal tools, and workflows a team is actually building an agent to operate. A model that performs well on WebArena says little about whether that same model can triage tickets inside a company's actual CRM. Public benchmarks also can't scale to cover the edge cases specific to a domain, because they were built once, for everyone, not for any one team's failure modes.
The practical consequence is that neither route gets a team what it needs: logs have the right domain but no ground truth, benchmarks have ground truth but the wrong domain. That gap is really a version of how much data a model — or an agent — actually needs to learn a task reliably — production logs simply don't contain enough of the right cases, and benchmarks don't contain any of a team's own cases at all. Generation is what closes both sides of that gap at once, which is why teams that hit this wall turn to it rather than waiting for either source to catch up.
Designing tasks with ground truth built in
The core idea behind generated agent tasks is that ground truth — the correct answer a model or agent is graded against — is written into the task at the moment it's created, not attached after the fact by a human reviewer. When you generate a task, you already know the answer, because you specified the scenario that produces it. That removes the annotation step that limits how much labeled data a team can realistically produce, and it's the same tradeoff that shows up in labeled-data generation generally — the generate-vs-annotate choice that determines whether ground truth is built in or bolted on.
In practice, tasks are designed as a graded hierarchy rather than a flat list. A single-hop task asks an agent to look up one fact — find the invoice number in this email thread. A multi-hop task, where multi-hop reasoning means chaining several such lookups or inferences together to reach an answer, asks the agent to combine several of those steps: find a customer's last three support tickets, determine which one is still open, and draft a follow-up that references the right order number. Graded difficulty means this hierarchy is built deliberately, from single-hop to multi-hop, so an evaluation can show not just whether an agent passes or fails, but where its reasoning starts to break down.
The Tonic Advantage: Tonic Fabricate designs hierarchies of verifiable tasks — from single-hop lookups to multi-hop reasoning chains — directly on top of a simulated environment, generating the correct answer alongside the task itself.
The evidence that this approach transfers to real performance is concrete. In a Tonic.ai benchmark, a model fine-tuned only on Fabricate-generated synthetic data — a simulated 100-person company with graded, multi-hop email tasks — improved on the real-world Enron email benchmark from 80.5% to 86%, outperforming o3 and gpt-4.1-mini, without training on a single real email (Tonic.ai research). When tasks are generated with their ground truth attached, the resulting training signal can match or beat what real-world data alone produces.
Populating environments with realistic structured context
Tasks need somewhere to happen. Beyond the tasks themselves, an agent environment needs the surrounding material an agent would encounter doing the job: emails, Slack-style messages, CRM records, calendar entries, and the APIs that connect them. Building that context by hand — scripting a fake inbox, faking a handful of CRM records — works for a demo, but breaks down at the scale and consistency an evaluation needs: the same customer name has to appear correctly across an email, a CRM record, and a calendar invite, and the timeline connecting them has to hold together across weeks of simulated activity, not just a handful of hand-placed examples.
Tonic Fabricate approaches this by simulating the environment as a whole rather than assembling pieces separately. From a single prompt, Fabricate can generate a company's activity over a defined timeline, with the same entities, IDs, and events staying consistent across every format the agent might touch — a message, a database row, a document. That consistency is what lets an environment stand in for production conditions: an agent tested against it is exercising the same kind of cross-system reasoning it would need on the job, without ever touching real systems or data.
The Tonic Advantage: From a single prompt, Fabricate can simulate a complete company's activity over a defined timeline — emails, messages, CRM records, calendar events — with the temporal and cross-dimensional consistency that hand-scripted mocks can't sustain at scale.
Scaling and maintaining environments as agents evolve
An environment that stresses today's model won't necessarily stress next quarter's. As models improve, the tasks that once separated a strong agent from a weak one start to saturate — every model clears them, and the evaluation stops telling you anything useful. Environments have to scale in both difficulty and volume alongside the models being tested against them, which means treating environment design as an ongoing practice rather than a one-time build.
The failure mode to watch for is drift: an environment that quietly stops being useful because it no longer contains anything a modern model finds hard. A fixed-difficulty benchmark built a year ago may still run, and may still report a high score, without that score meaning anything about how an agent will handle a genuinely novel case. Catching drift means periodically checking a benchmark's headroom — how much room is left between current model performance and a perfect score — and generating harder tasks, deeper multi-hop chains, and larger environments as that headroom closes.
This is part of the broader RL infrastructure question of how environments scale as a program matures — the same question that shapes how teams approach simulated environments for reinforcement learning more broadly. Generation helps directly here: because environments are specified rather than hand-built, scaling one up is a matter of adjusting the specification — more entities, deeper task chains, longer timelines — rather than re-authoring test fixtures from scratch.
From environments to evaluation: closing the loop
Building the environment and its tasks is necessary but not sufficient — an agent still has to be run against it and scored. Evaluation itself has two parts: correctness, whether the agent reached the right answer, and process, whether it took a reasonable path — the right tool calls, the right sequence — to get there. Because the tasks carry a ground-truth answer from generation, scoring correctness is comparatively simple. Scoring process is harder, and typically depends on comparing the agent's trace of actions against the reference path built into the task's design.
Running and monitoring that evaluation is its own layer, distinct from generating the environment in the first place. Trace and eval platforms — the tools teams use to log an agent's tool calls, replay its reasoning, and track scores across runs over time — solve a real and complementary problem: observability into what an agent actually did during a run. They depend on having something worth observing in the first place, which is where environment generation comes in; the two layers work together rather than compete for the same job.
Closing the loop means feeding evaluation failures back into environment design: when an agent consistently fails a particular multi-hop pattern, that pattern becomes the next environment's focus, generated with more variations and a wider spread of difficulty. Building the evaluation datasets and benchmarks themselves — the reference tasks a team scores against on an ongoing basis — is its own discipline, distinct from but dependent on the environment work described above. Treated as a cycle rather than a one-time setup, environment generation and evaluation compound: harder environments surface new failure modes, and new failure modes shape the next generation of environments.