Sooner or later, every team that tries Agile Data Agent (ADA) asks three questions. Can we run it on our own hardware and keep our data in-house? How much worse is a local model on our actual work? And what does a frontier model cost once it runs all day?
Benchmarks don't answer those questions, so I ran a test on real work. I took an ADE pipeline I had already built with ADA, froze its output as a golden reference, and had three agent setups rebuild it unattended, 45 times in total. One was Claude Sonnet 5 in the cloud. The other two were open-weight Qwen models running on my 48 GB MacBook Pro.
As for data, most ADA work involves none at all. The agent works with metadata: entity definitions, attributes, keys and loads, written as YAML that ADE turns into SQL. Data only reaches the model when you give the agent access to it, for example to sample tables or query the target database to check results. That's when it starts to matter where the model runs.
The short version:
-
A local model came within point of the frontier model on quality. Qwen3.8-27B scored 93.7 against Sonnet 5's 94.7 on phase-by-phase work, at zero marginal cost and about half the speed.
-
Sonnet 5 is still the best designer, and it's cheap. It passed all 15 of its runs, which cost $18.99 in total.
-
The biggest failures were shared. All three models tripped over the same gaps in skills and validation.
The test
The scenario is a four-layer ADE pipeline on NYC taxi data: staging from raw CSVs, a Data Vault design document, Data Vault metadata, and a publish layer with a fact and dimensions. In total it has 19 entities across six packages. The agents got the same casual one-line prompts I had originally used ("ok create data vault design", "awesome, now generate it"), plus an instruction to work autonomously and stop when ada validate passes.
There were five tasks. Four were phase-isolated: each started from the golden output of the previous phase, so a single early mistake couldn't cascade. The fifth built the whole pipeline end to end in one session. Every model got the same ADA harness: the agent definition, 20 skills, the project's AGENTS.md and the ADA CLI. There was no model-specific prompting.
Every run started in a fresh git repository with isolated _EVAL schemas. The local agents ran in a locked-down container with no ADE credentials at all. Copilot ran with deny rules on every command that writes to ADE. No run wrote to ADE.
Runs had two gates: ada validate had to pass, and nothing could be written to ADE. Within those gates, each run was scored from 0 to 100 on four kinds of check:
- Structure: similarity to the golden entities, covering attributes, loads, mappings and keys.
- Task checks: whether the specific requirements were met, such as views, load counts and template use.
- Server validation: a real
ada push --dry-runagainst ADE. - LLM judge: GPT-5.5 scored the design document against the golden design.
I tested the grader before the models: the golden reference scores 100 on every task, and deliberately broken copies fail.
The results
Staging was solved in this scenario. Every staging run from every model scored 100, and the 35B took about 30 seconds.
The 27B matched Sonnet phase by phase. It passed every phase-isolated run. It also scored slightly higher than Sonnet on Data Vault generation (96.8 vs 95.3) and the publish layer (95.8 vs 92.7), and passed ADE's server validation on every Data Vault generation run.
Sonnet is the best at design. The judge rated its Data Vault designs 0.87, against 0.68 for the 27B and 0.57 for the 35B. Sonnet was also the only model that never failed a run.
The 35B is fast but falls off after staging. From Data Vault generation onwards, it often produced metadata that passed local validation and was then rejected by the ADE server.
Speed and cost
The 35B stays above 100 tokens per second even at 128K of context, on par with the cloud model. The 27B is slowed down by prefill, the time it takes to read the prompt before answering. Once prompts pass 100K tokens, the slowest requests wait more than 25 seconds for the first token.
The context window never blocked a run, and prompts peaked at 154K out of 262K. Long context did make the 27B's end-to-end runs 2.5 times slower than Sonnet's.
On the cloud side, a full end-to-end build with Sonnet cost $3.09, and all 15 Sonnet runs cost $18.99 at GitHub Copilot's published rates.
At these prices, the reasons to go local are privacy for the tasks where the agent reads real data, offline work, and a flat cost for always-on jobs that would otherwise burn tokens around the clock.
The harness shows up in the numbers
Markus Heiskanen wrote recently that you rent the model, but the harness is your asset, and the token numbers back that up. pi is a deliberately minimal agent: four tools, a short system prompt, and skills loaded only when the agent opens them. Its first request was about 4.5K tokens, and its prompts averaged roughly 55K tokens against about 97K for Copilot CLI. Model and harness can't be fully separated, since Sonnet also verified its work more often. But prefill cost grows with context, so on local hardware a smaller prompt also means a faster run.
Where the models lost points
The lost points clustered in five places:
- Incomplete hub loading. A hub should be loaded from every source that carries its business key. All three models missed some of those loads (hub-loading scores ranged from 0.42 to 0.83). When every model misses the same rule, the problem is how the skill states the rule.
- Facts split per source. When asked for one fact loaded from two sources, the 35B usually created a separate fact for each source instead. ADA's publish skill hints at the right pattern but never shows it. In my runs, local models followed explicit rules and examples far better than hints.
- Common Data Vault anti-patterns. The weaker models modelled trips, which have no business key, as a hub instead of a link, and split the trip link per source instead of loading both sources into one link with source-specific satellites. Sonnet avoided these most reliably.
- Local validation weaker than the server. Every run passed
ada validate, yet ADE's server rejected several for dangling references and schema-shape errors. The agents stopped when told to. This is the single biggest lever: if the local validator caught what the server catches, every model would have corrected itself. - Errors compound end to end. Sonnet and the 27B dropped 12–16 points on the end-to-end task compared with the phase-isolated tasks. Some of that is genuine drift, and some is strict golden matching penalizing valid alternative designs.
Next changes to ADA:
- Close the gap between
ada validateand the server: reference integrity, attribute existence and schema shape. - State the hub-loading and fact-modelling rules explicitly, with worked examples.
- Add the anti-patterns to the analysis skill as rules: no business key means no hub, and one link per relationship, with satellites split by source.
- Run each phase in a fresh session, handing over files rather than chat history, which also keeps contexts small for local models.
Which model for which job
With the right model, a 48 GB laptop produced ADE metadata at frontier quality on phase-by-phase work. It ran offline, at no marginal cost, and anything the agent reads stays on the machine instead of going to a cloud model. Match the model to the job:
I didn't test top-tier models like Opus or Astra, and for this workload I didn't need to. The recommendation above comes from using them in my other work. Sonnet passed every run, and most of the points it lost came from missing rules in skills and a forgiving validator. Top-tier models earn their price on long-horizon sessions, very large contexts and deep design reasoning. That's where Sonnet and the 27B lost the most points, so that's where I'd expect them to help.
The setup I find most attractive is a hybrid. Sonnet writes the Data Vault design for well under a dollar, and the local 27B generates the metadata on the laptop. The frontier model handles the step that needs judgment, the local model handles the volume, and top-tier models cover the hardest design and long-horizon work.
Caveats: this was three runs per task per model, on one scenario and one machine. Sonnet ran through Copilot CLI and the Qwen models ran through pi, so some differences, especially in token usage, come from the harness rather than the model. The judge is an LLM, so its scores are good for ranking rather than as absolute truth.
Try it yourself
If you run ADA today, pick one task your agent keeps getting slightly wrong and write the missing rule into your AGENTS.md or CLAUDE.md before you look for a better model. If you want to try local models, start staging on a fast mixture-of-experts model and move the rule-heavy phases to a dense one. Before you settle on a setup, measure it on your own machine and your own project.
Henri Hemminki is Solutions Architect at Agile Data Engine and the founder and lead engineer of ADA. If you want to talk about running ADA with local models, find him on LinkedIn or through Agile Data Engine.