When Valid JSON Isn't Enough: 995 Real AI Tool-Calling Experiments

Every one of the 960 intent responses in this experiment parsed as valid JSON and passed its applicable validation path. The model still chose the wrong action in some cases—including whether to clear a value or leave it alone.

ArgBridge, the reversible codec being evaluated, achieved 91.1% exact intent accuracy. A simpler control using the original non-strict tool schema and local validation achieved 96.4%. ArgBridge also cost 46.1% more per correctly completed intent in this run.

The title's 995 experiments were 995 real API requests: 35 schema probes and 960 intent trials, built from 56 synthetic instructions. They were not 995 independent customer workflows. The engineering question was whether a representation that preserves data semantics also helps a model choose the right operation. Those are different properties.

The three states hidden inside a patch

Consider an update API with this contract:

{}                   # leave the existing name unchanged
{"name": None}       # clear the name
{"name": "Grace"}    # replace the name

This is application-defined patch behavior. JSON itself does not assign “clear” to null. Once an API adopts that meaning, however, member presence becomes part of its public contract.

OpenAI's strict function-calling requirements require object schemas to be closed with additionalProperties: false, and every declared property to appear in required. Adding null to a property's allowed types provides a nullable value, but cannot also represent omission when null already has a business meaning.

An adapter that fills a missing field with null can turn “do nothing” into “clear it.” An adapter that drops nulls can make an explicit clear disappear. Both transformations may produce valid JSON while changing the operation.

Typed dictionaries create another tension. A contract such as “arbitrary label names mapped to nullable strings” needs dynamic keys. Simply closing the object excludes those keys. The question is how to preserve the original contract through a stricter transport representation.

The initial hypothesis was about transport: within a supported subset, a reversible codec could preserve omitted fields, explicit nulls and typed maps while emitting strict schemas. Explicit presence tags and key/value entries made that plausible; the information could be represented without collapsing distinct states. Practical usefulness depended on whether a model could generate that representation reliably enough to justify its cost.

ArgBridge's architecture: encode presence, then validate twice

ArgBridge compiles a supported source schema into a wire schema and an internal transformation plan. An optional member becomes a tagged value:

{"state": "absent"}

{"state": "present", "value": null}

{"state": "present", "value": "Grace"}

These are field-level fragments; the complete wire message also has an arguments root. Required members retain their values through the child codec. Typed maps become an entries array of key/value pairs. Mixed fixed and dynamic objects use fields plus entries, and duplicate keys or entries that shadow declared fields are rejected.

For supported, valid input x, the target invariant is decode(encode(x)) == x, including which members exist. It does not preserve whitespace, key order or arbitrary-precision numeric spelling.

The model generates the wire representation directly. It does not first produce a correct original patch that the encoder then transports. This is where deterministic round-trip correctness stops answering the model-reliability question.

The implementation separates the boundaries:

1. Compilation validates the schema, refuses unsupported constructs and bounds expansion. The schema is copied defensively; remote references are disabled.
2. Encoding existing data validates the original value, transforms it and validates the emitted wire value.
3. Decoding model output checks the wire value, reconstructs the original JSON and validates the complete original contract again. The supplied loads parser additionally rejects duplicate raw JSON keys before a Python dictionary can hide them.
4. Application execution remains the caller's responsibility. guarded_call invokes a handler after decoding; it does not authenticate, authorize, sandbox or make the handler transactional.

The final source-schema check is necessary. A tagged branch can match one alternative of an original oneOf while also matching another, violating exclusivity. Validating the wire branch alone would miss that.

Resource limits are also part of the boundary: one million JSON bytes, 20,000 nodes, depth 64 and a bounded compilation expansion. The release added checks for expanded schemas, bundles and CLI serialization so an output could not exceed its own reader's budget. These are concrete safeguards, not a complete denial-of-service proof. See the architecture and scope.

What the deterministic tests could establish

The original offline corpus restored 40 valid inputs exactly, rejected 36 invalid source inputs and rejected 12 hostile wire/JSON cases. It exercised 87 assertions from an externally authored JSON Schema test suite. Another 256 assertions were explicitly excluded because their schemas were outside the supported subset.

That external suite provides independent authorship for those assertions. It does not make the locally designed workflows or the model benchmark independent. Property tests and fixed fixtures establish useful transport invariants; they cannot tell us whether a model will select the correct presence tag from an instruction.

As the project owner, I required real-model semantic evaluation before treating successful round-trip tests as evidence of reliable tool use. The comparison had to measure intended actions alongside schema acceptance, token overhead, latency and cost against current alternatives.

A frozen benchmark with six approaches

The corpus contains 56 English instructions across flat settings, nested settings, typed labels and a required-fields control. Seven primary categories cover omission, explicit null, ordinary updates, nesting, dictionaries, ambiguity and invalid/adversarial input, with eight instructions per category.

Instructions and workflow identifiers live in scenarios.json. Expected actions and exact patches live separately in expected.json. The oracle was fixed before inference and never included in model requests. That separation prevents answer leakage; it does not establish an independently authored ground truth.

Each supported approach received the same instruction, original contract, initial state, policy and model settings. The transport representation differed. The six arms were:

• ArgBridge: the original generated wire schema and reversible decoder.
• Factored ArgBridge: shared descriptions and repeated schema nodes through local references, with unchanged wire values and decoder.
• Handwritten strict operations: an operation-list wrapper written specifically for the four workflows.
• Original non-strict + validation: the original schema with explicit strict: false, followed by original-contract validation.
• OpenAI Agents strict conversion: the recorded openai-agents 0.24.0 transformation, followed by the original-contract guard.
• Mastra strict conversion: the recorded schema-compat 1.3.17 OpenAI compatibility path and exported null normalizer, followed by the same guard.

The SDK arms evaluate those exact transformation paths, not complete agent frameworks. Their recorded review included 2026 fixes; historical defects were not used as a substitute for testing the installed implementations. Both frameworks can host manually designed wrappers or non-strict tools. Exact comparator scope.

The pinned model was gpt-5.4-mini-2026-03-17 through the Responses API: reasoning effort none, temperature 0, maximum output 512 tokens, parallel tool calls disabled, store: false and default service tier. Three repetitions were interleaved in a deterministic randomized schedule. Temperature 0 was a setting, not a promise of identical responses.

Six arms × 56 scenarios × three repetitions would produce 1,008 intent trials. The Agents converter rejected the dictionary workflow locally, excluding 48 trials: 960 actual intent requests, plus 35 probes, equals 995. Those unsupported cases remain visible rather than disappearing from the coverage discussion.

There were no API errors or automatic retries. No real application handler was invoked; resulting state changes were simulated locally. The frozen protocol documents approval, budget reservations, request hashing and the treatment of unknown charges.

Validity was perfect in this sample. Intent was not.

All 35 probes were accepted: 31 strict requests and four non-strict requests. All 960 intent responses contained one tool call, parsed as JSON and passed the applicable local validation path. Non-strict outputs happened to validate in this sample too; non-strict generation has no schema-conformance guarantee.

Exact correctness required the intended action and exact patch, including member presence. Clarification and rejection were scored by action and a nonempty reason, not the quality of the explanation. A separate final-state metric distinguished an incorrect patch from an incorrect resulting state.

The full results, including cost per exact correct intent, were:

• Original non-strict + validation: 162/168, 96.4%, $0.00049550 per correct intent.
• Factored ArgBridge: 155/168, 92.3%, $0.00068952.
• ArgBridge: 153/168, 91.1%, $0.00072384.
• Handwritten strict: 138/168, 82.1%, $0.00065954, subject to the mapping flaw below.
• Mastra conversion: 93/168, 55.4%, $0.00088372, for the tested transformation path.
• Agents conversion: 39/120 observed trials, 32.5%, $0.00157413. Its 48 unsupported trials mean full-coverage correctness is 39/168, or 23.2%; this is a coverage adjustment, not another measured run.

The cost denominator includes correct updates, clarifications and rejections. The numerator includes all attempted intent requests in that arm, including errors of intent, and excludes separate schema probes. Using the unrounded costs, ArgBridge's cost per correct intent was 46.1% higher than the non-strict control. Total usage-derived cost for all 995 requests was $0.54992475, calculated from provider token usage and the recorded rates—not independently reconciled against an invoice.

On the 168 paired trials, original ArgBridge lost nine cases that the non-strict control got right and gained zero. Three repetitions share each scenario, so these counts do not establish statistical independence or a production-wide effect. Complete live results and raw evidence.

Three failures that a schema cannot explain away

A requested clear became an omission. In null-04, the instruction was “Clear the saved timezone so that the field is absent.” Under the frozen API policy, clearing a stored field requires the patch {"settings":{"timezone":null}}. ArgBridge instead decoded to {"settings":{}}, retaining the timezone. Both ArgBridge variants failed all three repetitions; the non-strict control passed all three. The codec faithfully preserved the wrong presence decision.

The final state was right, but the patch was wrong. In nested-08, the instruction explicitly retained the dark theme and changed only the timezone to UTC. Both ArgBridge variants also wrote the already-current theme on every repetition. That produced the intended final state but violated the exact patch oracle. In a real integration, redundant writes might matter for audit events or concurrency; those downstream consequences were not measured here.

Missing context became a guessed action. For “Increase the ticket priority” in ambiguous-06, the policy required clarification. Original ArgBridge inferred a ticket and target priority on one repetition. No type validator can recover the missing business context once the model has supplied a plausible, well-typed answer.

The non-strict control was not error-free. It also cleared a quota for “Set the quota to nothing” when the oracle required clarification. ArgBridge returned ten wrong nonempty patches: seven produced a wrong simulated final state and three were redundant theme writes. The non-strict control returned three wrong nonempty patches. No real-world harm was demonstrated because no live handler ran.

The manual baseline also needed criticism

The handwritten strict wrapper was intended as a strong comparator, but post-run inspection found an underdocumented mapping: it used leaf names such as timezone without explicitly identifying settings.timezone. Actual responses cited this mismatch while asking for clarification. It scored 29/48 on the nested workflow, versus 42/48 for ArgBridge and 48/48 for non-strict.

That flaw prevents a claim of superiority over a fully specified manual wrapper. It was documented rather than silently repaired in the frozen benchmark.

The release added a separate explicit-path comparator. It and ArgBridge both restored 41 expected update patches in deterministic tests. Its four schema representations used 817 content tokens versus ArgBridge's 1,281, and its reference arguments were smaller. These are offline encodings, not model-generated results. The revised comparator has no measured live acceptance or intent accuracy. Revised comparison and tradeoffs.

A reversible representation has a price

Across the original twelve-schema corpus, source schemas occupied 1,611 bytes and generated wire schemas 8,305 bytes: 5.16× expansion. Factoring reduced that to 7,527 bytes, or 4.67×, while retaining the decoder and wire values.

Bytes are not billed tokens. For those same canonical schema representations, token counts were 415 for the source, 1,874 for ArgBridge and 1,750 for the factored version. The separate four-workflow live evaluation recorded provider usage and tokenized the actual generated argument strings.

In that run, original ArgBridge averaged 34.7 argument tokens versus 14.9 for non-strict. Median provider latency was 1.030 seconds versus 0.850 seconds. These are observations from a sequential run and its network environment, not isolated causal estimates or production latency guarantees. Factoring reduced intent-request cost by 3.5% relative to original ArgBridge, but did not eliminate the semantic failures or overhead.

Engineering judgment after a negative result

The narrow capability remains useful to examine: automatic reversible transport when strict closed schemas are an integration requirement and the original contract needs presence or typed maps. The experiment did not establish that this convenience saves development time, improves model accuracy or creates customer demand.

For a small, stable tool set, a domain-specific operation wrapper may be clearer and smaller. If non-strict tools plus local validation meet the integration's requirements, that is a credible control to test. When a framework's native semantics already match the API, additional encoding may add complexity without value.

Based on the reported findings, I stopped product expansion and shifted the work from commercial exploration to an experimental open-source research release. The remaining scope covered packaging, documentation and fixes for demonstrable defects. Further product features would not answer the unresolved question of model reliability or supply missing evidence of demand.

The practical lessons for an AI systems team are concrete:

1. Define success at the operation boundary. Measure the intended action, exact patch and final state separately. JSON validity is one intermediate signal.
2. Keep the original contract as the final validator. Do not assume a lossy schema transformation preserves its meaning. Keep authorization and execution policy separate from validation.
3. Benchmark the simplest viable alternative. A generated protocol needs to earn its schema, argument and reasoning overhead against a well-specified manual control and the original contract.
4. Report coverage and cost denominators. Rejected schemas, wrong accepted patches and ambiguous instructions belong in the results. Cost per request alone can reward cheap mistakes.
5. Preserve failed hypotheses. Freeze inputs, retain actual outputs and label post-run fixes separately. A better comparator creates a new experiment; it cannot rewrite the old result.

The released package has meaningful engineering evidence—bounded parsing, deterministic and property tests, explicit refusal paths, reproducible artifacts and CI—but that evidence does not establish product-market fit or production-safe tool execution.

AI assistance and study limitations

AI assistance: Codex carried out substantial research, implementation, test development, evaluation tooling, analysis and release preparation. This article was also prepared with AI assistance.

The implementation, synthetic corpus and scorer came from the same development process. A separate expected-outcome file is not an independent reviewer. Recomputing 960 scores and 995 usage-cost calculations with zero discrepancies verifies consistency with the saved records and scorer; it does not independently validate the oracle. No external security review, customer adoption or commercial validation is claimed.

The study covers one model snapshot, four small workflows, 56 English instructions and correlated repeats. This German/English publication does not add a German-language model evaluation. There is no blinded holdout, multi-model study, load test or real downstream execution. The new release example was tested offline with mocked SDK transport; historical provider acceptance must not be attributed to a fresh live run of that example.

The compiler deliberately refuses many JSON Schema constructs. Decimal multipleOf inherits Python/jsonschema behavior, including the documented 0.3/0.1 edge case. macOS, alternate interpreters and every dependency combination remain unverified. Better evidence would come from independently supplied workflows and a new held-out comparison against the revised manual control.

Further feature development would require a concrete integration that needs strict generation and a new held-out evaluation against the revised manual wrapper demonstrating a useful advantage. The current results do not justify it.

Inspect the release and reproduce the evidence

• ArgBridge v0.1.0a1 experimental GitHub prerelease: wheel, source distribution, exact-commit source ZIP and SHA-256 manifest. It is not published to PyPI.
• Release source at 10556c99833149059ea9b86621a6b41acd7c0684: architecture, CLI, tests and examples.
• Full 995-request evaluation, preserved response records, analysis and audit.
• Offline reproduction instructions: use historical commit a9cd459c8cd58624af8102e1dc714c81678a79f9 and its pinned dependencies to recreate the historical plan and recompute the published analysis. Release code must not be silently substituted for the frozen experiment. A new live run requires separate credentials and paid inference.
• Final release-commit CI: all 13 jobs passed, covering Windows/Ubuntu Python 3.11–3.15, minimum dependencies, offline evaluation and packaging. The 113 distinct tests are not 113 independent model experiments.