Fine-tune a small open model into a reliable JSON tool caller, on one GPU, in about three hours, for about eight dollars. Every number on this page traces to a published artifact.
A reference implementation for adapting a small open language model to emit exactly one schema-valid JSON tool call per request, built on one premise: a fine-tuning claim is only as good as its evidence.
Given a user query and a set of tool schemas, the model must emit one JSON object: {"name": ..., "arguments": ...}. Nothing else counts. A conservative parser scores every output; every parse failure counts against every metric.
QLoRA training moves 41,943,040 adapter parameters, 0.52% of the base model's 8.03 billion. The frozen base stays byte-identical to NVIDIA's published checkpoint, so the adapter is the complete diff between the two compared systems.
Full-call exact match rises from 0.705 to 0.874 on 1,000 held-out prompts. The comparison report refuses to exist unless both sides provably used identical data, prompts, parser, and decoding, checked by SHA-256.
The base model loses 4.2 points on French queries. Mixing machine-translated French rows into training closes the gap to +0.3 points, with tool schemas and gold answers byte-identical across both languages.
The strategic argument for small open models plus task adapters, and the measurement that makes it concrete.
Most teams consume language models the way they consume electricity: a hosted API, a per-token bill, and no say over what happens behind the socket. That trade is often right. It stops being right when the capability is core to your product, when your data cannot leave, or when the model must work in your language, your regulation, your vocabulary. Then the question becomes: can you hold the capability yourself?
Post-training is how you hold it. A pinned open checkpoint plus a small trained adapter is a capability you own: you can pin it, audit it, re-run it, and ship it anywhere a GPU runs. NVIDIA frames this as sovereign AI. Announcing Nemotron 3, the company put it directly: organizations "from Europe to South Korea" are adopting open models "to build AI systems aligned to their own data, regulations and values." Jensen Huang at GTC Paris 2025: "You can now ask and get questions answered in the language, in the culture, in the sensibility of your country."
The economics point the same way. NVIDIA Research's position paper Small Language Models are the Future of Agentic AI argues that agentic systems mostly make small, repetitive, low-variation calls: route an input, pick a tool, format arguments. For that work, small specialized models are "sufficiently powerful, inherently more suitable, and necessarily more economical" than frontier generalists.
| property | hosted frontier api | pinned base + your adapter |
|---|---|---|
| weights | provider's, unversioned for you | pinned revision, byte-auditable |
| behavior over time | can change under you | frozen until you retrain |
| task capability | prompt, hope, re-prompt | a 168 MB artifact you version |
| your data | leaves the building | stays on your GPU |
| language coverage | whatever the provider tuned | measured, then trained where it lags |
| evidence | a marketing benchmark | re-runnable comparison report |
| marginal cost | per token, forever | about $8 of training, then your serving |
Sommelier's contribution to this argument is a worked, measured example of the regional-language case. Tool calling should work as well in French as in English; that is a claim worth measuring rather than assuming. Section 09 shows the measurement: the open base model quietly loses 4.2 points of full-call exact match when the user writes French, and one constrained-translation training run closes the gap to noise. If your users write French, or Korean, or Hindi, that gap is invisible until you measure it, and fixable once you can post-train.
Sommelier builds on nvidia/Llama-3.1-Nemotron-Nano-8B-v1. What makes the Nemotron effort unusual is not just open weights; it is that NVIDIA publishes the post-training data, the recipes, and the training code too.
| field | value |
|---|---|
| derived from | Meta Llama-3.1-8B-Instruct |
| architecture | dense decoder-only, 32 blocks, 8.03B params |
| post-training | SFT (math, code, reasoning, tool calling), then RL: REINFORCE (RLOO) + online RPO; final = SFT ⊕ RPO checkpoint merge |
| reasoning toggle | system prompt: detailed thinking on / off |
| context | 128K tokens |
| released | 2025-03-18 |
| license | NVIDIA Open Model License + Llama 3.1 Community License |
The Nano-8B is the small tier of the Llama-Nemotron family (Nano 8B, Super 49B, Ultra 253B). The larger tiers are Llama 3 checkpoints taken through neural architecture search, distillation, and continued pretraining; all three then share a reasoning-focused SFT + large-scale RL phase, with the Nano keeping the unmodified Llama 3.1 8B architecture. The paper claims these are the first open-source models with a runtime reasoning toggle.
Sommelier treats this model the way the license intends: as a foundation to specialize. Tool calling is one of the four capabilities its SFT phase explicitly targeted, which makes the base a fair, strong baseline; under Sommelier's evaluation policy (greedy decoding, conservative parser) it already gets 70.5% of full calls exactly right before any adapter.
| release | date | what is open | source |
|---|---|---|---|
| Llama-Nemotron (Nano 8B / Super 49B / Ultra 253B) | 2025-03 to 2025-05 | weights (NVIDIA Open Model License), full post-training dataset (CC-BY-4.0), NeMo / NeMo-Aligner / Megatron-LM code, paper | arXiv:2505.00949 |
| Nemotron Nano 2 (9B hybrid Mamba-Transformer) | 2025-08 | weights incl. base checkpoints, most pre/post-training data; 12B base pretrained on 20T tokens (FP8 recipe) | arXiv:2508.14444 |
| NVFP4 pretraining | 2025-09 | 4-bit floating-point pretraining recipe validated to a 10T-token horizon | arXiv:2509.25149 |
| Nemotron 3 (Nano / Super / Ultra, hybrid latent MoE) | 2025-12 | Nano weights at launch (Super and Ultra announced for H1 2026), training datasets, RL environments and libraries, aimed at multi-agent systems | nvidianews |
The source is Salesforce's xlam-function-calling-60k. Sommelier drops every excluded row under a declared, counted reason: what it keeps, what it drops, and why are all on record.
xlam-function-calling-60k (CC-BY-4.0, gated) contains 60,000 query-plus-tools examples produced by Salesforce's APIGen pipeline over 3,673 executable APIs in 21 categories, each verified by format checks, actual execution, and semantic review; a 600-sample human audit found over 95% correct. Sommelier trains and scores exactly one call per request, so the 52.6% of rows whose gold answer contains two or more calls are dropped under a declared reason, not silently. The docs put it plainly: "These rows are not bad data. They are out of scope."
sha256(normalize_query(q)), computed before splitting, so a duplicated query can never sit in train and test at once. The docs call the post-split re-assertion "belt and suspenders, but the belt is the mechanism."Both derived datasets are public under CC-BY-4.0. The English dataset publishes the exact prepared splits plus the chat-formatted training text; the French dataset publishes 14,936 paired rows whose tool schemas and gold answers are byte-identical to their English sources. Browse them here:
Stages communicate only through schema-versioned files under an artifact root; every transition writes a manifest with SHA-256 checksums of its inputs and outputs. Click any block to inspect it.
Each stage names what it consumes, what it produces, and which invariant protects it.
QLoRA composes three precisions on purpose: the frozen base stored in 4-bit NF4, compute in bfloat16, and a float32 adapter that holds the entire learned change.
LoRA constrains the weight update of each targeted projection to a low-rank product: instead of moving all 16.8 million entries of a q_proj, training learns two thin matrices A (16×4096) and B (4096×16) whose product, scaled by α/r = 2, is added to the frozen weight. B starts at zero, so the untrained adapter is exactly a no-op; every point of measured improvement is attributable to training and nothing else.
The adapter attaches to all seven linear projections of each of the 32 blocks (q, k, v, o, gate, up, down), following the QLoRA paper's finding that adapting every linear layer recovers full fine-tuning quality. Total: 448 tensors, 41,943,040 parameters, 0.52% of the base's 8.03 billion, 168 MB on disk. The docs' summary: "the correction that turns the base model into a reliable tool caller is, in parameter count, one thirty-second of one component of the network it corrects."
NF4 stores the frozen base in blocks of 64 weights with one absmax scale each, snapped to 16 levels placed at standard-normal quantiles; double quantization compresses the scales themselves. The 16 GB bfloat16 base fits in roughly 5.7 GB. Dequantization happens per matmul; the docs put it: "the quantization is a storage format, not a compute format."
| knob | value |
|---|---|
| quantization | nf4-4bit, double quant, bf16 compute |
| lora rank / alpha / dropout | 16 / 32 / 0.05 |
| target modules | q, k, v, o, gate, up, down proj |
| epochs | 2 |
| batch | 4 per device × 4 accumulation = 16 |
| learning rate | 2e-4, cosine, warmup 0.03 |
| max sequence length | 4,096 tokens |
| optimizer steps | 938 / epoch · 1,876 total · ~5.9 s each |
| wall clock / memory | 10,996 s · peak 26,306 MiB |
The evaluator extracts the first balanced JSON span from the model's output and never repairs it. Every parse failure counts against every metric. Try it below: this playground runs a faithful JavaScript port of sommelier.parser.v1.
A lenient parser that fixes trailing commas or picks the most plausible of several objects answers a different question: not "can this model emit a valid tool call" but "can this model plus my repair heuristics emit one". Repair moves work from the model to the harness by an amount you cannot measure, and that amount differs between base and adapter. So the parser is deliberately conservative, and its strictness cannot favor either side, because both face the identical parser.
| metric | numerator | denominator |
|---|---|---|
| valid_json_rate | records with parse_status ok | all records, including every parse failure (INV-DATA-005) |
| function_name_accuracy | ok ∧ predicted name equals gold name | |
| argument_exact_match | ok ∧ canonical JSON of arguments equals gold's | |
| full_call_exact_match | ok ∧ name equals ∧ arguments equal | |
| argument_f1 | 2 × matched (path, value) pairs | predicted pairs + gold pairs, pooled; failed parses still contribute their gold pairs |
Equality is canonical-JSON equality: sorted keys, compact separators. The docs are explicit about the incentive: "reports would look better if failures were dropped from the denominator, which is precisely why they are not."
--
JS port of sommelier/evaluation/parse.py: first balanced span, string-and-escape aware, shape must be exactly {"name", "arguments"} or a one-element array of it. The Python module is canonical; the published numbers come from it, not from this widget. The refusal, bare-arguments, invalid-JSON, and adapter presets are verbatim outputs from the reference run; the multi-call preset is a synthetic illustration of the one-call contract.
No opening brace or bracket anywhere. Typical base failure: a polite prose answer asking for clarification instead of calling the tool. 9 of 1,000 base outputs.
An opening bracket exists but no balanced span extracts, or json.loads rejects the span. 11 of 1,000 base outputs.
Valid JSON, wrong shape. The base's dominant failure at 64 of 1,000: in 50 cases it emitted the arguments object bare, without the name/arguments wrapper, and in 14 it added or renamed keys.
Exactly one call with exactly the keys name and arguments. Base: 916 of 1,000. Adapter: 1,000 of 1,000. "ok" only means parseable; the arguments can still be wrong, which is what the other four metrics measure.
Two full runs are published. v1 trained on 15,000 English rows (run nemotron-8b-full-3); v2 trained on the same English rows plus 13,113 French pairs (run nemotron-8b-fr-full-4). Both evaluated greedily on the same held-out prompts, by digest.
| metric | base | v1 adapter | Δ |
|---|---|---|---|
| valid JSON rate | 0.9160 (916/1000) | 1.0000 (1000/1000) | +0.0840 |
| function name accuracy | 0.9110 (911/1000) | 0.9960 (996/1000) | +0.0850 |
| argument exact match | 0.7070 (707/1000) | 0.8760 (876/1000) | +0.1690 |
| argument F1 | 0.7569 (3858/5097) | 0.9291 (5122/5513) | +0.1722 |
| full-call exact match | 0.7050 (705/1000) | 0.8740 (874/1000) | +0.1690 |
Read the columns top to bottom and the adapter's remaining errors localize. Validity saturates (1000/1000). Naming is nearly solved (996/1000). What remains is argument construction: 996 correct names, 876 exact argument matches, 874 fully exact calls. The metrics are ordered diagnostics, not redundancy.
The measurement changes exactly one variable: the language of the user's query. Tool schemas, gold answers, parser, and decoding stay byte-identical, so whatever moves in the numbers is the language.
The v2 experiment built a French paired variant of every selected row by constrained translation. Only the query is translated (Mistral-Nemo-Instruct-2407, greedy). Tool schemas and gold answers stay byte-identical to the English source, enforced at preparation time. Every gold argument value that appears verbatim in the English query is a protected span that must survive translation byte for byte, with boundary-aware matching so a protected "2" is not satisfied by "2026". Each French row inherits its English source's train/validation/test split, so a translation can never leak across a split boundary. Rows whose translation fails any audit are dropped with a counted reason: of 17,000 selected rows, 14,936 pairs survived (2,011 lost a protected span, 53 came back untranslated). The pairs inherit their sources’ splits, 13,113 train and 879 test with the rest validation, which is why training saw 13,113 rows while 14,936 are published.
| model | en | fr | gap (fr - en) |
|---|---|---|---|
| base Nemotron-Nano-8B | 0.7050 | 0.6633 | -4.2 pts |
| v1 adapter (English-only training) | 0.8740 | 0.8510 | -2.3 pts |
| v2 adapter (mixed en+fr training) | 0.8700 | 0.8726 | +0.3 pts |
Same tools, same gold calls, same parser; only the query language changed. The penalty concentrates in argument construction: argument exact match drops from 0.707 to 0.666.
Without a single French example, the v1 adapter lifts French full-call exact match 18.8 points (0.663 to 0.851), narrowing the gap from 4.2 to 2.3 points. Format discipline transfers across languages for free.
With 13,113 French pairs mixed in, French matches English on every metric to within a third of a point. Cost on English: at most 0.8 points, within one standard error at n=1,000; the regression is reported, not hidden.
The comparison is anchored end to end: the v2 run's English prompt set is byte-identical to the reference run's (digest a0da8fa2…), so both experiments scored exactly the same 1,000 English prompts. Training took 5 h 42 min (3,516 steps over 28,113 rows) at a nearly identical peak (26,369 vs 26,306 MiB); French rows cost the same tokens as English ones (p95 711 vs 716 tokens), so the bilingual run's extra bill is purely its extra rows.
The recipe generalizes: measure the regional gap under controlled conditions, then close it with data you can publish. The same method, unchanged, applies to any language whose gold arguments survive translation audits.
Aggregates hide texture. Below: what happened to each of the 1,000 held-out prompts, and a browser over real generations from the published runs. Nothing is paraphrased.
Samples are drawn verbatim from generations.jsonl of runs nemotron-8b-full-3 and nemotron-8b-fr-full-4, joined with the published test split. The selection covers every major cell of the matrix above, including the cases where the adapter is wrong. Verdicts use the same canonical-JSON comparison as the published metrics.
Both adapters load onto the unquantized bfloat16 base. Pick the English one for English-only traffic, the en+fr one for bilingual traffic. The prompt policy is part of the contract: reproduce it exactly.
import json, torch from peft import PeftModel from transformers import AutoModelForCausalLM, AutoTokenizer BASE = "nvidia/Llama-3.1-Nemotron-Nano-8B-v1" ADAPTER = "abdelstark/llama-3.1-nemotron-nano-8b-xlam-tool-calling-lora" tokenizer = AutoTokenizer.from_pretrained(BASE) model = AutoModelForCausalLM.from_pretrained(BASE, dtype="auto", device_map="auto") model = PeftModel.from_pretrained(model, ADAPTER) # A SYSTEM = ("You are a tool-calling model. Select the correct tool " "and return only the JSON tool call. Do not include explanations.") tools = [{"name": "lookup_weather", "description": "Look up the current weather for a city.", "parameters": {"city": {"description": "Name of the city.", "type": "str"}}}] # B tools_json = json.dumps(tools, separators=(",", ":"), sort_keys=True) messages = [ {"role": "system", "content": f"{SYSTEM}\n\nAvailable tools:\n{tools_json}"}, {"role": "user", "content": "What is the weather in Paris today?"}, ] prompt = tokenizer.apply_chat_template(messages, tokenize=False, add_generation_prompt=True) inputs = tokenizer(prompt, return_tensors="pt", add_special_tokens=False).to(model.device) # C out = model.generate(**inputs, do_sample=False, max_new_tokens=512) print(tokenizer.decode(out[0][inputs["input_ids"].shape[1]:], skip_special_tokens=True)) # [{"arguments":{"city":"Paris"},"name":"lookup_weather"}]
{"type": "object", "properties": ...} tools are out of distribution and typically fail to parse.add_special_tokens=False, exactly as the evaluator does.…-xlam-tool-calling-fr-en-lora and French queries work at English-level accuracy. Same base, same prompt policy, same license terms.
vllm serve nvidia/Llama-3.1-Nemotron-Nano-8B-v1 \
--dtype bfloat16 \
--enable-lora --max-lora-rank 16 \
--lora-modules sommelier-tool-caller=abdelstark/llama-3.1-nemotron-nano-8b-xlam-tool-calling-lora
curl -s http://localhost:8000/v1/chat/completions \ -H 'Content-Type: application/json' \ -d '{ "model": "sommelier-tool-caller", "messages": [ {"role": "system", "content": "You are a tool-calling model. Select the correct tool and return only the JSON tool call. Do not include explanations.\n\nAvailable tools:\n[{\"description\":\"Look up the current weather for a city.\",\"name\":\"lookup_weather\",\"parameters\":{\"city\":{\"description\":\"Name of the city.\",\"type\":\"str\"}}}]"}, {"role": "user", "content": "What is the weather in Paris today?"} ], "temperature": 0.0, "max_tokens": 512 }'
One server registers two model names, the base id and sommelier-tool-caller, so base-versus-adapter A/B requests hit the same endpoint. Tool schemas go in the system message as canonical JSON: sorted keys, compact separators, the exact policy the adapter saw in training.
# deploys vLLM + the published adapter behind an OpenAI-compatible URL # (scales to zero when idle; cold start takes a few minutes) git clone https://github.com/AbdelStark/sommelier && cd sommelier uv run modal deploy remote_serving.py # URL printed on deploy uv run modal run remote_serving.py # smoke: canonical request, parsed
The smoke entrypoint sends the canonical weather request and parses the reply with sommelier's own parser, so a green smoke run means the serving path agrees with the evaluation path. Set SOMMELIER_SERVE_API_KEY to require a Bearer token; without it the URL is open.
uv run sommelier serve adapter --config examples/config.full.yaml \ --adapter <path-to-adapter-dir> curl -s http://127.0.0.1:8000/v1/chat/completions \ -H 'Content-Type: application/json' \ -d '{"messages": [{"role": "user", "content": "What is the weather in Paris today?"}], "tools": [{"name": "lookup_weather", "description": "Look up the current weather for a city.", "parameters": {"city": {"description": "Name of the city.", "type": "str"}}}], "temperature": 0.0, "max_tokens": 256}' # response reports parse_status instead of repairing output: # {"raw_text": "[{\"arguments\":{\"city\":\"Paris\"},\"name\":\"lookup_weather\"}]", # "parsed_call": {"name": "lookup_weather", "arguments": {"city": "Paris"}}, # "parse_status": "ok", "model_kind": "adapter"}
Deliberately not a production system: no autoscaling, no multi-tenant isolation, no streaming. It reuses the evaluation prompt policy and parser, which is the point; the response tells you honestly whether the output parsed.
sommelier release preflight machine-checks the notices before any release.
Three ladders: a fixture run on a laptop (no GPU, no accounts), a smoke run (one GPU, about half an hour), the full run (one GPU, under four hours, about eight dollars).
| rung | needs | gets you |
|---|---|---|
| fixture | Python 3.13+, uv | data prepare + format build on synthetic rows; byte-identical rerun outputs |
| smoke | + Modal account, HF token, xlam gate accepted | the full six-stage chain at 100/20/20 (the docs: "a plumbing check, not a measurement") |
| full | same, ~4 GPU-hours on an L40S | the published numbers, within noise: bit-identical generations across GPU stacks are not guaranteed |
git clone https://github.com/AbdelStark/sommelier && cd sommelier uv sync --extra dev && uv run pytest # 380+ tests uv run sommelier data prepare --config examples/config.smoke.yaml --fixture \ --out examples/artifacts/runs/local/data --run-id local uv run sommelier format build --config examples/config.smoke.yaml \ --data examples/artifacts/runs/local/data \ --out examples/artifacts/runs/local/formatted --run-id local --fixture
# smoke first. always smoke before you spend hours. uv run modal run remote_pipeline.py \ --config examples/config.smoke.yaml --mode smoke --max-rows 2500 # the reference run (en) SOMMELIER_GPU=L40S SOMMELIER_TIMEOUT_SECONDS=28800 \ uv run modal run --detach remote_pipeline.py \ --config examples/config.full.yaml --mode full --max-rows 60000 # the bilingual run (en+fr): build the French pairs, then train on both SOMMELIER_GPU=L40S SOMMELIER_TIMEOUT_SECONDS=14400 \ uv run modal run --detach remote_translate.py \ --config examples/config.full.yaml --run-id fr-translate-2 SOMMELIER_GPU=L40S SOMMELIER_TIMEOUT_SECONDS=36000 \ uv run modal run --detach remote_pipeline.py \ --config examples/config.full.yaml --mode full --max-rows 60000 \ --translation-run-id fr-translate-2
config.resolved.yaml, per-stage manifests with SHA-256 checksums, splits, formatted text, the adapter, raw generations, and the two reports. A failed run leaves explicit failed manifests, never silent partial state.Scope discipline is part of the method. These boundaries come from the project's own docs.
52.6% of xlam rows answer with multiple calls and are filtered out under a declared reason. The published numbers say nothing about multi-call planning, multi-turn agents, tool execution, or retry logic.
"2" (string) does not match 2 (number); date spellings differ; lists compare by index. Semantically equivalent arguments count as misses, for base and adapter equally. Absolute numbers are pessimistic; the delta stays fair.
Reviewed on samples, not natively authored. 12.1% of pairs were excluded because gold arguments embed English text, so the French slice leans toward language-neutral arguments. Instruction-language effects are unmeasured: the system prompt stays English by design.
Nothing here ranks the model against public benchmarks or hosted models. The only claim is base versus adapter, under provably identical conditions, on this task. The v2-versus-v1 English difference (0.3 to 0.8 points) is within one standard error and stays undecided.
Determinism guarantees cover preparation, formatting, and evaluation. GPU training varies across driver stacks; reproducers should expect metrics within noise, not identical bytes.
Both serving paths exist to let you poke the adapter, not to carry production traffic. No autoscaling, no isolation, no streaming, no auth by default. The evaluation claim never depends on serving.
Ten questions on the core ideas. Tap an option to reveal the answer.