Description
Getting a watch, a smart home panel or a car head unit to understand "dim the living room and lock the back door" usually means either a round trip to a cloud LLM, with the latency and connectivity that implies, or a local model hundreds of megabytes large that the device cannot hold. Needle takes a third route: the whole model is a single 8 to 29 MB file that gives up general chat to focus on tool calls, structured extraction and embeddings, and beats models ten times its size on mobile tool calling.
It also tells you how sure it is. Every reply carries a calibrated confidence score, two requests in one sentence come back as two ordered calls, and a request no tool covers returns an empty list instead of a guess. Every depth from 2 to 20 layers is a deployable model, so smaller hardware simply takes a shallower slice, and the project site runs a smart home sandbox right in the browser on WebAssembly.
Tool calls: Hand it the functions your app exposes and it picks the right ones and fills every argument from what the user said. In Python, decorate a function with
Structured extraction: Declare a shape, pass in messy invoice, booking or notification text, and get typed fields back. Grammar-constrained decoding guarantees the output parses, and enum fields turn it into a classifier.
Text embeddings: The same model returns sentence vectors for on-device search, matching and routing, with no second model to ship.
Confidence routing: Use the per-reply score to decide whether to act, ask the user to confirm, or refuse, which adds a safety net for sensitive actions.
Depth ladder: Run 2 layers on a microcontroller and the full stack on a phone, all from the same weights.
Fine-tuning: Train LoRA adapters locally with
Engines for every platform: Each target ships a prebuilt engine under 1 MB, and
It also tells you how sure it is. Every reply carries a calibrated confidence score, two requests in one sentence come back as two ordered calls, and a request no tool covers returns an empty list instead of a guess. Every depth from 2 to 20 layers is a deployable model, so smaller hardware simply takes a shallower slice, and the project site runs a smart home sandbox right in the browser on WebAssembly.
Features
Tool calls: Hand it the functions your app exposes and it picks the right ones and fills every argument from what the user said. In Python, decorate a function with
@needle.tool: the signature supplies the types, the docstring the description, and run() executes the loop.Structured extraction: Declare a shape, pass in messy invoice, booking or notification text, and get typed fields back. Grammar-constrained decoding guarantees the output parses, and enum fields turn it into a classifier.
Text embeddings: The same model returns sentence vectors for on-device search, matching and routing, with no second model to ship.
Confidence routing: Use the per-reply score to decide whether to act, ask the user to confirm, or refuse, which adds a safety net for sensitive actions.
Depth ladder: Run 2 layers on a microcontroller and the full stack on a phone, all from the same weights.
Fine-tuning: Train LoRA adapters locally with
needle finetune, or run full fine-tuning on the Cactus platform with generated training data. On DroidCall every depth gains 18 to 36 points, and from 4 layers up it passes DeepSeek V4 Flash.Engines for every platform: Each target ships a prebuilt engine under 1 MB, and
needle build --platform macos-arm64 fetches the engine plus weights for macOS, Linux ARM, the browser, WASI or air-gapped setups.