Key takeaways
You.com tested their plugin on NeoSigma using GPT-6 Sol on the Pi agent harness, simulating how real users' agents interact with the plugin. The plugin boosted accuracy from 41% to 71% on FinSearchComp T2, 119 questions from FinSearchComp, a public benchmark of financial search questions annotated by financial experts. We also found a key cost-performance trade-off: You.com's general Search endpoint achieved nearly the same accuracy as its specialized Finance endpoint (69% vs. 71%) at 1/22 the cost.
One question from the benchmark illustrates why:
“As of the end of fiscal year 2022, what was the value of NVIDIA’s long-lived assets in Taiwan, in millions of dollars?”
No plugin: $463 million. The model answered from memory and was wrong.
You.com Search: $379 million. Correct.
You.com Finance: $379 million. Correct.
This wasn’t an isolated result. We found that in 28 of the 119 questions, GPT-6 Sol went from wrong without a plugin to right with either You.com endpoint, showing that the cheaper search endpoint delivered the same benefit.
Why evaluate your plugin on NeoSigma
A plugin's success depends on how agents actually use it. Testing the plugin without an agent shows how it responds to the inputs you chose, not how an agent will use it. In practice, your users' agents decide when to call your plugin, what inputs to send, and what to do with the results, and any of those steps can go wrong.
NeoSigma tests your tools inside live agents on different harnesses, connected through an MCP server, plugin or as a skill, to reveal its real impact before your users do. Our customers use the NeoSigma platform to answer questions like these before they ship new features:
Does the plugin help? NeoSigma recreates the same environment for each run, with the same agent harness, model, tasks and sandbox, once with your plugin and once without. The plugin is the only variable that changes, so any difference in results comes from the plugin.
How does it perform across agent harnesses? You can run the same tasks on different agent harnesses, such as Codex, Claude Code, Pi, etc. and compare how well each one uses your plugin. We support over 20 harnesses that you can choose from.
Does a new model still use it well? A harness tuned for an older model can perform worse with a newer one, which we call the scaffolding tax. Rerun your evals when a new model comes out to see whether performance drops, so you can update your skill or API before your users notice.
Where does it break? Every tool call is traced and every answer is graded against a reference, so you can open the failed tasks and see exactly what went wrong.
Is it getting better? NeoSigma turns failures into new evals and automatically reruns them, so each fix to your plugin is checked and a regression is caught before your users find it.
How You.com tested its plugin on NeoSigma
You.com provides search and inference APIs for AI. Its plugin gives agents tools that include Search, a general web search, and Finance, a research tool built for financial questions. The experiment tested which of these two tools helps agents answer financial questions best, and whether either beats having no plugin at all.
Setup
Tasks: FinSearchComp is a public benchmark that tests how well agents search for and reason about financial data. Financial experts wrote and checked its questions, and the answers come from company filings, government sources and financial databases. We used the 119 global questions of its T2 task set, Simple Historical Lookup, which asks for exact past figures, such as a company's reported value at the end of a fiscal year.
Agent: we ran You.com through its MCP server on the Pi harness with GPT-6 Sol.
Configurations: three experiments with the same agent, model and tasks.
No plugin: the agent answers from its own knowledge.
Search: the agent makes one You.com search and answers from the results, adding its own knowledge when the results are incomplete.
Finance: the agent runs one You.com Finance research job and answers from its result.
Grading: each answer was scored against the reference with FinSearchComp’s own grading prompt. Each question ran once per configuration.
Pricing: You.com’s published list prices per call: Search at $5 per 1,000 calls, and Finance (Deep tier) at $110 per 1,000 calls. Each question used at most one You.com call. Model costs are not included.
Experiment Results: General search nearly matched the finance tool at 1/22 of the price
GPT-6 Sol accuracy on 119 FinSearchComp T2 questions versus You.com price per 1,000 calls. With no plugin, the agent reached 41% accuracy. Search reached 69% at $5 per 1,000 calls, and Finance reached 71% at $110 per 1,000 calls. Each question ran once per configuration, and prices are You.com's list prices.
Our experiment shows that You.com Search came within 2 points of Finance (69% vs 71%) at 1/22 of the price per call.
For You.com, this analysis shows that Search is a strong default to recommend to agent builders, with Finance reserved for the questions where its small extra accuracy is worth the cost.
Try it on your plugin
Your users will find out how your plugin performs inside their agents. With NeoSigma, you find out first. Running these experiments takes only a few steps: connect your plugin's MCP server, use public benchmarks or let NeoSigma create evals from your production traces, and run the same tasks with and without your plugin on the harnesses and models your users run. Request a demo to run your first plugin eval.