Small LLMs for Enterprise Agents: Cost, Speed & Accuracy

Define the Small Model and Set a Fair Baseline

There is no single parameter-count cutoff that determines whether a model is “small” for every enterprise. Define the category in the context of your evaluation: models that are cheaper or faster to run than your larger-model baseline, or that meet specific infrastructure and deployment constraints.

Most importantly, distinguish the model from the whole agent system. An agent may combine a language model with prompts, business rules, tools, retrieval, orchestration, and human review. Strong retrieval can help a modest model answer questions accurately; poor tool integration can undermine even a capable model. Record which components are in use so you can understand what drives the result.

Compare models under the same conditions: prompts, tools, retrieved context, input data, and evaluation rules. Keep the settings that affect output consistent where possible. If one system gets a better search index or more tool access, the comparison will not show whether the model itself is responsible for the difference.

Build a Test Set from Real Enterprise Workflows

Use real tasks, not only general question-answering benchmarks. Ask teams to identify frequent and important workflows, then create representative test cases with expected outcomes. Remove sensitive information or use approved test data.

Include different kinds of work, such as:

• Multi-step requests, like checking an order, confirming its status, and drafting a response.
• Tool use, including choosing the right tool, supplying valid arguments, and interpreting results.
• Retrieval from policies, contracts, knowledge bases, or other approved sources.
• Structured outputs, such as a correctly populated JSON object or case-management form.
• Constraints, including permission limits, required fields, and rules about what the agent must not do.

Test failure cases as deliberately as routine tasks. Include ambiguous requests, unavailable tools, conflicting documents, missing information, and cases where the correct action is to ask a question or escalate. For example, an agent should not invent a refund approval when the policy is unclear or the user lacks authorization.

A test set should reflect the expected mix of work. If routine requests make up most production traffic, they should be represented accordingly, but rare high-impact scenarios also need explicit coverage. Keep a held-out set for later checks so repeated tuning does not make the evaluation misleading.

Measure Whether the Agent Completes the Task Correctly

Track agent task success rate : the share of tasks completed to the required standard, not simply the share of prompts that receive an answer. Define success before testing. For a refund workflow, that could mean retrieving the correct order, applying the policy, avoiding an unauthorized action, and returning the required response.

Score the parts of the work that matter:

• Correctness: Is the answer or action supported by the available information?
• Tool-call accuracy: Did the agent select the right tool and provide valid arguments?
• Constraint adherence: Did it follow permissions, format requirements, and business rules?
• Error recovery: Could it handle a failed tool call or incomplete result without looping or fabricating information?
• Escalation: Did it recognize when it lacked the confidence, authority, or information to proceed?

Use automated checks for objective requirements, such as valid JSON or correct tool arguments, and human review for judgment-heavy outcomes. Review failure examples, not just aggregate scores. A model with a high average success rate may still be unsuitable if it mishandles a small number of critical cases.

Measure End-to-End Latency and Cost per Successful Task

Model response time is only one part of enterprise agent latency . A task may involve several model calls, search, tool execution, network delays, and retries. Measure from the user’s request to the completed result, using the same workflow and infrastructure conditions you expect in production.

Report median latency and tail latency, such as the 95th percentile. The median describes a typical request; tail measurements show how slow the experience can become during more demanding tasks or busy periods. Break down the time by component so you can identify whether the model, retrieval, or a tool is the bottleneck.

Then calculate LLM inference cost and the broader cost per successful task. Include model calls and tokens, retries, retrieval, other agent services, and any human review that is part of the workflow. Divide the total cost by successfully completed tasks, rather than by all attempts. A cheaper model that needs repeated retries or often hands work to a person may cost more per useful outcome.

For example, compare two systems on the same 100 test tasks. If a small model costs less per attempt but completes fewer tasks, needs more retries, or triggers more review, its cost per successful task may be higher. Report both task quality and cost so that savings do not conceal a drop in reliability.

Use the Results to Decide What to Deploy

Set minimum thresholds for task success, latency, and cost before choosing a model. Also check privacy requirements, reliability, and throughput under expected demand. A system that performs well in a small offline test may not meet production needs when many users make requests at once.

Use a larger model or human review when tasks are ambiguous, consequential, or outside the small model’s demonstrated capability. Routing can be based on workflow type, explicit risk rules, or signals such as missing information and failed tool calls. Test the routing policy as part of the agent, since incorrect routing can erase the benefits of a model that performs well on its own.

Keep retrieval results separate from model results. For example, Box Content Retrieval was reported as targeting faster, more accurate, and less expensive agent queries, with expected improvements of four times faster responses, 20% fewer errors, and 90% fewer tokens. These are reported expectations for that retrieval feature, not evidence that a small LLM alone achieves those gains. Evaluate retrieval and the model as distinct components, then measure the combined system end to end.

Choose a representative set of enterprise workflows and run a controlled pilot. Compare small and larger models on task success, end-to-end latency, and cost per successful task. Set clear routing and escalation thresholds before deployment, then monitor real outcomes and retest when workflows, tools, or models change.

Step-by-Step Guide

1. 1

   Define the model and baseline

   Specify what counts as a small model for your environment, choose a larger-model baseline, and hold prompts, tools, context, data, and evaluation rules consistent across systems.

2. 2

   Build representative workflow tests

   Create approved test cases from frequent and important enterprise tasks. Include tool use, retrieval, structured outputs, permissions, ambiguous requests, missing information, and failure scenarios.

3. 3

   Score task completion

   Define success before testing, then score correctness, tool-call accuracy, constraint adherence, error recovery, and appropriate escalation using automated checks and human review.

4. 4

   Measure latency and cost

   Time each task from request to result, report median and 95th-percentile latency, and include model calls, tokens, retries, retrieval, services, and human review in cost per successful task.

5. 5

   Set deployment and routing thresholds

   Choose minimum quality, latency, and cost requirements, plus privacy and reliability checks. Route tasks outside the model's demonstrated capability to a larger model or human reviewer.

6. 6

   Pilot and monitor the system

   Run a controlled pilot on representative workflows, monitor real outcomes, review failures, and retest when the model, tools, retrieval, or workflows change.

Key Statistics

• Box Content Retrieval reported an expected 4x faster response time.This is a reported expectation for the Box Content Retrieval feature cited in the article, not a demonstrated result for small LLMs alone.
• Box Content Retrieval reported an expected 20% reduction in errors.The article attributes this expected improvement to the retrieval feature; it should not be interpreted as an isolated small-model performance gain.
• Box Content Retrieval reported an expected 90% reduction in tokens.This is a reported expectation for the retrieval feature. The article recommends evaluating retrieval separately from the model and measuring the combined system end to end.

Frequently Asked Questions