How Small Can a Private LLM Be? What 15 Models Taught Us About Knowledge-Base Assistants

“Our findings show that choosing an LLM for RAG is not simply a question of model size or answer quality. A reliable enterprise assistant must retrieve the right information, follow source-priority rules, and use tools as intended. These behaviors need to be tested explicitly and reinforced by the system architecture, rather than left to the model to decide.”

— Vadzim Tashlikovich, Head of ERP Solutions Department, SCAND

The smallest model that can summarize a document is not necessarily the smallest model you can trust with your company knowledge base. To find the best local LLM for RAG on real company documents, SCAND’s R&D team tested 15 open-weight model configurations on basic operations: summarization, knowledge-base retrieval, instruction following, and spreadsheet calculations.

While summarization is a relatively straightforward task, answering questions over internal documents requires a model to retrieve the right information, follow source-selection instructions, and provide reliable answers grounded in the available context.

In SCAND’s R&D tests, most open-weight LLMs handled basic document tasks, but only two successfully found information in internal documents while following the required source-selection instructions.

The results highlight why choosing a model for private RAG requires more than comparing parameter counts or benchmark scores.

The Challenge: Running AI on Private Company Data

Enterprises want AI assistants that can answer questions about internal policies, analyze business documents, summarize reports, and retrieve information from corporate knowledge bases. Yet many organizations cannot send confidential data to public AI APIs because of security, privacy, or regulatory requirements.

Self-hosted, open-weight large language models (LLMs) offer an alternative. They can run on infrastructure controlled by the organization, allowing companies to retain greater control over their data and AI workflows.

A more comprehensive deployment may require organizations to build a private LLM solution around their own data, infrastructure, and security requirements. Depending on the use case, smaller on-device LLMs may also be suitable when local processing, reduced data transfer, or limited connectivity is a priority.

However, private deployment comes with a cost. GPU memory requirements, hardware availability, inference performance, and regular maintenance all influence the total cost of ownership. A larger model may offer stronger capabilities, but it can also make deployment way more expensive.

This leads to a practical question for enterprise technology leaders: what is the smallest model that can perform everyday tasks over a company knowledge base without compromising security or accuracy?

At first glance, the answer might seem straightforward. Most modern LLMs can summarize text, extract information, and answer questions about documents they receive directly.

But a knowledge-base assistant must do more than process supplied text. It must decide where to look for information, select the right tools, follow instructions, and distinguish verified internal knowledge from general knowledge.

These are examples of low-agency AI tasks: operations that involve limited decision-making and a small number of tool calls rather than long, autonomous plans. Typical tasks include:

• Searching internal documents and retrieving relevant information.
• Summarizing reports and analyzing policies.
• Comparing tables and checking business rules.
• Reading spreadsheets and calculating balances.
• Drafting instructions and answering employee support questions.
• Tracking changes and retrieving historical information.

These tasks do not require a fully autonomous AI agent. They do, however, require a model that follows the application’s rules. The surrounding workflow is just as important as the model itself, which is why AI agent development involves designing tool access, instructions, and safeguards alongside model selection.

Our R&D team evaluated open-weight LLMs on representative knowledge-base operations to understand where smaller models are sufficient, where larger models make a difference, and which failure modes matter most in a private enterprise environment.

One finding stood out: the main risk was not simply an incorrect answer. It was a model that silently skipped the internal knowledge base, relied on its general knowledge, or attempted to use an external search tool when the task required internal sources.

How We Tested the Models

We compared open-weight models using the same set of internal corporate documents, seven everyday questions, and a unified infrastructure for agents equipped with auxiliary tools.

Each model was instructed to prioritize internal documents. Questions were posed multiple times to assess how consistently the models adhered to the required workflow. This matters because two AI agents with the same LLM can give different results when their prompts, tools, runtime settings, or surrounding workflows differ.

Our LLM tool-calling benchmark focused on whether models could use available tools appropriately, retrieve evidence from the correct source, and follow the application’s instructions rather than simply generate plausible answers.

The agent harness provided five capabilities:

• Internal knowledge-base search.
• File-system commands.
• Internet and Wikipedia search.
• Calculations.
• Excel file reading.

We grouped the seven questions into four categories.

Each question (several runs) was scored as a pass or fail, producing a maximum score of seven. The evaluation deliberately covered more than the quality of the final response.

We also examined the model’s behavior: Did it use the appropriate tool? Did it retrieve evidence from the correct source? Did it follow the standing instruction, or did it take a shortcut?

This distinction is important. An answer can be factually correct and still be unsuitable for an enterprise application if it was obtained through an unauthorized workflow.

The evaluation is a task-specific test rather than a universal ranking of every model’s tool-use capabilities. A broader tool calling benchmark LLM comparison would need to cover additional tools, task types, and runtime configurations.

Models and Hardware Requirements

We evaluated models of different sizes and architectures, including dense models and mixture-of-experts (MoE) models. We also compared different quantization settings for selected models.

Quantization reduces the precision used to represent model weights, often lowering memory requirements and making local deployment more accessible. However, it was determined that 4-bit quantization has the best balance between quality and size.

When a model struggles with domain-specific terminology or instructions, adaptation techniques such as parameter-efficient fine-tuning (PEFT) may also be worth evaluating. Fine-tuning is a separate consideration from the tests reported here, however, and should be assessed against the same task-specific criteria as the original model.

The following table summarizes the model configurations and memory requirements recorded during our tests.

Actual deployment requirements depend on the inference runtime, context length, KV cache, concurrent requests, and hardware configuration.

The memory comparison highlights an important consideration: parameter count alone does not determine the practical cost of running a model. Architecture, quantization, active parameters, and implementation all matter.

For enterprise deployments, the relevant question is not simply how large a model is. It is how much hardware it needs to deliver reliable performance on the intended workload.

Key Results: Summarization Is Easy; Reliable Retrieval Is Not

The results revealed a significant difference between processing information that is already available and deciding where to retrieve information from.

Fourteen of the 15 models in the scored comparison passed both summarization questions. In contrast, only two models passed both knowledge-base retrieval questions.

Finding 1: Summarization Is Largely Solved for This Workload

Fourteen models passed both summarization questions. Even relatively small models could condense supplied documents into useful summaries.

This makes summarization a poor standalone benchmark for selecting a private knowledge-base assistant. A model can perform well when the relevant text is placed directly in its context and still fail when it must retrieve that text independently.

For organizations that need only fixed-prompt document processing, a small model may be sufficient. For an assistant that must search across internal repositories, more extensive evaluation is necessary.

Finding 2: Knowledge-Base Retrieval Is the Real Dividing Line

Only Qwen 3.8 27B at 4-bit quantization (small model) and DeepSeek V4.1 Flash (medium-sized model) passed both retrieval questions.

The other models frequently failed to retrieve the expected information from internal documents, despite having access to a knowledge-base search tool and being instructed to prioritize it.

This is the difference between knowing how to answer a question and knowing how to obtain the evidence required to answer it.

In a corporate environment, an assistant must retrieve information from the correct source, not merely produce a plausible response. An answer drawn from general training data may be outdated, incomplete, or inconsistent with the company’s current policies. A purpose-built private RAG chatbot for an internal knowledge base illustrates the kind of workflow in which dependable retrieval is central to the result.

Finding 3: Multilingual Retrieval Can Work Across Language Boundaries

Qwen 3.8 27B successfully retrieved information about Poland from a document written in another language.

For international organizations, this capability can be particularly valuable. Corporate knowledge bases often contain documents created by different regional offices, local legal teams, or business units operating in different languages.

Language coverage, document quality, retrieval configuration, and the complexity of the question should all be included in a production evaluation.

Finding 4: Following Instructions Was Harder Than Answering Questions

Eleven of the 15 models passed neither instruction-following question. No model passed both instruction questions cleanly.

The models struggled when they were expected to use internal sources even for questions they appeared capable of answering from general knowledge.

This exposes a limitation of relying on system prompts alone. A standing instruction such as “Always search internal documents first” may influence behavior, but it does not guarantee that the model will follow the required workflow every time.

For enterprise systems, the application should enforce critical source-selection rules wherever possible rather than treating them as optional model behavior.

Finding 5: Spreadsheet Calculations Benefit From Dedicated Tools

Ten of the 15 models reportedly calculated the spreadsheet balance correctly, often by using a calculator.

This is a reminder that language models do not need to perform every operation themselves. For arithmetic, spreadsheet processing, and other deterministic tasks, specialized tools can be more reliable than asking a model to reason through raw values in text.

The model’s role is to identify the relevant data and invoke the appropriate operation. The tool should perform the calculation, and the application should validate the result where accuracy is important.

Finding 6: More Parameters Do Not Guarantee Better Results

Qwen 3.8 27B at 4-bit quantization passed six of seven tasks, while gpt-oss 120B (the golden industry standard) passed four.

Similarly, Gemma 4 26B MoE and Qwen 3.8 27B at 4-bit quantization had broadly comparable reported memory requirements but substantially different scores.

The practical lesson is straightforward: parameter count is not a substitute for task-specific evaluation.

A model with fewer parameters may be the better choice if it follows instructions more reliably, uses tools appropriately, and produces better results on the company’s actual workload.

Trade-Offs: Accuracy, Privacy, and Infrastructure Cost

The results suggest four broad groups of models for the tested workload. These tiers help narrow down the best open source LLM for RAG for a given workload, but they should be treated as evaluation-specific guidance rather than a universal ranking.

These tiers describe performance in our evaluation, not a universal ranking of model quality. When comparing the best LLM models for RAG, the right choice depends on whether the application prioritizes retrieval, summarization, tool use, or a combination of tasks.

Privacy Is an Application-Level Responsibility

One of the most consequential findings concerns external search. If a model can call an internet search tool and chooses to use it for a question about internal company data, the query may leave the organization’s controlled environment.

Depending on the search provider, the query could reveal confidential business context, customer information, or details about internal operations. Of course for production systems, internet search tools must be turned off.

For a company with strict data-handling requirements, that can be a compliance or security incident even if the final answer is correct.

This risk is not inherent to every open-weight model or every RAG architecture. It depends on how the application, tools, permissions, and network access are configured.

The requirements can be especially strict in regulated settings: for example, private AI for clinics must account for the confidentiality of patient information, while private AI for law firms must protect privileged communications and sensitive case materials.

A safer design makes the application responsible for source selection:

• Search the authorized internal knowledge base first.
• Restrict or disable external search for sensitive workflows.
• Apply access controls before retrieving documents.
• Keep tool permissions narrow and auditable.
• Require explicit authorization before any action that could disclose sensitive information.
• Log retrieval decisions and tool calls for evaluation and incident investigation.

If the application cannot find sufficient evidence internally, it should follow a defined fallback policy: ask for clarification, report that the information is unavailable, or request an authorized external lookup. It should not silently substitute public information for internal evidence.

Quantization and Configuration Matter

The same Qwen 3.8 27B model achieved six passes at 4-bit quantization and three at 8-bit quantization in the test results.

This is a noteworthy difference, but it should not be interpreted as proof that 4-bit quantization is inherently superior. Quantization alone does not explain the outcome. Runtime settings, prompts, sampling parameters, tool integration, and test variability may also influence results.

The correct approach is to benchmark the exact configuration intended for production, including its quantization method, inference engine, context length, and agent harness.

Hardware Costs Can Outweigh the Benefits

Qwen 3.8 27B at 4-bit quantization was the smaller of the two top-scoring models in this evaluation. DeepSeek V4.1 Flash achieved the same total score but had a much higher reported memory requirement.

That difference matters when designing a private AI deployment. A model that requires hundreds of gigabytes of memory may be difficult to justify for a workload involving document retrieval and simple calculations.

Infrastructure decisions should also account for inference latency, concurrent users, context length, availability requirements, and costs. The smallest model that passes the tests may not be the only viable option, but it is an important starting point. And it’s worthy to mention that for the cost of DeepSeek infrastructure, you can afford several Qwen models.

Recommendations: How to Choose a Private LLM

Knowing how to choose LLM for RAG starts with defining the tasks the model must perform and the consequences of failure. Rather than selecting a model by its parameter count, evaluate retrieval quality, instruction following, tool use, hardware requirements, and privacy controls together.

1. Start With Qwen 3.8 27B at 4-Bit Quantization for Evaluation

Of the tested configurations, Qwen 3.8 27B at 4-bit quantization offered the strongest balance between the reported task score and memory requirements.

DeepSeek V4.1 Flash achieved the same score, but its reported memory requirement was substantially higher. It may be worth evaluating when its particular capabilities justify the infrastructure cost.

gpt-oss 120B could also be considered for workloads where its strengths are relevant, but its retrieval and instruction-following behavior should be tested and the agent harness configured to match its capabilities.

These are starting points for evaluation, not universal production recommendations. If you’re choosing LLM for RAG, benchmark the model with your own documents and deployment configuration before making a final decision.

2. Use Smaller Models for Simpler Document Operations

For summarization and selected tasks involving a clearly identified document, Granite 4.2 8B and Qwen 3.5 9B are reasonable candidates to evaluate.

Smaller models may be sufficient when the application supplies the relevant text, limits the available tools, and does not depend on the model independently navigating a large knowledge base.

This can reduce infrastructure costs without paying for capabilities the workload does not need.

3. Evaluate the Best Local LLM for Tool Calling

The best local LLM for tool calling is not necessarily the largest or the most accurate on ordinary question-answering benchmarks. It must reliably select the right tool, provide valid inputs, interpret the tool’s output, and follow the application’s rules.

In our tests, the strongest retrieval results did not translate into perfect instruction following. This suggests that tool selection and source-selection behavior should be evaluated separately from answer quality.

For a production agent, consider restricting the available tools, validating arguments before execution, checking tool outputs, and recording each tool call for auditing.

4. Enforce Source Selection in the Application

Do not rely exclusively on a model’s promise to use internal documents. For sensitive workflows, the application should control retrieval order, access permissions, and external connectivity. It can be achieved by forced graph AI-agent architecture.

If external search is prohibited, the model should not have access to that tool in the first place. This architectural safeguard is more dependable than a prompt alone.

5. Use Dedicated Tools for Calculations

For balances, financial totals, spreadsheet comparisons, and other numerical operations, let a deterministic tool perform the calculation.

The model should identify the relevant rows and columns, invoke the appropriate operation, and explain the result. The application should validate inputs and outputs when mistakes could have material consequences.

6. Treat Very Small Models Cautiously in Agent Workflows

In this test, the smallest models struggled with tool use, retrieval, or calculations. Models around 8B parameters and above were generally more useful for document operations, but size alone did not guarantee reliable agent behavior.

Rather than adopting a universal parameter threshold, evaluate models against the exact operations required by the application. A smaller model with restricted responsibilities may be preferable to a larger model with poorly controlled access to tools.

7. Test With Real Company Documents Before Deployment

Public benchmarks and general-purpose evaluations cannot fully represent a company’s terminology, document structure, permissions, languages, or policies. A useful evaluation set should include:

• Questions whose answers are present in internal documents.
• Questions that require searching across multiple files.
• Documents written in different languages.
• Questions the model is likely to believe it can answer from general knowledge.
• Conflicting or outdated documents.
• Spreadsheet calculations and data extraction.
• Questions for which the correct response is that the evidence is insufficient.
• Attempts to access information the user is not authorized to see.

Evaluate not only the final answer but also the retrieval path, tool calls, source citations, access-control behavior, and consistency across repeated runs. This kind of validation is an important step when moving an AI pilot to production, because a promising demonstration does not by itself establish production readiness.

How SCAND Approaches Private LLM Selection

At SCAND, selecting an LLM for an enterprise solution is not simply a matter of choosing the largest model that fits the available hardware.

Through our private LLM development services, we evaluate candidate models against the client’s documents, target tasks, and infrastructure constraints. For retrieval-driven applications, this includes testing the architecture and retrieval pipeline as part of RAG development, not treating the language model as the only factor that determines quality.

The goal is to identify a configuration that offers the right balance of answer quality, retrieval reliability, privacy, performance, and cost. This evaluation helps answer practical questions before production deployment:

• Can the model retrieve the right information from the client’s knowledge base?
• Does it follow the application’s source-selection and security rules?
• Can it handle the client’s documents and terminology?
• Which tasks require specialized tools rather than language-model reasoning?
• What hardware is required to meet performance and concurrency expectations?

These considerations also apply to specialized use cases, such as a private LLM for source code documentation, where the assistant must retrieve accurate technical information from internal repositories rather than rely on generic answers.

The results then inform the architecture, model configuration, and safeguards needed for the specific use case. Organizations considering a similar approach can run this evaluation on your data to understand which model and deployment configuration best fit their documents, security requirements, and infrastructure.

Conclusion: The Smallest Model That Works Is the One That Follows the Rules

Our evaluation shows that summarization and reliable knowledge-base assistance are different capabilities.

Most tested models could summarize supplied text, but only two achieved the strongest overall score of six out of seven tasks. Neither passed both instruction-following questions cleanly. Even the strongest candidates therefore require application-level safeguards and further validation before production use.

For enterprises, the goal should not be to find the smallest LLM that can produce a convincing answer. It should be to find the smallest practical configuration that retrieves the right evidence, follows the required workflow, uses tools correctly, and respects data boundaries.

That is why private LLM selection should begin with the workload and the risks, not the parameter count. A carefully evaluated smaller model, supported by a well-designed RAG architecture, can be a more practical enterprise solution than a much larger model deployed without adequate controls.

The best private AI assistant is not the model that knows the most. It is the one that knows where to look, follows the rules, and stays within the boundaries of the system.

Frequently Asked Questions (FAQs)

Which LLM is best for RAG?

In our tests, Qwen 3.8 27B at 4-bit quantization and DeepSeek V4.1 Flash achieved the highest total score, with six out of seven tasks passed. Neither passed both instruction-following questions cleanly, so the best choice depends on the workload, hardware, and application-level safeguards.

What is the best LLM for a RAG chatbot?

The best LLM for RAG chatbot applications is one that retrieves the right evidence consistently, follows source-selection rules, and works within the available hardware budget. Qwen 3.8 27B at 4-bit quantization is a useful starting point from our evaluation, but it should be tested against the chatbot’s actual documents and questions.

What is the smallest LLM for RAG?

The smallest LLM for RAG depends on the complexity of the documents, the retrieval pipeline, and whether the model must use tools independently. In our evaluation, smaller models handled summarization better than knowledge-base retrieval, so a compact model may work for fixed-prompt tasks but require more validation for interactive RAG. In our tests 27B+ models is the right parameter size start.

What is the best local LLM for agents or agentic tasks?

The best local LLM for agents depends on the tools available, the complexity of the workflow, and how critical instruction following is. Our results show that strong retrieval alone is not enough: agents also need controlled tool access, clear source-selection rules, and evaluation across repeated runs.

Can a smaller local model outperform a larger model in RAG?

Yes. In our evaluation, Qwen 3.8 27B at 4-bit quantization achieved a higher score than gpt-oss 120B while requiring substantially less reported memory. This does not establish a universal ranking, but it demonstrates why real task performance matters more than parameter count alone.

How do you stop a RAG assistant from sending queries to the internet?

Remove the web search tool from sensitive workflows instead of relying on a prompt. In our tests, models sometimes chose external search even when told to use internal documents first. The application should control retrieval order, network access, and tool permissions, and log every tool call for review.