AI in Production: Why LLM Evaluation Is Still an Unsolved Problem

Introduction:

Every engineering team deploying a large language model faces the same fundamental question — how do you know if it is working? The question seems straightforward. Models have been evaluated for decades using well-established metrics. Accuracy, precision, recall, F1 score — the toolkit of machine learning evaluation is mature and widely understood.

But large language models produce outputs that these metrics were not designed to evaluate. A language model that answers a question correctly but verbosely, incorrectly but confidently, accurately but harmfully, or helpfully but inconsistently across similar inputs is not captured by any single number. The outputs are open-ended, context-dependent, and evaluated differently by different users in different situations.

The honest state of LLM evaluation in 2026 is that no team has fully solved it. The teams that deploy LLMs most effectively are not the ones that have found the right metric — they are the ones that have built evaluation systems that are honest about what they can and cannot measure.

Traditional Metrics Do Not Transfer to Open-Ended Outputs:

Classification metrics work because the output space is bounded — a model predicts one of a fixed set of classes and you can count how often it is right. Language model outputs are unbounded. A question can be answered correctly in thousands of different ways, and incorrectly in infinitely more. Counting correct answers requires defining what correct means — which for open-ended language generation is itself an unsolved problem.

Metrics like BLEU and ROUGE, borrowed from machine translation and summarisation research, measure the overlap between generated text and reference text. They can tell you whether a generated summary contains words that appear in a human-written reference summary. They cannot tell you whether the generated summary is accurate, whether it captures the most important information, or whether it would be useful to the person reading it.

Teams that report BLEU scores for language model outputs are measuring something. They are rarely measuring the thing that matters for the application they are building.

Human Evaluation Is the Gold Standard That Does Not Scale:

The most reliable way to evaluate language model outputs is to have humans read them and judge their quality. Human evaluation captures nuance, context-sensitivity, and usefulness in ways that automated metrics cannot. When you want to know whether a language model is producing good outputs, asking humans is the most honest answer available.

Human evaluation does not scale. Evaluating thousands of model outputs per day with human raters is expensive, slow, and introduces its own inconsistencies — different raters apply different standards, rater fatigue affects judgment quality, and the criteria for good outputs are difficult to specify precisely enough that raters apply them consistently.

Teams that rely on human evaluation as their primary quality signal can evaluate their models carefully but cannot evaluate them continuously. The model that was evaluated thoroughly before deployment may behave differently two months later as usage patterns evolve — and detecting that degradation requires evaluation at a frequency that human raters cannot sustain.

LLM-as-Judge Has Real Limitations:

Using a large language model to evaluate the outputs of another large language model — LLM-as-judge — has emerged as a popular approach to scaling evaluation beyond what human raters can handle. The evaluating model reads a prompt, a generated response, and an evaluation rubric, and produces a quality score or preference judgment.

This approach scales. It is significantly cheaper than human evaluation and can be run continuously at inference time. But it inherits the limitations of the model doing the evaluation. A judge model that shares biases with the model being evaluated will not catch those biases. A judge model that is susceptible to verbose, confident-sounding outputs will rate them highly regardless of their accuracy. A judge model that was trained on similar data to the model being evaluated may have the same blind spots.

LLM-as-judge correlates reasonably well with human judgment on average and diverges significantly on specific cases — particularly edge cases, nuanced quality distinctions, and domain-specific accuracy questions where the judge model lacks the expertise to evaluate correctly.

Benchmarks Measure the Wrong Distribution:

The benchmarks that are used to compare language models — MMLU, HumanEval, BIG-Bench — measure performance on curated datasets of questions and tasks. These datasets are carefully constructed to cover a range of capabilities and difficulty levels. They are not representative of the actual distribution of inputs a production system will receive.

A model that scores highly on MMLU may struggle with the specific domain knowledge your application requires. A model that excels at HumanEval coding tasks may produce poor outputs for the specific programming language and coding style your users expect. Benchmark performance predicts production performance only to the extent that the benchmark distribution resembles the production distribution — which is rarely as close as teams assume when making model selection decisions.

The models that perform best on public benchmarks are also the models that have been most extensively trained on data that resembles those benchmarks. This produces evaluation scores that overstate real-world capability and makes benchmark comparisons less informative than they appear.

Task-Specific Evaluation Is Necessary but Expensive to Build:

The most reliable approach to LLM evaluation for a specific application is building task-specific evaluation datasets that reflect the actual distribution of inputs the model will receive in production. These datasets should be collected from real usage, annotated by domain experts, and updated regularly as usage patterns evolve.

Building and maintaining these datasets is expensive. Domain expert annotation is slow and costly. Datasets collected from early production usage may not reflect the distribution of usage at scale. Keeping datasets current requires ongoing investment that competes with other engineering priorities.

Teams that invest in task-specific evaluation consistently produce better-aligned models and catch more production problems before they affect users. Teams that rely on generic benchmarks and automated metrics ship models that look good in evaluation and surprise them in production.

Evaluation Must Be Continuous, Not Periodic:

A model that was evaluated thoroughly before deployment is not necessarily performing well three months later. Data distributions shift, usage patterns evolve, and model behaviour can change through fine-tuning, prompt engineering updates, or changes in the data the model retrieves at inference time. Evaluation that happens only at deployment time misses all of these post-deployment changes.

Continuous evaluation — sampling production outputs regularly, running them through automated quality checks, routing a percentage to human review, and tracking quality metrics over time — provides the ongoing signal needed to detect degradation before it becomes significant. This requires instrumentation that is built alongside the model rather than added after deployment.

Teams that treat evaluation as a deployment gate rather than an ongoing operational practice consistently discover production quality problems through user complaints rather than through engineering instrumentation.

Conclusion:

LLM evaluation remains unsolved not because the problem is intractable but because the outputs language models produce are genuinely difficult to evaluate automatically, at scale, and across the full range of ways a model can succeed or fail. No single metric captures output quality. No benchmark fully predicts production performance. No automated approach fully replaces human judgment.

The teams deploying LLMs most effectively are the ones that combine multiple evaluation approaches — automated metrics for continuous monitoring, human evaluation for quality anchoring, task-specific datasets for domain alignment, and production outcome tracking for real-world feedback. They are honest about what each approach measures and what it misses. And they treat evaluation as an ongoing engineering investment rather than a problem to be solved once before deployment.