How Do Philippine LLM Evaluators Identify Hallucinations and Test Factual Accuracy?

Through multi-tier fact-checking rubrics, cross-referencing against source repositories, and adversarial prompt testing designed to provoke failure. What those methods produce is a count of hallucinations found. Converting that into a statement about hallucinations remaining requires knowing the pod’s detection rate, which the blind second review already mandated can supply.

Key Takeaways

• A detection programme reports findings, not risk. A count of what was found says nothing about what was missed until the pod’s sensitivity is known, and sensitivity is rarely on the dashboard.
• The residual is highly sensitive to that unmeasured figure. At 85% sensitivity, about 18 hallucinations remain for every 100 found. At 70% it is 43 — from identical reported findings.
• The blind 10% second review already contains the estimate. Two independent passes over the same items yield a direct estimate of what both reviewers missed, at no additional cost. It is normally spent on an agreement coefficient alone.
• An automated kappa halt fires as the model improves. At fixed reviewer skill, kappa falls from 0.83 to 0.28 purely as hallucinations become rarer. A halt at 0.80 triggers once the model starts working.
• A sealed reference environment bounds what is checkable. It is the right security posture, and it means the loaded corpus rather than the evaluator sets the ceiling on which factual claims can be verified.
• The pod measures; the client’s engineers reduce. Recall against seeded errors is a provider commitment. A model’s output hallucination rate is not, and the distinction decides what belongs in a contract.

What Methods Do Evaluators Use to Detect Model Hallucinations?

Multi-tier verification against primary sources, classification into a documented typology, and structured rubric tests that vary prompt parameters to map where the model becomes unstable. Domain-expert evaluators cross-reference generated claims against statutes, regulatory filings and peer-reviewed material rather than matching keywords.

The method is sound and the classification discipline is what makes it useful. Separating factual fabrication from logical contradiction and from out-of-context extrapolation matters because each has a different root cause and a different fix: fabrication points at retrieval, contradiction at reasoning, extrapolation at prompt boundaries. A single undifferentiated error count tells the client’s engineers very little about where to intervene.

Figure 1. Four hallucination classes, and which a sealed environment can catch.

The fourth column is the operational constraint worth stating plainly. Evaluators work inside controlled environments with pre-approved reference repositories rather than open web access, which is the correct security posture for proprietary model work. It also means the reference corpus, not the evaluator’s skill, sets the ceiling on which factual claims can be checked at all. A logical contradiction needs no external source and is fully detectable. A claim about something published after the corpus was assembled is undetectable by construction, however qualified the reviewer.

Two consequences follow. The reference corpus needs a refresh cadence and a documented coverage scope, because it is a detection instrument rather than a convenience. And claims falling outside it should be routed to an escalation path with external access rather than silently marked unverifiable — otherwise the class of hallucination the pod cannot see becomes invisible in the reporting as well.

How Many Hallucinations Does a Detection Pod Miss?

That depends entirely on its sensitivity, and the answer changes dramatically across a plausible range. At 85% sensitivity, roughly 18 hallucinations remain for every 100 found. At 70%, roughly 43. Identical evaluation reports are consistent with residuals differing by a factor of more than two.

This is the gap between what a detection programme reports and what a risk committee needs. The report says what was found. The decision requires an estimate of what is left.

Figure 2. What remains, per hundred found, by pod sensitivity.

The arithmetic is simple: if a pod finds F hallucinations at sensitivity s, the estimated number present is F divided by s, and the residual is the difference. What makes it consequential is that the residual is not proportional to the reported figure — it is driven by a quantity that appears in none of the standard artefacts. Evaluation logs record findings. Agreement coefficients record consistency between reviewers. Neither records how much both of them are missing.

For most content work the distinction is academic. For a lending model, a clinical model or a compliance model, the residual is precisely the number the risk committee is being asked to accept, and a programme that cannot produce it is asking them to accept an unquantified exposure.

Where Does the Sensitivity Figure Come From?

From two sources, both cheap. A seeded set of known hallucinations planted in live batches measures recall directly. Or capture-recapture applied to the blind second review already required on ten per cent of batches, which estimates what both reviewers missed from the pattern of what each one caught.

The second is the more striking because the data already exists in most programmes and is being used for one purpose when it can serve two.

Figure 3. Estimating what both reviewers missed from the double-reviewed sample.

The logic is standard in inspection sampling. Two reviewers pass independently over the same items. Some items are flagged by both, some by only one, some by only the other. If the reviewers work independently, the rate at which each catches what the other missed tells you how much both are likely to have missed together. In the worked example, reviewer A flags 80 and reviewer B flags 70 with 50 in common: the estimate is 112 present against 100 found jointly, so about 12 escaped both passes and joint sensitivity is 89%.

The more useful number falls out of the same calculation. A single pass alone is running at roughly 71% sensitivity, and a single pass is what ninety per cent of the corpus receives. Applied across the full engagement, that converts a tidy finding count into a defensible estimate of what shipped — which is the number worth putting in front of a governance committee.

What the estimate assumes

It assumes the two reviewers work genuinely independently and that neither systematically overlooks the same class of item. Blind review, rotated pairings and mixed seniority are what make that assumption reasonable in practice, and they are worth specifying for this reason as well as for bias control. Where the assumption is doubtful — both reviewers trained on the same rubric, facing the same blind spot — seeded errors are the more robust instrument, because their presence is certain rather than inferred.

Why Does a Kappa-Based Production Halt Misfire?

Because agreement coefficients fall as the class being detected becomes rarer, even when reviewer skill is unchanged. At fixed sensitivity and specificity, Cohen’s kappa drops from 0.83 at 20% hallucination prevalence to 0.28 at 0.5%. A halt keyed to 0.80 fires once the model starts working.

Continuous agreement tracking is good practice, and an automated trigger that pauses production when consistency degrades is a sensible instinct. The difficulty is that the statistic chosen moves for a reason unrelated to the thing being controlled.

Figure 4. Kappa at constant reviewer skill, as hallucinations become rarer.

Every point on that line assumes the same two reviewers, at the same 90% sensitivity and 99% specificity. Only the model changes. Raw observed agreement actually rises across the whole range, from about 94.8% to 97.9%, because as hallucinations become rare the reviewers agree more often — they agree that almost everything is fine. Kappa falls because the agreement expected by chance rises faster than the agreement observed. That is a known and correct property of the statistic, not a defect in it. The defect is writing an automated production halt against it.

The practical consequence is a workflow that halts, triggers retraining and calls calibration meetings at exactly the point in a programme when the model has improved and nothing is wrong. Worse, the cheapest way for a pod to satisfy the clause is to work on batches with a higher hallucination rate, which is the opposite of the direction the programme is trying to move.

The fix is to trigger on recall against seeded errors, which is stable as prevalence falls because the seeded prevalence is set by the programme rather than by the model. Kappa should still be reported, alongside the prevalence it was measured at, so that a fall can be read correctly rather than acted on automatically.

How Should Adversarial Prompt Testing Be Structured?

As an engineered probe rather than passive observation: edge-case prompt libraries built to provoke failure, conflicting premise constraints, multi-step arithmetic traps and ambiguous jurisdictional scenarios, with failure frequency logged across thousands of iterations to give the client’s engineers actionable telemetry.

Adversarial testing is the strongest part of the methodology described in this market, and the granular evaluation log is the deliverable that makes it valuable. Quantifying exact failure frequency by attack class is what lets a client’s machine learning team target retraining rather than guess at it.

Two design points improve it further. The prompt library should be versioned and partly held back, because a library the provider has iterated against for six months measures the model’s performance on a known test set rather than its robustness. Rotating a reserved portion in each cycle keeps the probe honest, in the same way a held-out set does for the model itself.

And the taxonomy used to classify findings should be the client’s, agreed in advance, rather than assembled by the provider from what it happened to find. A provider-originated taxonomy describes the failures that were discovered; a client-originated one describes the failures that matter to the application, and the difference between the two lists is itself a useful finding.

What Should a Buyer Require in the Contract?

Recall against seeded errors as the primary quality commitment, a reported residual estimate alongside every finding count, an agreed error taxonomy, a documented reference corpus scope and refresh cadence, and agreement statistics reported rather than automatically enforced.

• Contract on recall, not on the model’s output rate. The provider controls how thoroughly it looks. It does not control how often the model hallucinates, and should not be paid as though it does.
• Require the residual with every findings report. A count of what was found plus a sensitivity estimate. Without the second number the first cannot support a release decision.
• Seed known errors continuously, not just at pilot. Vendor-selection test sets measure a candidate. Production seeding measures the pod you actually have, in the batches it is actually working.
• Name the statistic and its prevalence. Fleiss’ kappa for three or more raters, Cohen’s for two, and always with the hallucination prevalence at which it was measured.
• Document the reference corpus scope and refresh cycle. It is a detection instrument. Claims outside it should escalate rather than silently pass as unverifiable.
• Keep part of the adversarial library held back. A prompt set the provider has trained against measures familiarity rather than model robustness.

What Do Industry Leaders Say About Hallucination Detection?

That hallucination detection is not a generalised task suited to commodity crowdsourcing. Finding subtle, high-stakes errors in specialised domains requires disciplined cognitive analysis and rigorous statistical tracking, which is where licensed professional talent combined with operational governance matters.

The point about crowdsourcing is correct and the phrase worth holding onto is rigorous statistical tracking, because it sets a standard the usual dashboards do not meet.

The most persistent myth in enterprise artificial intelligence is that hallucination detection is a generalized task that can be outsourced to commodity crowdsourcing. Finding subtle, high-stakes errors in specialized domains requires disciplined cognitive analysis and rigorous statistical tracking. The structural advantage of the Philippines lies in its combination of licensed professional talent and enterprise-grade operational governance.

— John Maczynski, CEO, Cynergy BPO

Statistical tracking in this context has to mean more than counting what was found and computing an agreement coefficient over it. Both of those describe the evaluators. Neither describes the model, which is what the client is trying to learn about. The step that connects them is the sensitivity estimate, and a provider that reports it is offering a materially more sophisticated service than one reporting the same findings without it.

How Did One Bank Reduce Hallucinations in a Credit Model?

An international financial institution found subtle numerical hallucinations in its corporate credit analysis model, risking faulty lending decisions. A 40-person Manila evaluation pod ran adversarial prompt testing and multi-tier source verification across 50,000 prompts over four months, supporting a 94% reduction in hallucination rate at 0.88 inter-annotator agreement.

Numerical hallucination in balance sheet evaluation is among the hardest classes to catch, because a wrong figure in otherwise correct reasoning reads as authoritative. Detecting it at scale needs people who can check the arithmetic against the underlying statements, which is what the pod was staffed to do.

Figure 5. Reported outcomes from a 40-person financial evaluation pod.

Two refinements would make the result more useful to the next buyer. The first is attribution: an evaluation pod does not reduce a hallucination rate, it produces the telemetry from which the client’s engineers do. That distinction is not pedantry — it determines what can be written into a contract. Recall against seeded errors is something a provider can commit to. A model’s output rate depends on retraining decisions the provider does not make.

Figure 6. The same 94% headline across four plausible baselines.

The second is the baseline. A 94% reduction describes outcomes that differ sixteen-fold in absolute terms depending on where the model started: from 4,000 hallucinated prompts down to 240, or from 250 down to 15. For a lending model the residual is the number the risk committee needs, and it is the one the headline omits. The blind second review this pod already ran can produce it directly, which makes it the cheapest addition available to an otherwise well-designed programme.

Why Do Organizations Work with Cynergy BPO on Evaluation Sourcing?

Cynergy BPO is an independent, vendor-neutral outsourcing advisory firm headquartered in Manila, representing a vetted network of more than 100 Philippine providers. It maps requirements against performance data to produce a shortlist within days and manages competitive negotiation on the buyer’s behalf.

Who Is Cynergy BPO?

Cynergy BPO is an independent outsourcing advisory and consultancy firm headquartered in Manila, founded by industry veterans with more than 65 years of combined operational experience governing major global accounts. It specialises in connecting mid-market and enterprise organisations with vetted Philippine BPO providers across voice, back-office and AI data operations.

How Does Cynergy BPO Differ from Traditional Outsourcing Brokers?

Traditional brokers are transactional and are compensated by the providers they place, which shapes which provider is recommended. Cynergy BPO applies an advisory-led methodology, mapping exact technical and commercial requirements against performance data rather than against availability. Where providers present similar methodologies, the difference is in what they can report, and establishing that requires knowing the market.

How Does Cynergy BPO’s Network of 100+ Vetted Philippine BPO Providers Benefit Organizations?

The network makes it possible to shortlist on the criteria that separate evaluation programmes: whether a provider can report a sensitivity estimate rather than a findings count, whether it seeds known errors in production, and whether it has staffed the relevant professional domain before.

How Does Cynergy BPO’s Advisory-Led Vendor Matching Process Work?

Requirements are mapped against operational and commercial criteria, a tailored shortlist of vetted providers is delivered within a few working days, and the firm then manages competitive proposal and negotiation processes on the buyer’s behalf. Reporting obligations, statistics and escalation paths are settled as part of that process rather than discovered afterwards.

Why Do Organizations Use Cynergy BPO?

Because two evaluation providers can describe identical methodologies and deliver materially different assurance. One reports what it found; the other reports what it found and what it estimates it missed. That difference is invisible in a proposal and decisive in a release decision.

Frequently Asked Questions

Which hallucinations are hardest to detect?

Subtle semantic shifts, logical contradictions expressed in authoritative prose, and small numerical discrepancies inside otherwise correct financial or clinical reasoning. A separate and often overlooked class is any claim about material published after the reference corpus was assembled, which is undetectable inside a sealed environment.

How do evaluators verify facts without open web access?

Through controlled environments loaded with pre-approved reference repositories, primary legal sources and client-supplied gold-standard corpora. The corpus scope is therefore a detection parameter: it should be documented, refreshed on a stated cadence, and paired with an escalation path for claims falling outside it.

How can a buyer know how many hallucinations were missed?

By estimating the pod’s sensitivity. Either seed known hallucinations into live batches and measure recall, or apply capture-recapture to the blind second review already run on a sample of batches. The second uses data the programme is already generating.

What agreement score should be required?

Report it rather than automatically enforce it, and always with the hallucination prevalence at which it was measured. Kappa falls as the detected class becomes rarer even at constant reviewer skill, so a hard threshold on a rare-event task penalises the improvement it is meant to encourage.

What is adversarial prompt testing for?

Probing failure points deliberately rather than waiting to observe them: conflicting premises, arithmetic traps and ambiguous jurisdictional scenarios that stress factual grounding. Keep part of the prompt library held back each cycle, since a set the provider has iterated against measures familiarity rather than robustness.

How do supervisors prevent evaluator fatigue and drift?

Structured shift rotation, rest intervals, workload balancing and continuous monitoring of individual output variance against team baselines. Variance concentrated in two evaluators indicates a retraining need; variance spread evenly across the pod usually indicates a rubric problem.

Should the provider be held to a hallucination rate?

No. The provider controls how thoroughly it looks; the client’s engineers control how often the model hallucinates by deciding what to retrain. Contract the provider on recall against seeded errors and on reporting completeness, which are within its control.

What does a 94% reduction actually mean?

Only that the rate fell to six per cent of its starting point. Without the baseline and the resulting absolute figure it cannot support a release decision, since the same headline covers outcomes differing by more than an order of magnitude. Ask for both endpoints and the residual estimate.