Quality Gates Reference

Version: 1.0
Status: Consolidated Reference
Date: February 3, 2026

1. Overview

Quality Gates are checkpoints in the FactHarbor analysis pipeline that enforce minimum standards for claim evaluation and verdict confidence. They ensure that:

• Only verifiable claims are analyzed (Gate 1)
• Verdicts have sufficient supporting evidence (Gate 4)
• Results meet minimum quality thresholds before publication

Target Audience: Developers, prompt engineers, and quality assurance reviewers.

Implemented Gates: Gate 1 (Claim Validation) and Gate 4 (Verdict Confidence Assessment)

Replaces scattered documentation in:

• Architecture Overview (Quality Gates section)
• Calculations.md (Gate 4 sections, still in Docs/ARCHITECTURE/)
• TriplePath Architecture (quality gates mentions)

2. Gate Architecture

2.1 Pipeline Integration

flowchart TB
    subgraph UNDERSTAND["Phase 1: UNDERSTAND"]
        Input[User Input] --> ClaimExtraction[Claim Extraction]
        ClaimExtraction --> GATE1["Gate 1: Claim Validation
━━━━━━━━━━━━━
Filter opinions,
predictions,
low-specificity"]
        GATE1 -->|Pass| ValidClaims[Valid Claims]
        GATE1 -.->|Fail| ExcludedClaims[Excluded Claims
with reasons]
    end

    subgraph RESEARCH["Phase 2: RESEARCH"]
        ValidClaims --> Search[Web Search]
        Search --> Sources[Source Documents]
        Sources --> EvidenceExtraction[Evidence Extraction]
    end

    subgraph VERDICT["Phase 3: VERDICT GENERATION"]
        EvidenceExtraction --> VerdictGeneration[Verdict Generation]
        VerdictGeneration --> GATE4["Gate 4: Confidence Assessment
━━━━━━━━━━━━━
Check source count,
fact count,
reasoning quality"]
        GATE4 -->|Pass| PublishableVerdicts[Publishable Verdicts]
        GATE4 -->|Warn| LowConfidenceVerdicts[Low Confidence
Verdicts]
    end

    style GATE1 fill:#fff9c4
    style GATE4 fill:#fff9c4
    style ExcludedClaims fill:#ffcdd2
    style ValidClaims fill:#c8e6c9
    style PublishableVerdicts fill:#c8e6c9
    style LowConfidenceVerdicts fill:#ffecb3

2.2 Gate States

2.3 Result Metadata

Every analysis result includes gate statistics:

3. Gate 1: Claim Validation

3.1 Purpose

Filter out non-verifiable claims before research begins, preventing wasted resources on opinions, predictions, or vague statements.

3.2 Criteria

Claims are EXCLUDED if:

1. Opinion/Editorial: Subjective judgment without factual basis
   • Example: "Policy X is the best approach"
   • Action: Exclude unless claim is central to thesis

1. Prediction/Speculation: Future-oriented claims that cannot be verified
   • Example: "Technology Y will dominate the market by 2030"
   • Action: Exclude unless claim is central to thesis

1. Low Specificity: Vague statements without concrete assertions
   • Example: "Some experts believe..."
   • Action: Exclude unless claim is central to thesis

Claims are KEPT if:

• Factual assertion: Verifiable statement about past/present
• Central claim: Core thesis claim (kept regardless of specificity)
• Attribution claim: Claims about what someone said/did

3.3 Implementation

File: apps/web/src/lib/analyzer/orchestrated.ts
Function: Applied during understandClaim() phase
Phase: UNDERSTAND (Phase 1)

Exclusion Process:

1. LLM extracts claims and marks each with role and type
2. Deterministic filter applies Gate 1 criteria
3. Excluded claims logged with reasons
4. Valid claims proceed to research phase

3.4 Configuration

UCM Pipeline Config:
{
 "gate1Enabled": true,  // Enable/disable Gate 1
 "gate1KeepCentralClaims": true  // Keep central claims regardless of specificity
}

3.5 Examples

Example 1: Opinion Excluded
Claim: "The Supreme Court's decision was unjust."
Role: evaluative
Result: EXCLUDED (opinion - no factual basis)
Reason: "Evaluative opinion without factual assertion"

Example 2: Central Claim Kept
Claim: "The policy will significantly improve outcomes."
Role: core
Result: KEPT (central to thesis, despite low specificity)
Reason: "Central claim kept for analysis"

Example 3: Factual Assertion Kept
Claim: "The court ruled in favor of Party A on January 15, 2025."
Role: core
Type: factual
Result: KEPT (verifiable factual assertion)

4. Gate 4: Verdict Confidence Assessment

4.1 Purpose

Ensure verdicts have sufficient supporting evidence before publication, preventing low-confidence judgments from misleading users.

4.2 Confidence Tiers

4.3 Implementation

File: apps/web/src/lib/analyzer/orchestrated.ts
Function: validateVerdictGate4()
Phase: VERDICT GENERATION (Phase 3)

Validation Process:

1. Count sources supporting verdict
2. Count facts extracted from sources
3. Measure reasoning length
4. Assign confidence tier
5. Apply context scoping for counter-evidence
6. Flag verdicts below threshold

4.4 Context Scoping

Counter-evidence is scoped to relevant analysis contexts:

This prevents criticism of one analysis context from penalizing claims about a different analysis context.

4.5 Central Claim Exception

Central claims remain publishable even if confidence is low, because they are core to the thesis and users need to see the verdict regardless of evidence sufficiency.

Rationale: Users submitted the input to understand the truth of central claims. Hiding low-confidence verdicts would be misleading.

4.6 Configuration

UCM Pipeline Config:
{
 "gate4Enabled": true,  // Enable/disable Gate 4
 "gate4MinSources": 2,  // Minimum sources for MEDIUM confidence
 "gate4MinFacts": 3,    // Minimum facts for MEDIUM confidence
 "gate4MinReasoningLength": 50  // Minimum reasoning length for MEDIUM
}

4.7 Examples

Example 1: HIGH Confidence
Verdict: "MOSTLY-TRUE" (85%)
Sources: 4 (Reuters, AP, BBC, Government site)
Facts: 12
Reasoning: 150 chars
Result: HIGH confidence tier
Action: Publish with full confidence

Example 2: LOW Confidence (Central Claim)
Verdict: "UNVERIFIED" (50%)
Sources: 1 (Blog post)
Facts: 2
Reasoning: 80 chars
Claim: Central
Result: LOW confidence tier
Action: Publish with warning (central claim exception)

Example 3: INSUFFICIENT (Non-Central)
Verdict: "UNVERIFIED" (50%)
Sources: 0
Facts: 0
Reasoning: 30 chars
Claim: Non-central
Result: INSUFFICIENT
Action: Exclude from report or mark as "No evidence found"

5. Confidence Impact on Verdict Calculation

5.1 Truth Percentage Modulation

Confidence modulates the final truth percentage within each verdict band:

Via truthFromBand() function:
function truthFromBand(band: "strong" | "partial" | "uncertain" | "refuted", confidence: number): number {
 const conf = normalizePercentage(confidence) / 100;
 switch (band) {
   case "strong":    return Math.round(72 + 28 * conf);  // 72-100%
   case "partial":   return Math.round(50 + 35 * conf);  // 50-85%
   case "uncertain": return Math.round(35 + 30 * conf);  // 35-65%
   case "refuted":   return Math.round(28 * (1 - conf)); // 0-28%
 }
}

5.2 Example Impact

"strong" band with varying confidence:

• High confidence (90%): 72 + 28x0.9 = 97% -> TRUE
• Medium confidence (60%): 72 + 28x0.6 = 89% -> TRUE
• Low confidence (30%): 72 + 28x0.3 = 80% -> MOSTLY-TRUE

Same evidence band, but lower confidence pulls verdict down within the band.

5.3 MIXED vs UNVERIFIED Distinction

Confidence determines whether 43-57% range is MIXED or UNVERIFIED:

• MIXED (confidence >= 60%): Evidence on both sides, high confidence in mixed state
• UNVERIFIED (confidence < 60%): Insufficient evidence, low confidence

6. Gate Statistics and Reporting

6.1 Gate Stats in Result JSON

Every analysis result includes gate statistics:

6.2 UI Display (Current Status)

Current: Gate stats included in JSON but not displayed in UI with per-item reasons

Planned: UI enhancements to show:

• Excluded claims with reasons in report
• Confidence tier badges on verdicts
• Warning indicators for low-confidence verdicts

7. Proposed Gates (Not Yet Implemented)

7.1 Gate 2: Source Quality (Proposed)

Purpose: Filter low-quality sources before evidence extraction
Criteria:

• Source reliability score > threshold
• Domain not in blocklist
• Content length > minimum

Status: Proposed but not implemented (Source Reliability system exists but not integrated as gate)

7.2 Gate 3: Evidence Relevance (Proposed)

Purpose: Filter tangential or low-probative-value evidence
Criteria:

• Thesis relevance score > threshold
• Recency appropriate for claim
• Geographic/jurisdictional match

Status: Proposed but not implemented (Evidence filtering exists but not formalized as gate)

8. Debugging and Diagnostics

8.1 Checking Gate Stats

In Result JSON:
const result = await analyzeJob(jobId);
console.log('Gate 1 excluded:', result.qualityGates.gate1Stats.excludedClaims);
console.log('Gate 4 confidence:', result.qualityGates.gate4Stats);

In Report Markdown:
Search for "Quality Gates" section (planned feature)

8.2 Common Issues

Issue 1: Too many claims excluded by Gate 1

• Symptom: Most claims marked as excluded
• Cause: Input is primarily opinion/editorial
• Solution: Clarify that Gate 1 is working correctly; input may not be fact-checkable

Issue 2: All verdicts marked LOW confidence

• Symptom: gate4Stats shows all verdicts in LOW tier
• Cause: Search returning few sources or sources have little relevant content
• Solution: Check search provider credentials, improve search queries, adjust Gate 4 thresholds

Issue 3: Central claims excluded by Gate 1

• Symptom: Core thesis claims not appearing in results
• Cause: gate1KeepCentralClaims=false in config
• Solution: Enable central claim exception in UCM Pipeline config

9. Configuration Reference

9.1 UCM Pipeline Config

9.2 Default Values

10. Testing Quality Gates

10.1 Unit Tests

File: apps/web/src/lib/analyzer/tests/quality-gates.test.ts

Coverage:

• Gate 1 exclusion scenarios (opinion, prediction, low-specificity)
• Gate 1 central claim exception
• Gate 4 confidence tier assignment
• Gate 4 context scoping
• Gate 4 central claim exception

10.2 Integration Tests

Test Analysis Inputs:

1. High-quality factual article -> Expect: Most claims pass Gate 1, high Gate 4 confidence
2. Opinion editorial -> Expect: Most claims excluded by Gate 1
3. Low-source analysis -> Expect: Low Gate 4 confidence tiers

10.3 Manual Testing

Steps:

1. Run analysis on test input
2. Check result JSON for qualityGates object
3. Verify exclusion reasons match expected criteria
4. Verify confidence tiers match source/fact counts

11. Related Documentation

• Calculations.md (see Calculations.md in local docs) - Verdict calculation methodology, confidence modulation
• Architecture Overview - Architecture overview, pipeline flow
• TriplePath Architecture - Pipeline variants and quality gate enforcement
• Evidence_Quality_Filtering.md (see Evidence_Quality_Filtering.md in local docs) - Evidence filtering (related to proposed Gate 3)

12. Conclusion

Quality Gates ensure that FactHarbor maintains high standards for claim evaluation and verdict confidence. The current implementation (Gate 1 and Gate 4) provides:

1. Input quality control (Gate 1) - Only verifiable claims analyzed
2. Output quality control (Gate 4) - Verdicts backed by sufficient evidence
3. Transparency - Gate stats included in every result
4. Configurability - Thresholds adjustable via UCM

Key Takeaways:

• Gate 1 filters non-verifiable claims before research
• Gate 4 assesses verdict confidence based on evidence
• Central claims receive special treatment (kept regardless of quality)
• Confidence modulates verdict truth percentages
• Gate stats available in result JSON for debugging

Last Updated: February 3, 2026
Document Status: Consolidated reference - combines scattered gate documentation