Wiki source code of POC1: Core Workflow with Quality Gates
Last modified by Robert Schaub on 2025/12/22 14:33
Show last authors
| author | version | line-number | content |
|---|---|---|---|
| 1 | = POC1: Core Workflow with Quality Gates = | ||
| 2 | |||
| 3 | **Phase Goal:** Prove AKEL can produce credible, quality outputs without manual intervention **Success Metric:** <10% hallucination rate, quality gates prevent low-confidence publications == 1. Overview == POC1 validates that the core AKEL workflow (Article → Claims → Verdicts) can produce trustworthy fact-checking analyses automatically. This phase implements **2 critical quality gates** to prevent low-quality outputs from being published. **Key Innovation:** Quality validation BEFORE publication, not after **What We're Proving:** | ||
| 4 | |||
| 5 | * AKEL can reliably extract factual claims from articles | ||
| 6 | * AKEL can generate credible verdicts with proper evidence | ||
| 7 | * Quality gates prevent hallucinations and low-confidence outputs | ||
| 8 | * Fully automated approach is viable == 2. Scope == === In Scope === * Core AKEL workflow (claim extraction, verdict generation) | ||
| 9 | * **Gate 1:** Claim Validation (factual vs. opinion/prediction) | ||
| 10 | * **Gate 4:** Verdict Confidence Assessment (minimum 2 sources, quality thresholds) | ||
| 11 | * Basic UI to display results | ||
| 12 | * Manual quality tracking === Out of Scope (Deferred to POC2+) === * User accounts / authentication | ||
| 13 | * Corrections system | ||
| 14 | * Search engine optimization (ClaimReview schema) | ||
| 15 | * Image verification | ||
| 16 | * API endpoints | ||
| 17 | * Archive.org integration | ||
| 18 | * Security hardening | ||
| 19 | * A/B testing | ||
| 20 | * Gates 2 & 3 (Evidence relevance, Scenario coherence) == 3. Requirements == === 3.1 NFR11: Quality Assurance Framework (POC1 Lite Version) === **Priority:** CRITICAL - Core POC1 Requirement **Fulfills:** AI safety, credibility, prevents embarrassing failures **Specification:** AKEL must validate outputs before displaying to users. POC1 implements a **2-gate subset** of the full NFR11 framework. ==== Gate 1: Claim Validation ==== **Purpose:** Ensure extracted claims are factual assertions, not opinions or predictions **Validation Checks:** | ||
| 21 | |||
| 22 | 1. **Factual Statement Test:** Can this be verified with evidence? | ||
| 23 | 2. **Opinion Detection:** Contains hedging language? ("I think", "probably", "best", "worst") | ||
| 24 | 3. **Specificity Score:** Contains concrete details? (names, numbers, dates, locations) | ||
| 25 | 4. **Future Prediction Test:** Makes claims about future events? **Pass Criteria:** | ||
| 26 | {{code}}- isFactual: true | ||
| 27 | - opinionScore: ≤ 0.3 | ||
| 28 | - specificityScore: ≥ 0.3 | ||
| 29 | - claimType: FACTUAL{{/code}} **Action if Failed:** | ||
| 30 | |||
| 31 | * Flag as "Non-verifiable: Opinion/Prediction/Ambiguous" | ||
| 32 | * Do NOT generate scenarios or verdicts | ||
| 33 | * Display explanation to user **Target:** 0% opinion statements processed as facts ==== Gate 4: Verdict Confidence Assessment ==== **Purpose:** Only publish verdicts with sufficient evidence and confidence **Validation Checks:** | ||
| 34 | |||
| 35 | 1. **Evidence Count:** Minimum 2 independent sources | ||
| 36 | 2. **Source Quality:** Average reliability ≥ 0.6 (on 0-1 scale) | ||
| 37 | 3. **Evidence Agreement:** % supporting vs. contradicting ≥ 0.6 | ||
| 38 | 4. **Uncertainty Factors:** Count of hedging statements in reasoning **Confidence Tiers:** | ||
| 39 | {{code}}HIGH (80-100%): - ≥3 sources - ≥0.7 average quality - ≥80% agreement MEDIUM (50-79%): - ≥2 sources - ≥0.6 average quality - ≥60% agreement LOW (0-49%): - ≥2 sources BUT low quality/agreement INSUFFICIENT: - <2 sources → DO NOT PUBLISH{{/code}} **POC1 Publication Rule:** | ||
| 40 | |||
| 41 | * Minimum **MEDIUM** confidence required | ||
| 42 | * Blocked verdicts show "Insufficient Evidence" message **Target:** 0% verdicts published with <2 sources === 3.2 Modified FR7: Automated Verdicts (Enhanced) === **Enhancement for POC1:** After AKEL generates a verdict, it must pass through the quality validation pipeline: {{code}}AKEL Workflow (POC1): 1. Extract claims from article ↓ | ||
| 43 | 2. [GATE 1] Validate each claim is fact-checkable ↓ (pass claims only) | ||
| 44 | 3. Generate verdicts for each claim ↓ | ||
| 45 | 4. [GATE 4] Validate verdict has sufficient evidence ↓ (pass verdicts only) | ||
| 46 | 5. Display to user Failed claims/verdicts: | ||
| 47 | - Store in database with failure reason | ||
| 48 | - Display explanatory message to user | ||
| 49 | - Log for quality metrics tracking{{/code}} **Updated Verdict States:** | ||
| 50 | * PUBLISHED - Passed all gates | ||
| 51 | * INSUFFICIENT_EVIDENCE - Failed Gate 4 | ||
| 52 | * NON_FACTUAL_CLAIM - Failed Gate 1 | ||
| 53 | * PROCESSING - In progress | ||
| 54 | * ERROR - System failure === 3.3 Modified FR4: Analysis Summary (Enhanced) === **Enhancement for POC1:** Analysis Summary must now display quality metadata: {{code}}Analysis Summary: Total Claims Found: 5 Verifiable Claims: 3 Non-verifiable (Opinion): 1 Non-verifiable (Prediction): 1 Verdicts Generated: 3 High Confidence: 1 Medium Confidence: 2 Insufficient Evidence: 0 Evidence Sources: 12 total Average Source Quality: 0.73 Quality Score: 8.5/10{{/code}} == 4. Success Criteria == POC1 is considered **SUCCESSFUL** if: **✅ Functional:** | ||
| 55 | * Processes diverse test articles without crashes | ||
| 56 | * Generates verdicts for all factual claims | ||
| 57 | * Blocks all non-factual claims (0% pass through) | ||
| 58 | * Blocks all insufficient-evidence verdicts (0% with <2 sources) **✅ Quality:** | ||
| 59 | * Hallucination rate <10% (manual verification) | ||
| 60 | * 0 verdicts with <2 sources published | ||
| 61 | * 0 opinion statements published as facts | ||
| 62 | * Average quality score ≥7.0/10 **✅ Performance:** | ||
| 63 | * Processing time reasonable for POC demonstration | ||
| 64 | * Quality gates execute efficiently | ||
| 65 | * UI displays results clearly **✅ Learnings:** | ||
| 66 | * Identified prompt engineering improvements | ||
| 67 | * Documented AKEL strengths/weaknesses | ||
| 68 | * Validated threshold values | ||
| 69 | * Clear path to POC2 defined == 5. Decision Gates == **POC1 → POC2 Decision:** * **IF** hallucination rate >10% → Pause, improve prompts before POC2 | ||
| 70 | * **IF** majority of claims non-processable → Rethink claim extraction approach | ||
| 71 | * **IF** quality gates too strict (excessive blocking) → Adjust thresholds | ||
| 72 | * **IF** quality gates too loose (hallucinations pass) → Tighten criteria **Only proceed to POC2 if all success criteria met** == 6. Architecture Notes == **POC1 Simplified Architecture:** {{code}}User Input → AKEL Processing → Quality Gates → Display (claim extraction (Gates 1 & 4) + verdicts){{/code}} **vs. Full System (Future):** {{code}}Input → Claim Extractor → Scenario Generator → Evidence Linker → Verdict Generator → All 4 Gates → Review Queue → Publication{{/code}} **POC1 Acceptable Simplifications:** | ||
| 73 | * Single AKEL call (not multi-component pipeline) | ||
| 74 | * No scenarios (implicit in verdicts) | ||
| 75 | * Basic evidence linking | ||
| 76 | * 2 gates instead of 4 | ||
| 77 | * No review queue **See:** [[Architecture>>Test.FactHarbor pre12 V0\.9\.70.Specification.Architecture.WebHome]] for details == Related Pages == * [[Roadmap Overview>>Test.FactHarbor pre12 V0\.9\.70.Roadmap.WebHome]] - All phases | ||
| 78 | * [[POC2 Requirements>>Test.FactHarbor pre12 V0\.9\.70.Roadmap.POC2.WebHome]] - Next phase | ||
| 79 | * [[Requirements>>Test.FactHarbor pre12 V0\.9\.70.Specification.Requirements.WebHome]] - Full system requirements | ||
| 80 | * [[Architecture>>Test.FactHarbor pre12 V0\.9\.70.Specification.Architecture.WebHome]] - System architecture | ||
| 81 | * [[NFR11 Full Specification>>Test.FactHarbor.Specification.Requirements.WebHome#NFR11]] - Complete quality framework **Document Status:** ✅ POC1 Specification Complete - Ready for Implementation **Version:** V0.9.70 |