Skip to Content
Last modified by Robert Schaub on 2025/12/22 14:16
From version 1.3
edited by Robert Schaub
on 2025/12/22 14:15
Change comment: Renamed back-links.
To version 1.2
edited by Robert Schaub
on 2025/12/22 14:15
Change comment: Update document after refactoring.

Summary

Details

Page properties
Content
... ... @@ -1,7 +1,4 @@
1 -= POC1: Core Workflow with Quality Gates =
2 -
3 - **Phase Goal:** Prove AKEL can produce credible, quality outputs without manual intervention **Success Metric:** <10% hallucination rate, quality gates prevent low-confidence publications == 1. Overview == POC1 validates that the core AKEL workflow (Article → Claims → Verdicts) can produce trustworthy fact-checking analyses automatically. This phase implements **2 critical quality gates** to prevent low-quality outputs from being published. **Key Innovation:** Quality validation BEFORE publication, not after **What We're Proving:**
4 -
1 += POC1: Core Workflow with Quality Gates = **Phase Goal:** Prove AKEL can produce credible, quality outputs without manual intervention **Success Metric:** <10% hallucination rate, quality gates prevent low-confidence publications == 1. Overview == POC1 validates that the core AKEL workflow (Article → Claims → Verdicts) can produce trustworthy fact-checking analyses automatically. This phase implements **2 critical quality gates** to prevent low-quality outputs from being published. **Key Innovation:** Quality validation BEFORE publication, not after **What We're Proving:**
5 5  * AKEL can reliably extract factual claims from articles
6 6  * AKEL can generate credible verdicts with proper evidence
7 7  * Quality gates prevent hallucinations and low-confidence outputs
... ... @@ -18,28 +18,29 @@
18 18  * Security hardening
19 19  * A/B testing
20 20  * Gates 2 & 3 (Evidence relevance, Scenario coherence) == 3. Requirements == === 3.1 NFR11: Quality Assurance Framework (POC1 Lite Version) === **Priority:** CRITICAL - Core POC1 Requirement **Fulfills:** AI safety, credibility, prevents embarrassing failures **Specification:** AKEL must validate outputs before displaying to users. POC1 implements a **2-gate subset** of the full NFR11 framework. ==== Gate 1: Claim Validation ==== **Purpose:** Ensure extracted claims are factual assertions, not opinions or predictions **Validation Checks:**
21 -
22 22  1. **Factual Statement Test:** Can this be verified with evidence?
23 23  2. **Opinion Detection:** Contains hedging language? ("I think", "probably", "best", "worst")
24 24  3. **Specificity Score:** Contains concrete details? (names, numbers, dates, locations)
25 25  4. **Future Prediction Test:** Makes claims about future events? **Pass Criteria:**
26 -{{code}}- isFactual: true
22 +{{code}}
23 +- isFactual: true
27 27  - opinionScore: ≤ 0.3
28 28  - specificityScore: ≥ 0.3
29 -- claimType: FACTUAL{{/code}} **Action if Failed:**
30 -
26 +- claimType: FACTUAL
27 +{{/code}} **Action if Failed:**
31 31  * Flag as "Non-verifiable: Opinion/Prediction/Ambiguous"
32 32  * Do NOT generate scenarios or verdicts
33 33  * Display explanation to user **Target:** 0% opinion statements processed as facts ==== Gate 4: Verdict Confidence Assessment ==== **Purpose:** Only publish verdicts with sufficient evidence and confidence **Validation Checks:**
34 -
35 35  1. **Evidence Count:** Minimum 2 independent sources
36 36  2. **Source Quality:** Average reliability ≥ 0.6 (on 0-1 scale)
37 37  3. **Evidence Agreement:** % supporting vs. contradicting ≥ 0.6
38 38  4. **Uncertainty Factors:** Count of hedging statements in reasoning **Confidence Tiers:**
39 -{{code}}HIGH (80-100%): - ≥3 sources - ≥0.7 average quality - ≥80% agreement MEDIUM (50-79%): - ≥2 sources - ≥0.6 average quality - ≥60% agreement LOW (0-49%): - ≥2 sources BUT low quality/agreement INSUFFICIENT: - <2 sources → DO NOT PUBLISH{{/code}} **POC1 Publication Rule:**
40 -
35 +{{code}}
36 +HIGH (80-100%): - ≥3 sources - ≥0.7 average quality - ≥80% agreement MEDIUM (50-79%): - ≥2 sources - ≥0.6 average quality - ≥60% agreement LOW (0-49%): - ≥2 sources BUT low quality/agreement INSUFFICIENT: - <2 sources → DO NOT PUBLISH
37 +{{/code}} **POC1 Publication Rule:**
41 41  * Minimum **MEDIUM** confidence required
42 -* Blocked verdicts show "Insufficient Evidence" message **Target:** 0% verdicts published with <2 sources === 3.2 Modified FR7: Automated Verdicts (Enhanced) === **Enhancement for POC1:** After AKEL generates a verdict, it must pass through the quality validation pipeline: {{code}}AKEL Workflow (POC1): 1. Extract claims from article ↓
39 +* Blocked verdicts show "Insufficient Evidence" message **Target:** 0% verdicts published with <2 sources === 3.2 Modified FR7: Automated Verdicts (Enhanced) === **Enhancement for POC1:** After AKEL generates a verdict, it must pass through the quality validation pipeline: {{code}}
40 +AKEL Workflow (POC1): 1. Extract claims from article ↓
43 43  2. [GATE 1] Validate each claim is fact-checkable ↓ (pass claims only)
44 44  3. Generate verdicts for each claim ↓
45 45  4. [GATE 4] Validate verdict has sufficient evidence ↓ (pass verdicts only)
... ... @@ -46,12 +46,15 @@
46 46  5. Display to user Failed claims/verdicts:
47 47  - Store in database with failure reason
48 48  - Display explanatory message to user
49 -- Log for quality metrics tracking{{/code}} **Updated Verdict States:**
47 +- Log for quality metrics tracking
48 +{{/code}} **Updated Verdict States:**
50 50  * PUBLISHED - Passed all gates
51 51  * INSUFFICIENT_EVIDENCE - Failed Gate 4
52 52  * NON_FACTUAL_CLAIM - Failed Gate 1
53 53  * PROCESSING - In progress
54 -* ERROR - System failure === 3.3 Modified FR4: Analysis Summary (Enhanced) === **Enhancement for POC1:** Analysis Summary must now display quality metadata: {{code}}Analysis Summary: Total Claims Found: 5 Verifiable Claims: 3 Non-verifiable (Opinion): 1 Non-verifiable (Prediction): 1 Verdicts Generated: 3 High Confidence: 1 Medium Confidence: 2 Insufficient Evidence: 0 Evidence Sources: 12 total Average Source Quality: 0.73 Quality Score: 8.5/10{{/code}} == 4. Success Criteria == POC1 is considered **SUCCESSFUL** if: **✅ Functional:**
53 +* ERROR - System failure === 3.3 Modified FR4: Analysis Summary (Enhanced) === **Enhancement for POC1:** Analysis Summary must now display quality metadata: {{code}}
54 +Analysis Summary: Total Claims Found: 5 Verifiable Claims: 3 Non-verifiable (Opinion): 1 Non-verifiable (Prediction): 1 Verdicts Generated: 3 High Confidence: 1 Medium Confidence: 2 Insufficient Evidence: 0 Evidence Sources: 12 total Average Source Quality: 0.73 Quality Score: 8.5/10
55 +{{/code}} == 4. Success Criteria == POC1 is considered **SUCCESSFUL** if: **✅ Functional:**
55 55  * Processes diverse test articles without crashes
56 56  * Generates verdicts for all factual claims
57 57  * Blocks all non-factual claims (0% pass through)
... ... @@ -69,13 +69,18 @@
69 69  * Clear path to POC2 defined == 5. Decision Gates == **POC1 → POC2 Decision:** * **IF** hallucination rate >10% → Pause, improve prompts before POC2
70 70  * **IF** majority of claims non-processable → Rethink claim extraction approach
71 71  * **IF** quality gates too strict (excessive blocking) → Adjust thresholds
72 -* **IF** quality gates too loose (hallucinations pass) → Tighten criteria **Only proceed to POC2 if all success criteria met** == 6. Architecture Notes == **POC1 Simplified Architecture:** {{code}}User Input → AKEL Processing → Quality Gates → Display (claim extraction (Gates 1 & 4) + verdicts){{/code}} **vs. Full System (Future):** {{code}}Input → Claim Extractor → Scenario Generator → Evidence Linker → Verdict Generator → All 4 Gates → Review Queue → Publication{{/code}} **POC1 Acceptable Simplifications:**
73 +* **IF** quality gates too loose (hallucinations pass) → Tighten criteria **Only proceed to POC2 if all success criteria met** == 6. Architecture Notes == **POC1 Simplified Architecture:** {{code}}
74 +User Input → AKEL Processing → Quality Gates → Display (claim extraction (Gates 1 & 4) + verdicts)
75 +{{/code}} **vs. Full System (Future):** {{code}}
76 +Input → Claim Extractor → Scenario Generator → Evidence Linker → Verdict Generator → All 4 Gates → Review Queue → Publication
77 +{{/code}} **POC1 Acceptable Simplifications:**
73 73  * Single AKEL call (not multi-component pipeline)
74 74  * No scenarios (implicit in verdicts)
75 75  * Basic evidence linking
76 76  * 2 gates instead of 4
77 77  * No review queue **See:** [[Architecture>>Test.FactHarbor.Specification.Architecture.WebHome]] for details == Related Pages == * [[Roadmap Overview>>Test.FactHarbor.Roadmap.WebHome]] - All phases
78 -* [[POC2 Requirements>>Test.FactHarbor pre11 V0\.9\.70.Roadmap.POC2.WebHome]] - Next phase
83 +* [[POC2 Requirements>>Test.FactHarbor.Roadmap.POC2.WebHome]] - Next phase
79 79  * [[Requirements>>Test.FactHarbor.Specification.Requirements.WebHome]] - Full system requirements
80 80  * [[Architecture>>Test.FactHarbor.Specification.Architecture.WebHome]] - System architecture
81 81  * [[NFR11 Full Specification>>Test.FactHarbor.Specification.Requirements.WebHome#NFR11]] - Complete quality framework **Document Status:** ✅ POC1 Specification Complete - Ready for Implementation **Version:** V0.9.70
87 +