Wiki source code of POC1: Core Workflow with Quality Gates
Last modified by Robert Schaub on 2025/12/24 20:35
Hide last authors
| author | version | line-number | content |
|---|---|---|---|
 |
1.1 | 1 | = POC1: Core Workflow with Quality Gates = |
| 2 | |||
| 3 | **Phase Goal:** Prove AKEL can produce credible, quality outputs without manual intervention | ||
| 4 | |||
| 5 | **Success Metric:** <10% hallucination rate, quality gates prevent low-confidence publications | ||
| 6 | |||
| |
1.4 | 7 | ---- |
| |
1.1 | 8 | |
| 9 | == 1. Overview == | ||
| 10 | |||
| 11 | POC1 validates that the core AKEL workflow (Article → Claims → Verdicts) can produce trustworthy fact-checking analyses automatically. This phase implements **2 critical quality gates** to prevent low-quality outputs from being published. | ||
| 12 | |||
| 13 | **Key Innovation:** Quality validation BEFORE publication, not after | ||
| 14 | |||
| 15 | **What We're Proving:** | ||
| |
1.3 | 16 | |
| |
1.1 | 17 | * AKEL can reliably extract factual claims from articles |
| 18 | * AKEL can generate credible verdicts with proper evidence | ||
| 19 | * Quality gates prevent hallucinations and low-confidence outputs | ||
| 20 | * Fully automated approach is viable | ||
| 21 | |||
| |
1.4 | 22 | ---- |
| |
1.1 | 23 | |
| 24 | == 2. Scope == | ||
| 25 | |||
| 26 | === In Scope === | ||
| 27 | |||
| 28 | * Core AKEL workflow (claim extraction, verdict generation) | ||
| 29 | * **Gate 1:** Claim Validation (factual vs. opinion/prediction) | ||
| 30 | * **Gate 4:** Verdict Confidence Assessment (minimum 2 sources, quality thresholds) | ||
| 31 | * Basic UI to display results | ||
| 32 | * Manual quality tracking | ||
| 33 | |||
| 34 | === Out of Scope (Deferred to POC2+) === | ||
| 35 | |||
| 36 | * User accounts / authentication | ||
| 37 | * Corrections system | ||
| 38 | * Search engine optimization (ClaimReview schema) | ||
| 39 | * Image verification | ||
| 40 | * API endpoints | ||
| 41 | * Archive.org integration | ||
| 42 | * Security hardening | ||
| 43 | * A/B testing | ||
| 44 | * Gates 2 & 3 (Evidence relevance, Scenario coherence) | ||
| 45 | |||
| |
1.4 | 46 | ---- |
| |
1.1 | 47 | |
| 48 | == 3. Requirements == | ||
| 49 | |||
| 50 | === 3.1 NFR11-POC1: Quality Assurance Framework (Lite) === | ||
| 51 | |||
| 52 | **Priority:** CRITICAL - Core POC1 Requirement | ||
| 53 | **Fulfills:** AI safety, credibility, prevents embarrassing failures | ||
| 54 | |||
| 55 | **Specification:** | ||
| 56 | |||
| 57 | AKEL must validate outputs before displaying to users. POC1 implements a **2-gate subset** of the full NFR11 framework. | ||
| 58 | |||
| 59 | ==== Gate 1: Claim Validation ==== | ||
| 60 | |||
| 61 | **Purpose:** Ensure extracted claims are factual assertions, not opinions or predictions | ||
| 62 | |||
| 63 | **Validation Checks:** | ||
| |
1.3 | 64 | |
| |
1.1 | 65 | 1. **Factual Statement Test:** Can this be verified with evidence? |
| 66 | 2. **Opinion Detection:** Contains hedging language? ("I think", "probably", "best", "worst") | ||
| 67 | 3. **Specificity Score:** Contains concrete details? (names, numbers, dates, locations) | ||
| 68 | 4. **Future Prediction Test:** Makes claims about future events? | ||
| 69 | |||
| 70 | **Pass Criteria:** | ||
| |
1.3 | 71 | {{code}}- isFactual: true |
| |
1.1 | 72 | - opinionScore: ≤ 0.3 |
| 73 | - specificityScore: ≥ 0.3 | ||
| |
1.3 | 74 | - claimType: FACTUAL{{/code}} |
| |
1.1 | 75 | |
| 76 | **Action if Failed:** | ||
| |
1.3 | 77 | |
| |
1.1 | 78 | * Flag as "Non-verifiable: Opinion/Prediction/Ambiguous" |
| 79 | * Do NOT generate scenarios or verdicts | ||
| 80 | * Display explanation to user | ||
| 81 | |||
| 82 | **Target:** 0% opinion statements processed as facts | ||
| 83 | |||
| |
1.4 | 84 | ---- |
| |
1.1 | 85 | |
| 86 | ==== Gate 4: Verdict Confidence Assessment ==== | ||
| 87 | |||
| 88 | **Purpose:** Only publish verdicts with sufficient evidence and confidence | ||
| 89 | |||
| 90 | **Validation Checks:** | ||
| |
1.3 | 91 | |
| |
1.1 | 92 | 1. **Evidence Count:** Minimum 2 independent sources |
| 93 | 2. **Source Quality:** Average reliability ≥ 0.6 (on 0-1 scale) | ||
| 94 | 3. **Evidence Agreement:** % supporting vs. contradicting ≥ 0.6 | ||
| 95 | 4. **Uncertainty Factors:** Count of hedging statements in reasoning | ||
| 96 | |||
| 97 | **Confidence Tiers:** | ||
| |
1.3 | 98 | {{code}}HIGH (80-100%): |
| |
1.1 | 99 | - ≥3 sources |
| 100 | - ≥0.7 average quality | ||
| 101 | - ≥80% agreement | ||
| 102 | |||
| 103 | MEDIUM (50-79%): | ||
| 104 | - ≥2 sources | ||
| 105 | - ≥0.6 average quality | ||
| 106 | - ≥60% agreement | ||
| 107 | |||
| 108 | LOW (0-49%): | ||
| 109 | - ≥2 sources BUT low quality/agreement | ||
| 110 | |||
| 111 | INSUFFICIENT: | ||
| |
1.3 | 112 | - <2 sources → DO NOT PUBLISH{{/code}} |
| |
1.1 | 113 | |
| 114 | **POC1 Publication Rule:** | ||
| |
1.3 | 115 | |
| |
1.1 | 116 | * Minimum **MEDIUM** confidence required |
| 117 | * Blocked verdicts show "Insufficient Evidence" message | ||
| 118 | |||
| 119 | **Target:** 0% verdicts published with <2 sources | ||
| 120 | |||
| |
1.4 | 121 | ---- |
| |
1.1 | 122 | |
| 123 | === 3.2 Modified FR7: Automated Verdicts (Enhanced) === | ||
| 124 | |||
| 125 | **Enhancement for POC1:** | ||
| 126 | |||
| 127 | After AKEL generates a verdict, it must pass through the quality validation pipeline: | ||
| 128 | |||
| 129 | {{code}} | ||
| 130 | AKEL Workflow (POC1): | ||
| 131 | |||
| 132 | 1. Extract claims from article | ||
| 133 | ↓ | ||
| 134 | 2. [GATE 1] Validate each claim is fact-checkable | ||
| 135 | ↓ (pass claims only) | ||
| 136 | 3. Generate verdicts for each claim | ||
| 137 | ↓ | ||
| 138 | 4. [GATE 4] Validate verdict has sufficient evidence | ||
| 139 | ↓ (pass verdicts only) | ||
| 140 | 5. Display to user | ||
| 141 | |||
| 142 | Failed claims/verdicts: | ||
| 143 | - Store in database with failure reason | ||
| 144 | - Display explanatory message to user | ||
| 145 | - Log for quality metrics tracking | ||
| 146 | {{/code}} | ||
| 147 | |||
| 148 | **Updated Verdict States:** | ||
| |
1.3 | 149 | |
| |
1.1 | 150 | * PUBLISHED - Passed all gates |
| 151 | * INSUFFICIENT_EVIDENCE - Failed Gate 4 | ||
| 152 | * NON_FACTUAL_CLAIM - Failed Gate 1 | ||
| 153 | * PROCESSING - In progress | ||
| 154 | * ERROR - System failure | ||
| 155 | |||
| |
1.4 | 156 | ---- |
| |
1.1 | 157 | |
| 158 | === 3.3 Modified FR4: Analysis Summary (Enhanced) === | ||
| 159 | |||
| 160 | **Enhancement for POC1:** | ||
| 161 | |||
| 162 | Analysis Summary must now display quality metadata: | ||
| 163 | |||
| 164 | {{code}} | ||
| 165 | Analysis Summary: | ||
| 166 | Total Claims Found: 5 | ||
| 167 | Verifiable Claims: 3 | ||
| 168 | Non-verifiable (Opinion): 1 | ||
| 169 | Non-verifiable (Prediction): 1 | ||
| 170 | |||
| 171 | Verdicts Generated: 3 | ||
| 172 | High Confidence: 1 | ||
| 173 | Medium Confidence: 2 | ||
| 174 | Insufficient Evidence: 0 | ||
| 175 | |||
| 176 | Evidence Sources: 12 total | ||
| 177 | Average Source Quality: 0.73 | ||
| 178 | |||
| 179 | Quality Score: 8.5/10 | ||
| 180 | {{/code}} | ||
| 181 | |||
| |
1.4 | 182 | ---- |
| |
1.1 | 183 | |
| 184 | == 4. Success Criteria == | ||
| 185 | |||
| 186 | POC1 is considered **SUCCESSFUL** if: | ||
| 187 | |||
| 188 | **✅ Functional:** | ||
| |
1.3 | 189 | |
| |
1.1 | 190 | * Processes diverse test articles without crashes |
| 191 | * Generates verdicts for all factual claims | ||
| 192 | * Blocks all non-factual claims (0% pass through) | ||
| 193 | * Blocks all insufficient-evidence verdicts (0% with <2 sources) | ||
| 194 | |||
| 195 | **✅ Quality:** | ||
| |
1.3 | 196 | |
| |
1.1 | 197 | * Hallucination rate <10% (manual verification) |
| 198 | * 0 verdicts with <2 sources published | ||
| 199 | * 0 opinion statements published as facts | ||
| 200 | * Average quality score ≥7.0/10 | ||
| 201 | |||
| 202 | **✅ Performance:** | ||
| |
1.3 | 203 | |
| |
1.1 | 204 | * Processing time reasonable for POC demonstration |
| 205 | * Quality gates execute efficiently | ||
| 206 | * UI displays results clearly | ||
| 207 | |||
| 208 | **✅ Learnings:** | ||
| |
1.3 | 209 | |
| |
1.1 | 210 | * Identified prompt engineering improvements |
| 211 | * Documented AKEL strengths/weaknesses | ||
| 212 | * Validated threshold values | ||
| 213 | * Clear path to POC2 defined | ||
| 214 | |||
| |
1.4 | 215 | ---- |
| |
1.1 | 216 | |
| 217 | == 5. Decision Gates == | ||
| 218 | |||
| 219 | **POC1 → POC2 Decision:** | ||
| 220 | |||
| 221 | * **IF** hallucination rate >10% → Pause, improve prompts before POC2 | ||
| 222 | * **IF** majority of claims non-processable → Rethink claim extraction approach | ||
| 223 | * **IF** quality gates too strict (excessive blocking) → Adjust thresholds | ||
| 224 | * **IF** quality gates too loose (hallucinations pass) → Tighten criteria | ||
| 225 | |||
| 226 | **Only proceed to POC2 if all success criteria met** | ||
| 227 | |||
| |
1.4 | 228 | ---- |
| |
1.1 | 229 | |
| 230 | == 6. Architecture Notes == | ||
| 231 | |||
| 232 | **POC1 Simplified Architecture:** | ||
| 233 | |||
| 234 | {{code}} | ||
| 235 | User Input → AKEL Processing → Quality Gates → Display | ||
| 236 | (claim extraction (Gates 1 & 4) | ||
| 237 | + verdicts) | ||
| 238 | {{/code}} | ||
| 239 | |||
| 240 | **vs. Full System (Future):** | ||
| 241 | |||
| 242 | {{code}} | ||
| 243 | Input → Claim Extractor → Scenario Generator → Evidence Linker | ||
| 244 | → Verdict Generator → All 4 Gates → Review Queue → Publication | ||
| 245 | {{/code}} | ||
| 246 | |||
| 247 | **POC1 Acceptable Simplifications:** | ||
| |
1.3 | 248 | |
| |
1.1 | 249 | * Single AKEL call (not multi-component pipeline) |
| 250 | * No scenarios (implicit in verdicts) | ||
| 251 | * Basic evidence linking | ||
| 252 | * 2 gates instead of 4 | ||
| 253 | * No review queue | ||
| 254 | |||
| |
1.10 | 255 | **See:** [[Architecture>>Archive.FactHarbor delta for V0\.9\.70.Specification.Architecture.WebHome]] for details |
| |
1.1 | 256 | |
| |
1.4 | 257 | ---- |
| |
1.1 | 258 | |
| 259 | == Related Pages == | ||
| 260 | |||
| |
1.9 | 261 | * [[Roadmap Overview>>Archive.FactHarbor delta for V0\.9\.70.Roadmap.WebHome]] - All phases |
| |
1.8 | 262 | * [[POC2 Requirements>>Archive.FactHarbor delta for V0\.9\.70.Roadmap.POC2.WebHome]] - Next phase |
| |
1.11 | 263 | * [[Requirements>>Archive.FactHarbor delta for V0\.9\.70.Specification.Requirements.WebHome]] - Full system requirements |
| |
1.10 | 264 | * [[Architecture>>Archive.FactHarbor delta for V0\.9\.70.Specification.Architecture.WebHome]] - System architecture |
| |
1.1 | 265 | * [[NFR11 Full Specification>>Test.FactHarbor.Specification.Requirements.WebHome#NFR11]] - Complete quality framework |
| 266 | |||
| |
1.4 | 267 | ---- |
| |
1.1 | 268 | |
| 269 | **Document Status:** ✅ POC1 Specification Complete - Ready for Implementation | ||
| 270 | **Version:** V0.9.70 |