Skip to Content
Last modified by Robert Schaub on 2025/12/22 14:32
From version 1.5
edited by Robert Schaub
on 2025/12/22 14:32
Change comment: Renamed back-links.
To version 1.1
edited by Robert Schaub
on 2025/12/22 14:22
Change comment: Imported from XAR

Summary

Details

Page properties
Parent
... ... @@ -1,1 +1,1 @@
1 -Test.FactHarbor pre12 V0\.9\.70.Roadmap.WebHome
1 +Test.FactHarbor.Roadmap.WebHome
Content
... ... @@ -1,25 +1,18 @@
1 -= POC2: Robust Quality & Reliability =
2 -
3 - **Phase Goal:** Prove AKEL produces high-quality outputs consistently at scale **Success Metric:** <5% hallucination rate, all 4 quality gates operational == 1. Overview == POC2 extends POC1 by implementing the full quality assurance framework (all 4 gates), adding evidence deduplication, and processing significantly more test articles to validate system reliability at scale. **Key Innovation:** Complete quality validation pipeline catches all categories of errors **What We're Proving:**
4 -
1 += POC2: Robust Quality & Reliability = **Phase Goal:** Prove AKEL produces high-quality outputs consistently at scale **Success Metric:** <5% hallucination rate, all 4 quality gates operational == 1. Overview == POC2 extends POC1 by implementing the full quality assurance framework (all 4 gates), adding evidence deduplication, and processing significantly more test articles to validate system reliability at scale. **Key Innovation:** Complete quality validation pipeline catches all categories of errors **What We're Proving:**
5 5  * All 4 quality gates work together effectively
6 6  * Evidence deduplication prevents artificial inflation
7 7  * System maintains quality at larger scale
8 8  * Quality metrics dashboard provides actionable insights == 2. New Requirements == === 2.1 NFR11: Complete Quality Assurance Framework === **Add Gates 2 & 3** (POC1 had only Gates 1 & 4) ==== Gate 2: Evidence Relevance Validation ==== **Purpose:** Ensure AI-linked evidence actually relates to the claim **Validation Checks:**
9 -
10 10  1. **Semantic Similarity:** Cosine similarity between claim and evidence embeddings ≥ 0.6
11 11  2. **Entity Overlap:** At least 1 shared named entity between claim and evidence
12 12  3. **Topic Relevance:** Evidence discusses the claim's subject matter (score ≥ 0.5) **Action if Failed:**
13 -
14 14  * Discard irrelevant evidence (don't count it)
15 15  * If <2 relevant evidence items remain → "Insufficient Evidence" verdict
16 16  * Log discarded evidence for quality review **Target:** 0% of evidence cited is off-topic ==== Gate 3: Scenario Coherence Check ==== **Purpose:** Validate scenarios are logical, complete, and meaningfully different **Validation Checks:**
17 -
18 18  1. **Completeness:** All required fields populated (assumptions, scope, evidence context)
19 19  2. **Internal Consistency:** Assumptions don't contradict each other (score <0.3)
20 20  3. **Distinctiveness:** Scenarios are meaningfully different (similarity <0.8)
21 21  4. **Minimum Detail:** At least 1 specific assumption per scenario **Action if Failed:**
22 -
23 23  * Merge duplicate scenarios
24 24  * Flag contradictory assumptions for review
25 25  * Reduce confidence score by 20%
... ... @@ -50,7 +50,9 @@
50 50  * Provenance tracked correctly **✅ Metrics Dashboard:**
51 51  * All metrics implemented and tracking
52 52  * Dashboard functional and useful
53 -* Alerts trigger appropriately == 4. Architecture Notes == **POC2 Enhanced Architecture:** {{code}}Input → AKEL Processing → All 4 Quality Gates → Display (claims + scenarios (1: Claim validation + evidence linking 2: Evidence relevance + verdicts) 3: Scenario coherence 4: Verdict confidence){{/code}} **Key Additions from POC1:**
46 +* Alerts trigger appropriately == 4. Architecture Notes == **POC2 Enhanced Architecture:** {{code}}
47 +Input → AKEL Processing → All 4 Quality Gates → Display (claims + scenarios (1: Claim validation + evidence linking 2: Evidence relevance + verdicts) 3: Scenario coherence 4: Verdict confidence)
48 +{{/code}} **Key Additions from POC1:**
54 54  * Scenario generation component
55 55  * Evidence deduplication system
56 56  * Gates 2 & 3 implementation
... ... @@ -57,7 +57,8 @@
57 57  * Quality metrics collection **Still Simplified vs. Full System:**
58 58  * Single AKEL orchestration (not multi-component pipeline)
59 59  * No review queue
60 -* No federation architecture **See:** [[Architecture>>Test.FactHarbor.Specification.Architecture.WebHome]] for details == Related Pages == * [[POC1>>Test.FactHarbor pre12 V0\.9\.70.Roadmap.POC1.WebHome]] - Previous phase
61 -* [[Beta 0>>Test.FactHarbor pre12 V0\.9\.70.Roadmap.Beta0.WebHome]] - Next phase
62 -* [[Roadmap Overview>>Test.FactHarbor pre12 V0\.9\.70.Roadmap.WebHome]]
55 +* No federation architecture **See:** [[Architecture>>Test.FactHarbor.Specification.Architecture.WebHome]] for details == Related Pages == * [[POC1>>Test.FactHarbor.Roadmap.POC1.WebHome]] - Previous phase
56 +* [[Beta 0>>Test.FactHarbor.Roadmap.Beta0.WebHome]] - Next phase
57 +* [[Roadmap Overview>>Test.FactHarbor.Roadmap.WebHome]]
63 63  * [[Architecture>>Test.FactHarbor.Specification.Architecture.WebHome]] **Document Status:** ✅ POC2 Specification Complete - Waiting for POC1 Completion **Version:** V0.9.70
59 +