Skip to Content
Last modified by Robert Schaub on 2025/12/24 20:33
From version 4.1
edited by Robert Schaub
on 2025/12/12 15:41
Change comment: Imported from XAR
To version 5.1
edited by Robert Schaub
on 2025/12/14 18:59
Change comment: Imported from XAR

Summary

Details

Page properties
Content
... ... @@ -1,9 +1,9 @@
1 1  = AKEL — AI Knowledge Extraction Layer =
2 2  
3 -AKEL is FactHarbor’s automated intelligence subsystem.
3 +AKEL is FactHarbor's automated intelligence subsystem.
4 4  Its purpose is to reduce human workload, enhance consistency, and enable scalable knowledge processing — **without ever replacing human judgment**.
5 5  
6 -All AKEL outputs are marked with **AuthorType = AI** and require human approval before publication.
6 +AKEL outputs are marked with **AuthorType = AI** and published according to risk-based review policies (see Publication Modes below).
7 7  
8 8  AKEL operates in two modes:
9 9  
... ... @@ -10,21 +10,23 @@
10 10  * **Single-node mode** (POC & Beta 0)
11 11  * **Federated multi-node mode** (Release 1.0+)
12 12  
13 -Human reviewers, experts, and moderators always retain final authority.
13 +Human reviewers, experts, and moderators always retain final authority over content marked as "Human-Reviewed."
14 14  
15 15  ----
16 16  
17 17  == Purpose and Role ==
18 18  
19 -AKEL transforms unstructured inputs into structured, review-ready drafts.
19 +AKEL transforms unstructured inputs into structured, publication-ready content.
20 20  
21 21  Core responsibilities:
22 22  
23 23  * Claim extraction from arbitrary text
24 -* Claim classification (domain, type, evaluability, safety)
24 +* Claim classification (domain, type, evaluability, safety, **risk tier**)
25 25  * Scenario generation (definitions, boundaries, assumptions, methodology)
26 26  * Evidence summarization and metadata extraction
27 -* Contradiction detection
27 +* **Contradiction detection and counter-evidence search**
28 +* **Reservation and limitation identification**
29 +* **Bubble detection** (echo chambers, conspiracy theories, isolated sources)
28 28  * Re-evaluation proposal generation
29 29  * Cross-node embedding exchange (Release 1.0+)
30 30  
... ... @@ -34,10 +34,12 @@
34 34  
35 35  * **AKEL Orchestrator** – central coordinator
36 36  * **Claim Extractor**
37 -* **Claim Classifier**
39 +* **Claim Classifier** (with risk tier assignment)
38 38  * **Scenario Generator**
39 39  * **Evidence Summarizer**
40 -* **Contradiction Detector**
42 +* **Contradiction Detector** (enhanced with counter-evidence search)
43 +* **Quality Gate Validator**
44 +* **Audit Sampling Scheduler**
41 41  * **Embedding Handler** (Release 1.0+)
42 42  * **Federation Sync Adapter** (Release 1.0+)
43 43  
... ... @@ -52,17 +52,220 @@
52 52  * External LLM API (optional)
53 53  * Embeddings (from local or federated peers)
54 54  
55 -=== Outputs (all require human approval) ===
56 -* ClaimVersion (draft)
57 -* ScenarioVersion (draft)
58 -* EvidenceVersion (summary + metadata draft)
59 -* VerdictVersion (draft; internal only)
59 +=== Outputs (publication mode varies by risk tier) ===
60 +* ClaimVersion (draft or AI-generated)
61 +* ScenarioVersion (draft or AI-generated)
62 +* EvidenceVersion (summary + metadata, draft or AI-generated)
63 +* VerdictVersion (draft, AI-generated, or human-reviewed)
60 60  * Contradiction alerts
65 +* Reservation and limitation notices
61 61  * Re-evaluation proposals
62 62  * Updated embeddings
63 63  
64 64  ----
65 65  
71 +== Publication Modes ==
72 +
73 +AKEL content is published according to three modes:
74 +
75 +=== Mode 1: Draft-Only (Never Public) ===
76 +
77 +**Used for:**
78 +* Failed quality gate checks
79 +* Sensitive topics flagged for expert review
80 +* Unclear scope or missing critical sources
81 +* High reputational risk content
82 +
83 +**Visibility:** Internal review queue only
84 +
85 +=== Mode 2: Published as AI-Generated (No Prior Human Review) ===
86 +
87 +**Requirements:**
88 +* All automated quality gates passed (see below)
89 +* Risk tier permits AI-draft publication (Tier B or C)
90 +* Contradiction search completed successfully
91 +* Clear labeling as "AI-Generated, Awaiting Human Review"
92 +
93 +**Label shown to users:**
94 +```
95 +[AI-Generated] This content was produced by AI and has not yet been human-reviewed.
96 +Source: AI | Review Status: Pending | Risk Tier: [B/C]
97 +Contradiction Search: Completed | Last Updated: [timestamp]
98 +```
99 +
100 +**User actions:**
101 +* Browse and read content
102 +* Request human review (escalates to review queue)
103 +* Flag for expert attention
104 +
105 +=== Mode 3: Published as Human-Reviewed ===
106 +
107 +**Requirements:**
108 +* Human reviewer or domain expert validated
109 +* All quality gates passed
110 +* Visible "Human-Reviewed" mark with reviewer role and timestamp
111 +
112 +**Label shown to users:**
113 +```
114 +[Human-Reviewed] This content has been validated by human reviewers.
115 +Source: AI+Human | Review Status: Approved | Reviewed by: [Role] on [timestamp]
116 +Risk Tier: [A/B/C] | Contradiction Search: Completed
117 +```
118 +
119 +----
120 +
121 +== Risk Tiers ==
122 +
123 +AKEL assigns risk tiers to all content to determine appropriate review requirements:
124 +
125 +=== Tier A — High Risk / High Impact ===
126 +
127 +**Domains:** Medical, legal, elections, safety/security, major reputational harm
128 +
129 +**Publication policy:**
130 +* Human review REQUIRED before "Human-Reviewed" status
131 +* AI-generated content MAY be published but:
132 + ** Clearly flagged as AI-draft with prominent disclaimer
133 + ** May have limited visibility
134 + ** Auto-escalated to expert review queue
135 + ** User warnings displayed
136 +
137 +**Audit rate:** Recommendation: 30-50% of published AI-drafts sampled in first 6 months
138 +
139 +=== Tier B — Medium Risk ===
140 +
141 +**Domains:** Contested public policy, complex science, causality claims, significant financial impact
142 +
143 +**Publication policy:**
144 +* AI-draft CAN publish immediately with clear labeling
145 +* Sampling audits conducted (see Audit System below)
146 +* High-engagement items auto-escalated to expert review
147 +* Users can request human review
148 +
149 +**Audit rate:** Recommendation: 10-20% of published AI-drafts sampled
150 +
151 +=== Tier C — Low Risk ===
152 +
153 +**Domains:** Definitions, simple factual lookups with strong primary sources, historical facts, established scientific consensus
154 +
155 +**Publication policy:**
156 +* AI-draft default publication mode
157 +* Sampling audits sufficient
158 +* Community flagging available
159 +* Human review on request
160 +
161 +**Audit rate:** Recommendation: 5-10% of published AI-drafts sampled
162 +
163 +----
164 +
165 +== Quality Gates (Mandatory Before AI-Draft Publication) ==
166 +
167 +All AI-generated content must pass these automated checks before Mode 2 publication:
168 +
169 +=== Gate 1: Source Quality ===
170 +* Primary sources identified and accessible
171 +* Source reliability scored against whitelist
172 +* Citation completeness verified
173 +* Publication dates checked
174 +* Author credentials validated (where applicable)
175 +
176 +=== Gate 2: Contradiction Search (MANDATORY) ===
177 +
178 +**The system MUST actively search for:**
179 +
180 +* **Counter-evidence** – Rebuttals, conflicting results, contradictory studies
181 +* **Reservations** – Caveats, limitations, boundary conditions, applicability constraints
182 +* **Alternative interpretations** – Different framings, definitions, contextual variations
183 +* **Bubble detection** – Conspiracy theories, echo chambers, ideologically isolated sources
184 +
185 +**Search coverage requirements:**
186 +* Academic literature (BOTH supporting AND opposing views)
187 +* Reputable media across diverse political/ideological perspectives
188 +* Official contradictions (retractions, corrections, updates, amendments)
189 +* Domain-specific skeptics, critics, and alternative expert opinions
190 +* Cross-cultural and international perspectives
191 +
192 +**Search must actively avoid algorithmic bubbles:**
193 +* Deliberately seek opposing viewpoints
194 +* Check for echo chamber patterns in source clusters
195 +* Identify tribal or ideological source clustering
196 +* Flag when search space appears artificially constrained
197 +* Verify diversity of perspectives represented
198 +
199 +**Outcomes:**
200 +* **Strong counter-evidence found** → Auto-escalate to Tier B or draft-only mode
201 +* **Significant uncertainty detected** → Require uncertainty disclosure in verdict
202 +* **Bubble indicators present** → Flag for expert review and human validation
203 +* **Limited perspective diversity** → Expand search or flag for human review
204 +
205 +=== Gate 3: Uncertainty Quantification ===
206 +* Confidence scores calculated for all claims and verdicts
207 +* Limitations explicitly stated
208 +* Data gaps identified and disclosed
209 +* Strength of evidence assessed
210 +* Alternative scenarios considered
211 +
212 +=== Gate 4: Structural Integrity ===
213 +* No hallucinations detected (fact-checking against sources)
214 +* Logic chain valid and traceable
215 +* References accessible and verifiable
216 +* No circular reasoning
217 +* Premises clearly stated
218 +
219 +**If any gate fails:**
220 +* Content remains in draft-only mode
221 +* Failure reason logged
222 +* Human review required before publication
223 +* Failure patterns analyzed for system improvement
224 +
225 +----
226 +
227 +== Audit System (Sampling-Based Quality Assurance) ==
228 +
229 +Instead of reviewing ALL AI output, FactHarbor implements stratified sampling audits:
230 +
231 +=== Sampling Strategy ===
232 +
233 +Audits prioritize:
234 +* **Risk tier** (higher tiers get more frequent audits)
235 +* **AI confidence score** (low confidence → higher sampling rate)
236 +* **Traffic and engagement** (high-visibility content audited more)
237 +* **Novelty** (new claim types, new domains, emerging topics)
238 +* **Disagreement signals** (user flags, contradiction alerts, community reports)
239 +
240 +=== Audit Process ===
241 +
242 +1. System selects content for audit based on sampling strategy
243 +2. Human auditor reviews AI-generated content against quality standards
244 +3. Auditor validates or corrects:
245 + * Claim extraction accuracy
246 + * Scenario appropriateness
247 + * Evidence relevance and interpretation
248 + * Verdict reasoning
249 + * Contradiction search completeness
250 +4. Audit outcome recorded (pass/fail + detailed feedback)
251 +5. Failed audits trigger immediate content review
252 +6. Audit results feed back into system improvement
253 +
254 +=== Feedback Loop (Continuous Improvement) ===
255 +
256 +Audit outcomes systematically improve:
257 +* **Query templates** – Refined based on missed evidence patterns
258 +* **Retrieval source weights** – Adjusted for accuracy and reliability
259 +* **Contradiction detection heuristics** – Enhanced to catch missed counter-evidence
260 +* **Model prompts and extraction rules** – Tuned for better claim extraction
261 +* **Risk tier assignments** – Recalibrated based on error patterns
262 +* **Bubble detection algorithms** – Improved to identify echo chambers
263 +
264 +=== Audit Transparency ===
265 +
266 +* Audit statistics published regularly
267 +* Accuracy rates by risk tier tracked and reported
268 +* System improvements documented
269 +* Community can view aggregate audit performance
270 +
271 +----
272 +
66 66  == Architecture Overview ==
67 67  
68 68  {{include reference="FactHarbor.Specification.Diagrams.AKEL Architecture.WebHome"/}}
... ... @@ -77,6 +77,7 @@
77 77  * Exchanges canonicalized claim forms
78 78  * Exchanges scenario templates
79 79  * Sends + receives contradiction alerts
287 +* Shares audit findings (with privacy controls)
80 80  * Never shares model weights
81 81  * Never overrides local governance
82 82  
... ... @@ -88,14 +88,39 @@
88 88  
89 89  ----
90 90  
91 -== Human Approval Workflow ==
299 +== Human Review Workflow (Mode 3 Publication) ==
92 92  
93 -1. AKEL generates draft outputs (AuthorType = AI)
94 -2. Reviewers inspect and approve/moderate the drafts
95 -3. Experts validate high-risk or domain-specific outputs
96 -4. Moderators finalize publication
97 -5. Version numbers increment, history preserved
301 +For content requiring human validation before "Human-Reviewed" status:
98 98  
99 -No AKEL output is ever published automatically.
303 +1. AKEL generates content and publishes as AI-draft (Mode 2) or keeps as draft (Mode 1)
304 +2. Reviewers inspect content in review queue
305 +3. Reviewers validate quality gates were correctly applied
306 +4. Experts validate high-risk (Tier A) or domain-specific outputs
307 +5. Moderators finalize "Human-Reviewed" publication
308 +6. Version numbers increment, full history preserved
100 100  
310 +**Note:** Most AI-generated content (Tier B and C) can remain in Mode 2 (AI-Generated) indefinitely. Human review is optional for these tiers unless users or audits flag issues.
311 +
101 101  ----
313 +
314 +== POC v1 Behavior ==
315 +
316 +The POC explicitly demonstrates AI-generated content publication:
317 +
318 +* Produces public AI-generated output (Mode 2)
319 +* No human data sources required
320 +* No human approval gate
321 +* Clear "AI-Generated - POC/Demo" labeling
322 +* All quality gates active (including contradiction search)
323 +* Users understand this demonstrates AI reasoning capabilities
324 +* Risk tier classification shown (demo purposes)
325 +
326 +----
327 +
328 +== Related Pages ==
329 +
330 +* [[Automation>>FactHarbor.Specification.Automation.WebHome]]
331 +* [[Requirements (Roles)>>FactHarbor.Specification.Requirements.WebHome]]
332 +* [[Workflows>>FactHarbor.Specification.Workflows.WebHome]]
333 +* [[Governance>>FactHarbor.Organisation.Governance]]
334 +