Wiki source code of Automation
Last modified by Robert Schaub on 2025/12/24 20:34
Hide last authors
| author | version | line-number | content |
|---|---|---|---|
 |
1.1 | 1 | = Automation = |
| 2 | |||
| |
6.1 | 3 | Automation in FactHarbor amplifies human capability while implementing risk-based oversight. |
| |
1.1 | 4 | |
| 5 | This chapter defines: | ||
| |
7.4 | 6 | |
| |
6.1 | 7 | * Risk-based publication model |
| 8 | * Quality gates for AI-generated content | ||
| |
1.1 | 9 | * What must remain human-only |
| |
6.1 | 10 | * What AI (AKEL) can draft and publish |
| |
1.1 | 11 | * What can be fully automated |
| 12 | * How automation evolves through POC → Beta 0 → Release 1.0 | ||
| 13 | |||
| |
6.1 | 14 | == POC v1 (AI-Generated Publication Demonstration) == |
| |
3.1 | 15 | |
| |
6.1 | 16 | The goal of POC v1 is to validate the automated reasoning capabilities and demonstrate AI-generated content publication. |
| |
3.1 | 17 | |
| 18 | === Workflow === | ||
| 19 | |||
| 20 | 1. **Input**: User pastes a block of raw text. | ||
| |
4.1 | 21 | 1. **Deep Analysis (Background)**: The system autonomously performs the full pipeline **before** displaying the text: |
| |
7.4 | 22 | |
| |
4.1 | 23 | * Extraction & Normalisation |
| 24 | * Scenario & Sub-query generation | ||
| |
6.1 | 25 | * Evidence retrieval with **contradiction search** |
| 26 | * Quality gate validation | ||
| 27 | * Verdict computation | ||
| |
7.4 | 28 | |
| |
4.1 | 29 | 1. **Visualisation (Extraction & Marking)**: The system displays the text with claims extracted and marked. |
| |
7.4 | 30 | |
| |
4.1 | 31 | * **Verdict-Based Coloring**: The extraction highlights (e.g. Orange/Green) are chosen **according to the computed verdict** for each claim. |
| |
6.1 | 32 | * **AI-Generated Label**: Clear indication that content is AI-produced |
| |
7.4 | 33 | |
| |
4.1 | 34 | 1. **Inspection**: User clicks a highlighted claim to see the **Reasoning Trail**, showing exactly which evidence and sub-queries led to that verdict. |
| |
3.1 | 35 | |
| 36 | === Technical Scope === | ||
| 37 | |||
| |
6.1 | 38 | * **AI-Generated Publication**: Content published as Mode 2 (AI-Generated, no prior human review) |
| 39 | * **Quality Gates Active**: All automated quality checks enforced | ||
| 40 | * **Contradiction Search Demonstrated**: Shows counter-evidence and reservation detection | ||
| 41 | * **Risk Tier Classification**: POC shows tier assignment (demo purposes) | ||
| 42 | * **No Human Approval Gate**: Demonstrates scalable AI publication | ||
| 43 | * **Structured Sub-Queries**: Logic generated by decomposing claims into the FactHarbor data model | ||
| |
3.1 | 44 | |
| |
1.1 | 45 | ---- |
| 46 | |||
| |
6.1 | 47 | == Publication Model == |
| |
1.1 | 48 | |
| |
6.1 | 49 | FactHarbor implements a risk-based publication model with three modes: |
| 50 | |||
| 51 | === Mode 1: Draft-Only === | ||
| |
7.4 | 52 | |
| |
6.1 | 53 | * Failed quality gates |
| 54 | * High-risk content pending expert review | ||
| 55 | * Internal review queue only | ||
| 56 | |||
| 57 | === Mode 2: AI-Generated (Public) === | ||
| |
7.4 | 58 | |
| |
6.1 | 59 | * Passed all quality gates |
| 60 | * Risk tier B or C | ||
| 61 | * Clear AI-generated labeling | ||
| 62 | * Users can request human review | ||
| 63 | |||
| 64 | === Mode 3: Human-Reviewed === | ||
| |
7.4 | 65 | |
| |
6.1 | 66 | * Validated by human reviewers/experts |
| 67 | * "Human-Reviewed" status badge | ||
| 68 | * Required for Tier A content publication | ||
| 69 | |||
| |
7.14 | 70 | See [[AKEL page>>Archive.FactHarbor V0\.9\.18 copy.Specification.AI Knowledge Extraction Layer (AKEL).WebHome]] for detailed publication mode descriptions. |
| |
6.1 | 71 | |
| 72 | ---- | ||
| 73 | |||
| 74 | == Risk Tiers and Automation Levels == | ||
| 75 | |||
| 76 | === Tier A (High Risk) === | ||
| |
7.4 | 77 | |
| |
6.1 | 78 | * **Domains**: Medical, legal, elections, safety, security |
| 79 | * **Automation**: AI can draft, human review required for "Human-Reviewed" status | ||
| 80 | * **AI publication**: Allowed with prominent disclaimers and warnings | ||
| 81 | * **Audit rate**: Recommendation: 30-50% | ||
| 82 | |||
| 83 | === Tier B (Medium Risk) === | ||
| |
7.4 | 84 | |
| |
6.1 | 85 | * **Domains**: Complex policy, science, causality claims |
| 86 | * **Automation**: AI can draft and publish (Mode 2) | ||
| 87 | * **Human review**: Optional, audit-based | ||
| 88 | * **Audit rate**: Recommendation: 10-20% | ||
| 89 | |||
| 90 | === Tier C (Low Risk) === | ||
| |
7.4 | 91 | |
| |
6.1 | 92 | * **Domains**: Definitions, established facts, historical data |
| 93 | * **Automation**: AI publication default | ||
| 94 | * **Human review**: On request or via sampling | ||
| 95 | * **Audit rate**: Recommendation: 5-10% | ||
| 96 | |||
| 97 | ---- | ||
| 98 | |||
| |
5.1 | 99 | == Human-Only Tasks == |
| |
1.1 | 100 | |
| |
6.1 | 101 | These require human judgment and cannot be automated: |
| |
1.1 | 102 | |
| |
6.1 | 103 | * **Ethical boundary decisions** (especially medical, political, psychological harm assessment) |
| 104 | * **Dispute resolution** between conflicting expert opinions | ||
| 105 | * **Governance policy** setting and enforcement | ||
| 106 | * **Final authority** on Tier A "Human-Reviewed" status | ||
| 107 | * **Audit system oversight** and quality standard definition | ||
| 108 | * **Risk tier policy** adjustments based on societal context | ||
| |
1.1 | 109 | |
| |
6.1 | 110 | ---- |
| |
1.1 | 111 | |
| |
6.1 | 112 | == AI-Draft with Audit (Semi-Automated) == |
| |
1.1 | 113 | |
| |
6.1 | 114 | AKEL drafts these; humans validate via sampling audits: |
| |
1.1 | 115 | |
| |
6.1 | 116 | * **Scenario structures** (definitions, assumptions, context) |
| 117 | * **Evaluation methods** and reasoning chains | ||
| 118 | * **Evidence relevance** assessment and ranking | ||
| 119 | * **Reliability scoring** and source evaluation | ||
| 120 | * **Verdict reasoning** with uncertainty quantification | ||
| 121 | * **Contradiction and reservation** identification | ||
| 122 | * **Scenario comparison** explanations | ||
| 123 | * **Public summaries** and accessibility text | ||
| 124 | |||
| 125 | Most Tier B and C content remains in AI-draft status unless: | ||
| |
7.4 | 126 | |
| |
6.1 | 127 | * Users request human review |
| 128 | * Audits identify errors | ||
| 129 | * High engagement triggers review | ||
| 130 | * Community flags issues | ||
| 131 | |||
| 132 | ---- | ||
| 133 | |||
| |
5.1 | 134 | == Fully Automated Structural Tasks == |
| |
1.1 | 135 | |
| 136 | These require no human interpretation: | ||
| 137 | |||
| |
6.1 | 138 | * **Claim normalization** (canonical form generation) |
| 139 | * **Duplicate detection** (vector embeddings, clustering) | ||
| 140 | * **Evidence metadata extraction** (dates, authors, publication info) | ||
| 141 | * **Basic reliability heuristics** (source reputation scoring) | ||
| 142 | * **Contradiction detection** (conflicting statements across sources) | ||
| 143 | * **Re-evaluation triggers** (new evidence, source updates) | ||
| 144 | * **Layout generation** (diagrams, summaries, UI presentation) | ||
| 145 | * **Federation integrity checks** (cross-node data validation) | ||
| |
1.1 | 146 | |
| |
5.1 | 147 | ---- |
| |
1.1 | 148 | |
| |
6.1 | 149 | == Quality Gates (Automated) == |
| |
5.1 | 150 | |
| |
6.1 | 151 | Before AI-draft publication (Mode 2), content must pass: |
| |
1.1 | 152 | |
| |
6.1 | 153 | 1. **Source Quality Gate** |
| |
1.1 | 154 | |
| |
7.4 | 155 | * Primary sources verified |
| 156 | * Citations complete and accessible | ||
| 157 | * Source reliability scored | ||
| 158 | |||
| |
6.1 | 159 | 2. **Contradiction Search Gate** (MANDATORY) |
| 160 | |||
| |
7.4 | 161 | * Counter-evidence actively sought |
| 162 | * Reservations and limitations identified | ||
| 163 | * Bubble detection (echo chambers, conspiracy theories) | ||
| 164 | * Diverse perspective verification | ||
| 165 | |||
| |
6.1 | 166 | 3. **Uncertainty Quantification Gate** |
| 167 | |||
| |
7.4 | 168 | * Confidence scores calculated |
| 169 | * Limitations stated | ||
| 170 | * Data gaps disclosed | ||
| 171 | |||
| |
6.1 | 172 | 4. **Structural Integrity Gate** |
| 173 | |||
| |
7.4 | 174 | * No hallucinations detected |
| 175 | * Logic chain valid | ||
| 176 | * References verifiable | ||
| 177 | |||
| |
7.14 | 178 | See [[AKEL page>>Archive.FactHarbor V0\.9\.18 copy.Specification.AI Knowledge Extraction Layer (AKEL).WebHome]] for detailed quality gate specifications. |
| |
6.1 | 179 | |
| 180 | ---- | ||
| 181 | |||
| 182 | == Audit System == | ||
| 183 | |||
| 184 | Instead of reviewing all AI output, systematic sampling audits ensure quality: | ||
| 185 | |||
| 186 | === Stratified Sampling === | ||
| |
7.4 | 187 | |
| |
6.1 | 188 | * Risk tier (A > B > C sampling rates) |
| 189 | * Confidence scores (low confidence → more audits) | ||
| 190 | * Traffic/engagement (popular content audited more) | ||
| 191 | * Novelty (new topics/claim types prioritized) | ||
| 192 | * User flags and disagreement signals | ||
| 193 | |||
| 194 | === Continuous Improvement Loop === | ||
| |
7.4 | 195 | |
| |
6.1 | 196 | Audit findings improve: |
| |
7.4 | 197 | |
| |
6.1 | 198 | * Query templates |
| 199 | * Source reliability weights | ||
| 200 | * Contradiction detection algorithms | ||
| 201 | * Risk tier assignment rules | ||
| 202 | * Bubble detection heuristics | ||
| 203 | |||
| 204 | === Transparency === | ||
| |
7.4 | 205 | |
| |
6.1 | 206 | * Audit statistics published |
| 207 | * Accuracy rates by tier reported | ||
| 208 | * System improvements documented | ||
| 209 | |||
| 210 | ---- | ||
| 211 | |||
| 212 | == Automation Roadmap == | ||
| 213 | |||
| 214 | Automation capabilities increase with system maturity while maintaining quality oversight. | ||
| 215 | |||
| 216 | === POC (Current Focus) === | ||
| 217 | |||
| 218 | **Automated:** | ||
| |
7.4 | 219 | |
| |
5.1 | 220 | * Claim normalization |
| |
6.1 | 221 | * Scenario template generation |
| |
5.1 | 222 | * Evidence metadata extraction |
| |
6.1 | 223 | * Simple verdict drafts |
| 224 | * **AI-generated publication** (Mode 2, with quality gates) | ||
| 225 | * **Contradiction search** | ||
| 226 | * **Risk tier assignment** | ||
| |
1.1 | 227 | |
| |
6.1 | 228 | **Human:** |
| |
7.4 | 229 | |
| |
6.1 | 230 | * High-risk content validation (Tier A) |
| 231 | * Sampling audits across all tiers | ||
| 232 | * Quality standard refinement | ||
| 233 | * Governance decisions | ||
| |
1.1 | 234 | |
| |
6.1 | 235 | === Beta 0 (Enhanced Automation) === |
| |
1.1 | 236 | |
| |
6.1 | 237 | **Automated:** |
| |
7.4 | 238 | |
| |
6.1 | 239 | * Detailed scenario generation |
| 240 | * Advanced evidence reliability scoring | ||
| |
5.1 | 241 | * Cross-scenario comparisons |
| |
6.1 | 242 | * Multi-source contradiction detection |
| 243 | * Internal Truth Landscape generation | ||
| 244 | * **Increased AI-draft coverage** (more Tier B content) | ||
| |
1.1 | 245 | |
| |
6.1 | 246 | **Human:** |
| |
7.4 | 247 | |
| |
6.1 | 248 | * Tier A final approval |
| 249 | * Audit sampling (continued) | ||
| 250 | * Expert validation of complex domains | ||
| 251 | * Quality improvement oversight | ||
| |
1.1 | 252 | |
| |
6.1 | 253 | === Release 1.0 (High Automation) === |
| |
1.1 | 254 | |
| |
6.1 | 255 | **Automated:** |
| |
7.4 | 256 | |
| |
6.1 | 257 | * Full scenario generation (comprehensive) |
| 258 | * Bayesian verdict scoring across scenarios | ||
| |
5.1 | 259 | * Multi-scenario summary generation |
| |
6.1 | 260 | * Anomaly detection across federated nodes |
| 261 | * AKEL-assisted cross-node synchronization | ||
| 262 | * **Most Tier B and all Tier C** auto-published | ||
| |
1.1 | 263 | |
| |
6.1 | 264 | **Human:** |
| |
7.4 | 265 | |
| |
6.1 | 266 | * Tier A oversight (still required) |
| 267 | * Strategic audits (lower sampling rates, higher value) | ||
| 268 | * Ethical decisions and policy | ||
| 269 | * Conflict resolution | ||
| |
5.1 | 270 | |
| 271 | ---- | ||
| 272 | |||
| |
6.1 | 273 | == Automation Levels Diagram == |
| |
5.1 | 274 | |
| |
7.17 | 275 | {{include reference="Archive.FactHarbor V0\.9\.23 Lost Data.Specification.Diagrams.Automation Level.WebHome"/}} |
| |
5.1 | 276 | |
| |
6.1 | 277 | ---- |
| |
5.1 | 278 | |
| |
6.1 | 279 | == Automation Roadmap Diagram == |
| |
5.1 | 280 | |
| |
7.18 | 281 | {{include reference="Archive.FactHarbor V0\.9\.23 Lost Data.Specification.Diagrams.Automation Roadmap.WebHome"/}} |
| |
5.1 | 282 | |
| 283 | ---- | ||
| 284 | |||
| |
6.1 | 285 | == Manual vs Automated Matrix == |
| |
5.1 | 286 | |
| |
7.19 | 287 | {{include reference="Archive.FactHarbor V0\.9\.23 Lost Data.Specification.Diagrams.Manual vs Automated matrix.WebHome"/}} |
| |
5.1 | 288 | |
| 289 | ---- | ||
| 290 | |||
| |
6.1 | 291 | == Related Pages == |
| |
5.1 | 292 | |
| |
7.14 | 293 | * [[AKEL (AI Knowledge Extraction Layer)>>Archive.FactHarbor V0\.9\.18 copy.Specification.AI Knowledge Extraction Layer (AKEL).WebHome]] |
| |
7.15 | 294 | * [[Requirements (Roles)>>Archive.FactHarbor V0\.9\.18 copy.Specification.Requirements.WebHome]] |
| |
7.16 | 295 | * [[Workflows>>Archive.FactHarbor V0\.9\.18 copy.Specification.Workflows.WebHome]] |
| |
6.1 | 296 | * [[Governance>>FactHarbor.Organisation.Governance]] |