Skip to Content

Changes for page Automation

Last modified by Robert Schaub on 2026/02/08 08:22
From version 1.1
edited by Robert Schaub
on 2025/12/18 12:03
Change comment: Imported from XAR
To version 4.4
edited by Robert Schaub
on 2026/01/20 20:27
Change comment: Renamed back-links.

Summary

Details

Page properties
Parent
... ... @@ -1,1 +1,1 @@
1 -FactHarbor.Specification.WebHome
1 +Archive.FactHarbor.Specification.WebHome
Content
... ... @@ -1,21 +1,31 @@
1 1  = Automation =
2 +
2 2  **How FactHarbor scales through automated claim evaluation.**
4 +
3 3  == 1. Automation Philosophy ==
6 +
4 4  FactHarbor is **automation-first**: AKEL (AI Knowledge Extraction Layer) makes all content decisions. Humans monitor system performance and improve algorithms.
5 5  **Why automation:**
9 +
6 6  * **Scale**: Can process millions of claims
7 7  * **Consistency**: Same evaluation criteria applied uniformly
8 8  * **Transparency**: Algorithms are auditable
9 9  * **Speed**: Results in <20 seconds typically
10 10  See [[Automation Philosophy>>FactHarbor.Organisation.Automation-Philosophy]] for detailed principles.
15 +
11 11  == 2. Claim Processing Flow ==
17 +
12 12  === 2.1 User Submits Claim ===
19 +
13 13  * User provides claim text + source URLs
14 14  * System validates format
15 15  * Assigns processing ID
16 16  * Queues for AKEL processing
24 +
17 17  === 2.2 AKEL Processing ===
26 +
18 18  **AKEL automatically:**
28 +
19 19  1. Parses claim into testable components
20 20  2. Extracts evidence from sources
21 21  3. Scores source credibility
... ... @@ -25,9 +25,12 @@
25 25  7. Publishes result
26 26  **Processing time**: Typically <20 seconds
27 27  **No human approval required** - publication is automatic
38 +
28 28  === 2.3 Publication States ===
40 +
29 29  **Processing**: AKEL working on claim (not visible to public)
30 30  **Published**: AKEL completed evaluation (public)
43 +
31 31  * Verdict displayed with confidence score
32 32  * Evidence and sources shown
33 33  * Risk tier indicated
... ... @@ -37,35 +37,128 @@
37 37  * Detected manipulation attempt
38 38  * Unusual pattern
39 39  * Moderator reviews and may take action
53 +
54 +== 2.5 LLM-Based Processing Architecture ==
55 +
56 +FactHarbor delegates complex reasoning and analysis tasks to Large Language Models (LLMs). The architecture evolves from POC to production:
57 +
58 +=== POC: Two-Phase Approach ===
59 +
60 +**Phase 1: Claim Extraction**
61 +
62 +* Single LLM call to extract all claims from submitted content
63 +* Light structure, focused on identifying distinct verifiable claims
64 +* Output: List of claims with context
65 +
66 +**Phase 2: Claim Analysis (Parallel)**
67 +
68 +* Single LLM call per claim (parallelizable)
69 +* Full structured output: Evidence, Scenarios, Sources, Verdict, Risk
70 +* Each claim analyzed independently
71 +
72 +**Advantages:**
73 +
74 +* Fast to implement ( to working POC)
75 +* Only 2-3 API calls total (1 + N claims)
76 +* Simple to debug (claim-level isolation)
77 +* Proves concept viability
78 +
79 +=== Production: Three-Phase Approach ===
80 +
81 +**Phase 1: Claim Extraction + Validation**
82 +
83 +* Extract distinct verifiable claims
84 +* Validate claim clarity and uniqueness
85 +* Remove duplicates and vague claims
86 +
87 +**Phase 2: Evidence Gathering (Parallel)**
88 +
89 +* For each claim independently:
90 +* Find supporting and contradicting evidence
91 +* Identify authoritative sources
92 +* Generate test scenarios
93 +* Validation: Check evidence quality and source validity
94 +* Error containment: Issues in one claim don't affect others
95 +
96 +**Phase 3: Verdict Generation (Parallel)**
97 +
98 +* For each claim:
99 +* Generate verdict based on validated evidence
100 +* Assess confidence and risk level
101 +* Flag low-confidence results for human review
102 +* Validation: Check verdict consistency with evidence
103 +
104 +**Advantages:**
105 +
106 +* Error containment between phases
107 +* Clear quality gates and validation
108 +* Observable metrics per phase
109 +* Scalable (parallel processing across claims)
110 +* Adaptable (can optimize each phase independently)
111 +
112 +=== LLM Task Delegation ===
113 +
114 +All complex cognitive tasks are delegated to LLMs:
115 +
116 +* **Claim Extraction**: Understanding context, identifying distinct claims
117 +* **Evidence Finding**: Analyzing sources, assessing relevance
118 +* **Scenario Generation**: Creating testable hypotheses
119 +* **Source Evaluation**: Assessing reliability and authority
120 +* **Verdict Generation**: Synthesizing evidence into conclusions
121 +* **Risk Assessment**: Evaluating potential impact
122 +
123 +=== Error Mitigation ===
124 +
125 +Research shows sequential LLM calls face compound error risks. FactHarbor mitigates this through:
126 +
127 +* **Validation gates** between phases
128 +* **Confidence thresholds** for quality control
129 +* **Parallel processing** to avoid error propagation across claims
130 +* **Human review queue** for low-confidence verdicts
131 +* **Independent claim processing** - errors in one claim don't cascade to others
132 +
40 40  == 3. Risk Tiers ==
134 +
41 41  Risk tiers classify claims by potential impact and guide audit sampling rates.
136 +
42 42  === 3.1 Tier A (High Risk) ===
138 +
43 43  **Domains**: Medical, legal, elections, safety, security
44 44  **Characteristics**:
141 +
45 45  * High potential for harm if incorrect
46 46  * Complex specialized knowledge required
47 47  * Often subject to regulation
48 48  **Publication**: AKEL publishes automatically with prominent risk warning
49 49  **Audit rate**: Higher sampling recommended
147 +
50 50  === 3.2 Tier B (Medium Risk) ===
149 +
51 51  **Domains**: Complex policy, science, causality claims
52 52  **Characteristics**:
152 +
53 53  * Moderate potential impact
54 54  * Requires careful evidence evaluation
55 55  * Multiple valid interpretations possible
56 56  **Publication**: AKEL publishes automatically with standard risk label
57 57  **Audit rate**: Moderate sampling recommended
158 +
58 58  === 3.3 Tier C (Low Risk) ===
160 +
59 59  **Domains**: Definitions, established facts, historical data
60 60  **Characteristics**:
163 +
61 61  * Low potential for harm
62 62  * Well-documented information
63 63  * Clear right/wrong answers typically
64 64  **Publication**: AKEL publishes by default
65 65  **Audit rate**: Lower sampling recommended
169 +
66 66  == 4. Quality Gates ==
171 +
67 67  AKEL applies quality gates before publication. If any fail, claim is **flagged** (not blocked - still published).
68 68  **Quality gates**:
174 +
69 69  * Sufficient evidence extracted (≥2 sources)
70 70  * Sources meet minimum credibility threshold
71 71  * Confidence score calculable
... ... @@ -72,14 +72,22 @@
72 72  * No detected manipulation patterns
73 73  * Claim parseable into testable form
74 74  **Failed gates**: Claim published with flag for moderator review
181 +
75 75  == 5. Automation Levels ==
76 -{{include reference="FactHarbor.Specification.Diagrams.Automation Level.WebHome"/}}
183 +
184 +{{include reference="Archive.FactHarbor.Specification.Diagrams.Automation Level.WebHome"/}}
77 77  FactHarbor progresses through automation maturity levels:
78 78  **Release 0.5** (Proof-of-Concept): Tier C only, human review required
79 79  **Release 1.0** (Initial): Tier B/C auto-published, Tier A flagged for review
80 80  **Release 2.0** (Mature): All tiers auto-published with risk labels, sampling audits
81 -See [[Automation Roadmap>>FactHarbor.Specification.Diagrams.Automation Roadmap.WebHome]] for detailed progression.
189 +See [[Automation Roadmap>>Archive.FactHarbor.Specification.Diagrams.Automation Roadmap.WebHome]] for detailed progression.
190 +
191 +== 5.5 Automation Roadmap ==
192 +
193 +{{include reference="Archive.FactHarbor.Specification.Diagrams.Automation Roadmap.WebHome"/}}
194 +
82 82  == 6. Human Role ==
196 +
83 83  Humans do NOT review content for approval. Instead:
84 84  **Monitoring**: Watch aggregate performance metrics
85 85  **Improvement**: Fix algorithms when patterns show issues
... ... @@ -86,7 +86,13 @@
86 86  **Exception handling**: Review AKEL-flagged items
87 87  **Governance**: Set policies AKEL applies
88 88  See [[Contributor Processes>>FactHarbor.Organisation.Contributor-Processes]] for how to improve the system.
203 +
204 +== 6.5 Manual vs Automated Matrix ==
205 +
206 +{{include reference="FactHarbor.Specification.Diagrams.Manual vs Automated matrix.WebHome"/}}
207 +
89 89  == 7. Moderation ==
209 +
90 90  Moderators handle items AKEL flags:
91 91  **Abuse detection**: Spam, manipulation, harassment
92 92  **Safety issues**: Content that could cause immediate harm
... ... @@ -94,7 +94,9 @@
94 94  **Action**: May temporarily hide content, ban users, or propose algorithm improvements
95 95  **Does NOT**: Routinely review claims or override verdicts
96 96  See [[Organisational Model>>FactHarbor.Organisation.Organisational-Model]] for moderator role details.
217 +
97 97  == 8. Continuous Improvement ==
219 +
98 98  **Performance monitoring**: Track AKEL accuracy, speed, coverage
99 99  **Issue identification**: Find systematic errors from metrics
100 100  **Algorithm updates**: Deploy improvements to fix patterns
... ... @@ -101,15 +101,21 @@
101 101  **A/B testing**: Validate changes before full rollout
102 102  **Retrospectives**: Learn from failures systematically
103 103  See [[Continuous Improvement>>FactHarbor.Organisation.How-We-Work-Together.Continuous-Improvement]] for improvement cycle.
226 +
104 104  == 9. Scalability ==
228 +
105 105  Automation enables FactHarbor to scale:
230 +
106 106  * **Millions of claims** processable
107 107  * **Consistent quality** at any volume
108 108  * **Cost efficiency** through automation
109 109  * **Rapid iteration** on algorithms
110 110  Without automation: Human review doesn't scale, creates bottlenecks, introduces inconsistency.
236 +
111 111  == 10. Transparency ==
238 +
112 112  All automation is transparent:
240 +
113 113  * **Algorithm parameters** documented
114 114  * **Evaluation criteria** public
115 115  * **Source scoring rules** explicit