Wiki source code of Automation Philosophy
Last modified by Robert Schaub on 2026/02/08 08:29
Hide last authors
| author | version | line-number | content |
|---|---|---|---|
 |
1.1 | 1 | = Automation Philosophy = |
| |
1.3 | 2 | |
| |
1.1 | 3 | **Core Principle**: AKEL is primary. Humans monitor, improve, and handle exceptions. |
| |
1.3 | 4 | |
| |
1.1 | 5 | == 1. The Principle == |
| |
1.3 | 6 | |
| |
1.1 | 7 | **FactHarbor is AI-first, not AI-assisted.** |
| 8 | This is not: | ||
| |
1.3 | 9 | |
| |
1.1 | 10 | * ❌ "AI helps humans make better decisions" |
| 11 | * ❌ "Humans review AI recommendations" | ||
| 12 | * ❌ "AI drafts, humans approve" | ||
| 13 | This is: | ||
| 14 | * ✅ "AI makes decisions, humans improve the AI" | ||
| 15 | * ✅ "Humans monitor metrics, not individual outputs" | ||
| 16 | * ✅ "Fix the system, not the data" | ||
| |
1.3 | 17 | |
| |
1.1 | 18 | == 2. Why This Matters == |
| |
1.3 | 19 | |
| |
1.1 | 20 | === 2.1 Scalability === |
| |
1.3 | 21 | |
| |
1.1 | 22 | **Human review doesn't scale**: |
| |
1.3 | 23 | |
| 24 | * 1 person can review 100 claims/day carefully | ||
| |
1.1 | 25 | * FactHarbor aims for millions of claims |
| 26 | * Would need 10,000+ reviewers | ||
| 27 | * Impossible to maintain consistency | ||
| 28 | **Algorithmic processing scales**: | ||
| 29 | * AKEL processes 1 claim or 1 million claims with same consistency | ||
| 30 | * Cost per claim approaches zero at scale | ||
| 31 | * Quality improves with more data | ||
| 32 | * 24/7 availability | ||
| |
1.3 | 33 | |
| |
1.1 | 34 | === 2.2 Consistency === |
| |
1.3 | 35 | |
| |
1.1 | 36 | **Human judgment varies**: |
| |
1.3 | 37 | |
| |
1.1 | 38 | * Different reviewers apply criteria differently |
| 39 | * Same reviewer makes different decisions on different days | ||
| 40 | * Influenced by fatigue, mood, recent examples | ||
| 41 | * Unconscious biases affect decisions | ||
| 42 | **Algorithmic processing is consistent**: | ||
| 43 | * Same input → same output, always | ||
| 44 | * Rules applied uniformly | ||
| 45 | * No mood, fatigue, or bias | ||
| 46 | * Predictable behavior | ||
| |
1.3 | 47 | |
| |
1.1 | 48 | === 2.3 Transparency === |
| |
1.3 | 49 | |
| |
1.1 | 50 | **Human judgment is opaque**: |
| |
1.3 | 51 | |
| |
1.1 | 52 | * "I just know" - hard to explain |
| 53 | * Expertise in human head | ||
| 54 | * Can't audit thought process | ||
| 55 | * Difficult to improve systematically | ||
| 56 | **Algorithmic processing is transparent**: | ||
| 57 | * Code can be audited | ||
| 58 | * Parameters are documented | ||
| 59 | * Decision logic is explicit | ||
| 60 | * Changes are tracked | ||
| 61 | * Can test "what if" scenarios | ||
| |
1.3 | 62 | |
| |
1.1 | 63 | === 2.4 Improvement === |
| |
1.3 | 64 | |
| |
1.1 | 65 | **Improving human judgment**: |
| |
1.3 | 66 | |
| |
1.1 | 67 | * Train each person individually |
| 68 | * Hope training transfers consistently | ||
| 69 | * Subjective quality assessment | ||
| 70 | * Slow iteration | ||
| 71 | **Improving algorithms**: | ||
| 72 | * Change code once, affects all decisions | ||
| 73 | * Test on historical data before deploying | ||
| 74 | * Measure improvement objectively | ||
| 75 | * Rapid iteration (deploy multiple times per week) | ||
| |
1.3 | 76 | |
| |
1.1 | 77 | == 3. The Human Role == |
| |
1.3 | 78 | |
| |
1.1 | 79 | Humans in FactHarbor are **system architects**, not **content judges**. |
| |
1.3 | 80 | |
| |
1.1 | 81 | === 3.1 What Humans Do === |
| |
1.3 | 82 | |
| |
1.1 | 83 | **Monitor** system performance: |
| |
1.3 | 84 | |
| |
1.1 | 85 | * Watch dashboards showing aggregate metrics |
| 86 | * Identify when metrics fall outside acceptable ranges | ||
| 87 | * Spot patterns in errors or edge cases | ||
| 88 | * Track user feedback trends | ||
| 89 | **Improve** algorithms and policies: | ||
| 90 | * Analyze systematic errors | ||
| 91 | * Propose algorithm improvements | ||
| 92 | * Update policies based on learning | ||
| 93 | * Test changes before deployment | ||
| 94 | * Document learnings | ||
| 95 | **Handle** exceptions: | ||
| 96 | * Items AKEL explicitly flags for review | ||
| 97 | * System gaming attempts | ||
| 98 | * Abuse and harassment | ||
| 99 | * Legal/safety emergencies | ||
| 100 | **Govern** the system: | ||
| 101 | * Set risk tier policies | ||
| 102 | * Define acceptable performance ranges | ||
| 103 | * Allocate resources | ||
| 104 | * Make strategic decisions | ||
| |
1.3 | 105 | |
| |
1.1 | 106 | === 3.2 What Humans Do NOT Do === |
| |
1.3 | 107 | |
| |
1.1 | 108 | **Review** individual claims for correctness: |
| |
1.3 | 109 | |
| |
1.1 | 110 | * ❌ "Let me check if this verdict is right" |
| 111 | * ❌ "I'll approve these before publication" | ||
| 112 | * ❌ "This needs human judgment" | ||
| 113 | **Override** AKEL decisions routinely: | ||
| 114 | * ❌ "AKEL got this wrong, I'll fix it" | ||
| 115 | * ❌ "I disagree with this verdict" | ||
| 116 | * ❌ "This source should be rated higher" | ||
| 117 | **Act as** approval gates: | ||
| 118 | * ❌ "All claims must be human-approved" | ||
| 119 | * ❌ "High-risk claims need review" | ||
| 120 | * ❌ "Quality assurance before publication" | ||
| 121 | **Why not?** Because this defeats the purpose and doesn't scale. | ||
| |
1.3 | 122 | |
| |
1.1 | 123 | == 4. When Humans Intervene == |
| |
1.3 | 124 | |
| |
1.1 | 125 | === 4.1 Legitimate Interventions === |
| |
1.3 | 126 | |
| |
1.1 | 127 | **Humans should intervene when**: |
| |
1.3 | 128 | |
| 129 | ==== AKEL explicitly flags for review ==== | ||
| 130 | |||
| 131 | : | ||
| 132 | |||
| |
1.1 | 133 | * AKEL's confidence is too low |
| 134 | * Detected potential manipulation | ||
| 135 | * Unusual pattern requiring human judgment | ||
| 136 | * Clear policy: "Flag if confidence <X" | ||
| |
1.3 | 137 | |
| 138 | ==== System metrics show problems ==== | ||
| 139 | |||
| 140 | : | ||
| 141 | |||
| |
1.1 | 142 | * Processing time suddenly increases |
| 143 | * Error rate jumps | ||
| 144 | * Confidence distribution shifts | ||
| 145 | * User feedback becomes negative | ||
| |
1.3 | 146 | |
| 147 | ==== Systematic bias detected ==== | ||
| 148 | |||
| 149 | : | ||
| 150 | |||
| |
1.1 | 151 | * Metrics show pattern of unfairness |
| 152 | * Particular domains consistently scored oddly | ||
| 153 | * Source types systematically mis-rated | ||
| |
1.3 | 154 | |
| 155 | ==== Legal/safety emergency ==== | ||
| 156 | |||
| 157 | : | ||
| 158 | |||
| |
1.1 | 159 | * Legal takedown required |
| 160 | * Imminent harm to individuals | ||
| 161 | * Security breach | ||
| 162 | * Compliance violation | ||
| |
1.3 | 163 | |
| |
1.1 | 164 | === 4.2 Illegitimate Interventions === |
| |
1.3 | 165 | |
| |
1.1 | 166 | **Humans should NOT intervene for**: |
| |
1.3 | 167 | |
| 168 | ==== "I disagree with this verdict" ==== | ||
| 169 | |||
| 170 | : | ||
| 171 | |||
| |
1.1 | 172 | * Problem: Your opinion vs AKEL's analysis |
| 173 | * Solution: If AKEL is systematically wrong, fix the algorithm | ||
| 174 | * Action: Gather data, propose algorithm improvement | ||
| |
1.3 | 175 | |
| 176 | ==== "This source should rank higher" ==== | ||
| 177 | |||
| 178 | : | ||
| 179 | |||
| |
1.1 | 180 | * Problem: Subjective preference |
| 181 | * Solution: Fix scoring rules systematically | ||
| 182 | * Action: Analyze why AKEL scored it lower, adjust scoring algorithm if justified | ||
| |
1.3 | 183 | |
| 184 | ==== "Manual quality gate" ==== | ||
| 185 | |||
| 186 | : | ||
| 187 | |||
| |
1.1 | 188 | * Problem: Creates bottleneck, defeats automation |
| 189 | * Solution: Improve AKEL's quality to not need human gate | ||
| 190 | * Action: Set quality thresholds in algorithm, not human review | ||
| |
1.3 | 191 | |
| 192 | ==== "I know better than the algorithm" ==== | ||
| 193 | |||
| 194 | : | ||
| 195 | |||
| |
1.1 | 196 | * Problem: Doesn't scale, introduces bias |
| 197 | * Solution: Teach the algorithm what you know | ||
| 198 | * Action: Update training data, adjust parameters, document expertise in policy | ||
| |
1.3 | 199 | |
| |
1.1 | 200 | == 5. Fix the System, Not the Data == |
| |
1.3 | 201 | |
| |
1.1 | 202 | **Fundamental principle**: When AKEL makes mistakes, improve AKEL, don't fix individual outputs. |
| |
1.3 | 203 | |
| |
1.1 | 204 | === 5.1 Why? === |
| |
1.3 | 205 | |
| |
1.1 | 206 | **Fixing individual outputs**: |
| |
1.3 | 207 | |
| |
1.1 | 208 | * Doesn't prevent future similar errors |
| 209 | * Doesn't scale (too many outputs) | ||
| 210 | * Creates inconsistency | ||
| 211 | * Hides systematic problems | ||
| 212 | **Fixing the system**: | ||
| 213 | * Prevents future similar errors | ||
| 214 | * Scales automatically | ||
| 215 | * Maintains consistency | ||
| 216 | * Surfaces and resolves root causes | ||
| |
1.3 | 217 | |
| |
1.1 | 218 | === 5.2 Process === |
| |
1.3 | 219 | |
| |
1.1 | 220 | **When you see a "wrong" AKEL decision**: |
| |
1.3 | 221 | |
| 222 | ==== Document it ==== | ||
| 223 | |||
| 224 | : | ||
| 225 | |||
| |
1.1 | 226 | * What was the claim? |
| 227 | * What did AKEL decide? | ||
| 228 | * What should it have decided? | ||
| 229 | * Why do you think it's wrong? | ||
| |
1.3 | 230 | |
| 231 | ==== Investigate ==== | ||
| 232 | |||
| 233 | : | ||
| 234 | |||
| |
1.1 | 235 | * Is this a one-off, or a pattern? |
| 236 | * Check similar claims - same issue? | ||
| 237 | * What caused AKEL to decide this way? | ||
| 238 | * What rule/parameter needs changing? | ||
| |
1.3 | 239 | |
| 240 | ==== Propose systematic fix ==== | ||
| 241 | |||
| 242 | : | ||
| 243 | |||
| |
1.1 | 244 | * Algorithm change? |
| 245 | * Policy clarification? | ||
| 246 | * Training data update? | ||
| 247 | * Parameter adjustment? | ||
| |
1.3 | 248 | |
| 249 | ==== Test the fix ==== | ||
| 250 | |||
| 251 | : | ||
| 252 | |||
| |
1.1 | 253 | * Run on historical data |
| 254 | * Does it fix this case? | ||
| 255 | * Does it break other cases? | ||
| 256 | * What's the overall impact? | ||
| |
1.3 | 257 | |
| 258 | ==== Deploy and monitor ==== | ||
| 259 | |||
| 260 | : | ||
| 261 | |||
| |
1.1 | 262 | * Gradual rollout |
| 263 | * Watch metrics closely | ||
| 264 | * Gather feedback | ||
| 265 | * Iterate if needed | ||
| |
1.3 | 266 | |
| |
1.1 | 267 | == 6. Balancing Automation and Human Values == |
| |
1.3 | 268 | |
| |
1.1 | 269 | === 6.1 Algorithms Embody Values === |
| |
1.3 | 270 | |
| |
1.1 | 271 | **Important**: Automation doesn't mean "value-free" |
| 272 | **Algorithms encode human values**: | ||
| |
1.3 | 273 | |
| |
1.1 | 274 | * Which evidence types matter most? |
| 275 | * How much weight to peer review? | ||
| 276 | * What constitutes "high risk"? | ||
| 277 | * When to flag for human review? | ||
| 278 | **These are human choices**, implemented in code. | ||
| |
1.3 | 279 | |
| |
1.1 | 280 | === 6.2 Human Governance of Automation === |
| |
1.3 | 281 | |
| |
1.1 | 282 | **Humans set**: |
| |
1.3 | 283 | |
| |
1.1 | 284 | * ✅ Risk tier policies (what's high-risk?) |
| 285 | * ✅ Evidence weighting (what types of evidence matter?) | ||
| 286 | * ✅ Source scoring criteria (what makes a source credible?) | ||
| 287 | * ✅ Moderation policies (what's abuse?) | ||
| 288 | * ✅ Bias mitigation strategies | ||
| 289 | **AKEL applies**: | ||
| 290 | * ✅ These policies consistently | ||
| 291 | * ✅ At scale | ||
| 292 | * ✅ Transparently | ||
| 293 | * ✅ Without subjective variation | ||
| |
1.3 | 294 | |
| |
1.1 | 295 | === 6.3 Continuous Value Alignment === |
| |
1.3 | 296 | |
| |
1.1 | 297 | **Ongoing process**: |
| |
1.3 | 298 | |
| |
1.1 | 299 | * Monitor: Are outcomes aligned with values? |
| 300 | * Analyze: Where do values and outcomes diverge? | ||
| 301 | * Adjust: Update policies or algorithms | ||
| 302 | * Test: Validate alignment improved | ||
| 303 | * Repeat: Values alignment is never "done" | ||
| |
1.3 | 304 | |
| |
1.1 | 305 | == 7. Cultural Implications == |
| |
1.3 | 306 | |
| |
1.1 | 307 | === 7.1 Mindset Shift Required === |
| |
1.3 | 308 | |
| |
1.1 | 309 | **From**: "I'm a content expert who reviews claims" |
| 310 | **To**: "I'm a system architect who improves algorithms" | ||
| 311 | **From**: "Good work means catching errors" | ||
| 312 | **To**: "Good work means preventing errors systematically" | ||
| 313 | **From**: "I trust my judgment" | ||
| 314 | **To**: "I make my judgment codifiable and testable" | ||
| |
1.3 | 315 | |
| |
1.1 | 316 | === 7.2 New Skills Needed === |
| |
1.3 | 317 | |
| |
1.1 | 318 | **Less emphasis on**: |
| |
1.3 | 319 | |
| |
1.1 | 320 | * Individual content judgment |
| 321 | * Manual review skills | ||
| 322 | * Subjective expertise application | ||
| 323 | **More emphasis on**: | ||
| 324 | * Data analysis and metrics interpretation | ||
| 325 | * Algorithm design and optimization | ||
| 326 | * Policy formulation | ||
| 327 | * Testing and validation | ||
| 328 | * Documentation and knowledge transfer | ||
| |
1.3 | 329 | |
| |
1.1 | 330 | === 7.3 Job Satisfaction Sources === |
| |
1.3 | 331 | |
| |
1.1 | 332 | **Satisfaction comes from**: |
| |
1.3 | 333 | |
| |
1.1 | 334 | * ✅ Seeing metrics improve after your changes |
| 335 | * ✅ Building systems that help millions | ||
| 336 | * ✅ Solving systematic problems elegantly | ||
| 337 | * ✅ Continuous learning and improvement | ||
| 338 | * ✅ Transparent, auditable impact | ||
| 339 | **Not from**: | ||
| 340 | * ❌ Being the expert who makes final call | ||
| 341 | * ❌ Manual review and approval | ||
| 342 | * ❌ Gatekeeping | ||
| 343 | * ❌ Individual heroics | ||
| |
1.3 | 344 | |
| |
1.1 | 345 | == 8. Trust and Automation == |
| |
1.3 | 346 | |
| |
1.1 | 347 | === 8.1 Building Trust in AKEL === |
| |
1.3 | 348 | |
| |
1.1 | 349 | **Users trust AKEL when**: |
| |
1.3 | 350 | |
| |
1.1 | 351 | * Transparent: How decisions are made is documented |
| 352 | * Consistent: Same inputs → same outputs | ||
| 353 | * Measurable: Performance metrics are public | ||
| 354 | * Improvable: Clear process for getting better | ||
| 355 | * Governed: Human oversight of policies, not outputs | ||
| |
1.3 | 356 | |
| |
1.1 | 357 | === 8.2 What Trust Does NOT Mean === |
| |
1.3 | 358 | |
| |
1.1 | 359 | **Trust in automation ≠**: |
| |
1.3 | 360 | |
| |
1.1 | 361 | * ❌ "Never makes mistakes" (impossible) |
| 362 | * ❌ "Better than any human could ever be" (unnecessary) | ||
| 363 | * ❌ "Beyond human understanding" (must be understandable) | ||
| 364 | * ❌ "Set it and forget it" (requires continuous improvement) | ||
| 365 | **Trust in automation =**: | ||
| 366 | * ✅ Mistakes are systematic, not random | ||
| 367 | * ✅ Mistakes can be detected and fixed systematically | ||
| 368 | * ✅ Performance continuously improves | ||
| 369 | * ✅ Decision process is transparent and auditable | ||
| |
1.3 | 370 | |
| |
1.1 | 371 | == 9. Edge Cases and Exceptions == |
| |
1.3 | 372 | |
| |
1.1 | 373 | === 9.1 Some Things Still Need Humans === |
| |
1.3 | 374 | |
| |
1.1 | 375 | **AKEL flags for human review when**: |
| |
1.3 | 376 | |
| |
1.1 | 377 | * Confidence below threshold |
| 378 | * Detected manipulation attempt | ||
| 379 | * Novel situation not seen before | ||
| 380 | * Explicit policy requires human judgment | ||
| 381 | **Humans handle**: | ||
| 382 | * Items AKEL flags | ||
| 383 | * Not routine review | ||
| |
1.3 | 384 | |
| |
1.1 | 385 | === 9.2 Learning from Exceptions === |
| |
1.3 | 386 | |
| |
1.1 | 387 | **When humans handle an exception**: |
| |
1.3 | 388 | |
| |
1.1 | 389 | 1. Resolve the immediate case |
| 390 | 2. Document: What made this exceptional? | ||
| 391 | 3. Analyze: Could AKEL have handled this? | ||
| 392 | 4. Improve: Update AKEL to handle similar cases | ||
| 393 | 5. Monitor: Did exception rate decrease? | ||
| 394 | **Goal**: Fewer exceptions over time as AKEL learns. | ||
| 395 | --- | ||
| |
1.3 | 396 | **Remember**: AKEL is primary. You improve the SYSTEM. The system improves the CONTENT.-- |
| 397 | |||
| |
1.1 | 398 | == 10. Related Pages == |
| |
1.3 | 399 | |
| 400 | * [[Governance>>Archive.FactHarbor 2026\.02\.08.Organisation.Governance.WebHome]] - How AKEL is governed | ||
| |
1.1 | 401 | * [[Contributor Processes>>FactHarbor.Organisation.Contributor-Processes]] - How to improve the system |
| 402 | * [[Organisational Model>>FactHarbor.Organisation.Organisational-Model]] - Team structure and roles | ||
| 403 | * [[System Performance Metrics>>FactHarbor.Specification.System-Performance-Metrics]] - What we monitor |