Changes for page Automation

Last modified by Robert Schaub on 2025/12/24 20:34

From 1.4 to 1.3 From 7.3 to 7.2

From version 7.2

edited by Robert Schaub
on 2025/12/16 20:28

Change comment: Renamed back-links.

To version 1.4

edited by Robert Schaub
on 2025/12/11 21:34

Change comment: Renamed back-links.

Raw
Rendered

Summary

Page properties (1 modified, 0 added, 0 removed)

Details

Page properties

Content

@@ -1,296 +1,167 @@
  = Automation =
--Automation in FactHarbor amplifies human capability while implementing risk-based oversight.
++Automation in FactHarbor amplifies human capability but never replaces human oversight.
++All automated outputs require human review before publication.
  This chapter defines:
--* Risk-based publication model
--* Quality gates for AI-generated content
  * What must remain human-only
--* What AI (AKEL) can draft and publish
++* What AI (AKEL) can draft
  * What can be fully automated
  * How automation evolves through POC → Beta 0 → Release 1.0
--== POC v1 (AI-Generated Publication Demonstration) ==
--
--The goal of POC v1 is to validate the automated reasoning capabilities and demonstrate AI-generated content publication.
--
--=== Workflow ===
--
--1. **Input**: User pastes a block of raw text.
--1. **Deep Analysis (Background)**: The system autonomously performs the full pipeline **before** displaying the text:
--
--* Extraction & Normalisation
--* Scenario & Sub-query generation
--* Evidence retrieval with **contradiction search**
--* Quality gate validation
--* Verdict computation
--
--1. **Visualisation (Extraction & Marking)**: The system displays the text with claims extracted and marked.
--
--* **Verdict-Based Coloring**: The extraction highlights (e.g. Orange/Green) are chosen **according to the computed verdict** for each claim.
--* **AI-Generated Label**: Clear indication that content is AI-produced
--
--1. **Inspection**: User clicks a highlighted claim to see the **Reasoning Trail**, showing exactly which evidence and sub-queries led to that verdict.
--
--=== Technical Scope ===
--
--* **AI-Generated Publication**: Content published as Mode 2 (AI-Generated, no prior human review)
--* **Quality Gates Active**: All automated quality checks enforced
--* **Contradiction Search Demonstrated**: Shows counter-evidence and reservation detection
--* **Risk Tier Classification**: POC shows tier assignment (demo purposes)
--* **No Human Approval Gate**: Demonstrates scalable AI publication
--* **Structured Sub-Queries**: Logic generated by decomposing claims into the FactHarbor data model
--
  ----
--== Publication Model ==
++= Manual vs Automated Responsibilities =
--FactHarbor implements a risk-based publication model with three modes:
--
--=== Mode 1: Draft-Only ===
--
--* Failed quality gates
--* High-risk content pending expert review
--* Internal review queue only
--
--=== Mode 2: AI-Generated (Public) ===
--
--* Passed all quality gates
--* Risk tier B or C
--* Clear AI-generated labeling
--* Users can request human review
--
--=== Mode 3: Human-Reviewed ===
--
--* Validated by human reviewers/experts
--* "Human-Reviewed" status badge
--* Required for Tier A content publication
--
--See [[AKEL page>>FactHarbor.Specification.AI Knowledge Extraction Layer (AKEL).WebHome]] for detailed publication mode descriptions.
--
------
--
--== Risk Tiers and Automation Levels ==
--
--=== Tier A (High Risk) ===
--
--* **Domains**: Medical, legal, elections, safety, security
--* **Automation**: AI can draft, human review required for "Human-Reviewed" status
--* **AI publication**: Allowed with prominent disclaimers and warnings
--* **Audit rate**: Recommendation: 30-50%
--
--=== Tier B (Medium Risk) ===
--
--* **Domains**: Complex policy, science, causality claims
--* **Automation**: AI can draft and publish (Mode 2)
--* **Human review**: Optional, audit-based
--* **Audit rate**: Recommendation: 10-20%
--
--=== Tier C (Low Risk) ===
--
--* **Domains**: Definitions, established facts, historical data
--* **Automation**: AI publication default
--* **Human review**: On request or via sampling
--* **Audit rate**: Recommendation: 5-10%
--
------
--
  == Human-Only Tasks ==
--These require human judgment and cannot be automated:
++These require human judgment, ethics, or contextual interpretation:
--* **Ethical boundary decisions** (especially medical, political, psychological harm assessment)
--* **Dispute resolution** between conflicting expert opinions
--* **Governance policy** setting and enforcement
--* **Final authority** on Tier A "Human-Reviewed" status
--* **Audit system oversight** and quality standard definition
--* **Risk tier policy** adjustments based on societal context
++* Definition of key terms in claims
++* Approval or rejection of scenarios
++* Interpretation of evidence in context
++* Final verdict approval
++* Governance decisions and dispute resolution
++* High-risk domain oversight
++* Ethical boundary decisions (especially medical, political, psychological)
------
++== Semi-Automated (AI Draft → Human Review) ==
--== AI-Draft with Audit (Semi-Automated) ==
++AKEL can draft these, but humans must refine/approve:
--AKEL drafts these; humans validate via sampling audits:
++* Scenario structures (definitions, assumptions, context)
++* Evaluation methods
++* Evidence relevance suggestions
++* Reliability hints
++* Verdict reasoning chains
++* Uncertainty and limitations
++* Scenario comparison explanations
++* Suggestions for merging or splitting scenarios
++* Draft public summaries
--* **Scenario structures** (definitions, assumptions, context)
--* **Evaluation methods** and reasoning chains
--* **Evidence relevance** assessment and ranking
--* **Reliability scoring** and source evaluation
--* **Verdict reasoning** with uncertainty quantification
--* **Contradiction and reservation** identification
--* **Scenario comparison** explanations
--* **Public summaries** and accessibility text
--
--Most Tier B and C content remains in AI-draft status unless:
--
--* Users request human review
--* Audits identify errors
--* High engagement triggers review
--* Community flags issues
--
------
--
  == Fully Automated Structural Tasks ==
  These require no human interpretation:
--* **Claim normalization** (canonical form generation)
--* **Duplicate detection** (vector embeddings, clustering)
--* **Evidence metadata extraction** (dates, authors, publication info)
--* **Basic reliability heuristics** (source reputation scoring)
--* **Contradiction detection** (conflicting statements across sources)
--* **Re-evaluation triggers** (new evidence, source updates)
--* **Layout generation** (diagrams, summaries, UI presentation)
--* **Federation integrity checks** (cross-node data validation)
++* Claim normalization
++* Duplicate & cluster detection (vector embeddings)
++* Evidence metadata extraction
++* Basic reliability heuristics
++* Contradiction detection
++* Re-evaluation triggers
++* Batch layout generation (diagrams, summaries)
++* Federation integrity checks
  ----
--== Quality Gates (Automated) ==
++= Automation Roadmap =
--Before AI-draft publication (Mode 2), content must pass:
++Automation increases with maturity.
--1. **Source Quality Gate**
++== POC (Low Automation) ==
--* Primary sources verified
--* Citations complete and accessible
--* Source reliability scored
++=== Automated ===
--2. **Contradiction Search Gate** (MANDATORY)
++* Claim normalization
++* Light scenario templates
++* Evidence metadata extraction
++* Simple verdict drafts (internal only)
--* Counter-evidence actively sought
--* Reservations and limitations identified
--* Bubble detection (echo chambers, conspiracy theories)
--* Diverse perspective verification
++=== Human ===
--3. **Uncertainty Quantification Gate**
++* All scenario definitions
++* Evidence interpretation
++* Verdict creation
++* Governance
--* Confidence scores calculated
--* Limitations stated
--* Data gaps disclosed
++== Beta 0 (Medium Automation) ==
--4. **Structural Integrity Gate**
++=== Automated ===
--* No hallucinations detected
--* Logic chain valid
--* References verifiable
++* Detailed scenario drafts
++* Evidence reliability scoring
++* Cross-scenario comparisons
++* Contradiction detection (local + remote nodes)
++* Internal Truth Landscape drafts
--See [[AKEL page>>FactHarbor.Specification.AI Knowledge Extraction Layer (AKEL).WebHome]] for detailed quality gate specifications.
++=== Human ===
------
++* Scenario approval
++* Final verdict validation
--== Audit System ==
++== Release 1.0 (High Automation) ==
--Instead of reviewing all AI output, systematic sampling audits ensure quality:
++=== Automated ===
--=== Stratified Sampling ===
++* Full scenario generation (definitions, assumptions, boundaries)
++* Evidence relevance scoring and ranking
++* Bayesian verdict scoring across scenario sets
++* Multi-scenario summary generation
++* Anomaly detection across nodes
++* AKEL-assisted federated synchronization
--* Risk tier (A > B > C sampling rates)
--* Confidence scores (low confidence → more audits)
--* Traffic/engagement (popular content audited more)
--* Novelty (new topics/claim types prioritized)
--* User flags and disagreement signals
++=== Human ===
--=== Continuous Improvement Loop ===
++* Final approval of all scenarios and verdicts
++* Ethical decisions
++* Oversight and conflict resolution
--Audit findings improve:
--
--* Query templates
--* Source reliability weights
--* Contradiction detection algorithms
--* Risk tier assignment rules
--* Bubble detection heuristics
--
--=== Transparency ===
--
--* Audit statistics published
--* Accuracy rates by tier reported
--* System improvements documented
--
  ----
--== Automation Roadmap ==
++= Automation Levels =
--Automation capabilities increase with system maturity while maintaining quality oversight.
++== Level 0 — Human-Centric (POC) ==
--=== POC (Current Focus) ===
++AI is purely advisory, nothing auto-published.
--**Automated:**
++== Level 1 — Assisted (Beta 0) ==
--* Claim normalization
--* Scenario template generation
--* Evidence metadata extraction
--* Simple verdict drafts
--* **AI-generated publication** (Mode 2, with quality gates)
--* **Contradiction search**
--* **Risk tier assignment**
++AI drafts structures; humans approve each part.
--**Human:**
++== Level 2 — Structured (Release 1.0) ==
--* High-risk content validation (Tier A)
--* Sampling audits across all tiers
--* Quality standard refinement
--* Governance decisions
++AI produces near-complete drafts; humans refine.
--=== Beta 0 (Enhanced Automation) ===
++== Level 3 — Distributed Intelligence (Future) ==
--**Automated:**
++Nodes exchange embeddings, contradiction alerts, and scenario templates.
++Humans still approve everything.
--* Detailed scenario generation
--* Advanced evidence reliability scoring
--* Cross-scenario comparisons
--* Multi-source contradiction detection
--* Internal Truth Landscape generation
--* **Increased AI-draft coverage** (more Tier B content)
++----
--**Human:**
++= Automation Matrix =
--* Tier A final approval
--* Audit sampling (continued)
--* Expert validation of complex domains
--* Quality improvement oversight
++== Always Human ==
--=== Release 1.0 (High Automation) ===
++* Final verdict approval
++* Scenario validity
++* Ethical decisions
++* Dispute resolution
--**Automated:**
++== Mostly AI (Human Validation Needed) ==
--* Full scenario generation (comprehensive)
--* Bayesian verdict scoring across scenarios
--* Multi-scenario summary generation
--* Anomaly detection across federated nodes
--* AKEL-assisted cross-node synchronization
--* **Most Tier B and all Tier C** auto-published
++* Claim normalization
++* Clustering
++* Evidence metadata
++* Reliability heuristics
++* Scenario drafts
++* Contradiction detection
--**Human:**
++== Mixed ==
--* Tier A oversight (still required)
--* Strategic audits (lower sampling rates, higher value)
--* Ethical decisions and policy
--* Conflict resolution
++* Definitions of ambiguous terms
++* Boundary choices
++* Assumption evaluation
++* Evidence selection
++* Verdict reasoning
  ----
--== Automation Levels Diagram ==
++= Diagram References =
--{{include reference="FactHarbor.Archive.FactHarbor V0\.9\.23 Lost Data.Specification.Diagrams.Automation Level.WebHome"/}}
++{{include reference="FactHarbor.Archive.Diagrams v0\.8q.Automation Roadmap.WebHome"/}}
------
++{{include reference="FactHarbor.Archive.Diagrams v0\.8q.Automation Level.WebHome"/}}
--== Automation Roadmap Diagram ==
--
--{{include reference="Test.FactHarborV09.Specification.Diagrams.Automation Roadmap.WebHome"/}}
--
------
--
--== Manual vs Automated Matrix ==
--
--{{include reference="Test.FactHarborV09.Specification.Diagrams.Manual vs Automated matrix.WebHome"/}}
--
------
--
--== Related Pages ==
--
--* [[AKEL (AI Knowledge Extraction Layer)>>FactHarbor.Specification.AI Knowledge Extraction Layer (AKEL).WebHome]]
--* [[Requirements (Roles)>>FactHarbor.Specification.Requirements.WebHome]]
--* [[Workflows>>FactHarbor.Specification.Workflows.WebHome]]
--* [[Governance>>FactHarbor.Organisation.Governance]]
++{{include reference="FactHarbor.Archive.Diagrams v0\.8q.Manual vs Automated matrix.WebHome"/}}

Changes for page Automation

Summary

Details

Applications

Navigation

Need help?