Automation

Automation in FactHarbor amplifies human capability while implementing risk-based oversight.

This chapter defines:

• Risk-based publication model
• Quality gates for AI-generated content
• What must remain human-only
• What AI (AKEL) can draft and publish
• What can be fully automated
• How automation evolves through POC → Beta 0 → Release 1.0

POC v1 (AI-Generated Publication Demonstration)

The goal of POC v1 is to validate the automated reasoning capabilities and demonstrate AI-generated content publication.

Workflow

1. Input: User pastes a block of raw text.
2. Deep Analysis (Background): The system autonomously performs the full pipeline before displaying the text:

• Extraction & Normalisation
• Scenario & Sub-query generation
• Evidence retrieval with contradiction search
• Quality gate validation
• Verdict computation

1. Visualisation (Extraction & Marking): The system displays the text with claims extracted and marked.

• Verdict-Based Coloring: The extraction highlights (e.g. Orange/Green) are chosen according to the computed verdict for each claim.
• AI-Generated Label: Clear indication that content is AI-produced

1. Inspection: User clicks a highlighted claim to see the Reasoning Trail, showing exactly which evidence and sub-queries led to that verdict.

Technical Scope

• AI-Generated Publication: Content published as Mode 2 (AI-Generated, no prior human review)
• Quality Gates Active: All automated quality checks enforced
• Contradiction Search Demonstrated: Shows counter-evidence and reservation detection
• Risk Tier Classification: POC shows tier assignment (demo purposes)
• No Human Approval Gate: Demonstrates scalable AI publication
• Structured Sub-Queries: Logic generated by decomposing claims into the FactHarbor data model

Publication Model

FactHarbor implements a risk-based publication model with three modes:

Mode 1: Draft-Only

• Failed quality gates
• High-risk content pending expert review
• Internal review queue only

Mode 2: AI-Generated (Public)

• Passed all quality gates
• Risk tier B or C
• Clear AI-generated labeling
• Users can request human review

Mode 3: Human-Reviewed

• Validated by human reviewers/experts
• "Human-Reviewed" status badge
• Required for Tier A content publication

See AKEL page for detailed publication mode descriptions.

Risk Tiers and Automation Levels

Tier A (High Risk)

• Domains: Medical, legal, elections, safety, security
• Automation: AI can draft, human review required for "Human-Reviewed" status
• AI publication: Allowed with prominent disclaimers and warnings
• Audit rate: Recommendation: 30-50%

Tier B (Medium Risk)

• Domains: Complex policy, science, causality claims
• Automation: AI can draft and publish (Mode 2)
• Human review: Optional, audit-based
• Audit rate: Recommendation: 10-20%

Tier C (Low Risk)

• Domains: Definitions, established facts, historical data
• Automation: AI publication default
• Human review: On request or via sampling
• Audit rate: Recommendation: 5-10%

Human-Only Tasks

These require human judgment and cannot be automated:

• Ethical boundary decisions (especially medical, political, psychological harm assessment)
• Dispute resolution between conflicting expert opinions
• Governance policy setting and enforcement
• Final authority on Tier A "Human-Reviewed" status
• Audit system oversight and quality standard definition
• Risk tier policy adjustments based on societal context

AI-Draft with Audit (Semi-Automated)

AKEL drafts these; humans validate via sampling audits:

• Scenario structures (definitions, assumptions, context)
• Evaluation methods and reasoning chains
• Evidence relevance assessment and ranking
• Reliability scoring and source evaluation
• Verdict reasoning with uncertainty quantification
• Contradiction and reservation identification
• Scenario comparison explanations
• Public summaries and accessibility text

Most Tier B and C content remains in AI-draft status unless:

• Users request human review
• Audits identify errors
• High engagement triggers review
• Community flags issues

Fully Automated Structural Tasks

These require no human interpretation:

• Claim normalization (canonical form generation)
• Duplicate detection (vector embeddings, clustering)
• Evidence metadata extraction (dates, authors, publication info)
• Basic reliability heuristics (source reputation scoring)
• Contradiction detection (conflicting statements across sources)
• Re-evaluation triggers (new evidence, source updates)
• Layout generation (diagrams, summaries, UI presentation)
• Federation integrity checks (cross-node data validation)

Quality Gates (Automated)

Before AI-draft publication (Mode 2), content must pass:

1. Source Quality Gate

• Primary sources verified
• Citations complete and accessible
• Source reliability scored

2. Contradiction Search Gate (MANDATORY)

• Counter-evidence actively sought
• Reservations and limitations identified
• Bubble detection (echo chambers, conspiracy theories)
• Diverse perspective verification

3. Uncertainty Quantification Gate

• Confidence scores calculated
• Limitations stated
• Data gaps disclosed

4. Structural Integrity Gate

• No hallucinations detected
• Logic chain valid
• References verifiable

See AKEL page for detailed quality gate specifications.

Audit System

Instead of reviewing all AI output, systematic sampling audits ensure quality:

Stratified Sampling

• Risk tier (A > B > C sampling rates)
• Confidence scores (low confidence → more audits)
• Traffic/engagement (popular content audited more)
• Novelty (new topics/claim types prioritized)
• User flags and disagreement signals

Continuous Improvement Loop

Audit findings improve:

• Query templates
• Source reliability weights
• Contradiction detection algorithms
• Risk tier assignment rules
• Bubble detection heuristics

Transparency

• Audit statistics published
• Accuracy rates by tier reported
• System improvements documented

Automation Roadmap

Automation capabilities increase with system maturity while maintaining quality oversight.

POC (Current Focus)

Automated:

• Claim normalization
• Scenario template generation
• Evidence metadata extraction
• Simple verdict drafts
• AI-generated publication (Mode 2, with quality gates)
• Contradiction search
• Risk tier assignment

Human:

• High-risk content validation (Tier A)
• Sampling audits across all tiers
• Quality standard refinement
• Governance decisions

Beta 0 (Enhanced Automation)

Automated:

• Detailed scenario generation
• Advanced evidence reliability scoring
• Cross-scenario comparisons
• Multi-source contradiction detection
• Internal Truth Landscape generation
• Increased AI-draft coverage (more Tier B content)

Human:

• Tier A final approval
• Audit sampling (continued)
• Expert validation of complex domains
• Quality improvement oversight

Release 1.0 (High Automation)

Automated:

• Full scenario generation (comprehensive)
• Bayesian verdict scoring across scenarios
• Multi-scenario summary generation
• Anomaly detection across federated nodes
• AKEL-assisted cross-node synchronization
• Most Tier B and all Tier C auto-published

Human:

• Tier A oversight (still required)
• Strategic audits (lower sampling rates, higher value)
• Ethical decisions and policy
• Conflict resolution

Automation Levels Diagram

Automation Level

This diagram shows the progression of automation levels in Test.FactHarborV09.

graph TD
    L0[Level 0: AI-Publication POC
    - AI-Generated Content Publication
    - Quality Gates Active
    - Contradiction Search Mandatory
    - Risk Tier Classification
    - Audit Sampling Started
    - Recommendations: 30-50% Tier A, 10-20% B, 5-10% C]
    
    L1[Level 1: Assisted Beta 0
    - AI drafts structures + publishes Tier C
    - Humans audit samples
    - Tier B mostly AI-published
    - Tier A requires human review
    - Feedback loop active
    - Sampling rates adjust based on performance]
    
    L2[Level 2: Structured Release 1.0
    - AI produces near-complete outputs
    - Most Tier B AI-published
    - Tier A human-reviewed for publication
    - Mature audit system
    - Lower sampling rates as confidence increases]
    
    L3[Level 3: Distributed Intelligence Future
    - Federated contradiction detection
    - Cross-node audit sharing
    - Advanced bubble detection
    - Tier A still human-reviewed
    - Strategic audits only]
    
    L0 --> L1
    L1 --> L2
    L2 --> L3
    
    subgraph "Constant Across All Levels"
        HumanAuth[Humans retain final authority
        Quality gates mandatory
        Tier A requires human review for Mode 3
        Audit system active
        All numbers are recommendations]
    end
    
    classDef current fill:#e3f2fd,stroke:#1976d2,stroke-width:3px
    classDef future fill:#f3e5f5,stroke:#7b1fa2,stroke-width:2px
    
    class L0 current
    class L3 future

Automation Roadmap Diagram

Automation Roadmap

This diagram shows the automation roadmap from POC to Release.

graph LR
    POC[POC: AI Publication Demo
    - Risk Tier Classification
    - Quality Gates Active
    - Contradiction Search
    - Mode 2 Publication Tier B/C
    - Audit Sampling Started]
    
    Beta[Beta 0: Scaled AI Publication
    - Most Tier B AI-Published
    - All Tier C AI-Published
    - Mature Contradiction Detection
    - Audit Feedback Loop Active
    - Cross-Scenario Analysis]
    
    R1[Release 1.0: High Automation
    - Comprehensive AI Publication
    - Strategic Audits Only
    - Federated Contradiction Detection
    - Cross-Node Audit Sharing
    - Tier A Human-Reviewed Required]
    
    Future[Future: Distributed Intelligence
    - Advanced Bubble Detection
    - Global Contradiction Network
    - Minimal Human Review Tier B/C
    - Tier A Oversight Continues]
    
    POC --> Beta
    Beta --> R1
    R1 --> Future
    
    subgraph "Quality Assurance Evolution"
        AuditHigh[Recommendation: 30-50% Tier A Sampling] --> AuditMed[Strategic High-Value Sampling]
        AuditMed --> AuditLow[Anomaly-Triggered Sampling]
    end
    
    classDef current fill:#e3f2fd,stroke:#1976d2,stroke-width:3px
    class POC current

Manual vs Automated Matrix

Manual vs Automated matrix

graph TD
    Human[Always Human
- Final Verdict Approval
- Ethics & Governance
- Dispute Resolution
- Scenario Validity]
    
    Mixed[Mixed / AI-Assisted
- Ambiguous Definitions
- Boundary Choices
- Verdict Reasoning Text]
    
    AI[Mostly AI + Human Check
- Claim Normalization
- Clustering
- Metadata Extraction
- Contradiction Alerts]

    Human --- Mixed
    Mixed --- AI

Related Pages

• AKEL (AI Knowledge Extraction Layer)
• Requirements (Roles)
• Workflows
• Governance