Automation

How FactHarbor scales through automated claim evaluation.

1. Automation Philosophy

FactHarbor is automation-first: AKEL (AI Knowledge Extraction Layer) makes all content decisions. Humans monitor system performance and improve algorithms.
Why automation:

• Scale: Can process millions of claims
• Consistency: Same evaluation criteria applied uniformly
• Transparency: Algorithms are auditable
• Speed: Results in <20 seconds typically
  See Automation Philosophy for detailed principles.

2. Claim Processing Flow

2.1 User Submits Claim

• User provides claim text + source URLs
• System validates format
• Assigns processing ID
• Queues for AKEL processing

2.2 AKEL Processing

AKEL automatically:

1. Parses claim into testable components
   2. Extracts evidence from sources
   3. Scores source credibility
   4. Evaluates claim against evidence
   5. Generates verdict with confidence score
   6. Assigns risk tier (A/B/C)
   7. Publishes result
   Processing time: Typically <20 seconds
   No human approval required - publication is automatic

2.3 Publication States

Processing: AKEL working on claim (not visible to public)
Published: AKEL completed evaluation (public)

• Verdict displayed with confidence score
• Evidence and sources shown
• Risk tier indicated
• Users can report issues
  Flagged: AKEL identified issue requiring moderator attention (still public)
• Low confidence below threshold
• Detected manipulation attempt
• Unusual pattern
• Moderator reviews and may take action

2.5 LLM-Based Processing Architecture

FactHarbor delegates complex reasoning and analysis tasks to Large Language Models (LLMs). The architecture evolves from POC to production:

POC: Two-Phase Approach

Phase 1: Claim Extraction

• Single LLM call to extract all claims from submitted content
• Light structure, focused on identifying distinct verifiable claims
• Output: List of claims with context

Phase 2: Claim Analysis (Parallel)

• Single LLM call per claim (parallelizable)
• Full structured output: Evidence, Scenarios, Sources, Verdict, Risk
• Each claim analyzed independently

Advantages:

• Fast to implement ( to working POC)
• Only 2-3 API calls total (1 + N claims)
• Simple to debug (claim-level isolation)
• Proves concept viability

Production: Three-Phase Approach

Phase 1: Claim Extraction + Validation

• Extract distinct verifiable claims
• Validate claim clarity and uniqueness
• Remove duplicates and vague claims

Phase 2: Evidence Gathering (Parallel)

• For each claim independently:
• Find supporting and contradicting evidence
• Identify authoritative sources
• Generate test scenarios
• Validation: Check evidence quality and source validity
• Error containment: Issues in one claim don't affect others

Phase 3: Verdict Generation (Parallel)

• For each claim:
• Generate verdict based on validated evidence
• Assess confidence and risk level
• Flag low-confidence results for human review
• Validation: Check verdict consistency with evidence

Advantages:

• Error containment between phases
• Clear quality gates and validation
• Observable metrics per phase
• Scalable (parallel processing across claims)
• Adaptable (can optimize each phase independently)

LLM Task Delegation

All complex cognitive tasks are delegated to LLMs:

• Claim Extraction: Understanding context, identifying distinct claims
• Evidence Finding: Analyzing sources, assessing relevance
• Scenario Generation: Creating testable hypotheses
• Source Evaluation: Assessing reliability and authority
• Verdict Generation: Synthesizing evidence into conclusions
• Risk Assessment: Evaluating potential impact

Error Mitigation

Research shows sequential LLM calls face compound error risks. FactHarbor mitigates this through:

• Validation gates between phases
• Confidence thresholds for quality control
• Parallel processing to avoid error propagation across claims
• Human review queue for low-confidence verdicts
• Independent claim processing - errors in one claim don't cascade to others

3. Risk Tiers

Risk tiers classify claims by potential impact and guide audit sampling rates.

3.1 Tier A (High Risk)

Domains: Medical, legal, elections, safety, security
Characteristics:

• High potential for harm if incorrect
• Complex specialized knowledge required
• Often subject to regulation
  Publication: AKEL publishes automatically with prominent risk warning
  Audit rate: Higher sampling recommended

3.2 Tier B (Medium Risk)

Domains: Complex policy, science, causality claims
Characteristics:

• Moderate potential impact
• Requires careful evidence evaluation
• Multiple valid interpretations possible
  Publication: AKEL publishes automatically with standard risk label
  Audit rate: Moderate sampling recommended

3.3 Tier C (Low Risk)

Domains: Definitions, established facts, historical data
Characteristics:

• Low potential for harm
• Well-documented information
• Clear right/wrong answers typically
  Publication: AKEL publishes by default
  Audit rate: Lower sampling recommended

4. Quality Gates

AKEL applies quality gates before publication. If any fail, claim is flagged (not blocked - still published).
Quality gates:

• Sufficient evidence extracted (≥2 sources)
• Sources meet minimum credibility threshold
• Confidence score calculable
• No detected manipulation patterns
• Claim parseable into testable form
  Failed gates: Claim published with flag for moderator review

5. Automation Levels

Automation Level
This diagram shows the progression of automation levels from POC through Release 1.0 and beyond.

Automation Level Mermaid

graph TD
 subgraph "Automation Maturity Progression"
 POC["Level 0: POC/Demo
- Tier C only
- AKEL generates, publishes with disclaimers
- High sampling audit
- Proof of concept"]
 
 R05["Release 0.5: Limited Production
- Tier B/C auto-published
- Tier A flagged for moderator review
- Higher sampling initially
- Algorithm improvement focus"]
 
 R10["Release 1.0: Full Production
- All tiers auto-published
- Clear risk labels on all content
- Reduced sampling as confidence grows
- Mature algorithm performance"]
 
 R20["Release 2.0+: Distributed Intelligence
- Federated multi-node operation
- Cross-node audit sharing
- Advanced pattern detection
- Strategic sampling only"]
 
 POC --> R05
 R05 --> R10
 R10 --> R20
 end
 
 style POC fill:#e1f5ff
 style R05 fill:#d4edff
 style R10 fill:#c7e5ff
 style R20 fill:#baddff

Key Principles Across All Levels:
- AKEL makes all publication decisions
- No human approval gates at any level
- Humans monitor metrics and improve algorithms
- Sampling audits inform improvements, don't block publication
- Risk tiers guide audit priorities, not publication permissions

FactHarbor progresses through automation maturity levels:
Release 0.5 (Proof-of-Concept): Tier C only, human review required
Release 1.0 (Initial): Tier B/C auto-published, Tier A flagged for review
Release 2.0 (Mature): All tiers auto-published with risk labels, sampling audits
See Automation Roadmap for detailed progression.

5.5 Automation Roadmap

Automation Roadmap
This diagram shows the automation roadmap from POC through Release 1.0.

Automation Roadmap Mermaid

graph LR
 subgraph "Quality Assurance Evolution"
 QA1["Initial: High Sampling
Higher rates for Tier A
Moderate rates for Tier B
Lower rates for Tier C"]
 QA2["Intermediate: Strategic Sampling
Focus on high-value learning
Sample new domains more
Reduce routine sampling"]
 QA3["Mature: Anomaly-Triggered
Sample based on metrics
Investigate unusual patterns
Strategic domain sampling"]
 
 QA1 --> QA2
 QA2 --> QA3
 end
 
 subgraph "POC: Proof of Concept"
 POC["POC Features
- Tier C Only
- Basic AKEL Processing
- Simple Risk Classification
- High Audit Sampling"]
 end
 
 subgraph "Release 0.5: Limited Production"
 R05["R0.5 Features
- Tier A/B/C Published
- All auto-publication
- Risk Labels Active
- Contradiction Detection
- Sampling-Based QA"]
 end
 
 subgraph "Release 1.0: Full Production"
 R10["R1.0 Features
- Comprehensive AI Publication
- Strategic Audits Only
- Federated Nodes (Beta)
- Cross-Node Data Sharing
- Mature Algorithm Performance"]
 end
 
 subgraph "Future: Distributed Intelligence"
 Future["Future Features
- Advanced Pattern Detection
- Global Contradiction Network
- Minimal Human QA (Anomalies Only)
- Full Federation"]
 end
 
 POC --> R05
 R05 --> R10
 R10 --> Future
 
 style POC fill:#e1f5ff
 style R05 fill:#d4edff
 style R10 fill:#c7e5ff
 style Future fill:#baddff

Automation Philosophy: At all stages, AKEL publishes automatically. Humans improve algorithms, not review content.

Sampling Rates: Start higher for learning, reduce as confidence grows. Rates are recommendations, not commitments.

6. Human Role

Humans do NOT review content for approval. Instead:
Monitoring: Watch aggregate performance metrics
Improvement: Fix algorithms when patterns show issues
Exception handling: Review AKEL-flagged items
Governance: Set policies AKEL applies
See Contributor Processes for how to improve the system.

6.5 Manual vs Automated Matrix

Manual vs Automated matrix

Manual vs Automated matrix Mermaid

graph TD
 subgraph "Automated by AKEL"
 A1["Claim Evaluation
- Evidence extraction
- Source scoring
- Verdict generation
- Risk classification
- Publication"]
 A2["Quality Assessment
- Contradiction detection
- Confidence scoring
- Pattern recognition
- Anomaly flagging"]
 A3["Content Management
- Scenario generation
- Evidence linking
- Source tracking
- Version control"]
 end
 subgraph "Human Responsibilities"
 H1["Algorithm Improvement
- Monitor performance metrics
- Identify systematic issues
- Propose fixes
- Test improvements
- Deploy updates"]
 H2["Policy Governance
- Set evaluation criteria
- Define risk tiers
- Establish thresholds
- Update guidelines"]
 H3["Exception Handling
- Review AKEL-flagged items
- Handle abuse/manipulation
- Address safety concerns
- Manage legal issues"]
 H4["Strategic Decisions
- Budget and resources
- Hiring and roles
- Major policy changes
- Partnership agreements"]
 end
 style A1 fill:#c7e5ff
 style A2 fill:#c7e5ff
 style A3 fill:#c7e5ff
 style H1 fill:#ffe5cc
 style H2 fill:#ffe5cc
 style H3 fill:#ffe5cc
 style H4 fill:#ffe5cc

Key Principle: AKEL handles all content decisions. Humans improve the system, not the data.
Never Manual:
- Individual claim approval
- Routine content review
- Verdict overrides (fix algorithm instead)
- Publication gates

7. Moderation

Moderators handle items AKEL flags:
Abuse detection: Spam, manipulation, harassment
Safety issues: Content that could cause immediate harm
System gaming: Attempts to manipulate scoring
Action: May temporarily hide content, ban users, or propose algorithm improvements
Does NOT: Routinely review claims or override verdicts
See Organisational Model for moderator role details.

8. Continuous Improvement

Performance monitoring: Track AKEL accuracy, speed, coverage
Issue identification: Find systematic errors from metrics
Algorithm updates: Deploy improvements to fix patterns
A/B testing: Validate changes before full rollout
Retrospectives: Learn from failures systematically
See Continuous Improvement for improvement cycle.

9. Scalability

Automation enables FactHarbor to scale:

• Millions of claims processable
• Consistent quality at any volume
• Cost efficiency through automation
• Rapid iteration on algorithms
  Without automation: Human review doesn't scale, creates bottlenecks, introduces inconsistency.

10. Transparency

All automation is transparent:

• Algorithm parameters documented
• Evaluation criteria public
• Source scoring rules explicit
• Confidence calculations explained
• Performance metrics visible
  See System Performance Metrics for what we measure.