Changes for page Federation & Decentralization

Last modified by Robert Schaub on 2025/12/24 20:33

From 1.2 to 1.1 From 7.4 to 7.3

From version 7.3

edited by Robert Schaub
on 2025/12/16 20:27

Change comment: Update document after refactoring.

To version 1.2

edited by Robert Schaub
on 2025/12/11 21:34

Change comment: Renamed back-links.

Raw
Rendered

Summary

Page properties (2 modified, 0 added, 0 removed)

Details

Page properties

Parent

@@ -1,1 +1,1 @@
--FactHarbor.Archive.FactHarbor V0\.9\.18.Specification.WebHome
++FactHarbor.Specification.WebHome

Content

@@ -1,431 +1,223 @@
  = Federation & Decentralization =
--FactHarbor is designed to operate as a **federated network of nodes** rather than a single central server.
++FactHarbor is designed as a network of independent nodes rather than a single centralized service.
++This provides resilience, local autonomy, and virtually unlimited scalability.
--Decentralization provides:
--* **Resilience** against censorship or political pressure
--* **Autonomy** for local governance and moderation
--* **Scalability** across many independent communities
--* **Trust** without centralized control
--* **Domain specialization** (health-focused nodes, energy-focused nodes, etc.)
++Each node remains fully functional on its own while participating in a shared global knowledge graph.
--FactHarbor draws inspiration from the Fediverse but uses stronger structure, versioning, and integrity guarantees.
--
  ----
--== Federated FactHarbor Nodes ==
++= Purpose of Federation =
--Each FactHarbor instance ("node") maintains:
--* Its own database
--* Its own AKEL instance
--* Its own reviewers, experts, and contributors
--* Its own governance rules
++A federated architecture enables:
--Nodes exchange structured information:
--* Claims
--* Scenarios
--* Evidence metadata (not necessarily full files)
--* Verdicts (optional)
--* Hashes and signatures for integrity
++* Resilience against censorship or political pressure
++* Local governance and moderation autonomy
++* Scalability by adding more nodes, not bigger servers
++* Shared knowledge structures without central authority
++* Domain specialization (e.g., health-focused node, energy-focused node)
++* Cross-node collaboration while preserving independence
--Nodes choose which external nodes they trust.
++FactHarbor takes inspiration from the Fediverse but uses stronger structure, integrity guarantees, and version lineage.
  ----
--== Global Identifiers ==
++= Federated Node Model =
--Every entity receives a globally unique, linkable identifier.
++Each FactHarbor node maintains:
--**Format**:
--`factharbor://node_url/type/local_id`
++* Its own PostgreSQL database
++* Its own vector database
++* Its own object storage
++* Its own AKEL instance
++* Its own reviewers, experts, and moderators
++* Its own trust and governance policies
--**Example**:
--`factharbor://factharbor.energy/claim/CLM-55812`
++Nodes exchange structured objects:
--**Supported types**:
--* `claim`
--* `scenario`
--* `evidence`
--* `verdict`
--* `user` (optional)
--* `cluster`
++* Claims
++* Scenarios
++* Evidence metadata (not raw files unless elected)
++* Verdicts (optional)
++* Integrity hashes and signatures
--**Properties**:
--* Globally consistent
--* Human-readable
--* Hash-derived
--* Independent of database internals
--* URL-resolvable (future enhancement)
++Nodes decide individually which external nodes to trust.
--This allows cross-node references and prevents identifier collisions in federated environments.
--
  ----
--== Trust Model ==
++= Global Identifiers =
--Each node maintains a **trust table** defining relationships with other nodes:
++Each entity receives a globally unique ID.
--=== Trust Levels ===
++Format:
++``factharbor://<node>/<type>/<local_id>``
--**Trusted Nodes**:
--* Claims auto-imported
--* Scenarios accepted without re-review
--* Evidence considered valid
--* Verdicts displayed to users
--* High synchronization priority
++Example:
++``factharbor://factharbor.energy/claim/CLM-55812``
--**Neutral Nodes**:
--* Claims imported but flagged for review
--* Scenarios require local validation
--* Evidence requires re-assessment
--* Verdicts shown with "external node" disclaimer
--* Normal synchronization priority
++Types include:
--**Untrusted Nodes**:
--* Claims quarantined, manual import only
--* Scenarios rejected by default
--* Evidence not accepted
--* Verdicts not displayed
--* No automatic synchronization
++* claim
++* scenario
++* evidence
++* verdict
++* user (optional)
++* cluster
--=== Trust Affects ===
++Identifiers are:
--* **Auto-import**: Whether claims/scenarios are automatically added
--* **Re-review requirements**: Whether local reviewers must validate
--* **Verdict display**: Whether external verdicts are shown to users
--* **Synchronization frequency**: How often data is exchanged
--* **Reputation signals**: How external reputation is interpreted
++* Globally consistent
++* Human-readable
++* Hash-derived
++* Independent from internal database IDs
--=== Local Trust Authority ===
--
--Each node's governance team decides:
--* Which nodes to trust
--* Trust level criteria
--* Trust escalation/degradation rules
--* Dispute resolution with partner nodes
--
--Trust is **local and autonomous** - no global trust registry exists.
--
  ----
--== Data Sharing Model ==
++= Data Sharing Model =
--=== What Nodes Share ===
++Nodes share:
--**Always shared** (if federation enabled):
--* Claims and claim clusters
--* Scenario structures
--* Evidence metadata and content hashes
--* Integrity signatures
++* Claims
++* Scenario structures
++* Evidence metadata + content hashes
++* Optional verdicts
++* Integrity metadata
--**Optionally shared**:
--* Full evidence files (large documents)
--* Verdicts (nodes may choose to keep verdicts local)
--* Vector embeddings
--* Scenario templates
--* AKEL distilled knowledge
++Nodes **do not** share:
--**Never shared**:
--* Internal user lists
--* Reviewer comments and internal discussions
--* Governance decisions and meeting notes
--* Access control data
--* Private or sensitive content marked as local-only
++* Local users
++* Review discussions
++* Internal moderation notes
++* Private evidence
--=== Large Evidence Files ===
++Large assets may use:
--Evidence files are:
--* Stored locally by default
--* Referenced via global content hash
--* Optionally served through IPFS
--* Accessible via direct peer-to-peer transfer
--* Can be stored in S3-compatible object storage
++* Local object storage
++* S3-compatible buckets
++* IPFS for cross-node replication (optional)
  ----
--== Synchronization Protocol ==
++= Trust Model =
--Nodes exchange data using multiple synchronization methods:
++Each node maintains a **trust table**:
--=== Push-Based Synchronization ===
++* Trusted nodes
++* Neutral nodes
++* Untrusted nodes
--**Mechanism**: Webhooks
++Trust influences:
--When local content changes:
--1. Node builds signed bundle
--2. Sends webhook notification to subscribed nodes
--3. Remote nodes fetch bundle
--4. Remote nodes validate and import
++* Whether claims are auto-imported
++* Whether scenarios are accepted without re-review
++* Whether evidence requires new validation
++* Whether external verdicts are visible to users
--**Use case**: Real-time updates for trusted partners
++Reputation is local but can be mapped with trust-weighting.
--=== Pull-Based Synchronization ===
++----
--**Mechanism**: Scheduled polling
++= Decentralized Processing =
--Nodes periodically:
--1. Query partner nodes for updates
--2. Fetch changed entities since last sync
--3. Validate and import
--4. Store sync checkpoint
++Each node independently performs:
--**Use case**: Regular batch updates, lower trust nodes
++* AKEL processing
++* Scenario drafting
++* Evidence review
++* Verdict calculation
++* Summary generation
++* Re-evaluation
--=== Subscription-Based Synchronization ===
++Nodes may specialize:
--**Mechanism**: WebSub-like protocol
++* medical node
++* psychology node
++* climate node
++* small node delegating expert review to trusted partners
--Nodes subscribe to:
--* Specific claim clusters
--* Specific domains (medical, energy, etc.)
--* Specific scenario types
--* Verdict updates
++Optional cross-node data sharing includes:
--Publisher pushes updates only to subscribers.
++* Embeddings
++* Claim clusters
++* Scenario templates
++* Verdict comparison metadata
++* Contradiction alerts
--**Use case**: Selective federation, domain specialization
--
--=== Large Asset Transfer ===
--
--For files >10MB:
--* S3-compatible object storage
--* IPFS (content-addressed)
--* Direct peer-to-peer transfer
--* Chunked HTTP transfer with resume support
--
  ----
--== Federation Sync Workflow ==
++= Cross-Node AI Knowledge Exchange =
--Complete synchronization sequence for creating and sharing new content:
++Nodes may exchange:
--=== Step 1: Local Node Creates New Versions ===
++* Embeddings for clustering
++* Canonical claim forms
++* Scenario templates
++* Reliability hints
++* Contradiction alerts
++* Lightweight model insights (NOT weights)
--User or AKEL creates:
--* New claim version
--* New scenario version
--* New evidence version
--* New verdict version
++AKEL **never**:
--All changes tracked with:
--* VersionID
--* ParentVersionID
--* AuthorType
--* Timestamp
--* JustificationText
++* Shares model weights
++* Automatically replaces local reviewer decisions
++* Accepts untrusted automated content
--=== Step 2: Federation Layer Builds Signed Bundle ===
--
--Federation layer packages:
--* Entity data (claim, scenario, evidence metadata, verdict)
--* Version lineage (ParentVersionID chain)
--* Cryptographic signatures
--* Node provenance information
--* Trust metadata
--
--Bundle format:
--* JSON-LD for structured data
--* Content-addressed hashes
--* Digital signatures for integrity
--
--=== Step 3: Bundle Includes Required Data ===
--
--Each bundle contains:
--* **Claims**: Full claim text, classification, domain
--* **Scenarios**: Definitions, assumptions, boundaries
--* **Evidence metadata**: Source URLs, hashes, reliability scores (not always full files)
--* **Verdicts**: Likelihood ranges, uncertainty, reasoning chains
--* **Lineage**: Version history, parent relationships
--* **Signatures**: Cryptographic proof of origin
--
--=== Step 4: Bundle Pushed to Trusted Neighbor Nodes ===
--
--Based on trust table:
--* Push to **trusted nodes** immediately
--* Queue for **neutral nodes** (batched)
--* Skip **untrusted nodes**
--
--Push methods:
--* Webhook notification
--* Direct API call
--* Pub/Sub message queue
--
--=== Step 5: Remote Nodes Validate Lineage and Signatures ===
--
--Receiving node:
--1. Verifies cryptographic signatures
--2. Validates version lineage (ParentVersionID chain)
--3. Checks for conflicts with local data
--4. Validates data structure and required fields
--5. Applies local trust policies
--
--Validation failures → reject or quarantine bundle
--
--=== Step 6: Accept or Branch Versions ===
--
--**Accept** (if validation passes):
--* Import new versions
--* Maintain provenance metadata
--* Link to local related entities
--* Update local indices
--
--**Branch** (if conflict detected):
--* Create parallel version tree
--* Mark as "external branch"
--* Allow local reviewers to merge or reject
--* Preserve both version histories
--
--**Reject** (if validation fails):
--* Log rejection reason
--* Notify source node (optional)
--* Quarantine for manual review (optional)
--
--=== Step 7: Local Re-evaluation Runs if Required ===
--
--After import, local node checks:
--* Does new evidence affect existing verdicts?
--* Do new scenarios require re-assessment?
--* Are there contradictions with local content?
--
--If yes:
--* Trigger AKEL re-evaluation
--* Queue for reviewer attention
--* Update affected verdicts
--* Notify users following related content
--
  ----
--== Cross-Node AI Knowledge Exchange ==
++= Synchronization Protocol =
--Each node runs its own AKEL instance and may exchange AI-derived knowledge:
++Nodes periodically exchange version bundles:
--=== What Can Be Shared ===
++* Claims
++* Scenarios
++* Evidence metadata + hashes
++* Optional verdicts
++* Templates
++* Embeddings (optional)
++* AKEL distilled knowledge summaries (optional)
--**Vector embeddings**:
--* For cross-node claim clustering
--* For semantic search alignment
--* Never includes training data
++Sync methods:
--**Canonical claim forms**:
--* Normalized claim text
--* Standard phrasing templates
--* Domain-specific formulations
++* Push (webhook)
++* Pull (cron)
++* Subscription (WebSub-like)
--**Scenario templates**:
--* Reusable scenario structures
--* Common assumption patterns
--* Evaluation method templates
++Large assets may be transferred via:
--**Contradiction alerts**:
--* Detected conflicts between claims
--* Evidence conflicts across nodes
--* Scenario incompatibilities
++* S3-compatible file transfer
++* IPFS
++* Peer-to-peer
--**Metadata and insights**:
--* Aggregate quality metrics
--* Reliability signal extraction
--* Bubble detection patterns
--
--=== What Can NEVER Be Shared ===
--
--**Model weights**: No sharing of trained model parameters
--
--**Training data**: No sharing of full training datasets
--
--**Local governance overrides**: AKEL suggestions can be overridden locally
--
--**User behavior data**: No cross-node tracking
--
--**Internal review discussions**: Private content stays private
--
--=== Benefits of AI Knowledge Exchange ===
--
--* Reduced duplication across nodes
--* Improved claim clustering accuracy
--* Faster contradiction detection
--* Shared scenario libraries
--* Cross-node quality improvements
--
--=== Local Control Maintained ===
--
--* Nodes accept or reject shared AI knowledge
--* Human reviewers can override any AKEL suggestion
--* Local governance always has final authority
--* No external AI control over local content
--* Privacy-preserving knowledge exchange
--
  ----
--== Decentralized Processing ==
++= Scaling to Thousands of Users =
--Each node independently performs:
--* AKEL processing
--* Scenario drafting and validation
--* Evidence review
--* Verdict calculation
--* Truth landscape summarization
++Nodes scale independently:
--Nodes can specialize:
--* Health-focused node with medical experts
--* Energy-focused node with domain knowledge
--* Small node delegating scenario libraries to partners
--* Regional node with language/culture specialization
++* horizontally scaled API servers
++* background worker pools
++* GPU queues for AKEL
++* caching (Redis)
++* sharded databases
--Optional data sharing includes:
--* Embeddings for clustering
--* Claim clusters for alignment
--* Scenario templates for efficiency
--* Verdict comparison metadata
++The network scales horizontally by adding more nodes.
------
++Communities can form:
--== Scaling to Thousands of Users ==
++* Domain-specific nodes
++* Region/language nodes
++* NGO/academic nodes
++* Private organization nodes
--Nodes scale independently through:
--* Horizontally scalable API servers
--* Worker pools for AKEL tasks
--* Hybrid storage (local + S3/IPFS)
--* Redis caching for performance
--* Sharded or partitioned databases
--
--Federation allows effectively unlimited horizontal scaling by adding new nodes.
--
--Communities may form:
--* Domain-specific nodes (epidemiology, energy, climate)
--* Language or region-based nodes
--* NGO or institutional nodes
--* Private organizational nodes
--* Academic research nodes
--
  Nodes cooperate through:
--* Scenario library sharing
--* Shared or overlapping claim clusters
--* Expert delegation between nodes
--* Distributed AKEL task support
--* Cross-node quality audits
------
++* Shared scenario libraries
++* Overlapping claim clusters
++* Expert delegation
++* Distributed AKEL tasks
--== Federation and Release 1.0 ==
--
--**POC**: Single node, optional federation experiments
--
--**Beta 0**: 2-3 nodes, basic federation protocol
--
--**Release 1.0**: Full federation support with:
--* Robust synchronization protocol
--* Trust model implementation
--* Cross-node AI knowledge exchange
--* Federated search and discovery
--* Distributed audit collaboration
--* Inter-node expert consultation
--
  ----
--== Related Pages ==
++= Diagram References =
--* [[AKEL (AI Knowledge Extraction Layer)>>FactHarbor.Specification.AI Knowledge Extraction Layer (AKEL).WebHome]]
--* [[Data Model>>FactHarbor.Specification.Data Model.WebHome]]
--* [[Architecture>>FactHarbor.Specification.Architecture.WebHome]]
--* [[Workflows>>FactHarbor.Specification.Workflows.WebHome]]
--
++{{include reference="FactHarbor.Archive.Diagrams v0\.8q.Federation Architecture.WebHome"/}}

Changes for page Federation & Decentralization

Summary

Details

Applications

Navigation

Need help?