Changes for page Data Model (From Specification Chat)

Last modified by Robert Schaub on 2025/12/24 20:35

From 1.1 to 1.2 From 4.1 to 5.1

From version 1.2

edited by Robert Schaub
on 2025/11/27 12:03

Change comment: Update document after refactoring.

To version 4.1

edited by Robert Schaub
on 2025/11/27 12:11

Change comment: There is no comment for this version

Raw
Rendered

Summary

Page properties (1 modified, 0 added, 0 removed)

Details

Page properties

Content

@@ -1,3 +1,171 @@
++== 1. Overall analysis & review of the data model ==
++
++=== 1.1 Strengths of the current design ===
++
++* (((
++**Identity vs. version pattern**
++Using base entities plus version entities (CLAIM + CLAIM_VERSION, SCENARIO + SCENARIO_VERSION, etc.) is exactly how modern knowledge systems handle:
++
++* auditability
++* time evolution
++* re-evaluation triggers
++* federation and partial replication
++)))
++* (((
++**Scenario-centric reasoning**
++Separating //Claim// (what people argue about) from //Scenario// (interpretive frame) is very aligned with “truth landscape” style systems:
++
++* Scenarios explain //why people disagree//.
++* Verdicts are tied to specific scenario versions → avoids mixing incompatible assumptions.
++)))
++* **Evidence and verdicts as first-class entities**
++Evidence is explicit, linked to scenarios, and verdicts are per scenario. This matches good practice from fact-checking, scientific assessment panels, and trust graphs.
++* (((
++**Cluster level (CLAIM_CLUSTER)**
++Grouping related claims avoids duplication and lets you:
++
++* reuse scenarios across paraphrases
++* share embeddings / semantic search
++* keep the system scalable as the corpus grows.
++)))
++* (((
++**Explicit review layer (REVIEW_ACTION, roles, etc.)**
++Separating “data” from “who reviewed what” keeps the model clean, and is exactly what you want for:
++
++* governance
++* permissions
++* audit trails
++* future trust scoring per user / role.
++)))
++
++----
++
++=== 1.2 Design decisions I’m locking in (based on our discussions) ===
++
++To make the model consistent and “state-of-the-art”, I will assume the following as //current intended design//:
++
++1. (((
++**Claims vs Scenarios**
++
++* CLAIM is the stable identity for “what people argue about”.
++* CLAIM_VERSION are individual phrasings / formulations / metadata.
++* (((
++SCENARIO belongs to a **CLAIM**, not to a specific CLAIM_VERSION.
++Rationale:
++
++* Many different phrasings share the //same// scenario.
++* You avoid duplicating scenarios per wording.
++)))
++* SCENARIO_VERSION holds detailed definitions, assumptions, boundaries, etc.
++)))
++1. (((
++**Version-specific reasoning**
++
++* **Verdicts** are always attached to SCENARIO_VERSION (not base SCENARIO).
++* **Evidence links** are between SCENARIO_VERSION and EVIDENCE_VERSION.
++→ This is what we agreed when we said //“SCENARIO_EVIDENCE_LINK should link the respective versions instead”//.
++)))
++1. (((
++**Clusters**
++
++* CLAIM_CLUSTER groups Claims (semantically close claims).
++* It is visible in **both diagrams** (Core Data Model and Data Use).
++)))
++1. (((
++**Review vs data**
++
++* (((
++All review happens **on versioned entities**:
++
++* CLAIM_VERSION
++* SCENARIO_VERSION
++* EVIDENCE_VERSION
++* SCENARIO_EVIDENCE_LINK_VERSION
++* VERDICT_VERSION
++)))
++* REVIEW_ACTION is the generic log of //who// did //what// on //which version//.
++)))
++1. (((
++**Users & roles**
++
++* USER has an attribute (or a linked entity) that distinguishes **technical users** from normal accounts.
++* We //keep// TECHNICAL_USER as a specialisation of USER (strictly technical accounts).
++* All human & technical accounts can hold roles via USER_ROLE_MEMBERSHIP.
++* (((
++Roles include:
++
++* READER
++* CONTRIBUTOR
++* TRUSTED_CONTRIBUTOR
++* REVIEWER
++* MODERATOR
++* SYSTEM_ADMIN / MAINTAINER
++* FEDERATION_OPERATOR
++* FEDERATION_ADMIN
++(all present in the Data Use ERD, but as rows of ROLE rather than separate entities).
++)))
++)))
++
++----
++
++=== 1.3 Gaps / potential problems ===
++
++These are the main issues & missing areas I see:
++
++1. (((
++**Versioning text in chapter 5 is currently too thin (‘…’ placeholders)**
++
++* (((
++The spec does not yet //verbally// spell out:
++
++* the identity vs version pattern, systematically
++* how re-evaluation triggers are derived from version changes
++* how this aligns with federation (which versions are replicated where).
++)))
++)))
++1. (((
++**No explicit “provenance granularity” in the model**
++
++* (((
++EVIDENCE is a single entity. For more advanced use cases, you may later want:
++
++* EVIDENCE_SOURCE (the whole article/report/video)
++* EVIDENCE_FRAGMENT (specific paragraph/clip with its own reliability, quote, etc.)
++)))
++* For now, I’ll keep EVIDENCE/EVIDENCE_VERSION as is, but I’ll mention this as a possible extension.
++)))
++1. (((
++**Review target polymorphism**
++
++* (((
++REVIEW_ACTION can apply to multiple entity types. In the diagram this shows as multiple relationships:
++
++* CLAIM_VERSION → REVIEW_ACTION
++* SCENARIO_VERSION → REVIEW_ACTION
++* etc.
++)))
++* A more “pure” relational modeling would use a generic “subjectType + subjectId” or an intermediate “REVIEW_TARGET” table.
++* For readability, I’ll keep the simpler multi-edge representation and mention the polymorphism in text.
++)))
++1. (((
++**Federation details missing from core ERD**
++
++* There is no explicit FEDERATION_NODE / REPLICATION_LOG in the Data Model chapter.
++* This is ok for “core logical data model”, but I’ll add a short note that federation metadata is handled in the Federation chapter and via additional entities.
++)))
++1. (((
++**Automation / AKEL artifacts left implicit**
++
++* (((
++The Data Model chapter currently doesn’t describe:
++
++* AKEL task queues
++* extraction runs
++* model versions
++)))
++* That’s fine for now; I’ll just clarify that those belong to a “Processing / AKEL” submodel, not the core logical data model.
++)))
++
  = 5. Data Model =
  The FactHarbor data model centers on four fully versioned, immutable entities:
@@ -41,10 +41,10 @@
    (scenarios live at the *claim* level, not per individual phrasing).
  * Verdicts and Scenario–Evidence links are always attached to **versions**:
  * {{code}}SCENARIO_VERSION{{/code}} +
--   {{code}}EVIDENCE_VERSION{{/code}} →
--   {{code}}SCENARIO_EVIDENCE_LINK_VERSION{{/code}}
++{{code}}EVIDENCE_VERSION{{/code}} →
++{{code}}SCENARIO_EVIDENCE_LINK_VERSION{{/code}}
  * {{code}}SCENARIO_VERSION{{/code}} →
--   {{code}}VERDICT_VERSION{{/code}}
++{{code}}VERDICT_VERSION{{/code}}
  This ensures that when a Scenario or Evidence changes, old verdicts and links
  remain intact as historical records and can be revisited.
@@ -59,124 +59,95 @@
  {{mermaid}}
  erDiagram
--
      CLAIM_CLUSTER {
--        string claimClusterId
--        string theme
--        string embeddingVectorRef
--        string language
--        datetime createdAt
++        string ClusterID PK
++        string EmbeddingVectorRef
++        string Theme
      }
      CLAIM {
--        string claimId
--        string claimClusterIdFk
--        string status
--        datetime createdAt
++        string ClaimID PK
++        string ClusterID FK
++        string Status
++        datetime CreatedAt
      }
      CLAIM_VERSION {
--        string claimVersionId
--        string claimIdFk
--        string text
--        string language
--        string claimType
--        string domain
--        string authorType
--        datetime createdAt
++        string ClaimVersionID PK
++        string ClaimID FK
++        string Text
++        string ClaimType
++        string Domain
++        datetime CreatedAt
      }
      SCENARIO {
--        string scenarioId
--        string claimIdFk
--        string key
--        string title
--        boolean isDeprecated
++        string ScenarioID PK
++        string ClaimID FK
++        string Name
++        datetime CreatedAt
      }
      SCENARIO_VERSION {
--        string scenarioVersionId
--        string scenarioIdFk
--        string versionTag
--        string definitionsJson
--        string assumptionsJson
--        string boundariesJson
--        string notes
--        datetime createdAt
++        string ScenarioVersionID PK
++        string ScenarioID FK
++        string Definitions
++        string Assumptions
++        string Boundaries
++        datetime CreatedAt
      }
      EVIDENCE {
--        string evidenceId
--        string canonicalSourceId
--        string mainUrl
--        string evidenceType
--        string language
++        string EvidenceID PK
++        string SourceType
++        string URL
++        float ReliabilityScore
      }
      EVIDENCE_VERSION {
--        string evidenceVersionId
--        string evidenceIdFk
--        string snapshotLocation
--        string extractionSummary
--        string reliabilityModel
--        datetime collectedAt
--        datetime createdAt
++        string EvidenceVersionID PK
++        string EvidenceID FK
++        string Summary
++        float ReliabilityScore
++        datetime CreatedAt
      }
      SCENARIO_EVIDENCE_LINK {
--        string scenarioEvidenceLinkId
--        string scenarioIdFk
--        string evidenceIdFk
++        string LinkID PK
++        string ScenarioVersionID FK
++        string EvidenceVersionID FK
++        float Relevance
++        string Direction
      }
--    SCENARIO_EVIDENCE_LINK_VERSION {
--        string scenarioEvidenceLinkVersionId
--        string scenarioEvidenceLinkIdFk
--        string scenarioVersionIdFk
--        string evidenceVersionIdFk
--        float relevance
--        string direction   %% SUPPORTS / CONTRADICTS / MIXED / CONTEXT
--        string rationale
--        datetime createdAt
--    }
--
      VERDICT {
--        string verdictId
--        string scenarioIdFk
--        string verdictType    %% e.g. likelihood, classification
++        string VerdictID PK
++        string ScenarioID FK
      }
      VERDICT_VERSION {
--        string verdictVersionId
--        string verdictIdFk
--        string scenarioVersionIdFk
--        float probability
--        float confidence
--        string reasoningSummary
--        string uncertaintyFactorsJson
--        datetime createdAt
++        string VerdictVersionID PK
++        string VerdictID FK
++        float Verdict
++        float Confidence
++        string Reasoning
++        datetime CreatedAt
      }
--    %% Relationships
--
      CLAIM_CLUSTER ||--o{ CLAIM : contains
--    CLAIM ||--o{ CLAIM_VERSION : has_versions
--    CLAIM ||--o{ SCENARIO : has_scenarios
--    SCENARIO ||--o{ SCENARIO_VERSION : has_versions
++    CLAIM ||--o{ CLAIM_VERSION : versions
--    EVIDENCE ||--o{ EVIDENCE_VERSION : has_versions
++    CLAIM ||--o{ SCENARIO : has
++    SCENARIO ||--o{ SCENARIO_VERSION : versions
--    SCENARIO ||--o{ SCENARIO_EVIDENCE_LINK : may_link
--    EVIDENCE ||--o{ SCENARIO_EVIDENCE_LINK : may_link
++    EVIDENCE ||--o{ EVIDENCE_VERSION : versions
--    SCENARIO_EVIDENCE_LINK ||--o{ SCENARIO_EVIDENCE_LINK_VERSION : has_versions
++    SCENARIO_VERSION ||--o{ SCENARIO_EVIDENCE_LINK : links
++    EVIDENCE_VERSION ||--o{ SCENARIO_EVIDENCE_LINK : linked
--    SCENARIO_VERSION ||--o{ SCENARIO_EVIDENCE_LINK_VERSION : uses_evidence
--    EVIDENCE_VERSION ||--o{ SCENARIO_EVIDENCE_LINK_VERSION : is_used_in
++    SCENARIO ||--o{ VERDICT : assessed
++    VERDICT ||--o{ VERDICT_VERSION : versions
--    SCENARIO ||--o{ VERDICT : has_verdicts
--    VERDICT ||--o{ VERDICT_VERSION : has_versions
--    SCENARIO_VERSION ||--o{ VERDICT_VERSION : assessed_in
  {{/mermaid}}
  **Important points:**
@@ -201,83 +201,145 @@
  {{mermaid}}
  erDiagram
++    %% Core clusters shown for context
++    CLAIM_CLUSTER {
++        string ClusterID PK
++        string EmbeddingVectorRef
++        string Theme
++    }
--    USER {
--        string userId
--        string displayName
--        string email
--        string userType      %% "human" or "technical"
--        datetime createdAt
++    CLAIM {
++        string ClaimID PK
++        string ClusterID FK
++        string Status
++        datetime CreatedAt
      }
--    TECHNICAL_USER {
--        string technicalUserId
--        string userIdFk
--        string description
--        string systemIdentifier
++    CLAIM_VERSION {
++        string ClaimVersionID PK
++        string ClaimID FK
++        string Text
++        string ClaimType
++        string Domain
++        datetime CreatedAt
      }
--    ROLE {
--        string roleId
--        string code          %% e.g. READER, CONTRIBUTOR, REVIEWER, TRUSTED_CONTRIBUTOR, MODERATOR, SYSTEM_ADMIN, FEDERATION_OPERATOR, FEDERATION_ADMIN
--        string description
--        boolean isFederationRole
++    SCENARIO {
++        string ScenarioID PK
++        string ClaimID FK
++        string Name
++        datetime CreatedAt
      }
--    USER_ROLE_MEMBERSHIP {
--        string membershipId
--        string userIdFk
--        string roleIdFk
--        datetime grantedAt
--        string grantedByUserIdFk
++    SCENARIO_VERSION {
++        string ScenarioVersionID PK
++        string ScenarioID FK
++        string Definitions
++        string Assumptions
++        string Boundaries
++        datetime CreatedAt
      }
--    REVIEW_ACTION {
--        string reviewActionId
--        string subjectType         %% e.g. CLAIM_VERSION, SCENARIO_VERSION...
--        string subjectVersionId
--        string actionType          %% APPROVE, REJECT, FLAG, COMMENT, REQUEST_CHANGES...
--        string outcome             %% ACCEPTED, REJECTED, ESCALATED...
--        string comment
--        string createdByUserIdFk
--        datetime createdAt
++    EVIDENCE {
++        string EvidenceID PK
++        string SourceType
++        string URL
++        float ReliabilityScore
      }
--    %% Versioned data entities (references from the core model)
++    EVIDENCE_VERSION {
++        string EvidenceVersionID PK
++        string EvidenceID FK
++        string Summary
++        float ReliabilityScore
++        datetime CreatedAt
++    }
--    CLAIM_VERSION {
--        string claimVersionId
++    VERDICT {
++        string VerdictID PK
++        string ScenarioID FK
      }
--    SCENARIO_VERSION {
--        string scenarioVersionId
++    VERDICT_VERSION {
++        string VerdictVersionID PK
++        string VerdictID FK
++        float Verdict
++        float Confidence
++        string Reasoning
++        datetime CreatedAt
      }
--    EVIDENCE_VERSION {
--        string evidenceVersionId
++    %% Users and roles
++    USER {
++        string UserID PK
++        string Handle
++        string Email
      }
--    SCENARIO_EVIDENCE_LINK_VERSION {
--        string scenarioEvidenceLinkVersionId
++    TECHNICAL_USER {
++        string UserID PK
++        string SystemName
      }
--    VERDICT_VERSION {
--        string verdictVersionId
++    CONTRIBUTING_USER {
++        string UserID PK
++        string DisplayName
      }
--    %% Relationships
++    TRUSTED_CONTRIBUTOR {
++        string UserID PK
++        string TrustLevel
++    }
--    USER ||--o{ TECHNICAL_USER : may_be
--    USER ||--o{ USER_ROLE_MEMBERSHIP : has_role
--    ROLE ||--o{ USER_ROLE_MEMBERSHIP : assigned_to
++    REVIEWER {
++        string UserID PK
++        string Domain
++    }
++    EXPERT {
++        string UserID PK
++        string ExpertiseArea
++    }
++
++    FEDERATION_NODE {
++        string NodeID PK
++        string Region
++    }
++
++    FEDERATION_ADMIN {
++        string UserID PK
++        string Permissions
++    }
++
++    REVIEW_ACTION {
++        string ReviewActionID PK
++        string UserID FK
++        string TargetEntityType
++        string TargetEntityVersionID
++        string ActionType
++        string Comment
++        datetime Timestamp
++    }
++
++    %% Inheritance / specialization (modelled as relationships)
++    USER ||--o{ TECHNICAL_USER : "is a"
++    USER ||--o{ CONTRIBUTING_USER : "is a"
++
++    CONTRIBUTING_USER ||--o{ TRUSTED_CONTRIBUTOR : "subset"
++    CONTRIBUTING_USER ||--o{ REVIEWER : "subset"
++    CONTRIBUTING_USER ||--o{ EXPERT : "subset"
++
++    TECHNICAL_USER ||--o{ FEDERATION_NODE : "operates"
++    TECHNICAL_USER ||--o{ FEDERATION_ADMIN : "administers"
++
++    %% Review actions on versioned entities
      USER ||--o{ REVIEW_ACTION : performs
--    CLAIM_VERSION ||--o{ REVIEW_ACTION : is_reviewed_in
--    SCENARIO_VERSION ||--o{ REVIEW_ACTION : is_reviewed_in
--    EVIDENCE_VERSION ||--o{ REVIEW_ACTION : is_reviewed_in
--    SCENARIO_EVIDENCE_LINK_VERSION ||--o{ REVIEW_ACTION : is_reviewed_in
--    VERDICT_VERSION ||--o{ REVIEW_ACTION : is_reviewed_in
++    REVIEW_ACTION }o--|| CLAIM_VERSION : reviews
++    REVIEW_ACTION }o--|| SCENARIO_VERSION : reviews
++    REVIEW_ACTION }o--|| EVIDENCE_VERSION : reviews
++    REVIEW_ACTION }o--|| VERDICT_VERSION : reviews
++
  {{/mermaid}}
  Notes:
@@ -309,7 +309,7 @@
  * It may inherit some links from earlier scenarios, or start empty depending
      on the change classification (cosmetic vs. conceptual).
  * All verdicts for that scenario are recalculated and stored as new
--   {{code}}VERDICT_VERSION{{/code}} entries.
++{{code}}VERDICT_VERSION{{/code}} entries.
  * REVIEW_ACTIONs are always attached to the **exact version** that was seen by
    the reviewer. This preserves a faithful audit trail if data later changes.

Changes for page Data Model (From Specification Chat)

Summary

Details

Applications

Navigation

Need help?