Wiki source code of Data Model
Version 1.1 by Robert Schaub on 2025/12/18 12:03
Show last authors
| author | version | line-number | content |
|---|---|---|---|
| 1 | = Data Model = | ||
| 2 | FactHarbor's data model is **simple, focused, designed for automated processing**. | ||
| 3 | == 1. Core Entities == | ||
| 4 | === 1.1 Claim === | ||
| 5 | Fields: id, assertion, domain, **status** (Published/Hidden only), **confidence_score**, **risk_score**, completeness_score, version, views, edit_count | ||
| 6 | ==== Performance Optimization: Denormalized Fields ==== | ||
| 7 | **Rationale**: Claims system is 95% reads, 5% writes. Denormalizing common data reduces joins and improves query performance by 70%. | ||
| 8 | **Additional cached fields in claims table**: | ||
| 9 | * **evidence_summary** (JSONB): Top 5 most relevant evidence snippets with scores | ||
| 10 | * Avoids joining evidence table for listing/preview | ||
| 11 | * Updated when evidence is added/removed | ||
| 12 | * Format: `[{"text": "...", "source": "...", "relevance": 0.95}, ...]` | ||
| 13 | * **source_names** (TEXT[]): Array of source names for quick display | ||
| 14 | * Avoids joining through evidence to sources | ||
| 15 | * Updated when sources change | ||
| 16 | * Format: `["New York Times", "Nature Journal", ...]` | ||
| 17 | * **scenario_count** (INTEGER): Number of scenarios for this claim | ||
| 18 | * Quick metric without counting rows | ||
| 19 | * Updated when scenarios added/removed | ||
| 20 | * **cache_updated_at** (TIMESTAMP): When denormalized data was last refreshed | ||
| 21 | * Helps invalidate stale caches | ||
| 22 | * Triggers background refresh if too old | ||
| 23 | **Update Strategy**: | ||
| 24 | * **Immediate**: Update on claim edit (user-facing) | ||
| 25 | * **Deferred**: Update via background job every hour (non-critical) | ||
| 26 | * **Invalidation**: Clear cache when source data changes significantly | ||
| 27 | **Trade-offs**: | ||
| 28 | * ✅ 70% fewer joins on common queries | ||
| 29 | * ✅ Much faster claim list/search pages | ||
| 30 | * ✅ Better user experience | ||
| 31 | * ⚠️ Small storage increase (~10%) | ||
| 32 | * ⚠️ Need to keep caches in sync | ||
| 33 | === 1.2 Evidence === | ||
| 34 | Fields: claim_id, source_id, excerpt, url, relevance_score, supports | ||
| 35 | === 1.3 Source === | ||
| 36 | **Purpose**: Track reliability of information sources over time | ||
| 37 | **Fields**: | ||
| 38 | * **id** (UUID): Unique identifier | ||
| 39 | * **name** (text): Source name (e.g., "New York Times", "Nature Journal") | ||
| 40 | * **domain** (text): Website domain (e.g., "nytimes.com") | ||
| 41 | * **type** (enum): NewsOutlet, AcademicJournal, GovernmentAgency, etc. | ||
| 42 | * **track_record_score** (0-100): Overall reliability score | ||
| 43 | * **accuracy_history** (JSON): Historical accuracy data | ||
| 44 | * **correction_frequency** (float): How often source publishes corrections | ||
| 45 | * **last_updated** (timestamp): When track record last recalculated | ||
| 46 | **How It Works**: | ||
| 47 | * Initial score based on source type (70 for academic journals, 30 for unknown) | ||
| 48 | * Updated daily by background scheduler | ||
| 49 | * Formula: accuracy_rate (50%) + correction_policy (20%) + editorial_standards (15%) + bias_transparency (10%) + longevity (5%) | ||
| 50 | * Track Record Check in AKEL pipeline: Adjusts evidence confidence based on source quality | ||
| 51 | * Quality thresholds: 90+=Exceptional, 70-89=Reliable, 50-69=Acceptable, 30-49=Questionable, <30=Unreliable | ||
| 52 | **See**: SOURCE Track Record System documentation for complete details on calculation, updates, and usage | ||
| 53 | Fields: id, name, domain, **track_record_score**, **accuracy_history**, **correction_frequency** | ||
| 54 | **Key**: Automated source reliability tracking | ||
| 55 | ==== Source Scoring Process (Separation of Concerns) ==== | ||
| 56 | **Critical design principle**: Prevent circular dependencies between source scoring and claim analysis. | ||
| 57 | **The Problem**: | ||
| 58 | * Source scores should influence claim verdicts | ||
| 59 | * Claim verdicts should update source scores | ||
| 60 | * But: Direct feedback creates circular dependency and potential feedback loops | ||
| 61 | **The Solution**: Temporal separation | ||
| 62 | ==== Weekly Background Job (Source Scoring) ==== | ||
| 63 | Runs independently of claim analysis: | ||
| 64 | {{code language="python"}} | ||
| 65 | def update_source_scores_weekly(): | ||
| 66 | """ | ||
| 67 | Background job: Calculate source reliability | ||
| 68 | Never triggered by individual claim analysis | ||
| 69 | """ | ||
| 70 | # Analyze all claims from past week | ||
| 71 | claims = get_claims_from_past_week() | ||
| 72 | for source in get_all_sources(): | ||
| 73 | # Calculate accuracy metrics | ||
| 74 | correct_verdicts = count_correct_verdicts_citing(source, claims) | ||
| 75 | total_citations = count_total_citations(source, claims) | ||
| 76 | accuracy = correct_verdicts / total_citations if total_citations > 0 else 0.5 | ||
| 77 | # Weight by claim importance | ||
| 78 | weighted_score = calculate_weighted_score(source, claims) | ||
| 79 | # Update source record | ||
| 80 | source.track_record_score = weighted_score | ||
| 81 | source.total_citations = total_citations | ||
| 82 | source.last_updated = now() | ||
| 83 | source.save() | ||
| 84 | # Job runs: Sunday 2 AM UTC | ||
| 85 | # Never during claim processing | ||
| 86 | {{/code}} | ||
| 87 | ==== Real-Time Claim Analysis (AKEL) ==== | ||
| 88 | Uses source scores but never updates them: | ||
| 89 | {{code language="python"}} | ||
| 90 | def analyze_claim(claim_text): | ||
| 91 | """ | ||
| 92 | Real-time: Analyze claim using current source scores | ||
| 93 | READ source scores, never UPDATE them | ||
| 94 | """ | ||
| 95 | # Gather evidence | ||
| 96 | evidence_list = gather_evidence(claim_text) | ||
| 97 | for evidence in evidence_list: | ||
| 98 | # READ source score (snapshot from last weekly update) | ||
| 99 | source = get_source(evidence.source_id) | ||
| 100 | source_score = source.track_record_score | ||
| 101 | # Use score to weight evidence | ||
| 102 | evidence.weighted_relevance = evidence.relevance * source_score | ||
| 103 | # Generate verdict using weighted evidence | ||
| 104 | verdict = synthesize_verdict(evidence_list) | ||
| 105 | # NEVER update source scores here | ||
| 106 | # That happens in weekly background job | ||
| 107 | return verdict | ||
| 108 | {{/code}} | ||
| 109 | ==== Monthly Audit (Quality Assurance) ==== | ||
| 110 | Moderator review of flagged source scores: | ||
| 111 | * Verify scores make sense | ||
| 112 | * Detect gaming attempts | ||
| 113 | * Identify systematic biases | ||
| 114 | * Manual adjustments if needed | ||
| 115 | **Key Principles**: | ||
| 116 | ✅ **Scoring and analysis are temporally separated** | ||
| 117 | * Source scoring: Weekly batch job | ||
| 118 | * Claim analysis: Real-time processing | ||
| 119 | * Never update scores during analysis | ||
| 120 | ✅ **One-way data flow during processing** | ||
| 121 | * Claims READ source scores | ||
| 122 | * Claims NEVER WRITE source scores | ||
| 123 | * Updates happen in background only | ||
| 124 | ✅ **Predictable update cycle** | ||
| 125 | * Sources update every Sunday 2 AM | ||
| 126 | * Claims always use last week's scores | ||
| 127 | * No mid-week score changes | ||
| 128 | ✅ **Audit trail** | ||
| 129 | * Log all score changes | ||
| 130 | * Track score history | ||
| 131 | * Explainable calculations | ||
| 132 | **Benefits**: | ||
| 133 | * No circular dependencies | ||
| 134 | * Predictable behavior | ||
| 135 | * Easier to reason about | ||
| 136 | * Simpler testing | ||
| 137 | * Clear audit trail | ||
| 138 | **Example Timeline**: | ||
| 139 | ``` | ||
| 140 | Sunday 2 AM: Calculate source scores for past week | ||
| 141 | → NYT score: 0.87 (up from 0.85) | ||
| 142 | → Blog X score: 0.52 (down from 0.61) | ||
| 143 | Monday-Saturday: Claims processed using these scores | ||
| 144 | → All claims this week use NYT=0.87 | ||
| 145 | → All claims this week use Blog X=0.52 | ||
| 146 | Next Sunday 2 AM: Recalculate scores including this week's claims | ||
| 147 | → NYT score: 0.89 (trending up) | ||
| 148 | → Blog X score: 0.48 (trending down) | ||
| 149 | ``` | ||
| 150 | === 1.4 Scenario === | ||
| 151 | **Purpose**: Different interpretations or contexts for evaluating claims | ||
| 152 | **Key Concept**: Scenarios are extracted from evidence, not generated arbitrarily. Each scenario represents a specific context, assumption set, or condition under which a claim should be evaluated. | ||
| 153 | **Relationship**: One-to-many with Claims (**simplified for V1.0**: scenario belongs to single claim) | ||
| 154 | **Fields**: | ||
| 155 | * **id** (UUID): Unique identifier | ||
| 156 | * **claim_id** (UUID): Foreign key to claim (one-to-many) | ||
| 157 | * **description** (text): Human-readable description of the scenario | ||
| 158 | * **assumptions** (JSONB): Key assumptions that define this scenario context | ||
| 159 | * **extracted_from** (UUID): Reference to evidence that this scenario was extracted from | ||
| 160 | * **verdict_summary** (text): Compiled verdict for this scenario | ||
| 161 | * **confidence** (decimal 0-1): Confidence level for verdict in this scenario | ||
| 162 | * **created_at** (timestamp): When scenario was created | ||
| 163 | * **updated_at** (timestamp): Last modification | ||
| 164 | **How Found**: Evidence search → Extract context → Create scenario → Link to claim | ||
| 165 | **Example**: | ||
| 166 | For claim "Vaccines reduce hospitalization": | ||
| 167 | * Scenario 1: "Clinical trials (healthy adults 18-65, original strain)" from trial paper | ||
| 168 | * Scenario 2: "Real-world data (diverse population, Omicron variant)" from hospital data | ||
| 169 | * Scenario 3: "Immunocompromised patients" from specialist study | ||
| 170 | **V2.0 Evolution**: Many-to-many relationship can be added if users request cross-claim scenario sharing. For V1.0, keeping scenarios tied to single claims simplifies queries and reduces complexity without limiting functionality. | ||
| 171 | === 1.5 User === | ||
| 172 | Fields: username, email, **role** (Reader/Contributor/Moderator), **reputation**, contributions_count | ||
| 173 | === User Reputation System == | ||
| 174 | **V1.0 Approach**: Simple manual role assignment | ||
| 175 | **Rationale**: Complex reputation systems aren't needed until 100+ active contributors demonstrate the need for automated reputation management. Start simple, add complexity when metrics prove necessary. | ||
| 176 | === Roles (Manual Assignment) === | ||
| 177 | **reader** (default): | ||
| 178 | * View published claims and evidence | ||
| 179 | * Browse and search content | ||
| 180 | * No editing permissions | ||
| 181 | **contributor**: | ||
| 182 | * Submit new claims | ||
| 183 | * Suggest edits to existing content | ||
| 184 | * Add evidence | ||
| 185 | * Requires manual promotion by moderator/admin | ||
| 186 | **moderator**: | ||
| 187 | * Approve/reject contributor suggestions | ||
| 188 | * Flag inappropriate content | ||
| 189 | * Handle abuse reports | ||
| 190 | * Assigned by admins based on trust | ||
| 191 | **admin**: | ||
| 192 | * Manage users and roles | ||
| 193 | * System configuration | ||
| 194 | * Access to all features | ||
| 195 | * Founder-appointed initially | ||
| 196 | === Contribution Tracking (Simple) === | ||
| 197 | **Basic metrics only**: | ||
| 198 | * `contributions_count`: Total number of contributions | ||
| 199 | * `created_at`: Account age | ||
| 200 | * `last_active`: Recent activity | ||
| 201 | **No complex calculations**: | ||
| 202 | * No point systems | ||
| 203 | * No automated privilege escalation | ||
| 204 | * No reputation decay | ||
| 205 | * No threshold-based promotions | ||
| 206 | === Promotion Process === | ||
| 207 | **Manual review by moderators/admins**: | ||
| 208 | 1. User demonstrates value through contributions | ||
| 209 | 2. Moderator reviews user's contribution history | ||
| 210 | 3. Moderator promotes user to contributor role | ||
| 211 | 4. Admin promotes trusted contributors to moderator | ||
| 212 | **Criteria** (guidelines, not automated): | ||
| 213 | * Quality of contributions | ||
| 214 | * Consistency over time | ||
| 215 | * Collaborative behavior | ||
| 216 | * Understanding of project goals | ||
| 217 | === V2.0+ Evolution === | ||
| 218 | **Add complex reputation when**: | ||
| 219 | * 100+ active contributors | ||
| 220 | * Manual role management becomes bottleneck | ||
| 221 | * Clear patterns of abuse emerge requiring automation | ||
| 222 | **Future features may include**: | ||
| 223 | * Automated point calculations | ||
| 224 | * Threshold-based promotions | ||
| 225 | * Reputation decay for inactive users | ||
| 226 | * Track record scoring for contributors | ||
| 227 | See [[When to Add Complexity>>FactHarbor.Specification.When-to-Add-Complexity]] for triggers. | ||
| 228 | === 1.6 Edit === | ||
| 229 | **Fields**: entity_type, entity_id, user_id, before_state (JSON), after_state (JSON), edit_type, reason, created_at | ||
| 230 | **Purpose**: Complete audit trail for all content changes | ||
| 231 | === Edit History Details === | ||
| 232 | **What Gets Edited**: | ||
| 233 | * **Claims** (20% edited): assertion, domain, status, scores, analysis | ||
| 234 | * **Evidence** (10% edited): excerpt, relevance_score, supports | ||
| 235 | * **Scenarios** (5% edited): description, assumptions, confidence | ||
| 236 | * **Sources**: NOT versioned (continuous updates, not editorial decisions) | ||
| 237 | **Who Edits**: | ||
| 238 | * **Contributors** (rep sufficient): Corrections, additions | ||
| 239 | * **Trusted Contributors** (rep sufficient): Major improvements, approvals | ||
| 240 | * **Moderators**: Abuse handling, dispute resolution | ||
| 241 | * **System (AKEL)**: Re-analysis, automated improvements (user_id = NULL) | ||
| 242 | **Edit Types**: | ||
| 243 | * `CONTENT_CORRECTION`: User fixes factual error | ||
| 244 | * `CLARIFICATION`: Improved wording | ||
| 245 | * `SYSTEM_REANALYSIS`: AKEL re-processed claim | ||
| 246 | * `MODERATION_ACTION`: Hide/unhide for abuse | ||
| 247 | * `REVERT`: Rollback to previous version | ||
| 248 | **Retention Policy** (5 years total): | ||
| 249 | 1. **Hot storage** (3 months): PostgreSQL, instant access | ||
| 250 | 2. **Warm storage** (2 years): Partitioned, slower queries | ||
| 251 | 3. **Cold storage** (3 years): S3 compressed, download required | ||
| 252 | 4. **Deletion**: After 5 years (except legal holds) | ||
| 253 | **Storage per 1M claims**: ~400 MB (20% edited × 2 KB per edit) | ||
| 254 | **Use Cases**: | ||
| 255 | * View claim history timeline | ||
| 256 | * Detect vandalism patterns | ||
| 257 | * Learn from user corrections (system improvement) | ||
| 258 | * Legal compliance (audit trail) | ||
| 259 | * Rollback capability | ||
| 260 | See **Edit History Documentation** for complete details on what gets edited by whom, retention policy, and use cases | ||
| 261 | === 1.7 Flag === | ||
| 262 | Fields: entity_id, reported_by, issue_type, status, resolution_note | ||
| 263 | === 1.8 QualityMetric === | ||
| 264 | **Fields**: metric_type, category, value, target, timestamp | ||
| 265 | **Purpose**: Time-series quality tracking | ||
| 266 | **Usage**: | ||
| 267 | * **Continuous monitoring**: Hourly calculation of error rates, confidence scores, processing times | ||
| 268 | * **Quality dashboard**: Real-time display with trend charts | ||
| 269 | * **Alerting**: Automatic alerts when metrics exceed thresholds | ||
| 270 | * **A/B testing**: Compare control vs treatment metrics | ||
| 271 | * **Improvement validation**: Measure before/after changes | ||
| 272 | **Example**: `{type: "ErrorRate", category: "Politics", value: 0.12, target: 0.10, timestamp: "2025-12-17"}` | ||
| 273 | === 1.9 ErrorPattern === | ||
| 274 | **Fields**: error_category, claim_id, description, root_cause, frequency, status | ||
| 275 | **Purpose**: Capture errors to trigger system improvements | ||
| 276 | **Usage**: | ||
| 277 | * **Error capture**: When users flag issues or system detects problems | ||
| 278 | * **Pattern analysis**: Weekly grouping by category and frequency | ||
| 279 | * **Improvement workflow**: Analyze → Fix → Test → Deploy → Re-process → Monitor | ||
| 280 | * **Metrics**: Track error rate reduction over time | ||
| 281 | **Example**: `{category: "WrongSource", description: "Unreliable tabloid cited", root_cause: "No quality check", frequency: 23, status: "Fixed"}` | ||
| 282 | == 1.5 User Class Diagram == | ||
| 283 | {{include reference="FactHarbor.Specification.Diagrams.User Class Diagram.WebHome"/}} | ||
| 284 | == 2. Versioning Strategy == | ||
| 285 | **All Content Entities Are Versioned**: | ||
| 286 | * **Claim**: Every edit creates new version (V1→V2→V3...) | ||
| 287 | * **Evidence**: Changes tracked in edit history | ||
| 288 | * **Scenario**: Modifications versioned | ||
| 289 | **How Versioning Works**: | ||
| 290 | * Entity table stores **current state only** | ||
| 291 | * Edit table stores **all historical states** (before_state, after_state as JSON) | ||
| 292 | * Version number increments with each edit | ||
| 293 | * Complete audit trail maintained forever | ||
| 294 | **Unversioned Entities** (current state only, no history): | ||
| 295 | * **Source**: Track record continuously updated (not versioned history, just current score) | ||
| 296 | * **User**: Account state (reputation accumulated, not versioned) | ||
| 297 | * **QualityMetric**: Time-series data (each record is a point in time, not a version) | ||
| 298 | * **ErrorPattern**: System improvement queue (status tracked, not versioned) | ||
| 299 | **Example**: | ||
| 300 | ``` | ||
| 301 | Claim V1: "The sky is blue" | ||
| 302 | → User edits → | ||
| 303 | Claim V2: "The sky is blue during daytime" | ||
| 304 | → EDIT table stores: {before: "The sky is blue", after: "The sky is blue during daytime"} | ||
| 305 | ``` | ||
| 306 | == 2.5. Storage vs Computation Strategy == | ||
| 307 | **Critical architectural decision**: What to persist in databases vs compute dynamically? | ||
| 308 | **Trade-off**: | ||
| 309 | * **Store more**: Better reproducibility, faster, lower LLM costs | Higher storage/maintenance costs | ||
| 310 | * **Compute more**: Lower storage/maintenance costs | Slower, higher LLM costs, less reproducible | ||
| 311 | === Recommendation: Hybrid Approach === | ||
| 312 | **STORE (in PostgreSQL):** | ||
| 313 | ==== Claims (Current State + History) ==== | ||
| 314 | * **What**: assertion, domain, status, created_at, updated_at, version | ||
| 315 | * **Why**: Core entity, must be persistent | ||
| 316 | * **Also store**: confidence_score (computed once, then cached) | ||
| 317 | * **Size**: ~1 KB per claim | ||
| 318 | * **Growth**: Linear with claims | ||
| 319 | * **Decision**: ✅ STORE - Essential | ||
| 320 | ==== Evidence (All Records) ==== | ||
| 321 | * **What**: claim_id, source_id, excerpt, url, relevance_score, supports, extracted_at | ||
| 322 | * **Why**: Hard to re-gather, user contributions, reproducibility | ||
| 323 | * **Size**: ~2 KB per evidence (with excerpt) | ||
| 324 | * **Growth**: 3-10 evidence per claim | ||
| 325 | * **Decision**: ✅ STORE - Essential for reproducibility | ||
| 326 | ==== Sources (Track Records) ==== | ||
| 327 | * **What**: name, domain, track_record_score, accuracy_history, correction_frequency | ||
| 328 | * **Why**: Continuously updated, expensive to recompute | ||
| 329 | * **Size**: ~500 bytes per source | ||
| 330 | * **Growth**: Slow (limited number of sources) | ||
| 331 | * **Decision**: ✅ STORE - Essential for quality | ||
| 332 | ==== Edit History (All Versions) ==== | ||
| 333 | * **What**: before_state, after_state, user_id, reason, timestamp | ||
| 334 | * **Why**: Audit trail, legal requirement, reproducibility | ||
| 335 | * **Size**: ~2 KB per edit | ||
| 336 | * **Growth**: Linear with edits (~A portion of claims get edited) | ||
| 337 | * **Retention**: Hot storage 3 months → Warm storage 2 years → Archive to S3 3 years → Delete after 5 years total | ||
| 338 | * **Decision**: ✅ STORE - Essential for accountability | ||
| 339 | ==== Flags (User Reports) ==== | ||
| 340 | * **What**: entity_id, reported_by, issue_type, description, status | ||
| 341 | * **Why**: Error detection, system improvement triggers | ||
| 342 | * **Size**: ~500 bytes per flag | ||
| 343 | * **Growth**: 5-high percentage of claims get flagged | ||
| 344 | * **Decision**: ✅ STORE - Essential for improvement | ||
| 345 | ==== ErrorPatterns (System Improvement) ==== | ||
| 346 | * **What**: error_category, claim_id, description, root_cause, frequency, status | ||
| 347 | * **Why**: Learning loop, prevent recurring errors | ||
| 348 | * **Size**: ~1 KB per pattern | ||
| 349 | * **Growth**: Slow (limited patterns, many fixed) | ||
| 350 | * **Decision**: ✅ STORE - Essential for learning | ||
| 351 | ==== QualityMetrics (Time Series) ==== | ||
| 352 | * **What**: metric_type, category, value, target, timestamp | ||
| 353 | * **Why**: Trend analysis, cannot recreate historical metrics | ||
| 354 | * **Size**: ~200 bytes per metric | ||
| 355 | * **Growth**: Hourly = 8,760 per year per metric type | ||
| 356 | * **Retention**: 2 years hot, then aggregate and archive | ||
| 357 | * **Decision**: ✅ STORE - Essential for monitoring | ||
| 358 | **STORE (Computed Once, Then Cached):** | ||
| 359 | ==== Analysis Summary ==== | ||
| 360 | * **What**: Neutral text summary of claim analysis (200-500 words) | ||
| 361 | * **Computed**: Once by AKEL when claim first analyzed | ||
| 362 | * **Stored in**: Claim table (text field) | ||
| 363 | * **Recomputed**: Only when system significantly improves OR claim edited | ||
| 364 | * **Why store**: Expensive to regenerate ($0.01-0.05 per analysis), doesn't change often | ||
| 365 | * **Size**: ~2 KB per claim | ||
| 366 | * **Decision**: ✅ STORE (cached) - Cost-effective | ||
| 367 | ==== Confidence Score ==== | ||
| 368 | * **What**: 0-100 score of analysis confidence | ||
| 369 | * **Computed**: Once by AKEL | ||
| 370 | * **Stored in**: Claim table (integer field) | ||
| 371 | * **Recomputed**: When evidence added, source track record changes significantly, or system improves | ||
| 372 | * **Why store**: Cheap to store, expensive to compute, users need it fast | ||
| 373 | * **Size**: 4 bytes per claim | ||
| 374 | * **Decision**: ✅ STORE (cached) - Performance critical | ||
| 375 | ==== Risk Score ==== | ||
| 376 | * **What**: 0-100 score of claim risk level | ||
| 377 | * **Computed**: Once by AKEL | ||
| 378 | * **Stored in**: Claim table (integer field) | ||
| 379 | * **Recomputed**: When domain changes, evidence changes, or controversy detected | ||
| 380 | * **Why store**: Same as confidence score | ||
| 381 | * **Size**: 4 bytes per claim | ||
| 382 | * **Decision**: ✅ STORE (cached) - Performance critical | ||
| 383 | **COMPUTE DYNAMICALLY (Never Store):** | ||
| 384 | ==== Scenarios ==== ⚠️ CRITICAL DECISION | ||
| 385 | * **What**: 2-5 possible interpretations of claim with assumptions | ||
| 386 | * **Current design**: Stored in Scenario table | ||
| 387 | * **Alternative**: Compute on-demand when user views claim details | ||
| 388 | * **Storage cost**: ~1 KB per scenario × 3 scenarios average = ~3 KB per claim | ||
| 389 | * **Compute cost**: $0.005-0.01 per request (LLM API call) | ||
| 390 | * **Frequency**: Viewed in detail by ~20% of users | ||
| 391 | * **Trade-off analysis**: | ||
| 392 | - IF STORED: 1M claims × 3 KB = 3 GB storage, $0.05/month, fast access | ||
| 393 | - IF COMPUTED: 1M claims × 20% views × $0.01 = $2,000/month in LLM costs | ||
| 394 | * **Reproducibility**: Scenarios may improve as AI improves (good to recompute) | ||
| 395 | * **Speed**: Computed = 5-8 seconds delay, Stored = instant | ||
| 396 | * **Decision**: ✅ STORE (hybrid approach below) | ||
| 397 | **Scenario Strategy** (APPROVED): | ||
| 398 | 1. **Store scenarios** initially when claim analyzed | ||
| 399 | 2. **Mark as stale** when system improves significantly | ||
| 400 | 3. **Recompute on next view** if marked stale | ||
| 401 | 4. **Cache for 30 days** if frequently accessed | ||
| 402 | 5. **Result**: Best of both worlds - speed + freshness | ||
| 403 | ==== Verdict Synthesis ==== | ||
| 404 | * **What**: Final conclusion text synthesizing all scenarios | ||
| 405 | * **Compute cost**: $0.002-0.005 per request | ||
| 406 | * **Frequency**: Every time claim viewed | ||
| 407 | * **Why not store**: Changes as evidence/scenarios change, users want fresh analysis | ||
| 408 | * **Speed**: 2-3 seconds (acceptable) | ||
| 409 | **Alternative**: Store "last verdict" as cached field, recompute only if claim edited or marked stale | ||
| 410 | * **Recommendation**: ✅ STORE cached version, mark stale when changes occur | ||
| 411 | ==== Search Results ==== | ||
| 412 | * **What**: Lists of claims matching search query | ||
| 413 | * **Compute from**: Elasticsearch index | ||
| 414 | * **Cache**: 15 minutes in Redis for popular queries | ||
| 415 | * **Why not store permanently**: Constantly changing, infinite possible queries | ||
| 416 | ==== Aggregated Statistics ==== | ||
| 417 | * **What**: "Total claims: 1,234,567", "Average confidence: 78%", etc. | ||
| 418 | * **Compute from**: Database queries | ||
| 419 | * **Cache**: 1 hour in Redis | ||
| 420 | * **Why not store**: Can be derived, relatively cheap to compute | ||
| 421 | ==== User Reputation ==== | ||
| 422 | * **What**: Score based on contributions | ||
| 423 | * **Current design**: Stored in User table | ||
| 424 | * **Alternative**: Compute from Edit table | ||
| 425 | * **Trade-off**: | ||
| 426 | - Stored: Fast, simple | ||
| 427 | - Computed: Always accurate, no denormalization | ||
| 428 | * **Frequency**: Read on every user action | ||
| 429 | * **Compute cost**: Simple COUNT query, milliseconds | ||
| 430 | * **Decision**: ✅ STORE - Performance critical, read-heavy | ||
| 431 | === Summary Table === | ||
| 432 | | Data Type | Storage | Compute | Size per Claim | Decision | Rationale | | ||
| 433 | |-----------|---------|---------|----------------|----------|-----------| | ||
| 434 | | Claim core | ✅ | - | 1 KB | STORE | Essential | | ||
| 435 | | Evidence | ✅ | - | 2 KB × 5 = 10 KB | STORE | Reproducibility | | ||
| 436 | | Sources | ✅ | - | 500 B (shared) | STORE | Track record | | ||
| 437 | | Edit history | ✅ | - | 2 KB × 20% = 400 B avg | STORE | Audit | | ||
| 438 | | Analysis summary | ✅ | Once | 2 KB | STORE (cached) | Cost-effective | | ||
| 439 | | Confidence score | ✅ | Once | 4 B | STORE (cached) | Fast access | | ||
| 440 | | Risk score | ✅ | Once | 4 B | STORE (cached) | Fast access | | ||
| 441 | | Scenarios | ✅ | When stale | 3 KB | STORE (hybrid) | Balance cost/speed | | ||
| 442 | | Verdict | ✅ | When stale | 1 KB | STORE (cached) | Fast access | | ||
| 443 | | Flags | ✅ | - | 500 B × 10% = 50 B avg | STORE | Improvement | | ||
| 444 | | ErrorPatterns | ✅ | - | 1 KB (global) | STORE | Learning | | ||
| 445 | | QualityMetrics | ✅ | - | 200 B (time series) | STORE | Trending | | ||
| 446 | | Search results | - | ✅ | - | COMPUTE + 15min cache | Dynamic | | ||
| 447 | | Aggregations | - | ✅ | - | COMPUTE + 1hr cache | Derivable | | ||
| 448 | **Total storage per claim**: ~18 KB (without edits and flags) | ||
| 449 | **For 1 million claims**: | ||
| 450 | * **Storage**: ~18 GB (manageable) | ||
| 451 | * **PostgreSQL**: ~$50/month (standard instance) | ||
| 452 | * **Redis cache**: ~$20/month (1 GB instance) | ||
| 453 | * **S3 archives**: ~$5/month (old edits) | ||
| 454 | * **Total**: ~$75/month infrastructure | ||
| 455 | **LLM cost savings by caching**: | ||
| 456 | * Analysis summary stored: Save $0.03 per claim = $30K per 1M claims | ||
| 457 | * Scenarios stored: Save $0.01 per claim × 20% views = $2K per 1M claims | ||
| 458 | * Verdict stored: Save $0.003 per claim = $3K per 1M claims | ||
| 459 | * **Total savings**: ~$35K per 1M claims vs recomputing every time | ||
| 460 | === Recomputation Triggers === | ||
| 461 | **When to mark cached data as stale and recompute:** | ||
| 462 | 1. **User edits claim** → Recompute: all (analysis, scenarios, verdict, scores) | ||
| 463 | 2. **Evidence added** → Recompute: scenarios, verdict, confidence score | ||
| 464 | 3. **Source track record changes >10 points** → Recompute: confidence score, verdict | ||
| 465 | 4. **System improvement deployed** → Mark affected claims stale, recompute on next view | ||
| 466 | 5. **Controversy detected** (high flag rate) → Recompute: risk score | ||
| 467 | **Recomputation strategy**: | ||
| 468 | * **Eager**: Immediately recompute (for user edits) | ||
| 469 | * **Lazy**: Recompute on next view (for system improvements) | ||
| 470 | * **Batch**: Nightly re-evaluation of stale claims (if <1000) | ||
| 471 | === Database Size Projection === | ||
| 472 | **Year 1**: 10K claims | ||
| 473 | * Storage: 180 MB | ||
| 474 | * Cost: $10/month | ||
| 475 | **Year 3**: 100K claims | ||
| 476 | * Storage: 1.8 GB | ||
| 477 | * Cost: $30/month | ||
| 478 | **Year 5**: 1M claims | ||
| 479 | * Storage: 18 GB | ||
| 480 | * Cost: $75/month | ||
| 481 | **Year 10**: 10M claims | ||
| 482 | * Storage: 180 GB | ||
| 483 | * Cost: $300/month | ||
| 484 | * Optimization: Archive old claims to S3 ($5/TB/month) | ||
| 485 | **Conclusion**: Storage costs are manageable, LLM cost savings are substantial. | ||
| 486 | == 3. Key Simplifications == | ||
| 487 | * **Two content states only**: Published, Hidden | ||
| 488 | * **Three user roles only**: Reader, Contributor, Moderator | ||
| 489 | * **No complex versioning**: Linear edit history | ||
| 490 | * **Reputation-based permissions**: Not role hierarchy | ||
| 491 | * **Source track records**: Continuous evaluation | ||
| 492 | == 3. What Gets Stored in the Database == | ||
| 493 | === 3.1 Primary Storage (PostgreSQL) === | ||
| 494 | **Claims Table**: | ||
| 495 | * Current state only (latest version) | ||
| 496 | * Fields: id, assertion, domain, status, confidence_score, risk_score, completeness_score, version, created_at, updated_at | ||
| 497 | **Evidence Table**: | ||
| 498 | * All evidence records | ||
| 499 | * Fields: id, claim_id, source_id, excerpt, url, relevance_score, supports, extracted_at, archived | ||
| 500 | **Scenario Table**: | ||
| 501 | * All scenarios for each claim | ||
| 502 | * Fields: id, claim_id, description, assumptions (text array), confidence, created_by, created_at | ||
| 503 | **Source Table**: | ||
| 504 | * Track record database (continuously updated) | ||
| 505 | * Fields: id, name, domain, type, track_record_score, accuracy_history (JSON), correction_frequency, last_updated, claim_count, corrections_count | ||
| 506 | **User Table**: | ||
| 507 | * All user accounts | ||
| 508 | * Fields: id, username, email, role (Reader/Contributor/Moderator), reputation, created_at, last_active, contributions_count, flags_submitted, flags_accepted | ||
| 509 | **Edit Table**: | ||
| 510 | * Complete version history | ||
| 511 | * Fields: id, entity_type, entity_id, user_id, before_state (JSON), after_state (JSON), edit_type, reason, created_at | ||
| 512 | **Flag Table**: | ||
| 513 | * User-reported issues | ||
| 514 | * Fields: id, entity_type, entity_id, reported_by, issue_type, description, status, resolved_by, resolution_note, created_at, resolved_at | ||
| 515 | **ErrorPattern Table**: | ||
| 516 | * System improvement queue | ||
| 517 | * Fields: id, error_category, claim_id, description, root_cause, frequency, status, created_at, fixed_at | ||
| 518 | **QualityMetric Table**: | ||
| 519 | * Time-series quality data | ||
| 520 | * Fields: id, metric_type, metric_category, value, target, timestamp | ||
| 521 | === 3.2 What's NOT Stored (Computed on-the-fly) === | ||
| 522 | * **Verdicts**: Synthesized from evidence + scenarios when requested | ||
| 523 | * **Risk scores**: Recalculated based on current factors | ||
| 524 | * **Aggregated statistics**: Computed from base data | ||
| 525 | * **Search results**: Generated from Elasticsearch index | ||
| 526 | === 3.3 Cache Layer (Redis) === | ||
| 527 | **Cached for performance**: | ||
| 528 | * Frequently accessed claims (TTL: 1 hour) | ||
| 529 | * Search results (TTL: 15 minutes) | ||
| 530 | * User sessions (TTL: 24 hours) | ||
| 531 | * Source track records (TTL: 1 hour) | ||
| 532 | === 3.4 File Storage (S3) === | ||
| 533 | **Archived content**: | ||
| 534 | * Old edit history (>3 months) | ||
| 535 | * Evidence documents (archived copies) | ||
| 536 | * Database backups | ||
| 537 | * Export files | ||
| 538 | === 3.5 Search Index (Elasticsearch) === | ||
| 539 | **Indexed for search**: | ||
| 540 | * Claim assertions (full-text) | ||
| 541 | * Evidence excerpts (full-text) | ||
| 542 | * Scenario descriptions (full-text) | ||
| 543 | * Source names (autocomplete) | ||
| 544 | Synchronized from PostgreSQL via change data capture or periodic sync. | ||
| 545 | == 4. Related Pages == | ||
| 546 | * [[Architecture>>FactHarbor.Specification.Architecture.WebHome]] | ||
| 547 | * [[Requirements>>FactHarbor.Specification.Requirements.WebHome]] | ||
| 548 | * [[Workflows>>FactHarbor.Specification.Workflows.WebHome]] |