Changes for page FactHarbor POC1 Architecture Analysis 1.Jan.26
Last modified by Robert Schaub on 2026/02/08 08:12
From version 1.1
edited by Robert Schaub
on 2026/01/02 09:59
on 2026/01/02 09:59
Change comment:
There is no comment for this version
To version 2.1
edited by Robert Schaub
on 2026/01/02 10:01
on 2026/01/02 10:01
Change comment:
There is no comment for this version
Summary
-
Page properties (1 modified, 0 added, 0 removed)
Details
- Page properties
-
- Content
-
... ... @@ -1,16 +1,17 @@ 1 += FactHarbor POC1 Architecture Analysis = 1 1 2 -= FactHarbor POC1 Architecture Analysis= 3 3 4 - 5 5 **Version:** 2.6.17 6 6 **Analysis Date:** January 2026 7 7 **Document Purpose:** Technical diagrams, gap analysis, and optimization recommendations 8 8 9 ---- 8 +----- 10 10 11 - == 1. AKEL Flow Diagram (with LLM and WebSearch Interactions)==10 +---- 12 12 12 +== 1. AKEL Flow Diagram (with LLM and WebSearch Interactions) == 13 13 14 + 14 14 {{mermaid}} 15 15 flowchart TB 16 16 subgraph Input["📥 Input Layer"] ... ... @@ -92,10 +92,10 @@ 92 92 class UNDERSTAND,DECIDE,FETCHSRC,EXTRACT,VERDICT,REPORT step 93 93 {{/mermaid}} 94 94 95 ---- 96 +----- 96 96 97 97 98 -== 2. ERD Data Model (Current POC1 Implementation)== 99 +== 2. ERD Data Model (Current POC1 Implementation) == 99 99 100 100 101 101 {{mermaid}} ... ... @@ -187,10 +187,10 @@ 187 187 } 188 188 {{/mermaid}} 189 189 190 ---- 191 +----- 191 191 192 192 193 -== 3. Overall Architecture with Interactions== 194 +== 3. Overall Architecture with Interactions == 194 194 195 195 196 196 {{mermaid}} ... ... @@ -286,77 +286,74 @@ 286 286 class ANALYZE_API,JOBS_API,JOB_API,EVENTS_API,RUN_JOB api 287 287 {{/mermaid}} 288 288 289 ---- 290 +----- 290 290 291 291 292 -== 4. Specification vs Implementation Gap Analysis== 293 +== 4. Specification vs Implementation Gap Analysis == 293 293 294 294 295 295 296 -=== 4.1 Data Model Gaps=== 297 +=== 4.1 Data Model Gaps === 297 297 298 298 299 -| Specification Entity | POC1 Status | Gap Description | 300 -|- --------------------|-------------|-----------------|301 -| **Claim** | ⚠️ Partial | No persistent storage; claims exist only in JSON result. Missing: `status`, `confidence_score`, `risk_score`, `completeness_score`, `version`, `views`, `edit_count` | 302 -| **Evidence** | ⚠️ Partial | Implemented as `ExtractedFact` but lacks: `supports` enum, proper `relevance_score` | 303 -| **Source** | ⚠️ Partial | `FetchedSource` exists but missing: `type` enum, `accuracy_history`, `correction_frequency`, weekly update scheduler | 304 -| **Scenario** | ❌ Missing | Not implemented. Claims are evaluated directly without scenario contexts | 305 -| **Verdict** | ⚠️ Partial | `ClaimVerdict` exists but missing: `likelihood_range`, `uncertainty_factors` array, proper `explanation_summary` | 306 -| **User** | ❌ Missing | No user authentication or role system | 307 -| **Edit** | ❌ Missing | No audit trail for changes | 300 +| Specification Entity | POC1 Status | Gap Description | 301 +|-|-|-| 302 +| **Claim** | ⚠️ Partial | No persistent storage; claims exist only in JSON result. Missing: `status`, `confidence_score`, `risk_score`, `completeness_score`, `version`, `views`, `edit_count` | 303 +| **Evidence** | ⚠️ Partial | Implemented as `ExtractedFact` but lacks: `supports` enum, proper `relevance_score` | 304 +| **Source** | ⚠️ Partial | `FetchedSource` exists but missing: `type` enum, `accuracy_history`, `correction_frequency`, weekly update scheduler | 305 +| **Scenario** | ❌ Missing | Not implemented. Claims are evaluated directly without scenario contexts | 306 +| **Verdict** | ⚠️ Partial | `ClaimVerdict` exists but missing: `likelihood_range`, `uncertainty_factors` array, proper `explanation_summary` | 307 +| **User** | ❌ Missing | No user authentication or role system | 308 +| **Edit** | ❌ Missing | No audit trail for changes | 308 308 309 - 310 310 === 4.2 AKEL Component Gaps === 311 311 312 -| Spec Component | POC1 Status | Gap Description | 313 -| ----------------|-------------|-----------------|314 -| **AKEL Orchestrator** | ✅ Implemented | `runAnalysis()` function serves this role | 315 -| **Claim Extractor** | ✅ Implemented | `understandClaim()` with claim role/dependency tracking | 316 -| **Claim Classifier** | ⚠️ Partial | Risk tier (A/B/C) assigned, but no domain classification | 317 -| **Scenario Generator** | ❌ Missing | Claims evaluated without scenario extraction | 318 -| **Evidence Summarizer** | ✅ Implemented | `extractFacts()` function | 319 -| **Contradiction Detector** | ⚠️ Partial | `isContestedClaim` flag exists but no active contradiction search | 320 -| **Quality Gate Validator** | ❌ Missing | No source quality gates, no mandatory checks | 321 -| **Audit Sampling Scheduler** | ❌ Missing | No audit system | 322 -| **Embedding Handler** | ❌ Missing | Not needed for POC | 323 -| **Federation Sync** | ❌ Missing | Not needed for POC | 312 +| Spec Component | POC1 Status | Gap Description | 313 +| |-|-| 314 +| **AKEL Orchestrator** | ✅ Implemented | `runAnalysis()` function serves this role | 315 +| **Claim Extractor** | ✅ Implemented | `understandClaim()` with claim role/dependency tracking | 316 +| **Claim Classifier** | ⚠️ Partial | Risk tier (A/B/C) assigned, but no domain classification | 317 +| **Scenario Generator** | ❌ Missing | Claims evaluated without scenario extraction | 318 +| **Evidence Summarizer** | ✅ Implemented | `extractFacts()` function | 319 +| **Contradiction Detector** | ⚠️ Partial | `isContestedClaim` flag exists but no active contradiction search | 320 +| **Quality Gate Validator** | ❌ Missing | No source quality gates, no mandatory checks | 321 +| **Audit Sampling Scheduler** | ❌ Missing | No audit system | 322 +| **Embedding Handler** | ❌ Missing | Not needed for POC | 323 +| **Federation Sync** | ❌ Missing | Not needed for POC | 324 324 325 +=== 4.3 Architecture Gaps === 325 325 326 -=== 4.3 Architecture Gaps=== 327 327 328 +| Spec Requirement | POC1 Status | Gap Description | 329 +||-|-| 330 +| **Three-Layer Architecture** | ✅ Implemented | Interface (Next.js) → Processing (AKEL) → Data (SQLite) | 331 +| **LLM Abstraction Layer** | ✅ Implemented | AI SDK supports multiple providers with failover | 332 +| **PostgreSQL Primary DB** | ⚠️ Different | Using SQLite for simplicity (acceptable for POC) | 333 +| **Redis Caching** | ❌ Missing | No caching layer | 334 +| **S3 Archival** | ❌ Missing | No long-term storage | 335 +| **Background Jobs** | ❌ Missing | No scheduler for source updates, cache warming | 336 +| **Quality Monitoring** | ⚠️ Partial | LLM call counting exists, but no anomaly detection | 328 328 329 -| Spec Requirement | POC1 Status | Gap Description | 330 -|------------------|-------------|-----------------| 331 -| **Three-Layer Architecture** | ✅ Implemented | Interface (Next.js) → Processing (AKEL) → Data (SQLite) | 332 -| **LLM Abstraction Layer** | ✅ Implemented | AI SDK supports multiple providers with failover | 333 -| **PostgreSQL Primary DB** | ⚠️ Different | Using SQLite for simplicity (acceptable for POC) | 334 -| **Redis Caching** | ❌ Missing | No caching layer | 335 -| **S3 Archival** | ❌ Missing | No long-term storage | 336 -| **Background Jobs** | ❌ Missing | No scheduler for source updates, cache warming | 337 -| **Quality Monitoring** | ⚠️ Partial | LLM call counting exists, but no anomaly detection | 338 +=== 4.4 Publication & Review Gaps === 338 338 339 339 340 -=== 4.4 Publication & Review Gaps=== 341 +| Spec Feature | POC1 Status | Gap Description | 342 +||-|-| 343 +| **Risk Tier Publication Rules** | ❌ Missing | All results published immediately regardless of tier | 344 +| **Human Review Queue** | ❌ Missing | No review workflow | 345 +| **AI-Generated Labeling** | ⚠️ Partial | Results show "AI analysis" but no formal labeling system | 346 +| **Audit Rate Sampling** | ❌ Missing | No sampling audits | 341 341 348 +----- 342 342 343 -| Spec Feature | POC1 Status | Gap Description | 344 -|--------------|-------------|-----------------| 345 -| **Risk Tier Publication Rules** | ❌ Missing | All results published immediately regardless of tier | 346 -| **Human Review Queue** | ❌ Missing | No review workflow | 347 -| **AI-Generated Labeling** | ⚠️ Partial | Results show "AI analysis" but no formal labeling system | 348 -| **Audit Rate Sampling** | ❌ Missing | No sampling audits | 349 349 350 - ---351 +== 5. Optimization Recommendations == 351 351 352 352 353 -== 5. Optimization Recommendations== 354 354 355 +=== 5.1 Cost Optimizations === 355 355 356 356 357 -=== 5.1 Cost Optimizations=== 358 - 359 - 360 360 {{mermaid}} 361 361 pie title Current LLM Cost Distribution (Estimated per Analysis) 362 362 "Step 1: Understand" : 15 ... ... @@ -364,18 +364,17 @@ 364 364 "Step 3: Verdicts" : 25 365 365 {{/mermaid}} 366 366 367 -| Optimization | Estimated Savings | Implementation Effort | 368 -| --------------|-------------------|----------------------|369 -| **Cache claim understanding** | 30-50% on repeated claims | Medium | 370 -| **Use Haiku for fact extraction** | 40% on Step 2 costs | Low (config change) | 371 -| **Batch fact extraction** | 20% fewer API calls | Medium | 372 -| **Skip search for known claims** | 50%+ for cached claims | High (needs claim DB) | 373 -| **Reduce max iterations** | Linear reduction | Low (config change) | 365 +| Optimization | Estimated Savings | Implementation Effort | 366 +||-----|| 367 +| **Cache claim understanding** | 30-50% on repeated claims | Medium | 368 +| **Use Haiku for fact extraction** | 40% on Step 2 costs | Low (config change) | 369 +| **Batch fact extraction** | 20% fewer API calls | Medium | 370 +| **Skip search for known claims** | 50%+ for cached claims | High (needs claim DB) | 371 +| **Reduce max iterations** | Linear reduction | Low (config change) | 374 374 373 +=== 5.2 Timing Optimizations === 375 375 376 -=== 5.2 Timing Optimizations=== 377 377 378 - 379 379 {{mermaid}} 380 380 gantt 381 381 title Current Analysis Timeline (Typical) ... ... @@ -401,18 +401,17 @@ 401 401 Generate Verdicts :b5, after b4, 10s 402 402 {{/mermaid}} 403 403 404 -| Optimization | Time Savings | Notes | 405 -| --------------|--------------|-------|406 -| **Parallel source fetching** | Already implemented | Currently fetches 3 sources in parallel | 407 -| **Streaming LLM responses** | 20-30% perceived | User sees progress faster | 408 -| **Search query batching** | 10-15% | Send multiple queries to search API | 409 -| **Reduce prompt size** | 5-10% per call | Optimize system prompts | 410 -| **Use faster models for extraction** | 30-40% on Step 2 | Claude Haiku vs Sonnet | 401 +| Optimization | Time Savings | Notes | 402 +|||-----| 403 +| **Parallel source fetching** | Already implemented | Currently fetches 3 sources in parallel | 404 +| **Streaming LLM responses** | 20-30% perceived | User sees progress faster | 405 +| **Search query batching** | 10-15% | Send multiple queries to search API | 406 +| **Reduce prompt size** | 5-10% per call | Optimize system prompts | 407 +| **Use faster models for extraction** | 30-40% on Step 2 | Claude Haiku vs Sonnet | 411 411 409 +=== 5.3 Priority Recommendations === 412 412 413 -=== 5.3 Priority Recommendations=== 414 414 415 - 416 416 1. **HIGH PRIORITY - Implement Claim Caching** 417 417 - Cache claim verdicts by content hash 418 418 - Reduces costs for repeated/similar claims ... ... @@ -428,14 +428,14 @@ 428 428 - Cache search results (1h TTL) 429 429 - Reduces external API calls 430 430 431 ---- 427 +----- 432 432 433 433 434 -== 6. Separated Verdict Architecture Proposal== 430 +== 6. Separated Verdict Architecture Proposal == 435 435 436 436 437 437 438 -=== 6.1 Current Architecture=== 434 +=== 6.1 Current Architecture === 439 439 440 440 441 441 {{mermaid}} ... ... @@ -454,7 +454,7 @@ 454 454 - Article verdict tightly coupled to claim extraction 455 455 456 456 457 -=== 6.2 Proposed Separated Architecture=== 453 +=== 6.2 Proposed Separated Architecture === 458 458 459 459 460 460 {{mermaid}} ... ... @@ -510,29 +510,26 @@ 510 510 {{/mermaid}} 511 511 512 512 513 -=== 6.3 Benefits Analysis=== 509 +=== 6.3 Benefits Analysis === 514 514 515 515 516 -| Benefit | Impact | Rationale | 517 -|- --------|--------|-----------|518 -| **Cost Reduction** | 40-70% for repeated claims | Many articles share common claims (e.g., "COVID vaccines are safe") | 519 -| **Faster Analysis** | 50%+ for cached claims | Skip research + LLM calls for known claims | 520 -| **Consistency** | High | Same claim always gets same verdict (until cache expires) | 521 -| **Freshness Control** | Configurable TTL | Balance consistency vs. new evidence | 522 -| **Scalability** | Linear improvement | More users = higher cache hit rate | 512 +| Benefit | Impact | Rationale | 513 +|-| |-----| 514 +| **Cost Reduction** | 40-70% for repeated claims | Many articles share common claims (e.g., "COVID vaccines are safe") | 515 +| **Faster Analysis** | 50%+ for cached claims | Skip research + LLM calls for known claims | 516 +| **Consistency** | High | Same claim always gets same verdict (until cache expires) | 517 +| **Freshness Control** | Configurable TTL | Balance consistency vs. new evidence | 518 +| **Scalability** | Linear improvement | More users = higher cache hit rate | 523 523 524 - 525 525 === 6.4 Implementation Considerations === 526 526 527 527 **Claim Hashing Strategy:** 528 -{{code language="typescript"}} 529 -function getClaimHash(claim: string): string { 523 +{{code language="typescript"}}function getClaimHash(claim: string): string { 530 530 // Normalize: lowercase, remove punctuation, stem words 531 531 const normalized = normalize(claim); 532 532 // Hash for cache key 533 533 return crypto.createHash('sha256').update(normalized).digest('hex').slice(0, 16); 534 -} 535 -{{/code}} 528 +}{{/code}} 536 536 537 537 **Cache Invalidation Triggers:** 538 538 - TTL expiration (default 7 days) ... ... @@ -545,7 +545,7 @@ 545 545 - Same claims in different article contexts may yield different article verdicts 546 546 - Example: "Vaccines are safe" + "Vaccines cause autism" → article may be misleading even if first claim is true 547 547 548 -### 6.5 Recommendation 541 +### 6.5 Recommendation## 549 549 550 550 **YES, separating is beneficial** with the following caveats: 551 551 ... ... @@ -561,14 +561,14 @@ 561 561 - Phase 2: Semantic similarity caching (embedding-based) 562 562 - Phase 3: Federated claim sharing across instances 563 563 564 ---- 557 +----- 565 565 566 566 567 -== 7. Summary== 560 +== 7. Summary == 568 568 569 569 570 570 571 -=== Current State=== 564 +=== Current State === 572 572 573 573 - POC1 implements core AKEL pipeline successfully 574 574 - Claim dependency tracking is implemented ... ... @@ -576,7 +576,7 @@ 576 576 - No persistent claim storage or caching 577 577 578 578 579 -=== Key Gaps from Specification=== 572 +=== Key Gaps from Specification === 580 580 581 581 - No scenario extraction 582 582 - No user/role system ... ... @@ -585,7 +585,7 @@ 585 585 - No review queue 586 586 587 587 588 -=== Recommended Next Steps=== 581 +=== Recommended Next Steps === 589 589 590 590 1. Implement claim caching layer 591 591 2. Separate claim vs article verdict generation ... ... @@ -592,5 +592,3 @@ 592 592 3. Add Redis for source/search caching 593 593 4. Implement tiered model selection 594 594 5. Add basic audit logging 595 - 596 -