Skip to Content
Last modified by Robert Schaub on 2026/02/08 08:12
From version 1.1
edited by Robert Schaub
on 2026/01/02 09:59
Change comment: There is no comment for this version
To version 6.1
edited by Robert Schaub
on 2026/01/02 10:06
Change comment: There is no comment for this version

Summary

Details

Page properties
Content
... ... @@ -1,14 +1,12 @@
1 += FactHarbor POC1 Architecture Analysis =
1 1  
2 -= FactHarbor POC1 Architecture Analysis=
3 -
4 -
5 5  **Version:** 2.6.17
6 6  **Analysis Date:** January 2026
7 7  **Document Purpose:** Technical diagrams, gap analysis, and optimization recommendations
8 8  
9 ----
7 +----
10 10  
11 -== 1. AKEL Flow Diagram (with LLM and WebSearch Interactions)==
9 +== 1. AKEL Flow Diagram (with LLM and WebSearch Interactions) ==
12 12  
13 13  
14 14  {{mermaid}}
... ... @@ -92,12 +92,10 @@
92 92   class UNDERSTAND,DECIDE,FETCHSRC,EXTRACT,VERDICT,REPORT step
93 93  {{/mermaid}}
94 94  
95 ----
93 +----
96 96  
95 +== 2. ERD Data Model (Current POC1 Implementation) ==
97 97  
98 -== 2. ERD Data Model (Current POC1 Implementation)==
99 -
100 -
101 101  {{mermaid}}
102 102  erDiagram
103 103   JOB ||--o{ JOB_EVENT : "has"
... ... @@ -187,12 +187,10 @@
187 187   }
188 188  {{/mermaid}}
189 189  
190 ----
186 +----
191 191  
188 +== 3. Overall Architecture with Interactions ==
192 192  
193 -== 3. Overall Architecture with Interactions==
194 -
195 -
196 196  {{mermaid}}
197 197  flowchart TB
198 198   subgraph Client["🖥️ Client Layer"]
... ... @@ -286,77 +286,64 @@
286 286   class ANALYZE_API,JOBS_API,JOB_API,EVENTS_API,RUN_JOB api
287 287  {{/mermaid}}
288 288  
289 ----
283 +----
290 290  
285 +== 4. Specification vs Implementation Gap Analysis ==
291 291  
292 -== 4. Specification vs Implementation Gap Analysis==
287 +=== 4.1 Data Model Gaps ===
293 293  
289 +| Specification Entity | POC1 Status | Gap Description |
290 +|-|-|-|
291 +| **Claim** | ⚠️ Partial | No persistent storage; claims exist only in JSON result. Missing: `status`, `confidence_score`, `risk_score`, `completeness_score`, `version`, `views`, `edit_count` |
292 +| **Evidence** | ⚠️ Partial | Implemented as `ExtractedFact` but lacks: `supports` enum, proper `relevance_score` |
293 +| **Source** | ⚠️ Partial | `FetchedSource` exists but missing: `type` enum, `accuracy_history`, `correction_frequency`, weekly update scheduler |
294 +| **Scenario** | ❌ Missing | Not implemented. Claims are evaluated directly without scenario contexts |
295 +| **Verdict** | ⚠️ Partial | `ClaimVerdict` exists but missing: `likelihood_range`, `uncertainty_factors` array, proper `explanation_summary` |
296 +| **User** | ❌ Missing | No user authentication or role system |
297 +| **Edit** | ❌ Missing | No audit trail for changes |
294 294  
295 -
296 -=== 4.1 Data Model Gaps===
297 -
298 -
299 -| Specification Entity | POC1 Status | Gap Description |
300 -|---------------------|-------------|-----------------|
301 -| **Claim** | ⚠️ Partial | No persistent storage; claims exist only in JSON result. Missing: `status`, `confidence_score`, `risk_score`, `completeness_score`, `version`, `views`, `edit_count` |
302 -| **Evidence** | ⚠️ Partial | Implemented as `ExtractedFact` but lacks: `supports` enum, proper `relevance_score` |
303 -| **Source** | ⚠️ Partial | `FetchedSource` exists but missing: `type` enum, `accuracy_history`, `correction_frequency`, weekly update scheduler |
304 -| **Scenario** | ❌ Missing | Not implemented. Claims are evaluated directly without scenario contexts |
305 -| **Verdict** | ⚠️ Partial | `ClaimVerdict` exists but missing: `likelihood_range`, `uncertainty_factors` array, proper `explanation_summary` |
306 -| **User** | ❌ Missing | No user authentication or role system |
307 -| **Edit** | ❌ Missing | No audit trail for changes |
308 -
309 -
310 310  === 4.2 AKEL Component Gaps ===
311 311  
312 -| Spec Component | POC1 Status | Gap Description |
313 -|----------------|-------------|-----------------|
314 -| **AKEL Orchestrator** | ✅ Implemented | `runAnalysis()` function serves this role |
315 -| **Claim Extractor** | ✅ Implemented | `understandClaim()` with claim role/dependency tracking |
316 -| **Claim Classifier** | ⚠️ Partial | Risk tier (A/B/C) assigned, but no domain classification |
317 -| **Scenario Generator** | ❌ Missing | Claims evaluated without scenario extraction |
318 -| **Evidence Summarizer** | ✅ Implemented | `extractFacts()` function |
319 -| **Contradiction Detector** | ⚠️ Partial | `isContestedClaim` flag exists but no active contradiction search |
320 -| **Quality Gate Validator** | ❌ Missing | No source quality gates, no mandatory checks |
321 -| **Audit Sampling Scheduler** | ❌ Missing | No audit system |
322 -| **Embedding Handler** | ❌ Missing | Not needed for POC |
323 -| **Federation Sync** | ❌ Missing | Not needed for POC |
301 +| Spec Component | POC1 Status | Gap Description |
302 +| |-|-|
303 +| **AKEL Orchestrator** | ✅ Implemented | `runAnalysis()` function serves this role |
304 +| **Claim Extractor** | ✅ Implemented | `understandClaim()` with claim role/dependency tracking |
305 +| **Claim Classifier** | ⚠️ Partial | Risk tier (A/B/C) assigned, but no domain classification |
306 +| **Scenario Generator** | ❌ Missing | Claims evaluated without scenario extraction |
307 +| **Evidence Summarizer** | ✅ Implemented | `extractFacts()` function |
308 +| **Contradiction Detector** | ⚠️ Partial | `isContestedClaim` flag exists but no active contradiction search |
309 +| **Quality Gate Validator** | ❌ Missing | No source quality gates, no mandatory checks |
310 +| **Audit Sampling Scheduler** | ❌ Missing | No audit system |
311 +| **Embedding Handler** | ❌ Missing | Not needed for POC |
312 +| **Federation Sync** | ❌ Missing | Not needed for POC |
324 324  
314 +=== 4.3 Architecture Gaps ===
325 325  
326 -=== 4.3 Architecture Gaps===
316 +| Spec Requirement | POC1 Status | Gap Description |
317 +| |-|-|
318 +| **Three-Layer Architecture** | ✅ Implemented | Interface (Next.js) → Processing (AKEL) → Data (SQLite) |
319 +| **LLM Abstraction Layer** | ✅ Implemented | AI SDK supports multiple providers with failover |
320 +| **PostgreSQL Primary DB** | ⚠️ Different | Using SQLite for simplicity (acceptable for POC) |
321 +| **Redis Caching** | ❌ Missing | No caching layer |
322 +| **S3 Archival** | ❌ Missing | No long-term storage |
323 +| **Background Jobs** | ❌ Missing | No scheduler for source updates, cache warming |
324 +| **Quality Monitoring** | ⚠️ Partial | LLM call counting exists, but no anomaly detection |
327 327  
326 +=== 4.4 Publication & Review Gaps ===
328 328  
329 -| Spec Requirement | POC1 Status | Gap Description |
330 -|------------------|-------------|-----------------|
331 -| **Three-Layer Architecture** | ✅ Implemented | Interface (Next.js) → Processing (AKEL) → Data (SQLite) |
332 -| **LLM Abstraction Layer** | ✅ Implemented | AI SDK supports multiple providers with failover |
333 -| **PostgreSQL Primary DB** | ⚠️ Different | Using SQLite for simplicity (acceptable for POC) |
334 -| **Redis Caching** | ❌ Missing | No caching layer |
335 -| **S3 Archival** | ❌ Missing | No long-term storage |
336 -| **Background Jobs** | ❌ Missing | No scheduler for source updates, cache warming |
337 -| **Quality Monitoring** | ⚠️ Partial | LLM call counting exists, but no anomaly detection |
328 +| Spec Feature | POC1 Status | Gap Description |
329 +| |-|-|
330 +| **Risk Tier Publication Rules** | ❌ Missing | All results published immediately regardless of tier |
331 +| **Human Review Queue** | ❌ Missing | No review workflow |
332 +| **AI-Generated Labeling** | ⚠️ Partial | Results show "AI analysis" but no formal labeling system |
333 +| **Audit Rate Sampling** | ❌ Missing | No sampling audits |
338 338  
335 +----
339 339  
340 -=== 4.4 Publication & Review Gaps===
337 +== 5. Optimization Recommendations ==
341 341  
339 +=== 5.1 Cost Optimizations ===
342 342  
343 -| Spec Feature | POC1 Status | Gap Description |
344 -|--------------|-------------|-----------------|
345 -| **Risk Tier Publication Rules** | ❌ Missing | All results published immediately regardless of tier |
346 -| **Human Review Queue** | ❌ Missing | No review workflow |
347 -| **AI-Generated Labeling** | ⚠️ Partial | Results show "AI analysis" but no formal labeling system |
348 -| **Audit Rate Sampling** | ❌ Missing | No sampling audits |
349 -
350 ----
351 -
352 -
353 -== 5. Optimization Recommendations==
354 -
355 -
356 -
357 -=== 5.1 Cost Optimizations===
358 -
359 -
360 360  {{mermaid}}
361 361  pie title Current LLM Cost Distribution (Estimated per Analysis)
362 362   "Step 1: Understand" : 15
... ... @@ -364,18 +364,16 @@
364 364   "Step 3: Verdicts" : 25
365 365  {{/mermaid}}
366 366  
367 -| Optimization | Estimated Savings | Implementation Effort |
368 -|--------------|-------------------|----------------------|
369 -| **Cache claim understanding** | 30-50% on repeated claims | Medium |
370 -| **Use Haiku for fact extraction** | 40% on Step 2 costs | Low (config change) |
371 -| **Batch fact extraction** | 20% fewer API calls | Medium |
372 -| **Skip search for known claims** | 50%+ for cached claims | High (needs claim DB) |
373 -| **Reduce max iterations** | Linear reduction | Low (config change) |
348 +| Optimization | Estimated Savings | Implementation Effort |
349 +| |-| |
350 +| **Cache claim understanding** | 30-50% on repeated claims | Medium |
351 +| **Use Haiku for fact extraction** | 40% on Step 2 costs | Low (config change) |
352 +| **Batch fact extraction** | 20% fewer API calls | Medium |
353 +| **Skip search for known claims** | 50%+ for cached claims | High (needs claim DB) |
354 +| **Reduce max iterations** | Linear reduction | Low (config change) |
374 374  
356 +=== 5.2 Timing Optimizations ===
375 375  
376 -=== 5.2 Timing Optimizations===
377 -
378 -
379 379  {{mermaid}}
380 380  gantt
381 381   title Current Analysis Timeline (Typical)
... ... @@ -401,18 +401,16 @@
401 401   Generate Verdicts :b5, after b4, 10s
402 402  {{/mermaid}}
403 403  
404 -| Optimization | Time Savings | Notes |
405 -|--------------|--------------|-------|
406 -| **Parallel source fetching** | Already implemented | Currently fetches 3 sources in parallel |
407 -| **Streaming LLM responses** | 20-30% perceived | User sees progress faster |
408 -| **Search query batching** | 10-15% | Send multiple queries to search API |
409 -| **Reduce prompt size** | 5-10% per call | Optimize system prompts |
410 -| **Use faster models for extraction** | 30-40% on Step 2 | Claude Haiku vs Sonnet |
383 +| Optimization | Time Savings | Notes |
384 +| | |-|
385 +| **Parallel source fetching** | Already implemented | Currently fetches 3 sources in parallel |
386 +| **Streaming LLM responses** | 20-30% perceived | User sees progress faster |
387 +| **Search query batching** | 10-15% | Send multiple queries to search API |
388 +| **Reduce prompt size** | 5-10% per call | Optimize system prompts |
389 +| **Use faster models for extraction** | 30-40% on Step 2 | Claude Haiku vs Sonnet |
411 411  
391 +=== 5.3 Priority Recommendations ===
412 412  
413 -=== 5.3 Priority Recommendations===
414 -
415 -
416 416  1. **HIGH PRIORITY - Implement Claim Caching**
417 417   - Cache claim verdicts by content hash
418 418   - Reduces costs for repeated/similar claims
... ... @@ -428,16 +428,12 @@
428 428   - Cache search results (1h TTL)
429 429   - Reduces external API calls
430 430  
431 ----
408 +----
432 432  
410 +== 6. Separated Verdict Architecture Proposal ==
433 433  
434 -== 6. Separated Verdict Architecture Proposal==
412 +=== 6.1 Current Architecture ===
435 435  
436 -
437 -
438 -=== 6.1 Current Architecture===
439 -
440 -
441 441  {{mermaid}}
442 442  flowchart LR
443 443   subgraph Current["Current: Monolithic Analysis"]
... ... @@ -453,10 +453,8 @@
453 453  - No caching of individual claim verdicts
454 454  - Article verdict tightly coupled to claim extraction
455 455  
429 +=== 6.2 Proposed Separated Architecture ===
456 456  
457 -=== 6.2 Proposed Separated Architecture===
458 -
459 -
460 460  {{mermaid}}
461 461  flowchart TB
462 462   subgraph Input["Input Processing"]
... ... @@ -509,30 +509,25 @@
509 509   class CONTEXT,ARTICLE_VERDICT dynamic
510 510  {{/mermaid}}
511 511  
483 +=== 6.3 Benefits Analysis ===
512 512  
513 -=== 6.3 Benefits Analysis===
485 +| Benefit | Impact | Rationale |
486 +|-| |-|
487 +| **Cost Reduction** | 40-70% for repeated claims | Many articles share common claims (e.g., "COVID vaccines are safe") |
488 +| **Faster Analysis** | 50%+ for cached claims | Skip research + LLM calls for known claims |
489 +| **Consistency** | High | Same claim always gets same verdict (until cache expires) |
490 +| **Freshness Control** | Configurable TTL | Balance consistency vs. new evidence |
491 +| **Scalability** | Linear improvement | More users = higher cache hit rate |
514 514  
515 -
516 -| Benefit | Impact | Rationale |
517 -|---------|--------|-----------|
518 -| **Cost Reduction** | 40-70% for repeated claims | Many articles share common claims (e.g., "COVID vaccines are safe") |
519 -| **Faster Analysis** | 50%+ for cached claims | Skip research + LLM calls for known claims |
520 -| **Consistency** | High | Same claim always gets same verdict (until cache expires) |
521 -| **Freshness Control** | Configurable TTL | Balance consistency vs. new evidence |
522 -| **Scalability** | Linear improvement | More users = higher cache hit rate |
523 -
524 -
525 525  === 6.4 Implementation Considerations ===
526 526  
527 527  **Claim Hashing Strategy:**
528 -{{code language="typescript"}}
529 -function getClaimHash(claim: string): string {
496 +{{code language="typescript"}}function getClaimHash(claim: string): string {
530 530   // Normalize: lowercase, remove punctuation, stem words
531 531   const normalized = normalize(claim);
532 532   // Hash for cache key
533 533   return crypto.createHash('sha256').update(normalized).digest('hex').slice(0, 16);
534 -}
535 -{{/code}}
501 +}{{/code}}
536 536  
537 537  **Cache Invalidation Triggers:**
538 538  - TTL expiration (default 7 days)
... ... @@ -545,7 +545,7 @@
545 545  - Same claims in different article contexts may yield different article verdicts
546 546  - Example: "Vaccines are safe" + "Vaccines cause autism" → article may be misleading even if first claim is true
547 547  
548 -### 6.5 Recommendation
514 +### 6.5 Recommendation##
549 549  
550 550  **YES, separating is beneficial** with the following caveats:
551 551  
... ... @@ -561,23 +561,19 @@
561 561   - Phase 2: Semantic similarity caching (embedding-based)
562 562   - Phase 3: Federated claim sharing across instances
563 563  
564 ----
530 +----
565 565  
532 +== 7. Summary ==
566 566  
567 -== 7. Summary==
534 +=== Current State ===
568 568  
569 -
570 -
571 -=== Current State===
572 -
573 573  - POC1 implements core AKEL pipeline successfully
574 574  - Claim dependency tracking is implemented
575 575  - Multiple LLM providers supported
576 576  - No persistent claim storage or caching
577 577  
541 +=== Key Gaps from Specification ===
578 578  
579 -=== Key Gaps from Specification===
580 -
581 581  - No scenario extraction
582 582  - No user/role system
583 583  - No audit trail
... ... @@ -584,13 +584,10 @@
584 584  - No source track record updates
585 585  - No review queue
586 586  
549 +=== Recommended Next Steps ===
587 587  
588 -=== Recommended Next Steps===
589 -
590 590  1. Implement claim caching layer
591 591  2. Separate claim vs article verdict generation
592 592  3. Add Redis for source/search caching
593 593  4. Implement tiered model selection
594 594  5. Add basic audit logging
595 -
596 -