Skip to Content
Version 4.1 by Robert Schaub on 2025/12/24 16:55
Show last authors
1 = POC1 API & Schemas Specification =
2
3 ----
4
5 == Version History ==
6
7 |=Version|=Date|=Changes
8 |0.4.1|2025-12-24|Applied 9 critical fixes: file format notice, verdict taxonomy, canonicalization algorithm, Stage 1 cost policy, BullMQ fix, language in cache key, historical claims TTL, idempotency, copyright policy
9 |0.4|2025-12-24|**BREAKING:** 3-stage pipeline with claim-level caching, user tier system, cache-only mode for free users, Redis cache architecture
10 |0.3.1|2025-12-24|Fixed single-prompt strategy, SSE clarification, schema canonicalization, cost constraints
11 |0.3|2025-12-24|Added complete API endpoints, LLM config, risk tiers, scraping details
12
13 ----
14
15 == 1. Core Objective (POC1) ==
16
17 The primary technical goal of POC1 is to validate **Approach 1 (Single-Pass Holistic Analysis)** while implementing **claim-level caching** to achieve cost sustainability.
18
19 The system must prove that AI can identify an article's **Main Thesis** and determine if supporting claims logically support that thesis without committing fallacies.
20
21 === Success Criteria: ===
22
23 * Test with 30 diverse articles
24 * Target: ≥70% accuracy detecting misleading articles
25 * Cost: <$0.25 per NEW analysis (uncached)
26 * Cost: $0.00 for cached claim reuse
27 * Cache hit rate: ≥50% after 1,000 articles
28 * Processing time: <2 minutes (standard depth)
29
30 === Economic Model: ===
31
32 * **Free tier:** $10 credit per month (~~40-140 articles depending on cache hits)
33 * **After limit:** Cache-only mode (instant, free access to cached claims)
34 * **Paid tier:** Unlimited new analyses
35
36 ----
37
38 == 2. Architecture Overview ==
39
40 === 2.1 3-Stage Pipeline with Caching ===
41
42 FactHarbor POC1 uses a **3-stage architecture** designed for claim-level caching and cost efficiency:
43
44 {{mermaid}}
45 graph TD
46 A[Article Input] --> B[Stage 1: Extract Claims]
47 B --> C{For Each Claim}
48 C --> D[Check Cache]
49 D -->|Cache HIT| E[Return Cached Verdict]
50 D -->|Cache MISS| F[Stage 2: Analyze Claim]
51 F --> G[Store in Cache]
52 G --> E
53 E --> H[Stage 3: Holistic Assessment]
54 H --> I[Final Report]
55 {{/mermaid}}
56
57 ==== Stage 1: Claim Extraction (Haiku, no cache) ====
58
59 * **Input:** Article text
60 * **Output:** 5 canonical claims (normalized, deduplicated)
61 * **Model:** Claude Haiku 4
62 * **Cost:** $0.003 per article
63 * **Cache strategy:** No caching (article-specific)
64
65 ==== Stage 2: Claim Analysis (Sonnet, CACHED) ====
66
67 * **Input:** Single canonical claim
68 * **Output:** Scenarios + Evidence + Verdicts
69 * **Model:** Claude Sonnet 3.5
70 * **Cost:** $0.081 per NEW claim
71 * **Cache strategy:** Redis, 90-day TTL
72 * **Cache key:** claim:v1norm1:{language}:{sha256(canonical_claim)}
73
74 ==== Stage 3: Holistic Assessment (Sonnet, no cache) ====
75
76 * **Input:** Article + Claim verdicts (from cache or Stage 2)
77 * **Output:** Article verdict + Fallacies + Logic quality
78 * **Model:** Claude Sonnet 3.5
79 * **Cost:** $0.030 per article
80 * **Cache strategy:** No caching (article-specific)
81
82 === Total Cost Formula: ===
83
84 {{{Cost = $0.003 (extraction) + (N_new_claims × $0.081) + $0.030 (holistic)
85
86 Examples:
87 - 0 new claims (100% cache hit): $0.033
88 - 1 new claim (80% cache hit): $0.114
89 - 3 new claims (40% cache hit): $0.276
90 - 5 new claims (0% cache hit): $0.438
91 }}}
92
93 ----
94
95 === 2.2 User Tier System ===
96
97 |=Tier|=Monthly Credit|=After Limit|=Cache Access|=Analytics
98 |**Free**|$10|Cache-only mode|✅ Full|Basic
99 |**Pro** (future)|$50|Continues|✅ Full|Advanced
100 |**Enterprise** (future)|Custom|Continues|✅ Full + Priority|Full
101
102 **Free Tier Economics:**
103
104 * $10 credit = 40-140 articles analyzed (depending on cache hit rate)
105 * Average 70 articles/month at 70% cache hit rate
106 * After limit: Cache-only mode
107
108 ----
109
110 === 2.3 Cache-Only Mode (Free Tier Feature) ===
111
112 When free users reach their $10 monthly limit, they enter **Cache-Only Mode**:
113
114 ==== What Cache-Only Mode Provides: ====
115
116 ✅ **Claim Extraction (Platform-Funded):**
117
118 * Stage 1 extraction runs at $0.003 per article
119 * **Cost: Absorbed by platform** (not charged to user credit)
120 * Rationale: Extraction is necessary to check cache, and cost is negligible
121 * Rate limit: Max 50 extractions/day in cache-only mode (prevents abuse)
122
123 ✅ **Instant Access to Cached Claims:**
124
125 * Any claim that exists in cache → Full verdict returned
126 * Cost: $0 (no LLM calls)
127 * Response time: <100ms
128
129 ✅ **Partial Article Analysis:**
130
131 * Check each claim against cache
132 * Return verdicts for ALL cached claims
133 * For uncached claims: Return "status": "cache_miss"
134
135 ✅ **Cache Coverage Report:**
136
137 * "3 of 5 claims available in cache (60% coverage)"
138 * Links to cached analyses
139 * Estimated cost to complete: $0.162 (2 new claims)
140
141 ❌ **Not Available in Cache-Only Mode:**
142
143 * New claim analysis (Stage 2 LLM calls blocked)
144 * Full holistic assessment (Stage 3 blocked if any claims missing)
145
146 ==== User Experience Example: ====
147
148 {{{{
149 "status": "cache_only_mode",
150 "message": "Monthly credit limit reached. Showing cached results only.",
151 "cache_coverage": {
152 "claims_total": 5,
153 "claims_cached": 3,
154 "claims_missing": 2,
155 "coverage_percent": 60
156 },
157 "cached_claims": [
158 {"claim_id": "C1", "verdict": "Likely", "confidence": 0.82},
159 {"claim_id": "C2", "verdict": "Highly Likely", "confidence": 0.91},
160 {"claim_id": "C4", "verdict": "Unclear", "confidence": 0.55}
161 ],
162 "missing_claims": [
163 {"claim_id": "C3", "claim_text": "...", "estimated_cost": "$0.081"},
164 {"claim_id": "C5", "claim_text": "...", "estimated_cost": "$0.081"}
165 ],
166 "upgrade_options": {
167 "top_up": "$5 for 20-70 more articles",
168 "pro_tier": "$50/month unlimited"
169 }
170 }
171 }}}
172
173 **Design Rationale:**
174
175 * Free users still get value (cached claims often answer their question)
176 * Demonstrates FactHarbor's value (partial results encourage upgrade)
177 * Sustainable for platform (no additional cost)
178 * Fair to all users (everyone contributes to cache)
179
180 ----
181
182 == 3. REST API Contract ==
183
184 === 3.1 User Credit Tracking ===
185
186 **Endpoint:** GET /v1/user/credit
187
188 **Response:** 200 OK
189
190 {{{{
191 "user_id": "user_abc123",
192 "tier": "free",
193 "credit_limit": 10.00,
194 "credit_used": 7.42,
195 "credit_remaining": 2.58,
196 "reset_date": "2025-02-01T00:00:00Z",
197 "cache_only_mode": false,
198 "usage_stats": {
199 "articles_analyzed": 67,
200 "claims_from_cache": 189,
201 "claims_newly_analyzed": 113,
202 "cache_hit_rate": 0.626
203 }
204 }
205 }}}
206
207 ----
208
209 === 3.2 Create Analysis Job (3-Stage) ===
210
211 **Endpoint:** POST /v1/analyze
212
213 ==== Idempotency Support: ====
214
215 To prevent duplicate job creation on network retries, clients SHOULD include:
216
217 {{{POST /v1/analyze
218 Idempotency-Key: {client-generated-uuid}
219 }}}
220
221 OR use the client.request_id field:
222
223 {{{{
224 "input_url": "...",
225 "client": {
226 "request_id": "client-uuid-12345",
227 "source_label": "optional"
228 }
229 }
230 }}}
231
232 **Server Behavior:**
233
234 * If Idempotency-Key or request_id seen before (within 24 hours):
235 ** Return existing job (200 OK, not 202 Accepted)
236 ** Do NOT create duplicate job or charge twice
237 * Idempotency keys expire after 24 hours (matches job retention)
238
239 **Example Response (Idempotent):**
240
241 {{{{
242 "job_id": "01J...ULID",
243 "status": "RUNNING",
244 "idempotent": true,
245 "original_request_at": "2025-12-24T10:31:00Z",
246 "message": "Returning existing job (idempotency key matched)"
247 }
248 }}}
249
250 ==== Request Body: ====
251
252 {{{{
253 "input_type": "url",
254 "input_url": "https://example.com/medical-report-01",
255 "input_text": null,
256 "options": {
257 "browsing": "on",
258 "depth": "standard",
259 "max_claims": 5,
260 "scenarios_per_claim": 2,
261 "max_evidence_per_scenario": 6,
262 "context_aware_analysis": true
263 },
264 "client": {
265 "request_id": "optional-client-tracking-id",
266 "source_label": "optional"
267 }
268 }
269 }}}
270
271 **Options:**
272
273 * browsing: on | off (retrieve web sources or just output queries)
274 * depth: standard | deep (evidence thoroughness)
275 * max_claims: 1-10 (default: **5** for cost control)
276 * scenarios_per_claim: 1-5 (default: **2** for cost control)
277 * max_evidence_per_scenario: 3-10 (default: **6**)
278 * context_aware_analysis: true | false (experimental)
279
280 **Response:** 202 Accepted
281
282 {{{{
283 "job_id": "01J...ULID",
284 "status": "QUEUED",
285 "created_at": "2025-12-24T10:31:00Z",
286 "estimated_cost": 0.114,
287 "cost_breakdown": {
288 "stage1_extraction": 0.003,
289 "stage2_new_claims": 0.081,
290 "stage2_cached_claims": 0.000,
291 "stage3_holistic": 0.030
292 },
293 "cache_info": {
294 "claims_to_extract": 5,
295 "estimated_cache_hits": 4,
296 "estimated_new_claims": 1
297 },
298 "links": {
299 "self": "/v1/jobs/01J...ULID",
300 "result": "/v1/jobs/01J...ULID/result",
301 "report": "/v1/jobs/01J...ULID/report",
302 "events": "/v1/jobs/01J...ULID/events"
303 }
304 }
305 }}}
306
307 **Error Responses:**
308
309 402 Payment Required - Free tier limit reached, cache-only mode
310
311 {{{{
312 "error": "credit_limit_reached",
313 "message": "Monthly credit limit reached. Entering cache-only mode.",
314 "cache_only_mode": true,
315 "credit_remaining": 0.00,
316 "reset_date": "2025-02-01T00:00:00Z",
317 "action": "Resubmit with cache_preference=allow_partial for cached results"
318 }
319 }}}
320
321 ----
322
323 == 4. Data Schemas ==
324
325 === 4.1 Stage 1 Output: ClaimExtraction ===
326
327 {{{{
328 "job_id": "01J...ULID",
329 "stage": "stage1_extraction",
330 "article_metadata": {
331 "title": "Article title",
332 "source_url": "https://example.com/article",
333 "extracted_text_length": 5234,
334 "language": "en"
335 },
336 "claims": [
337 {
338 "claim_id": "C1",
339 "claim_text": "Original claim text from article",
340 "canonical_claim": "Normalized, deduplicated phrasing",
341 "claim_hash": "sha256:abc123...",
342 "is_central_to_thesis": true,
343 "claim_type": "causal",
344 "evaluability": "evaluable",
345 "risk_tier": "B",
346 "domain": "public_health"
347 }
348 ],
349 "article_thesis": "Main argument detected",
350 "cost": 0.003
351 }
352 }}}
353
354 ----
355
356 === 4.5 Verdict Label Taxonomy ===
357
358 FactHarbor uses **three distinct verdict taxonomies** depending on analysis level:
359
360 ==== 4.5.1 Scenario Verdict Labels (Stage 2) ====
361
362 Used for individual scenario verdicts within a claim.
363
364 **Enum Values:**
365
366 * Highly Likely - Probability 0.85-1.0, high confidence
367 * Likely - Probability 0.65-0.84, moderate-high confidence
368 * Unclear - Probability 0.35-0.64, or low confidence
369 * Unlikely - Probability 0.16-0.34, moderate-high confidence
370 * Highly Unlikely - Probability 0.0-0.15, high confidence
371 * Unsubstantiated - Insufficient evidence to determine probability
372
373 ==== 4.5.2 Claim Verdict Labels (Rollup) ====
374
375 Used when summarizing a claim across all scenarios.
376
377 **Enum Values:**
378
379 * Supported - Majority of scenarios are Likely or Highly Likely
380 * Refuted - Majority of scenarios are Unlikely or Highly Unlikely
381 * Inconclusive - Mixed scenarios or majority Unclear/Unsubstantiated
382
383 **Mapping Logic:**
384
385 * If ≥60% scenarios are (Highly Likely | Likely) → Supported
386 * If ≥60% scenarios are (Highly Unlikely | Unlikely) → Refuted
387 * Otherwise → Inconclusive
388
389 ==== 4.5.3 Article Verdict Labels (Stage 3) ====
390
391 Used for holistic article-level assessment.
392
393 **Enum Values:**
394
395 * WELL-SUPPORTED - Article thesis logically follows from supported claims
396 * MISLEADING - Claims may be true but article commits logical fallacies
397 * REFUTED - Central claims are refuted, invalidating thesis
398 * UNCERTAIN - Insufficient evidence or highly mixed claim verdicts
399
400 **Note:** Article verdict considers **claim centrality** (central claims override supporting claims).
401
402 ==== 4.5.4 API Field Mapping ====
403
404 |=Level|=API Field|=Enum Name
405 |Scenario|scenarios[].verdict.label|scenario_verdict_label
406 |Claim|claims[].rollup_verdict (optional)|claim_verdict_label
407 |Article|article_holistic_assessment.overall_verdict|article_verdict_label
408
409 ----
410
411 == 5. Cache Architecture ==
412
413 === 5.1 Redis Cache Design ===
414
415 **Technology:** Redis 7.0+ (in-memory key-value store)
416
417 **Cache Key Schema:**
418
419 {{{claim:v1norm1:{language}:{sha256(canonical_claim)}
420 }}}
421
422 **Example:**
423
424 {{{Claim (English): "COVID vaccines are 95% effective"
425 Canonical: "covid vaccines are 95 percent effective"
426 Language: "en"
427 SHA256: abc123...def456
428 Key: claim:v1norm1:en:abc123...def456
429 }}}
430
431 **Rationale:** Prevents cross-language collisions and enables per-language cache analytics.
432
433 **Data Structure:**
434
435 {{{SET claim:v1norm1:en:abc123...def456 '{...ClaimAnalysis JSON...}'
436 EXPIRE claim:v1norm1:en:abc123...def456 7776000 # 90 days
437 }}}
438
439 ----
440
441 === 5.1.1 Canonical Claim Normalization (v1) ===
442
443 The cache key depends on deterministic claim normalization. All implementations MUST follow this algorithm exactly.
444
445 **Algorithm: Canonical Claim Normalization v1**
446
447 {{{def normalize_claim_v1(claim_text: str, language: str) -> str:
448 """
449 Normalizes claim to canonical form for cache key generation.
450 Version: v1norm1 (POC1)
451 """
452 import re
453 import unicodedata
454
455 # Step 1: Unicode normalization (NFC)
456 text = unicodedata.normalize('NFC', claim_text)
457
458 # Step 2: Lowercase
459 text = text.lower()
460
461 # Step 3: Remove punctuation (except hyphens in words)
462 text = re.sub(r'[^\w\s-]', '', text)
463
464 # Step 4: Normalize whitespace (collapse multiple spaces)
465 text = re.sub(r'\s+', ' ', text).strip()
466
467 # Step 5: Numeric normalization
468 text = text.replace('%', ' percent')
469 # Spell out single-digit numbers
470 num_to_word = {'0':'zero', '1':'one', '2':'two', '3':'three',
471 '4':'four', '5':'five', '6':'six', '7':'seven',
472 '8':'eight', '9':'nine'}
473 for num, word in num_to_word.items():
474 text = re.sub(rf'\b{num}\b', word, text)
475
476 # Step 6: Common abbreviations (English only in v1)
477 if language == 'en':
478 text = text.replace('covid-19', 'covid')
479 text = text.replace('u.s.', 'us')
480 text = text.replace('u.k.', 'uk')
481
482 # Step 7: NO entity normalization in v1
483 # (Trump vs Donald Trump vs President Trump remain distinct)
484
485 return text
486
487 # Version identifier (include in cache namespace)
488 CANONICALIZER_VERSION = "v1norm1"
489 }}}
490
491 **Cache Key Formula (Updated):**
492
493 {{{language = "en"
494 canonical = normalize_claim_v1(claim_text, language)
495 cache_key = f"claim:{CANONICALIZER_VERSION}:{language}:{sha256(canonical)}"
496
497 Example:
498 claim: "COVID-19 vaccines are 95% effective"
499 canonical: "covid vaccines are 95 percent effective"
500 sha256: abc123...def456
501 key: "claim:v1norm1:en:abc123...def456"
502 }}}
503
504 **Cache Metadata MUST Include:**
505
506 {{{{
507 "canonical_claim": "covid vaccines are 95 percent effective",
508 "canonicalizer_version": "v1norm1",
509 "language": "en",
510 "original_claim_samples": ["COVID-19 vaccines are 95% effective"]
511 }
512 }}}
513
514 **Version Upgrade Path:**
515
516 * v1norm1 → v1norm2: Cache namespace changes, old keys remain valid until TTL
517 * v1normN → v2norm1: Major version bump, invalidate all v1 caches
518
519 ----
520
521 === 5.1.2 Copyright & Data Retention Policy ===
522
523 **Evidence Excerpt Storage:**
524
525 To comply with copyright law and fair use principles:
526
527 **What We Store:**
528
529 * **Metadata only:** Title, author, publisher, URL, publication date
530 * **Short excerpts:** Max 25 words per quote, max 3 quotes per evidence item
531 * **Summaries:** AI-generated bullet points (not verbatim text)
532 * **No full articles:** Never store complete article text beyond job processing
533
534 **Total per Cached Claim:**
535
536 * Scenarios: 2 per claim
537 * Evidence items: 6 per scenario (12 total)
538 * Quotes: 3 per evidence × 25 words = 75 words per item
539 * **Maximum stored verbatim text:** ~~900 words per claim (12 × 75)
540
541 **Retention:**
542
543 * Cache TTL: 90 days
544 * Job outputs: 24 hours (then archived or deleted)
545 * No persistent full-text article storage
546
547 **Rationale:**
548
549 * Short excerpts for citation = fair use
550 * Summaries are transformative (not copyrightable)
551 * Limited retention (90 days max)
552 * No commercial republication of excerpts
553
554 **DMCA Compliance:**
555
556 * Cache invalidation endpoint available for rights holders
557 * Contact: dmca@factharbor.org
558
559 ----
560
561 == Summary ==
562
563 This WYSIWYG preview shows the **structure and key sections** of the 1,515-line API specification.
564
565 **Full specification includes:**
566
567 * Complete API endpoints (7 total)
568 * All data schemas (ClaimExtraction, ClaimAnalysis, HolisticAssessment, Complete)
569 * Quality gates & validation rules
570 * LLM configuration for all 3 stages
571 * Implementation notes with code samples
572 * Testing strategy
573 * Cross-references to other pages
574
575 **The complete specification is available in:**
576
577 * FactHarbor_POC1_API_and_Schemas_Spec_v0_4_1_PATCHED.md (45 KB standalone)
578 * Export files (TEST/PRODUCTION) for xWiki import