Changes for page POC2: Robust Quality & Reliability
Last modified by Robert Schaub on 2025/12/24 09:59
Summary
-
Page properties (1 modified, 0 added, 0 removed)
Details
- Page properties
-
- Content
-
... ... @@ -160,95 +160,6 @@ 160 160 **See:** [[Architecture>>Test.FactHarbor pre10 V0\.9\.70.Specification.Architecture.WebHome]] for details 161 161 162 162 163 -== 5. Context-Aware Analysis (Conditional Feature) == 164 - 165 -**Status:** Depends on POC1 experimental test results 166 - 167 -**Background:** 168 - 169 -POC1 tested context-aware analysis as an experimental feature using Approach 1 (Single-Pass Holistic Analysis). The goal is to detect when articles use accurate individual claims but reach misleading conclusions through faulty logic or selective presentation. 170 - 171 -**See:** [[Article Verdict Problem>>Test.FactHarbor.Specification.POC.Article-Verdict-Problem]] for complete investigation 172 - 173 -=== 5.1 POC2 Implementation Path === 174 - 175 -**Decision based on POC1 test results (30-article test set):** 176 - 177 -==== If POC1 Accuracy ≥70% (Success) ==== 178 - 179 -**Action:** Implement as standard feature (no longer experimental) 180 - 181 -**Enhancement to FR4:** 182 -* Context-aware analysis becomes part of standard Analysis Summary 183 -* Article verdict may differ from simple claim average 184 -* AI evaluates logical structure and reasoning quality 185 - 186 -**Potential Upgrade to Approach 6 (Hybrid):** 187 -* Add weighted claim importance (some claims more central than others) 188 -* Add rule-based fallacy detection alongside AI reasoning 189 -* Combine AI judgment with heuristic checks for robustness 190 - 191 -**Target:** Maintain ≥70% accuracy at detecting misleading articles 192 - 193 -==== If POC1 Accuracy 50-70% (Promising) ==== 194 - 195 -**Action:** Implement alternative Approach 4 (Weighted Aggregation) 196 - 197 -**Instead of holistic analysis:** 198 -* AI assigns importance weights (0-1) to each claim 199 -* Weight based on: claim centrality, evidence strength, logical role 200 -* Article verdict = weighted average of claim verdicts 201 -* More structured than pure AI reasoning 202 - 203 -**Rationale:** If holistic reasoning is inconsistent, structured weighting may work better 204 - 205 -==== If POC1 Accuracy <50% (Insufficient) ==== 206 - 207 -**Action:** Defer context-aware analysis to post-POC2 208 - 209 -**Fallback:** 210 -* Focus on individual claim accuracy only 211 -* Article verdict = simple average of claim verdicts 212 -* Note limitation: May miss misleading articles built from accurate claims 213 - 214 -**Future consideration:** Try Approach 7 (LLM-as-Judge) with better models in future releases 215 - 216 -=== 5.2 Testing in POC2 === 217 - 218 -**If context-aware feature is implemented:** 219 - 220 -* Expand test set from 30 to 100 articles 221 -* Include more diverse article types (op-eds, news, analysis, advocacy) 222 -* Track false positive rate (flagging good articles as misleading) 223 -* Validate with subject matter experts when possible 224 - 225 -**Success Metrics:** 226 -* ≥70% accuracy on misleading article detection 227 -* <15% false positive rate 228 -* Reasoning is comprehensible to users 229 - 230 -=== 5.3 Architecture Notes === 231 - 232 -**Context-aware analysis adds NO additional API calls** 233 - 234 -The enhanced analysis happens within the existing AKEL workflow: 235 - 236 -{{code}} 237 -Standard Flow: Context-Aware Enhancement: 238 -1. Extract claims 1. Extract claims + mark central claims 239 -2. Find evidence 2. Find evidence 240 -3. Generate verdicts 3. Generate verdicts 241 -4. Write summary 4. Write context-aware summary 242 - (evaluates article structure) 243 -{{/code}} 244 - 245 -**Cost:** $0 increase (same API calls, enhanced prompt only) 246 - 247 -**See:** [[POC Requirements>>Test.FactHarbor.Specification.POC.Requirements]] Component 1 for implementation details 248 - 249 - 250 - 251 - 252 252 == Related Pages == 253 253 254 254 * [[POC1>>Test.FactHarbor pre10 V0\.9\.70.Roadmap.POC1.WebHome]] - Previous phase