Skip to Content
Last modified by Robert Schaub on 2025/12/24 09:59
From version 1.1
edited by Robert Schaub
on 2025/12/23 18:19
Change comment: Imported from XAR
To version 2.1
edited by Robert Schaub
on 2025/12/24 09:44
Change comment: Imported from XAR

Summary

Details

Page properties
Content
... ... @@ -160,6 +160,95 @@
160 160  **See:** [[Architecture>>Test.FactHarbor pre10 V0\.9\.70.Specification.Architecture.WebHome]] for details
161 161  
162 162  
163 +== 5. Context-Aware Analysis (Conditional Feature) ==
164 +
165 +**Status:** Depends on POC1 experimental test results
166 +
167 +**Background:**
168 +
169 +POC1 tested context-aware analysis as an experimental feature using Approach 1 (Single-Pass Holistic Analysis). The goal is to detect when articles use accurate individual claims but reach misleading conclusions through faulty logic or selective presentation.
170 +
171 +**See:** [[Article Verdict Problem>>Test.FactHarbor.Specification.POC.Article-Verdict-Problem]] for complete investigation
172 +
173 +=== 5.1 POC2 Implementation Path ===
174 +
175 +**Decision based on POC1 test results (30-article test set):**
176 +
177 +==== If POC1 Accuracy ≥70% (Success) ====
178 +
179 +**Action:** Implement as standard feature (no longer experimental)
180 +
181 +**Enhancement to FR4:**
182 +* Context-aware analysis becomes part of standard Analysis Summary
183 +* Article verdict may differ from simple claim average
184 +* AI evaluates logical structure and reasoning quality
185 +
186 +**Potential Upgrade to Approach 6 (Hybrid):**
187 +* Add weighted claim importance (some claims more central than others)
188 +* Add rule-based fallacy detection alongside AI reasoning
189 +* Combine AI judgment with heuristic checks for robustness
190 +
191 +**Target:** Maintain ≥70% accuracy at detecting misleading articles
192 +
193 +==== If POC1 Accuracy 50-70% (Promising) ====
194 +
195 +**Action:** Implement alternative Approach 4 (Weighted Aggregation)
196 +
197 +**Instead of holistic analysis:**
198 +* AI assigns importance weights (0-1) to each claim
199 +* Weight based on: claim centrality, evidence strength, logical role
200 +* Article verdict = weighted average of claim verdicts
201 +* More structured than pure AI reasoning
202 +
203 +**Rationale:** If holistic reasoning is inconsistent, structured weighting may work better
204 +
205 +==== If POC1 Accuracy <50% (Insufficient) ====
206 +
207 +**Action:** Defer context-aware analysis to post-POC2
208 +
209 +**Fallback:**
210 +* Focus on individual claim accuracy only
211 +* Article verdict = simple average of claim verdicts
212 +* Note limitation: May miss misleading articles built from accurate claims
213 +
214 +**Future consideration:** Try Approach 7 (LLM-as-Judge) with better models in future releases
215 +
216 +=== 5.2 Testing in POC2 ===
217 +
218 +**If context-aware feature is implemented:**
219 +
220 +* Expand test set from 30 to 100 articles
221 +* Include more diverse article types (op-eds, news, analysis, advocacy)
222 +* Track false positive rate (flagging good articles as misleading)
223 +* Validate with subject matter experts when possible
224 +
225 +**Success Metrics:**
226 +* ≥70% accuracy on misleading article detection
227 +* <15% false positive rate
228 +* Reasoning is comprehensible to users
229 +
230 +=== 5.3 Architecture Notes ===
231 +
232 +**Context-aware analysis adds NO additional API calls**
233 +
234 +The enhanced analysis happens within the existing AKEL workflow:
235 +
236 +{{code}}
237 +Standard Flow: Context-Aware Enhancement:
238 +1. Extract claims 1. Extract claims + mark central claims
239 +2. Find evidence 2. Find evidence
240 +3. Generate verdicts 3. Generate verdicts
241 +4. Write summary 4. Write context-aware summary
242 + (evaluates article structure)
243 +{{/code}}
244 +
245 +**Cost:** $0 increase (same API calls, enhanced prompt only)
246 +
247 +**See:** [[POC Requirements>>Test.FactHarbor.Specification.POC.Requirements]] Component 1 for implementation details
248 +
249 +
250 +
251 +
163 163  == Related Pages ==
164 164  
165 165  * [[POC1>>Test.FactHarbor pre10 V0\.9\.70.Roadmap.POC1.WebHome]] - Previous phase