Wiki source code of Workflows
Version 7.1 by Robert Schaub on 2025/12/15 16:56
Show last authors
| author | version | line-number | content |
|---|---|---|---|
| 1 | = Workflows = | ||
| 2 | |||
| 3 | This page describes the core workflows for content creation, review, and publication in FactHarbor. | ||
| 4 | |||
| 5 | == Overview == | ||
| 6 | |||
| 7 | FactHarbor workflows support three publication modes with risk-based review: | ||
| 8 | |||
| 9 | * **Mode 1 (Draft)**: Internal only, failed quality gates or pending review | ||
| 10 | * **Mode 2 (AI-Generated)**: Public with AI-generated label, passed quality gates | ||
| 11 | * **Mode 3 (Human-Reviewed)**: Public with human-reviewed status, highest trust | ||
| 12 | |||
| 13 | Workflows vary by **Risk Tier** (A/B/C) and **Content Type** (Claim, Scenario, Evidence, Verdict). | ||
| 14 | |||
| 15 | ---- | ||
| 16 | |||
| 17 | == Claim Submission & Publication Workflow == | ||
| 18 | |||
| 19 | === Step 1: Claim Submission === | ||
| 20 | |||
| 21 | **Actor**: Contributor or AKEL | ||
| 22 | |||
| 23 | **Actions**: | ||
| 24 | * Submit claim text | ||
| 25 | * Provide initial sources (optional for human contributors, mandatory for AKEL) | ||
| 26 | * System assigns initial AuthorType (Human or AI) | ||
| 27 | |||
| 28 | **Output**: Claim draft created | ||
| 29 | |||
| 30 | === Step 2: AKEL Processing === | ||
| 31 | |||
| 32 | **Automated Steps**: | ||
| 33 | 1. Claim extraction and normalization | ||
| 34 | 2. Classification (domain, type, evaluability) | ||
| 35 | 3. Risk tier assignment (A/B/C suggested) | ||
| 36 | 4. Initial scenario generation | ||
| 37 | 5. Evidence search | ||
| 38 | 6. **Contradiction search** (mandatory) | ||
| 39 | 7. Quality gate validation | ||
| 40 | |||
| 41 | **Output**: Processed claim with risk tier and quality gate results | ||
| 42 | |||
| 43 | === Step 3: Quality Gate Checkpoint === | ||
| 44 | |||
| 45 | **Gates Evaluated**: | ||
| 46 | * Source quality | ||
| 47 | * Contradiction search completion | ||
| 48 | * Uncertainty quantification | ||
| 49 | * Structural integrity | ||
| 50 | |||
| 51 | **Outcomes**: | ||
| 52 | * **All gates pass** → Proceed to Mode 2 publication (if Tier B or C) | ||
| 53 | * **Any gate fails** → Mode 1 (Draft), flag for human review | ||
| 54 | * **Tier A** → Mode 2 with warnings + auto-escalate to expert queue | ||
| 55 | |||
| 56 | === Step 4: Publication (Risk-Tier Dependent) === | ||
| 57 | |||
| 58 | **Tier C (Low Risk)**: | ||
| 59 | * **Direct to Mode 2**: AI-generated, public, clearly labeled | ||
| 60 | * User can request human review | ||
| 61 | * Sampling audit applies | ||
| 62 | |||
| 63 | **Tier B (Medium Risk)**: | ||
| 64 | * **Direct to Mode 2**: AI-generated, public, clearly labeled | ||
| 65 | * Higher audit sampling rate | ||
| 66 | * High-engagement content may auto-escalate | ||
| 67 | |||
| 68 | **Tier A (High Risk)**: | ||
| 69 | * **Mode 2 with warnings**: AI-generated, public, prominent disclaimers | ||
| 70 | * **Auto-escalated** to expert review queue | ||
| 71 | * User warnings displayed | ||
| 72 | * Highest audit sampling rate | ||
| 73 | |||
| 74 | === Step 5: Human Review (Optional for B/C, Escalated for A) === | ||
| 75 | |||
| 76 | **Triggers**: | ||
| 77 | * User requests review | ||
| 78 | * Audit flags issues | ||
| 79 | * High engagement (Tier B) | ||
| 80 | * Automatic (Tier A) | ||
| 81 | |||
| 82 | **Process**: | ||
| 83 | 1. Reviewer/Expert examines claim | ||
| 84 | 2. Validates quality gates | ||
| 85 | 3. Checks contradiction search results | ||
| 86 | 4. Assesses risk tier appropriateness | ||
| 87 | 5. Decision: Approve, Request Changes, or Reject | ||
| 88 | |||
| 89 | **Outcomes**: | ||
| 90 | * **Approved** → Mode 3 (Human-Reviewed) | ||
| 91 | * **Changes Requested** → Back to contributor or AKEL for revision | ||
| 92 | * **Rejected** → Rejected status with reasoning | ||
| 93 | |||
| 94 | ---- | ||
| 95 | |||
| 96 | == Scenario Creation Workflow == | ||
| 97 | |||
| 98 | === Step 1: Scenario Generation === | ||
| 99 | |||
| 100 | **Automated (AKEL)**: | ||
| 101 | * Generate scenarios for claim | ||
| 102 | * Define boundaries, assumptions, context | ||
| 103 | * Identify evaluation methods | ||
| 104 | |||
| 105 | **Manual (Expert/Reviewer)**: | ||
| 106 | * Create custom scenarios | ||
| 107 | * Refine AKEL-generated scenarios | ||
| 108 | * Add domain-specific nuances | ||
| 109 | |||
| 110 | === Step 2: Scenario Validation === | ||
| 111 | |||
| 112 | **Quality Checks**: | ||
| 113 | * Completeness (definitions, boundaries, assumptions clear) | ||
| 114 | * Relevance to claim | ||
| 115 | * Evaluability | ||
| 116 | * No circular logic | ||
| 117 | |||
| 118 | **Risk Tier Assignment**: | ||
| 119 | * Inherits from parent claim | ||
| 120 | * Can be overridden by expert if scenario increases/decreases risk | ||
| 121 | |||
| 122 | === Step 3: Scenario Publication === | ||
| 123 | |||
| 124 | **Mode 2 (AI-Generated)**: | ||
| 125 | * Tier B/C scenarios can publish immediately | ||
| 126 | * Subject to sampling audits | ||
| 127 | |||
| 128 | **Mode 1 (Draft)**: | ||
| 129 | * Tier A scenarios default to draft | ||
| 130 | * Require expert validation for Mode 2 or Mode 3 | ||
| 131 | |||
| 132 | ---- | ||
| 133 | |||
| 134 | == Evidence Evaluation Workflow == | ||
| 135 | |||
| 136 | === Step 1: Evidence Search & Retrieval === | ||
| 137 | |||
| 138 | **AKEL Actions**: | ||
| 139 | * Search academic databases, reputable media | ||
| 140 | * **Mandatory contradiction search** (counter-evidence, reservations) | ||
| 141 | * Extract metadata (author, date, publication, methodology) | ||
| 142 | * Assess source reliability | ||
| 143 | |||
| 144 | **Quality Requirements**: | ||
| 145 | * Primary sources preferred | ||
| 146 | * Diverse perspectives included | ||
| 147 | * Echo chambers flagged | ||
| 148 | * Conflicting evidence acknowledged | ||
| 149 | |||
| 150 | === Step 2: Evidence Summarization === | ||
| 151 | |||
| 152 | **AKEL Generates**: | ||
| 153 | * Summary of evidence | ||
| 154 | * Relevance assessment | ||
| 155 | * Reliability score | ||
| 156 | * Limitations and caveats | ||
| 157 | * Conflicting evidence summary | ||
| 158 | |||
| 159 | **Quality Gate**: Structural integrity, source quality | ||
| 160 | |||
| 161 | === Step 3: Evidence Review === | ||
| 162 | |||
| 163 | **Reviewer/Expert Validates**: | ||
| 164 | * Accuracy of summaries | ||
| 165 | * Appropriateness of sources | ||
| 166 | * Completeness of contradiction search | ||
| 167 | * Reliability assessments | ||
| 168 | |||
| 169 | **Outcomes**: | ||
| 170 | * **Mode 2**: Evidence summaries published as AI-generated | ||
| 171 | * **Mode 3**: After human validation | ||
| 172 | * **Mode 1**: Failed quality checks or pending expert review | ||
| 173 | |||
| 174 | ---- | ||
| 175 | |||
| 176 | == Verdict Generation Workflow == | ||
| 177 | |||
| 178 | === Step 1: Verdict Computation === | ||
| 179 | |||
| 180 | **AKEL Computes**: | ||
| 181 | * Verdict across scenarios | ||
| 182 | * Confidence scores | ||
| 183 | * Uncertainty quantification | ||
| 184 | * Key assumptions | ||
| 185 | * Limitations | ||
| 186 | |||
| 187 | **Inputs**: | ||
| 188 | * Claim text | ||
| 189 | * Scenario definitions | ||
| 190 | * Evidence assessments | ||
| 191 | * Contradiction search results | ||
| 192 | |||
| 193 | === Step 2: Verdict Validation === | ||
| 194 | |||
| 195 | **Quality Gates**: | ||
| 196 | * All four gates apply (source, contradiction, uncertainty, structure) | ||
| 197 | * Reasoning chain must be traceable | ||
| 198 | * Assumptions must be explicit | ||
| 199 | |||
| 200 | **Risk Tier Check**: | ||
| 201 | * Tier A: Always requires expert validation for Mode 3 | ||
| 202 | * Tier B: Mode 2 allowed, audit sampling | ||
| 203 | * Tier C: Mode 2 default | ||
| 204 | |||
| 205 | === Step 3: Verdict Publication === | ||
| 206 | |||
| 207 | **Mode 2 (AI-Generated Verdict)**: | ||
| 208 | * Clear labeling with confidence scores | ||
| 209 | * Uncertainty disclosure | ||
| 210 | * Links to reasoning trail | ||
| 211 | * User can request expert review | ||
| 212 | |||
| 213 | **Mode 3 (Expert-Validated Verdict)**: | ||
| 214 | * Human reviewer/expert stamp | ||
| 215 | * Additional commentary (optional) | ||
| 216 | * Highest trust level | ||
| 217 | |||
| 218 | ---- | ||
| 219 | |||
| 220 | == Audit Workflow == | ||
| 221 | |||
| 222 | === Step 1: Audit Sampling Selection === | ||
| 223 | |||
| 224 | **Stratified Sampling**: | ||
| 225 | * Risk tier priority (A > B > C) | ||
| 226 | * Low confidence scores | ||
| 227 | * High traffic content | ||
| 228 | * Novel topics | ||
| 229 | * User flags | ||
| 230 | |||
| 231 | **Sampling Rates** (Recommendations): | ||
| 232 | * Tier A: 30-50% | ||
| 233 | * Tier B: 10-20% | ||
| 234 | * Tier C: 5-10% | ||
| 235 | |||
| 236 | === Step 2: Audit Execution === | ||
| 237 | |||
| 238 | **Auditor Actions**: | ||
| 239 | 1. Review sampled AI-generated content | ||
| 240 | 2. Validate quality gates were properly applied | ||
| 241 | 3. Check contradiction search completeness | ||
| 242 | 4. Assess reasoning quality | ||
| 243 | 5. Identify errors or hallucinations | ||
| 244 | |||
| 245 | **Audit Outcome**: | ||
| 246 | * **Pass**: Content remains in Mode 2, logged as validated | ||
| 247 | * **Fail**: Content flagged for review, system improvement triggered | ||
| 248 | |||
| 249 | === Step 3: Feedback Loop === | ||
| 250 | |||
| 251 | **System Improvements**: | ||
| 252 | * Failed audits analyzed for patterns | ||
| 253 | * AKEL parameters adjusted | ||
| 254 | * Quality gates refined | ||
| 255 | * Risk tier assignments recalibrated | ||
| 256 | |||
| 257 | **Transparency**: | ||
| 258 | * Audit statistics published periodically | ||
| 259 | * Patterns shared with community | ||
| 260 | * System improvements documented | ||
| 261 | |||
| 262 | ---- | ||
| 263 | |||
| 264 | == Mode Transition Workflow == | ||
| 265 | |||
| 266 | === Mode 1 → Mode 2 === | ||
| 267 | |||
| 268 | **Requirements**: | ||
| 269 | * All quality gates pass | ||
| 270 | * Risk tier B or C (or A with warnings) | ||
| 271 | * Contradiction search completed | ||
| 272 | |||
| 273 | **Trigger**: Automatic upon quality gate validation | ||
| 274 | |||
| 275 | === Mode 2 → Mode 3 === | ||
| 276 | |||
| 277 | **Requirements**: | ||
| 278 | * Human reviewer/expert validation | ||
| 279 | * Quality standards confirmed | ||
| 280 | * For Tier A: Expert approval required | ||
| 281 | * For Tier B/C: Reviewer approval sufficient | ||
| 282 | |||
| 283 | **Trigger**: Human review completion | ||
| 284 | |||
| 285 | === Mode 3 → Mode 1 (Demotion) === | ||
| 286 | |||
| 287 | **Rare - Only if**: | ||
| 288 | * New evidence contradicts verdict | ||
| 289 | * Error discovered in reasoning | ||
| 290 | * Source retraction | ||
| 291 | |||
| 292 | **Process**: | ||
| 293 | 1. Content flagged for re-evaluation | ||
| 294 | 2. Moved to draft (Mode 1) | ||
| 295 | 3. Re-processed through workflow | ||
| 296 | 4. Reason for demotion documented | ||
| 297 | |||
| 298 | ---- | ||
| 299 | |||
| 300 | == User Actions Across Modes == | ||
| 301 | |||
| 302 | === On Mode 1 (Draft) Content === | ||
| 303 | |||
| 304 | **Contributors**: | ||
| 305 | * Edit their own drafts | ||
| 306 | * Submit for review | ||
| 307 | |||
| 308 | **Reviewers/Experts**: | ||
| 309 | * View and comment | ||
| 310 | * Request changes | ||
| 311 | * Approve for Mode 2 or Mode 3 | ||
| 312 | |||
| 313 | === On Mode 2 (AI-Generated) Content === | ||
| 314 | |||
| 315 | **All Users**: | ||
| 316 | * Read and use content | ||
| 317 | * Request human review | ||
| 318 | * Flag for expert attention | ||
| 319 | * Provide feedback | ||
| 320 | |||
| 321 | **Reviewers/Experts**: | ||
| 322 | * Validate for Mode 3 transition | ||
| 323 | * Edit and refine | ||
| 324 | * Adjust risk tier if needed | ||
| 325 | |||
| 326 | === On Mode 3 (Human-Reviewed) Content === | ||
| 327 | |||
| 328 | **All Users**: | ||
| 329 | * Read with highest confidence | ||
| 330 | * Still can flag if new evidence emerges | ||
| 331 | |||
| 332 | **Reviewers/Experts**: | ||
| 333 | * Update if needed | ||
| 334 | * Trigger re-evaluation if new evidence | ||
| 335 | |||
| 336 | ---- | ||
| 337 | |||
| 338 | == Diagram References == | ||
| 339 | |||
| 340 | === Claim and Scenario Lifecycle (Overview) === | ||
| 341 | |||
| 342 | {{include reference="Test.FactHarborV09.Organisation.Diagrams.Claim and Scenario Lifecycle (Overview).WebHome"/}} | ||
| 343 | |||
| 344 | === Claim and Scenario Workflow === | ||
| 345 | |||
| 346 | {{include reference="Test.FactHarborV09.Specification.Diagrams.Claim and Scenario Workflow.WebHome"/}} | ||
| 347 | |||
| 348 | === Evidence and Verdict Workflow === | ||
| 349 | |||
| 350 | {{include reference="Test.FactHarborV09.Specification.Diagrams.Evidence and Verdict Workflow.WebHome"/}} | ||
| 351 | |||
| 352 | === Quality and Audit Workflow === | ||
| 353 | |||
| 354 | {{include reference="Test.FactHarborV09.Specification.Diagrams.Quality and Audit Workflow.WebHome"/}} | ||
| 355 | |||
| 356 | |||
| 357 | |||
| 358 | {{include reference="Test.FactHarborV09.Specification.Diagrams.Manual vs Automated matrix.WebHome"/}} | ||
| 359 | |||
| 360 | ---- | ||
| 361 | |||
| 362 | == Related Pages == | ||
| 363 | |||
| 364 | * [[AKEL (AI Knowledge Extraction Layer)>>FactHarbor.Specification.AI Knowledge Extraction Layer (AKEL).WebHome]] | ||
| 365 | * [[Automation>>FactHarbor.Specification.Automation.WebHome]] | ||
| 366 | * [[Requirements (Roles)>>FactHarbor.Specification.Requirements.WebHome]] | ||
| 367 | * [[Governance>>FactHarbor.Organisation.Governance]] |