Use case · Verification

Verify claims and citations against a source

For each claim an LLM (or a writer) makes, check whether the cited source supports it, contradicts it, or says nothing, and only auto-accept confident supports.

The problem

LLMs cite sources that don't say what they claim, and human-written reports drift from their references too. Checking each claim by hand doesn't scale. An automatic support/contradict/unrelated check catches fabricated and misread citations and sends only the doubtful ones to a reviewer.

How to frame it

First do the cheap, exact part in code: if the answer quotes text, check the quote actually appears in the source; if it doesn't, the citation is fabricated and no model call is needed.

Then send the claim and the cited section as the state and ask one choice question: does the section support the claim, contradict it, or say nothing about it? This is precisely the kind of judgement our model is built on (entailment), which makes it fast and well calibrated. Auto-accept only confident supports; everything else goes to review.

The questions

• relation choice How does the section relate to the claim?

Example request

{
  "model": "ora1",
  "state": {
    "claim": "The company's revenue grew 24% year over year in Q3.",
    "section": "Third-quarter revenue was $3.1 million, up 24% from $2.5 million in the third quarter of last year, driven by new enterprise contracts."
  },
  "questions": {
    "relation": {
      "type": "choice",
      "instructions": "How does the section relate to the claim?",
      "criteria": {
        "supports": "The section states the claim or directly implies that it is true",
        "contradicts": "The section states something that makes the claim false",
        "says_nothing": "The section does not address the claim either way"
      }
    }
  }
}

Turn answers into actions

a = response["answers"]["relation"]
if a["choice"] == "supports" and a["confidence"] >= 0.8:
    status = "verified"
elif a["choice"] == "contradicts" and a["confidence"] >= 0.8:
    status = "contradicted"           # flag loudly: the source says the opposite
else:
    status = "needs_review"           # unsupported or unsure

Tips

• Split long answers into one claim per sentence and check each separately.
• Pass only the cited section, not the whole document; shorter sources give sharper answers.
• Keep the exact-quote check in code; it's free and catches fabrications with certainty.