Docs / Questions & answers
Questions & answers
Choosing thresholds and calibration
An answer becomes a decision at a line you choose. Pick lines per question, start from how often the outcome normally happens, and check them against what really happened.
Forecasts: start from the base rate
Every outcome has a usual frequency. A 5% drawdown within a month is common for a volatile stock and rare for an index; a volume spike is rare for everyone. So the same number means different things: SPY's drop5 was 0.203 and AMD's 0.763 on the same session. Compare an answer with what the model says on quiet, ordinary names before you call it high.
Rules: a line on strength
A rule belief is about how clearly a fact holds. 0.6 is a sensible default for "clearly true" and 0.4 for "clearly false"; the band between is a reading close to its line. Raise the line for filters where a near miss is costly.
Lines scale with what is at stake
| Action | Example line | Why |
|---|---|---|
| Add to a watchlist | up_week >= 0.55 | Cheap to be wrong |
| Post to the desk | drop5 >= 0.6 | People read it |
| Reduce a position | drop5 >= 0.65 and below_50d >= 0.6 | A fact and a forecast agree |
| Block an order | drop5 >= 0.75 | Expensive both ways; few, clear cases |
Check calibration yourself
Log each answer with the outcome once it is known. Group answers into bands (0.5–0.6, 0.6–0.7, …) and compare each band's average answer with how often the outcome happened. Where they agree, the numbers can be read as odds; where they do not, move your line.
import pandas as pd
# rows: symbol, as_of, question, p (the yes_no answer), happened (0/1, filled in later)
df = pd.read_csv("ora_log.csv")
df["band"] = (df.p * 10).astype(int) / 10
print(df.groupby(["question", "band"]).agg(n=("p", "size"), avg_p=("p", "mean"), hit_rate=("happened", "mean")))
The Examples with a real outcome tag (gap fills, paraphrases) let you try this on cases where the answer is already known.
choice and score
Use confidence (0 to 1) to gate them: see Confidence.
Information, not investment advice. Beliefs and probabilities on this page are real responses for the session shown and change with every session.