Benchmarks

Ora vs JEV on verified financial questions

Head-to-head runs on questions with a checkable answer: labels computed from real bars by exact rules, never by a model. Every number below comes from a saved report file; methodology, sample sizes and the places where JEV wins are on the page.

Technical questions: Ora 86.5% · JEV 80.7% with indicator values · JEV 70.2% with raw bars (1,000 questions, 2026-10-31)

Latest run · 2026-10-31

Technical knowledge, 1,000 held-out questions

Questions drawn from held-out splits of the technical knowledge bank: unseen wordings (474) and unseen stocks (526). This set is now burned: it is published here and no longer used to test Ora.

86.5%Ora accuracy, all 1,000 (Brier 0.127)

97.2%Ora on the 778 it answered with a rule

80.7%JEV with indicator values (Brier 0.129)

70.2%JEV with raw bars (Brier 0.197)

Ora (rule engine + question compiler)86.5%

JEV, given indicator values80.7%

JEV, given raw bars70.2%

By subcategory

Accuracy, then Brier score (lower is better). The best accuracy in each row is marked. JEV is ahead on Volume, Volatility, Levels, Performance.

Questions	n	Ora (rule engine + question compiler)	JEV, given indicator values	JEV, given raw bars
All	1,000	86.5% 0.127	80.7% 0.129	70.2% 0.197
Ora answered with a rule	778	97.2% 0.091	78.8% 0.139	69.0% 0.203
Ora declined (scored 0.5)	222	49.1% 0.250	87.4% 0.094	74.3% 0.177
Basics	125	95.2% 0.093	84.0% 0.140	72.0% 0.179
Trend	125	92.8% 0.111	76.8% 0.147	75.2% 0.191
Momentum	125	88.8% 0.103	88.8% 0.076	65.6% 0.211
Volume	125	79.2% 0.130	93.6% 0.065	67.2% 0.198
Volatility	125	80.8% 0.160	84.8% 0.104	68.8% 0.204
Levels	125	82.4% 0.138	72.8% 0.167	87.2% 0.116
Patterns	125	98.4% 0.118	50.4% 0.289	48.0% 0.327
Performance	125	74.4% 0.160	94.4% 0.043	77.6% 0.149

What each model received

Ora got the question in English and the symbol's daily bars up to the question date (the same data its API reads). It compiles the question to a rule (Ora Rules v1) and evaluates it. When it cannot compile a question it declines (no rule); a decline is scored as 0.5: wrong for accuracy, 0.25 for Brier. Ora declined 222 of 1,000.
JEV, raw bars: the same question with the last 260 daily bars (date, open, high, low, close, volume) as its state; mean 13,569 input tokens, median latency 194 ms.
JEV, indicator values: the same question with the precomputed indicator values for that day and a legend of what each key means; mean 4,319 input tokens, median latency 145 ms. JEV has no market data of its own, so it is given the numbers.
Labels are computed from the bars by exact rules (verified against TA-Lib), not by a model. Accuracy counts a probability above 0.5 as yes.

Download the questions

All 1,000 questions with their labels and every model's answer (645 symbols, sessions 2011-04-05 to 2026-10-30): CSV · JSONL. Fields: id, symbol, as_of, subcategory, question, label, ora_belief (empty = declined), ora_rule, jev_raw_bars_p, jev_indicators_p.

2026-10-29

Forecasts: 800 held-out questions about what happens next

Will it be up in five sessions, draw down 5% within a month, beat SPY? Questions whose answer is unknown at the time, scored against what then happened. fin_test (800 held-out forecast questions, 16 question families x 50).

66.0%Ora accuracy (Brier 0.196)

60.9%JEV accuracy (Brier 0.225)

4.6 msOra median latency (JEV 55.4 ms)

65%of answers on the same side

Ora66.0%

JEV60.9%

By question

50 questions each. Accuracy, then Brier. Forecasting is hard: a few families are below a coin flip for both models. JEV is ahead on 2 of 16: outperform_spy_5d, up_1d.

Question family	n	Ora	JEV
`big_move_5d` Did the stock make an unusually large move (more than one standard deviation) over the following five sessions?	50	68% 0.219	62% 0.209
`drawdown_5pct_20d` Did the price fall 5% or more below the current close at any point over the following twenty sessions?	50	66% 0.198	58% 0.236
`gap_down_1d` Will it gap down tomorrow?	50	92% 0.079	92% 0.111
`gap_up_1d` Did the next session open more than 1% above the current close (gap up)?	50	92% 0.071	90% 0.093
`new_high_20d_5d` Will it make a new 20-day high in the next week?	50	78% 0.139	68% 0.197
`new_low_20d_5d` Did the price break below its 20-session low within the following five sessions?	50	74% 0.160	72% 0.182
`outperform_spy_20d` Did the stock outperform the market (SPY) over the following twenty sessions?	50	38% 0.266	32% 0.299
`outperform_spy_5d` Did the stock outperform the market (SPY) over the following five sessions?	50	38% 0.265	42% 0.320
`rally_5pct_20d` Will it rally at least 5% at some point in the next month?	50	64% 0.231	58% 0.236
`up_1d` Does it close up the next day?	50	50% 0.250	60% 0.234
`up_20d` Will the stock be higher in a month?	50	74% 0.221	52% 0.322
`up_5d` Did the price close higher after five sessions?	50	60% 0.247	58% 0.278
`vol_down_5d` Did realized volatility decrease over the following five sessions?	50	64% 0.217	54% 0.239
`vol_up_20d` Will volatility be higher over the next month?	50	46% 0.249	40% 0.270
`vol_up_5d` Does realised volatility go up in the next week?	50	60% 0.253	48% 0.267
`volume_spike_1d` Was next session's volume more than twice the 20-session average?	50	92% 0.073	88% 0.105

What each model received

Both models got the same request: the question and the symbol's context for that day (returns, volatility, volume and trend measures). Ora answered with the ora-fin-0.1.1 forecast model; JEV with TypeSafe jev-latest.
accuracy = the provider's own yes/no label vs the realised outcome (a probability of exactly 0.50 is labelled yes); Brier = mean (p - outcome)^2. JEV answered exactly 0.50 on 37 questions; those count as yes.
The 800 items are not published: this forecast test set is still used to test new versions.

Methodology

Held out. Technical questions come from held-out splits of Ora's knowledge bank: wordings and stocks Ora was never trained on. Forecast questions are dated after the training window.
Verified labels. A technical label is the exact rule applied to the bars (for example the close against its 20-day SMA), checked against TA-Lib and an independent pure-Python implementation. A forecast label is what the bars did afterwards.
Same information. JEV has no market data, so it receives the bars or the indicator values in its state. Ora reads the same bars from its own store.
Scoring. Accuracy (a probability above 0.5 is yes) and Brier score, the mean squared error of the probability (0 is perfect; always answering 0.5 scores 0.25). Ora's declines are scored as 0.5.
Where this does not apply. These are market questions. On general text, JEV is ahead of Ora's text model; that is not what this page measures.
Ora rule beliefs are strengths, not probabilities (spec section 6); they are scored as probabilities here, which, if anything, penalises Ora's Brier.

Rerun it

The scripts that produced these numbers, in the Ora source tree:

# technical head-to-head (needs bars + a TypeSafe API key)
ORA_BARS=data/fin/knowledge/bars.parquet python fin/knowledge/h2h_jev_prep.py --n 1000
python fin/knowledge/h2h_jev_run.py --workers 6            # -> data/fin/knowledge/h2h_jev_report.json
# a fresh run on new questions writes h2h_jev_<tag>_report.json; it appears here on the next build
# forecasts: Admin > Benchmarks (deploy/bench.sh) on fin_test.jsonl
node web/bin/build-growth.js                               # rebuild this page

Answers are model outputs for information only, not investment advice. JEV is a product of TypeSafe; results are from its public API on the dates shown.

Create an account Open the Playground Use Ora from AI agents