Three weeks against the market, second time round: how the model's Vuelta went

How to read this page

This is written for someone who follows cycling and has no reason to know how a forecasting model or a betting market works. The market's odds are used here as a yardstick for information, not as advice: they are the best public summary of what everyone paying attention thinks will happen, so matching them means the model is capturing about as much as the crowd, and beating them means it knows something the crowd has not priced. Six terms carry the page.

What is and is not in the count. Stage 3 was cancelled in a hailstorm on the Col de Mont-Louis and is not scored. Stages 13, 18 and 19 were missed mornings: no key was chosen and no board was published before the start, so they are excluded rather than reconstructed. That leaves 17 scored stages and 50 published reads. Hidden calibration anchors and after-the-fact "hindsight" runs are never in a settlement number.

Level with the market on the winner, and a very different race from July

The fairest single comparison is the winner market: how much probability each board put on the rider who actually won, averaged over the 17 scored stages, with the model scored the way it was published (equal weight across that morning's reads).

Average winner-market log loss per stage, lower is better. At the Tour the market beat us by 0.46 nats a stage. At the Vuelta the gap is +0.023 in the model's favour, which is a tie. Our favourite won on 22 reads to the market's 19.

Notice the second thing in that table before the first. The market's own score was 2.942 at the Vuelta against 1.944 at the Tour. This was a much harder race to forecast for everyone: the overwhelming favourite left it on stage 8, five stages went to escapes, and three were won by riders the market had at under 1%. So "level with the market" here is level with a market that was itself struggling. That matters for how much credit to take, and it comes back in the section on hard days.

Here is the whole race, day by day: how far above or below the market the model priced each actual winner.

Per-stage winner-market difference in nats: how much more (green) or less (red) probability the model put on the actual winner than the market did, equal weight across that day's published reads. Tags mark the stage classes discussed below. Each stage links to its preview page.

Two things stand out. The good days are big: the model's four best days (stage 1, stage 6, stage 21, stage 2) are each worth more than half a nat, and only one stage all Tour had done that. And the bad days are smaller than the Tour's. The worst day here is −1.40; the Tour had four days worse than that.

Where the model saw something the market did not

The same pattern shows up in the small print. Among riders who finished on a stage podium, these are the ones where the model had been furthest above the market:

Win probability on the rider, model (equal weight across reads) against the devigged market, for podium finishers. Roglič third at Aitana, Johannessen and Debruyne second and third from the escape on stage 20, Gall third in Andorra: the model kept finding the right riders in the places behind the winner, especially on GC days.

Where the model was wrong, and what kind of wrong it was

And the mirror image of the table above: podium finishers where the market had been furthest above the model.

Almost every name here is a rider who finishes at the front of reduced or escape groups: Romele, Nys, Cort, Coquard, Buitrago, Cristian Rodríguez. The market rates them from watching races; the model's features, built on results, kept leaving them outside its top fifteen. This group is the Vuelta's version of the Tour's "hard-man" problem, and it is where the depth deficit below comes from.

Better at the very top, behind the market further down

At the Tour the pattern was clear: the model was behind the market on the winner and the podium, level by ten riders deep and slightly ahead by fifteen. The Vuelta flipped it. For each depth, count how many of the actual top-k finishers each board had in its own top k, summed over the scored stages:

Actual top-k riders captured inside each board's own top k, summed over 17 stages (Tour: 20). Model = its published reads averaged per stage. Rank correlation over the whole field: model 0.445, market 0.465 (Tour: 0.507 against 0.532). On the podium market, priced on 8 stages, the two were level: 0.0520 against 0.0524 mean per-rider log loss.

So the Tour's diagnosis, "we read the race as well as the market but will not commit to our favourite", does not describe the Vuelta. Here the model committed, and was often right to. What it lost was the middle of the board: the 16-rider deficit at fifteen deep is concentrated on the escape days: on stages 7, 14 and 15 the market's card of break specialists found three to six of the top fifteen and our production board found one or two. The next two sections split that out.

Did the model's shyness come back?

The Tour's most fixable defect was under-confidence: when the model said 38% it happened 40% of the time, when it said 60% on a top-five place it happened 82%. Here is the same table for the Vuelta. Each cell reads probability we stated → rate it actually happened, and bold marks a gap of more than eight points.

Calibration of the published boards over all visible reads. The winner head is still a little shy in the 0.30–0.50 band, on a small sample, and is roughly honest above 0.50. The depth heads have gone the other way: when we said 83% for a top-five place it happened 62% of the time. That is over-confidence, not shyness, and it tracks the thinning field: with Pogačar, Pedersen, Groves and twenty others gone, the riders the model kept rating highly for the places were not all there at the end.

We also re-ran the Tour's cheapest fix, a one-parameter "sharpening" of the board fitted leave-one-stage-out. At the Tour it was worth about 0.08 nats a stage on ordinary days. At the Vuelta it is worth nothing: the best fitted setting on all 17 stages is "leave the board alone", and on the ten peloton-settled days sharpening made the held-out score slightly worse (1.444 → 1.495). One race said sharpen, the next said do not. The honest reading is that the miscalibration is not a stable constant to correct but depends on the race and the state of the field, which is an argument for a calibration that watches the current race rather than one number fitted in July.

Escape days still cost us, but far less than in July

After the Tour we named one class of day as the model's structural hole: a shattered, non-mountain stage won by a punchy non-climber from a break, a late move, or a reduced hard-men sprint. The broader version was any day a breakaway decided the stage. The same carves, drawn the same way (by day shape and winner type, never by how we scored), applied to the Vuelta:

• Hard-man days: stages 10, 15, 21. Tronchon from a 68-rider group with the sprinters gone; van Aert from a five-man escape on a rolling day; Johannessen on the Granada circuit with van Aert 65th. Stage 13, van Aert from a two-up, belongs here too but was a missed morning.
• Climbers' breaks: stages 7, 12, 14, 20. Leknessund, Omrzel, Brenner and Landa, all from the escape on mountain days. Stage 19 (Dunbar) belongs here and was also unscored.
• Peloton or GC-settled: the other ten, including the Monaco time trial.

Win LL = average winner log loss, model / market, with the difference in brackets (+ means the model was closer). Podium LL where the market priced a podium. Top-k columns are model-minus-market percentage points of the actual top k captured.

Three things to take from that.

• On the days it is built for, the model beat the market outright. Peloton and GC-settled days: 1.444 against 1.745, a 0.30-nat lead per stage, on ten stages. At the Tour the same frame was a dead heat (+0.033). This is the strongest result of the campaign.
• The escape days hurt about a third as much as they did in July. Break-won days ran −0.43 against the market here, against −1.38 at the Tour; hard-man days −0.39 against −1.00. The reason is not that we got better at them. It is that the market could not see these winners either: Tronchon at 0.7%, Landa 0.8%, Brenner 0.3%, Leknessund 0.4%. At the Tour the break winners were Carapaz twice, Philipsen and Schmid, all riders the market had priced. When the surprise is a genuine surprise, being blind costs little. When the market can see it and we cannot, it costs a lot.
• The depth deficit lives on the escape days. Take the break-won days out and the model is ahead at three and ten deep and within four points at fifteen. On the break days themselves it is nine to twelve points behind at every depth: the market's card of stage hunters populates its top fifteen with the people who actually get in the move, and ours does not.

So the summary sentence from July still stands, with one amendment: we price a bike race at or above market level whenever it is decided by the group we can see, and we lose ground on the days a breakaway decides it, in proportion to how well the market can see the escape.

The day key: right about the kind of day two times in three, and not yet the fix

The Tour review ended with a priority list, and the top item was day-class detection: know what kind of stage it is before it starts, so the model can pull the right kind of rider into its shortlist and so we can say in advance which days to trust. Between the races we built that as a menu of eight day types plus three "macro" types (bunch sprint, GC day, time trial), a labeller that grades each finished stage against the menu, and a serving channel that takes a morning key and uses it to re-rank the shortlist of riders the model considers. Two design decisions matter for reading the results. The key only changes who is in the top fifteen and in what order; the prices still come from the untouched production model, so a wrong key cannot corrupt a probability. And the key is chosen by judgment each morning, from the road book, the news and the market card, then graded that evening. The Vuelta was the first live test.

How good were the keys?

Of 17 keyed stages the key's heaviest item matched the oracle on 8, the oracle's item had some weight on 15, and the average weight on the true item was 0.49. The oracle's own labels across the twenty raced stages were: bunch 8, gc_day 4, res_2 3, res_4 3, itt 2.

One row per scored stage. Model and market columns are the win probability on the actual winner; Δ is the log-loss difference (+ = model closer). "Winner's rank" is where the winner sat on each board. "Top-15 found" is how many of the actual top fifteen each board had in its own top fifteen. The key is the morning's weights; "graded" is the evening verdict with the weight placed on the true item.

The misses have a shape, and it was visible by the first week. On stages 2, 5 and 6 the key put the "interesting" residual item ahead of the boring macro one, and the road went the boring way each time. Corrected for that, stage 7 went the interesting way and the key had 60% on a GC day. From there the rule became a question, "who has a reason to chase?", and it worked on the sprint stages: 8, 9, 10, 11, 16 and 17 were all exact. It did not work on the mountain escapes. Stages 7, 12, 14 and 20 all went to the break; on two of them (7 and 14) the key's majority was still the GC day, and on the other two the GC day still carried 40–45%. The written lesson from La Pandera was the sharpest: a breakaway with the leader's own team in it will not be chased, however good the launchpad, and the market had read that by lunchtime. There was also a labelling quirk: three consecutive escape wins (stages 15, 19 and 20) were graded as the "hard puncheur" item rather than the "mountain escape" one, even on stage 20 where the front fifteen were nearly all climbers. That cost stage 20 its grade.

Did the key help the boards?

This is the question the whole program turns on, and it can be measured cleanly because every prediction file carries both rankings: the production model's order and the key-adjusted order that was actually published. Count the actual top fifteen found by each.

Actual top-fifteen riders inside each board's top fifteen, averaged across that morning's reads (production and published are the same riders priced the same way; only the order differs). These use the model's published ranking, which is a separate ranking head; the depth table earlier orders riders by their win price, as the Tour review did, and that ordering finds fewer of the top fifteen (103) than the ranking head does. "Best single read" is the highest of the morning's reads. The hindsight run re-ran the day afterwards knowing its real shape; on the 13 stages that have one it found 85 against the published boards' 77, the best single read's 93 and the market's 83.

Read plainly: the key moved the shortlist by +5 riders in 255 over the whole race. That is a small positive, and at ten deep it was a small negative (−3). The gains came exactly where the theory said they would, on the two mountain escapes (stages 7 and 14, +1.7 and +1.7) where the front group looked nothing like the production ranking's default, and the losses came on days where the key was wrong and pushed the right riders out (stage 6, −1.0). The market still found more of the top fifteen than either of our boards.

The more encouraging number is the "best single read" column. Across the race the best of our own morning reads found 132 of the actual top fifteen, against the market's 119. The information is inside our own scenario structure. What we could not do, yet, was say in the morning which read to trust. That is the same finding the Tour produced about scenario weighting, now with a mechanism attached.

Did the key tell us which days to trust?

At the Tour the days we had flagged in advance as hard-man days were the days we lost most heavily, which suggested a simple rule: publish with less confidence on flagged days. The Vuelta's version of a flag is a key that puts most of its weight on a residual item. Six days were keyed that way (stages 2, 5, 6, 12, 20, 21); eleven were keyed macro-majority.

The Tour's rule inverted. The days we flagged as unusual were our better days against the market; the days we called ordinary were where we lost. The reason is recall: of the seven escape and hard-man days, the key put its majority on an unusual item on only three (12, 20 and 21). Stages 7, 10, 14 and 15 were all keyed as ordinary days and were not.

So the honest verdict on the fix: the machinery works as designed (the prices were never touched, the re-ranking helped exactly where the key was right about an unusual front), the keying rules improved measurably through the race, and neither yet catches the days that matter most. The mountain-escape question, "will the bunch let it go?", turned out to be about team incentives rather than terrain, and that is now written down as a rule rather than a feeling.

The shadow agent

From stage 16 an automated agent chose its own key each morning, blind to the human one, as a rehearsal for running the daily cycle unattended. It produced 4 live keys. It also replayed the stages before it went live from frozen morning bundles after the fact (a calibration exercise, not a live record), where it was exact on nine of eleven graded stages against the human's five on the same days.

Live, blind keys only. ⚑ marks a key the agent flagged for human review itself. On stage 18 there was no human key at all, because the morning was missed; the agent's key was the only prospective read of the day and it was exact.

GC: the race changed on stage 8, and only one of our heads noticed

For seven stages this was a procession. Pogačar won the prologue, Andorra and Castellón and led by 3:34. Then he crashed on the flat run into Xeraco and abandoned, and the Vuelta became a four-man race that Enric Mas won from Roglič and Gall. That makes the GC boards a test of one thing above all: does the board respond to the race? We had two answers running side by side all month. The main blend, which prices riders from their history and form and does not look at the standings, and the state-aware "GC outlook" head, which does.

Left: the market's board converges on the final order as the race resolves; the blend's stays flat near 0.71, exactly the Tour's frozen-board defect again; the outlook head sits between them. Right: the same story in the price on the winner. From the morning after Aitana, when Mas took red, the blend had him at 7% and never moved; the outlook head had him at 44% that day and 58% going into the last mountain stage; the market went from 33% to 83%. The market's GC prices closed after stage 19, so its line stops there.

Boards paired with the market snapshot that knew the same stages (the market closed its GC prices after stage 19, so the board after stage 20 is not in this table). The outlook head matched the market on the winner, beat it on the podium and at fifteen deep, and trailed it on whole-field ordering because it collapses to near zero on the chasers once it has picked its two leaders.

The last board with a market price to compare against, going into the final mountain stage:

GC win probability after stage 19. The blend had the race leader fourth on its own board, behind Roglič, Gall and Onley, because it never saw the standings and Mas's history is that of a rider who finishes second. The outlook head knew who was winning. Neither head is right alone: the outlook over-concentrates on the leaders and gives third-placed Gall a 1% chance; the blend knows who is good and ignores the race. The Tour review said this needed a corrector over the shared board rather than either head on its own, and the Vuelta says it twice as loudly.

Same defect, two races. The blend's ordering of the GC does not learn from the race in either; the market's does. The outlook head was the best podium estimator on the board at both Grand Tours, market included, and both times it was displayed as a caption rather than used as the price.

What changed between the two Grand Tours, and what did not

The programme this Vuelta wrote for us

In priority order, with the priority argued by the numbers above.

A closing note in the same spirit as July's. The story we expected to tell coming into this race was that the day key would show up as a measurable improvement on the hard days. It did not, and the same records that would have shown the improvement show exactly why not. The story we did not expect was that the model would be level with the market on winners and ahead on ordinary days by a clear margin, and that our best single days would come from refusing the market's breakaway cards on stages where the strongest rider simply rode away. We only know either of those things because every read, every board and every key was written down before the road answered.