Measuring Line Accuracy

What's the best way to measure line (or win probability) accuracy? The baseball model I use calculates a win probability for each team and places wagers if my calculated line varies from the available line. So in theory, my +100 line plays should win 50% of the time, +110 should win 47.62%, and so on...and obviously I can track this by breaking it down and analyzing each individual number but how do I track the overall accuracy of the model?

Easier with totals or runlines, you can use rmse to measure how far your model is from actual outcomes. For moneylines, an easy way is to sum the expected number of wins of favourites (just average the expected win chance), and compare it with reality.

This question has been asked in so many different ways that it's getting almost humorous. If, and I mean if,some one actually had something meaningful to say, they wouldn't give it up.....end of story. Now sometimes, and it's rare, you might be able to get a scrap if you are paying attention.

Originally Posted by Gingervitis

What's the best way to measure line (or win probability) accuracy? The baseball model I use calculates a win probability for each team and places wagers if my calculated line varies from the available line. So in theory, my +100 line plays should win 50% of the time, +110 should win 47.62%, and so on...and obviously I can track this by breaking it down and analyzing each individual number but how do I track the overall accuracy of the model?

Here's how I do it:

Code: [View]

n:  800, ao: 1.921, oo: 1.995, z: -1.10
n:  800, ao: 1.772, oo: 1.626, z: +2.90
n:  800, ao: 1.645, oo: 1.578, z: +1.49
n:  800, ao: 1.533, oo: 1.476, z: +1.50
n:  800, ao: 1.433, oo: 1.386, z: +1.44
n:  800, ao: 1.343, oo: 1.260, z: +3.18
n:  800, ao: 1.260, oo: 1.223, z: +1.65
n:  800, ao: 1.183, oo: 1.140, z: +2.50
n:  800, ao: 1.111, oo: 1.097, z: +1.06
n:  800, ao: 1.038, oo: 1.032, z: +0.83

p_value: 5.611e-05

In this example, the model provided odds for 8k matches. The odds for the favorite were then grouped into 10 odds ranges, each one receiving 800 matches. Here is the legend for the above table:

n: number of matches in group
ao: average model odds for group
oo: observed odds for group (i.e. n / h, with h being the number of times that the model favorite won)
z: (h - n * p) / sqrt(n*p*q), with p = 1/ao, q = 1-p
p_value: 1 - F(sum(z²)), with sum(z²) being the sum of the squared z values for all groups and F being the CDF of the chi-squared distribution with 10 degrees of freedom, 10 because we used 10 groups.

A p_value less than 0.05 leads to the model being rejected. A rule of thumb: accurate models should have p_value in the range of 0.2 to 0.8.

The motivation for this process is that, if the model is entirely correct then the match results are independent, thus the zs are independent standard normal variates and the sum of their squares has the chi-squared distribution.

Keep in mind that the odds ranges should be wide enough to contain a sufficient number of matches so that the normal approximation to the binomial distribution is acceptable, but narrow enough for each group to be approximated reasonably well by its ao.

I realize that this method is not without its flaws, in fact I have in mind a way that a model can use to "trick" the method into accepting it, but I won't go into detail here, unless other people start.

HeeHaw...so basically calculate the expected number of wins for each wager (ie a +100 line equals 0.5 expected wins), sum these up for all wagers and compare to actual number of wins?

HUY...great stuff much appreciated, I'll give it a shot.