Logit model evaluation

[Juret]

Hey,

How do you go about evaluating the predictive ability of a logistic model for Money Line betting? (on out-of-sample data, of course)

As the outcome is always 0 or 1 (away or home win) there is no, for me, intuitive error measurement. Would you simulate flat or Kelly staking? Or simply compare win rates, perhaps slicing up the data in comparable groups, such as: evenly-matched games, large home favorites etc.

Thanks

[bztips]

You're on the right track. If you're a flat bettor, then simulate flat betting; ditto for Kelly. I have also done what you've mentioned -- looking at data subsets to see if my model does better with some subsets relative to others. I do that to try to discover where my model is weak and possibly give insights into how it might be improved. Of course if you do too much of that there's data-mining risk.

[brettd]

Logit models should come up with a win % probability. If you're not coming up with a probability to achieve either side of a binary event, you're doing something wrong.

Dissect the money line into sub groups that will each have sufficient sample sizes, and then examine ROI% in each sub group.

[Juret]

Thank you, I will proceed with a simulation versus closing lines.

brettd, of course the equation spits out a probability. my concern lied in determining the errors of prediction.

[mathdotcom]

If you have 2009 raw data and 2010 moneyline data then generate your logit predictions for each game using the 2009 data, and force the model to follow a rule that it will bet any game where the model has an edge of x% over the moneyline. From this you can see how well the model would have done in terms of win%, units won, etc. As you increase x your model should do better provided the sample size does not drop significantly.

If you're just talking about goodness of fit, there are a number of Pseudo R-squareds for logit/probit models.