Backtesting, theoretical and empirical data

I have a question regarding backtesting and introducing a correlation factor for empirical and theoretical data. The testing is for Soccer and currently compiling data to back test for the past 15 seasons (roughly before that time there was a rule change that changed the amount of points for a win).

Let's say in the past 15 years in the English Premier League, teams that had a theoretical probability of winning of 0.62 (based on my spreadsheet score/result predictor), actually won 59% of the time (0.59). In other words you would need to multiply the theoretical probability by 0.952 (0.59/0.62) to convert the theoretical value into its real, empirical value. I called this a correlation factor.

Would it be realistic to apply this correlation factor to the current season, ie. whenever my spreadsheet favours a team to win 62% of the time, should I adjust this figure by the correlation factor of 0.952 for the value of 62%, all the time. Of course, if I did this, I would need to know how much of the difference was distributed to the draw result and how much was distributed to the loss result and multiply them accordingly.

But will this increase the accuracy of my spreadsheet predictions?

To answer that, I think you need to answer: does the correlation factor stay fairly constant throughout each different historical season. Or does it vary greatly from season to season?

I'll have the answer to that once I finish compiling all the data of my 15 year backtest. However, I would love a second opinion. What does everyone think about all this and has anyone done any tests on this themselves? Does my idea sound good in principle or it is seriously flawed in a way that I've yet to discover?

Even if I can't use the data to improve the accuracy of current season predictions I suppose it will still be useful to backtest the accuracy of the spreadsheet predictor in past seasons.

Cheers for any input into this.

Based on the numbers you have given, it seems to me that your model is not very accurate. That is not necessarily too bad. The crucial piece of the puzzle that you did not post is whether or not your estimates were closer than the market's estimates.

How close your estimates were to the markets estimates would be extremely important. If your model is more accurate than what the market predicts theoretically it should be profitable.

Originally Posted by Data

Based on the numbers you have given, it seems to me that your model is not very accurate. That is not necessarily too bad. The crucial piece of the puzzle that you did not post is whether or not your estimates were closer than the market's estimates.

Data, the numbers I gave were random numbers that I made up. I just invented some numbers to give as an example to try to explain what I was trying to do. I haven't actually finished compiling all of the data to backtest for the past 15 years yet.

Was just looking for a second opinion to see whether adding a correlation factor into current season predictions, based on previous seasons data, would be a good idea or not.

Once I've finished the backtest, I'll have a good idea of that myself but just wondering if anyone else has an opinion on the idea of adjusting win/loss/draw % by a correlation factor in principle and whether or not they've tested it themselves before.

As for market estimates, compared to my estimates, I only have this data for the current soccer season and it is not enough to make a conclusion because so few games have been played in the current European Soccer season. Profitable so far but way too early to make a conclusion.

My opinion is no you should not introduce your "correlation factor" into your model. If I understand this correctly, what you are calling the correlation factor is the difference between your prediction and the observed results. Even if your prediction is very very good, just the randomness factor is going to blow the empirical observed results in a distribution around your pedicted value (i.e. above and below in terms of percentage of wins).

To take this difference from one year and multiply it with your prediction is in effect changing your model, and the result could just as easily (due to randomness) have been a little higher instead of lower (in win percentage) than your prediction.

So I would say you should track this difference that you are calling "correlation factor" over multiple trials or years and the closer it gets to zero is an indication that your model is getting stronger.

However, you have to be extremely careful not to overfit your model to the data (often called data mining), that is you could just play with the numbers until you get this "correlation factor" to zero but have a model that is actually ineffective because you have basically used the data to predict itself.

I think a better word for the computation you are describing is "error." That is the difference between the predicted and the actual -- and the real question is how much of this error is due to randomness and how much is due to a shortcoming of the model. If the observed result continues to fall on one side or the other of the observed results then this would indicate that some of the error is not random, and you should look for factors to add, or take away, or weight differently to adjust the model, but you shouldn't just blindly alter your prediction in the direction that would bring your prediction closer to the observed result.

just my two cents!

Thanks Pedro.

That's exactly the sort of opinion I was looking for.

Once I finish my backtest, the main thing I was looking to find out was how to interpret the data I find, how to react to it. Thanks for your advice on this issue. I've done about 11 seasons of the backtest, just got to finish compiling the data for the final 4 seasons.

Thanks for the points! I am saving up for Justin7's new book and I almost have enough!

here is something from wikipedia about systematic vs random error that I was trying to write about above


http://en.wikipedia.org/wiki/Type_I_error#Type_I_error

Originally Posted by wikipedia

Statistical error vs. systematic error


Scientists recognize two different sorts of error:[Note 1]

• Statistical error: the difference between a computed, estimated, or measured value and the true, specified, or theoretically correct value (see errors and residuals in statistics) that is caused by random, and inherently unpredictable fluctuations in the measurement apparatus or the system being studied.[Note 2]
• Systematic error: the difference between a computed, estimated, or measured value and the true, specified, or theoretically correct value that is caused by non-random fluctuations from an unknown source (see uncertainty), and which, once identified, can usually be eliminated.[Note 2]

"can usually be eliminated" with regard to systematic error is probably too strong in our case with sports handicapping, but it is the error that we hope to eliminate.

I have some more that I hope to add later about matching up the metric that I think might also relate to your original question.

To identify Systematic error, compare to closing market lines. Justin's book does talk about these issues.

Go get that book Pedro.

Flight, thank you so much for all them points

Looks like I can go order that book now!

I really appreciate it

backtesting is freaking awesome. Make it work.