Can Someone Confirm My Happiness?

[Grind-It-Out]

I think I just created a frickin sweet model, but I'm paranoid that I did something wrong with testing that invalidates the data. If someone could verify that I tested correctly that would be hugely appreciated.

The process:
I created a model, backtested using data from April 1st, 2009 - April 30th, 2010, and then made repeated changes to the model until backtesting on that data set yielded the best result. Then, once I was confident in my model, I "forward tested" from May 1st, 2010 through yesterday.

The results (from May 1st onward):
494 - 292, +285.99 units, Z-Score = 6.05 (All bets were between 1 and 5 units)

Since I didn't backtest with the data I used for "forward testing" does that mean that these results are valid? They almost seem too good to be true, which has me worried.

[hutennis]

What is "forward testing" ?

[Grind-It-Out]

What is "forward testing" ?

I don't know if there is such a thing. I didn't know what to called it, so I made up a name. Basically, I just mean that the data set I used to create the model and the data set I used to come up with the results are two different sets.

[sharpcat]

Kind of alarming that you came up with 786 plays in 92 days if this is baseball alone you would have bet 60-70% of the games on the board.

[hutennis]

So it looks like you have not actually tried to place any bets.

Why don't you do that?

Start betting money. Small money. 1 unit = $1.

If your model is good then you'll be quietly making a lot of money gradually increasing stakes and soon will become financially independent.
A lot of money in a bank will be the best possible conformation of your happiness.
No amount of guessing (or trying to find black cat in a dark room when there is no cat there in a first place) you are asking for in your post can even come close.

If after placing real bets you'll see that sigma 6 is nowhere in sight, money are lost and happy dreams of financial independence vanished like fart in a wind then you can come back to forums, disclose details of your now worthless model and ask for help on figuring out why it worked great on paper and failed miserably in real life.
I'm sure, you gonna get all the help you need.

Yep, looks reasonable to me.

[roasthawg]

Sounds solid to me.

[u21c3f6]

I think I just created a frickin sweet model, but I'm paranoid that I did something wrong with testing that invalidates the data. If someone could verify that I tested correctly that would be hugely appreciated.

The process:
I created a model, backtested using data from April 1st, 2009 - April 30th, 2010, and then made repeated changes to the model until backtesting on that data set yielded the best result. Then, once I was confident in my model, I "forward tested" from May 1st, 2010 through yesterday.

The results (from May 1st onward):
494 - 292, +285.99 units, Z-Score = 6.05 (All bets were between 1 and 5 units)

Since I didn't backtest with the data I used for "forward testing" does that mean that these results are valid? They almost seem too good to be true, which has me worried.

I think you need to either base your stats on a flat bet basis or separate your stats by the number of units and then test those results for confidence levels. You may have hit a larger % of 5 unit than 1 unit wagers which may skew your results if you comingle them.

Joe.

PS. You may also have to separate by odds ranges as a winner or two or three at very high odds may also skew your results.

[bztips]

I think you need to either base your stats on a flat bet basis or separate your stats by the number of units and then test those results for confidence levels. You may have hit a larger % of 5 unit than 1 unit wagers which may skew your results if you comingle them.

Joe.

PS. You may also have to separate by odds ranges as a winner or two or three at very high odds may also skew your results.

If you're using Kelly, you would EXPECT to hit a higher % on your larger bets (that's why you bet more )

[Grind-It-Out]

I think you need to either base your stats on a flat bet basis or separate your stats by the number of units and then test those results for confidence levels. You may have hit a larger % of 5 unit than 1 unit wagers which may skew your results if you comingle them. Joe. PS. You may also have to separate by odds ranges as a winner or two or three at very high odds may also skew your results.

The z-score would account for any outliers like that.

That being said, the breakdown by unit size is:

1: 359 - 242 (+101.18)
2: 99 - 41 (+97.28)
3: 23 - 8 (+35.73)
4: 9 - 1 (+31.80)
5: 4 - 0 (+20.00)

Also, there aren't any ridiculously large lines. They all pretty much fall between -200 and +200.

If you're using Kelly, you would EXPECT to hit a higher % on your larger bets (that's why you bet more )

Precisely! Obviously Kelly doesn't recommend a flat 1-5 unit scale, but I'm not good enough (yet) to predict with any more precision than that.

[Indecent]

The process:
I created a model, backtested using data from April 1st, 2009 - April 30th, 2010, and then made repeated changes to the model until backtesting on that data set yielded the best result. Then, once I was confident in my model, I "forward tested" from May 1st, 2010 through yesterday.

Ideally, for each tweak you would use a new set of games (validation set) to compare the results (forward testing as you called it). Tweaking the same set of data would make me concerned of data-mining/overtraining issues. If you had more games to test the model on you would be able to get a better idea of what it could do moving forward. Even games that occurred in previous seasons are new to your model and could be use as a validation set to further verify the model.

With that said, I can't say for sure if you would suffer from data-mining issues given what you've provided.

Gl

[sharpcat]

Ideally, for each tweak you would use a new set of games (validation set) to compare the results (forward testing as you called it). Tweaking the same set of data would make me concerned of data-mining/overtraining issues. If you had more games to test the model on you would be able to get a better idea of what it could do moving forward. Even games that occurred in previous seasons are new to your model and could be use as a validation set to further verify the model.

With that said, I can't say for sure if you would suffer from data-mining issues given what you've provided.

Gl

This is a very good point I completely missed this when I first read this post.

Every time you alter a system you are testing you have to use a new set of data since a pattern noticed in the original set of data may have contributed to your new improved system.

If possible I would look into backtesting prior seasons.

[MarketMaker]

My guess would be that if you made an error it would be using data from a date past the date of the game to find your bets.

For example, if you had data from April 1st 2009 to yesterday and then used all of that data to try to predict a game on May 1st, 2010 you would seemingly have a very successful model. It is important that you only use data up to and not including the date of the event to come up with your prediction.

It is also possible that you just have a really successful model. A win rate of 62.8% is not something I have ever heard of in baseball if your median line is +/-100 but I know it is possible in other sports.

[MarketMaker]

Since I didn't backtest with the data I used for "forward testing" does that mean that these results are valid? They almost seem too good to be true, which has me worried.

This has me worried. So if you didn't backtest with the data you used for "forward testing", and you stated that you backtested with data from April 1st, 2009 - April 30th, 2010, what data did you use in your model beginning on May 1st, 2010?

[Grind-It-Out]

Since I didn't backtest with the data I used for "forward testing" does that mean that these results are valid? They almost seem too good to be true, which has me worried.

This has me worried. So if you didn't backtest with the data you used for "forward testing", and you stated that you backtested with data from April 1st, 2009 - April 30th, 2010, what data did you use in your model beginning on May 1st, 2010?

By "forward testing", I suppose I still meant bactesting, but on a different data set.

I used data from April 1st, 2009 - April 30th, 2010 to build and test the model. Data from May 1st onward was not used in any way, until I was 100% done development. Then, I backtested again from May 1st onward. What I was trying to convey was that I didn't alter the model based on this data.

[MarketMaker]

So from May 1st onward what data did you use?

[Grind-It-Out]

So from May 1st onward what data did you use?

I had data up until August 1st. I knew I didn't want to compromise the results by backtesting on the same data I used for final testing, so I simply separated one data set into two data sets.

[MarketMaker]

I don't think you are understanding what I am getting at. For your predictions for the games specifically on May 1st, 2010 what data did you use?

[djiddish98]

Here's a rephrasing of my own: You're running a test for games on June 1, 2010. Did you use data from April 1, 2009 to May 31, 2010 to test the games on June 1? Or are you using data from April 1, 2009 to August 1, 2010 or May 1, 2010 to August 1, 2010 to test on June 1?

[Grind-It-Out]

I don't think you are understanding what I am getting at. For your predictions for the games specifically on May 1st, 2010 what data did you use?

On May 1st, 2010, I used data from April 1st 2009 - April 30th, 2010.
On May 2nd, 2010, I used data from April 1st 2009 - May 1st, 2010.
etc.

[MarketMaker]

If that is the case your data is valid.

[djiddish98]

Also, is this just moneylines or ML, RL, Total, other derivatives, etc?

[Grind-It-Out]

Also, is this just moneylines or ML, RL, Total, other derivatives, etc?

ML, RL and total.

For the ML and RL, I take the ML if it is -200 or better, and the RL otherwise.

[MarketMaker]

ML, RL and total.

For the ML and RL, I take the ML if it is -200 or better, and the RL otherwise.

What exactly does your model output? Predicted runs for each team?

[djiddish98]

Why is there a ceiling of -200? I would think a model wouldn't really see a RL as more valuable since it is merely derived from the ml, unless the market was somehow over pricing long dog lose by 1 probability.

Also, what line / book are you grading against? Opener, closer, etc?

[mathdotcom]

Sounds rogue

[Grind-It-Out]

Why is there a ceiling of -200? I would think a model wouldn't really see a RL as more valuable since it is merely derived from the ml, unless the market was somehow over pricing long dog lose by 1 probability.

Also, what line / book are you grading against? Opener, closer, etc?

The idea is that I want to introduce as little juice as possible. There is also a ceiling of +200 for the same reason.

I'd rather bet a runline with lines of +120/-130 than a moneyline with lines of +240/-280.

[djiddish98]

I get what you're saying, but what book is offering 40 cent spreads on the ml and 10 cent spreads on the rl?

There is a small window where you might think the ml should be -278, so the rl could become +ev based on your model. However, I would think typically if a ml isn't valuable, the rl isn't either.

What is the average odds bet? How many bets on totals, rl, ml?

[wrongturn]

check if your code has bug. It is too good to be true.

[mathdotcom]

The idea is that I want to introduce as little juice as possible. There is also a ceiling of +200 for the same reason.

I'd rather bet a runline with lines of +120/-130 than a moneyline with lines of +240/-280.

I hope you know why this should not be an absolute statement, otherwise I'm sure your model is wrong.

[Grind-It-Out]

I hope you know why this should not be an absolute statement, otherwise I'm sure your model is wrong.

I don't know how you can say that without knowing anything about my model. My model doesn't tell me how much of an edge a team has, only if they have one or not. The unit size tells me the confidence that an edge exists, not the size of the edge.

So, armed with limited information, I believe my attempt to reduce vig is a valid one. My thought is that the oddsmakers will do a very good job (most of the time) converting the moneyline into a comparable runline, and vice versa.

[lasker]

I can't give technical advice, but I would say just bet small at first and see how it goes. Or don't bet it at all yet, just track the results... If your model can truly produce so many picks at such a high percentage, you'll have your answer soon enough

And let us know if it turns out to be gold or fool's gold!

[CrimsonQueen]

I disagree that the oddsmakers will make the RL and ML comparable all (or even most) of the time. What I'm saying is there's two games on the board RIGHT NOW:
Game 1: (Phillies)
ML = -200 RL -120
Game 2: (Cardinals)
ML = -225 RL -105

My point is sometimes a team is more likely to win the game but not to cover the RL.

[jgilmartin]

Home favorites and away favorites will have different runlines.

[Grind-It-Out]

I disagree that the oddsmakers will make the RL and ML comparable all (or even most) of the time. What I'm saying is there's two games on the board RIGHT NOW:
Game 1: (Phillies)
ML = -200 RL -120
Game 2: (Cardinals)
ML = -225 RL -105

My point is sometimes a team is more likely to win the game but not to cover the RL.

Right, and the books account for this with the lines they set, thus making them comparable.

[sharpcat]

I think books basically convert ML to RL using historical Home/Away 1 run win statistics, but because these are still 2 separate markets even though they move closely together they do not have to move together. Whether to bet the ML or the RL or ML or spread should come down to which side has more value and just because a market has high juice does not mean that you can not find value in it.