How would you evaluate these results?

**Dark Horse** · 02-12-09, 06:28 PM

Welcome to this place.

Ganch is the man to answer this question in the correct mathematical format.

In general, if you're betting mostly dogs (which you probably do to get +111), and if you didn't data-mine, I would think your numbers look good enough to bet.

Would I personally bet it? No. I had to rethink, because my initial reply would be yes. It sounds too much like a blanket approach to me, in which a little error can be magnified dramatically. The past offers remarkably little assurance for the future. So, not knowing the precision of the method, I would prefer to use it as a filter, and get more specific on a game-by-game basis. But mathematically, I'm sure that's not the right response.

**Sinister Cat** · 02-12-09, 06:30 PM

Your Kelly calculation is right. Note that a lot of people bet less than a full kelly.

Whether it should be bet or not depends at least partly on how it was developed. If it is the only system you developed, and then backtested it against pristine data (i.e., not the data used to develop the system), then it is probably good.

And yeah, you can calculate the likelihood that your results could have been achieved by chance by a breakeven handicapper using the method outlined in one of Ganchrow's thread's ( title starts with "a better way to measure..." or something like that). But it seems like a pretty big sample.

Edit: I should add that although your kelly calculation is correct for 52.84% at +111, you need to calculate it on a bet-by-bet basis. With the same edge you will bet less on a game with longer odds, and more on a favorite.

**reno cool** · 02-12-09, 06:46 PM

I'm a bit surprised to see such consistency year in and year out over 250-300 or so games, which worries me a little about how you developed your strategy.

**roasthawg** · 02-12-09, 06:53 PM

I would take that to the bank.

**josie88** · 02-12-09, 08:08 PM

Terrific feedback. The approach started as a challenge to find value with dogs. I looked at obvious things, past performance, home/away, rivalries, etc and came up blank for quite awhile. The problem was I had no way of measuring how much, say, home field advantage or pitcher rest had on a game. If the specific starter had 3 days rest, how did that affect their play? How about starters in general? How about 4 days? What about 5 or 6? Was there any impact at all? If I could just impact win/loss % by a few points!

I found a database that had stats like no other. I learned to write queries and began to learn which stats in general and which specific team stats really affect game outcomes. My intuition told me to keep sample sizes as large as you could, and then apply stats one at a time and see how they affect the ROI. I am quick to throw out small sample sizes. I can't emphasize that enough.

I had a breakthrough when I realized I needed to be looking in ranges of data as well as yes/no type data to find those extra percentage points here and there to tip the approach to a profitable one. From there I manually look at each game to find out reasons to throw it out. Key injuries, key player just back from surgery, new pitcher, really anything that is an unknown to me and I sit those out.

**Data** · 02-12-09, 08:21 PM

This describes developing a system that explains past outcomes. It is as predictive as coin flipping.

**josie88** · 02-12-09, 08:23 PM

I should add that the manual handicapping was only for last year and had little impact anyway. I didn't have any way to apply that to the prior year's games.

**josie88** · 02-12-09, 08:26 PM

Thanks data. How could I avoid that pitfall?

**Data** · 02-12-09, 09:05 PM

You need to test your system against a few prior seasons which were not part of the original dataset. Only those seasons results matter.

**josie88** · 02-12-09, 10:10 PM

By proxy I think I've already done that.

Let me reason this out - please bear with me. I'm still a student of this.

Scenario one is that the system is created with a dataset of one season and applied to fresh data and produces profitable results.

Scenario two is that the system is created with a dataset of five seasons and applied to the same five season dataset and produces similar profitable results over the five seasons.

So, what if the technique to devise the system was scenario two, but upon further examination there was no distinction in results? That is to say, if I had devised the system from scenario one's strategy or two the results are the same?

You see, I can take the system, apply it to any one season and produce the results. My reasoning is, perhaps by coincidence, since the results can be duplicated with smaller datasets and by the whole dataset as well, then I have accomplished my task. The point is (sorry for rambling) had i never incorporated the other season's data, the results would be the same. And we still have the benefit of a nice sample size.

What do you think?

**Data** · 02-12-09, 10:17 PM

Originally posted by josie88

What do you think?

Thus far it is just a wishful thinking. See my previous post.

**Peep** · 02-12-09, 10:40 PM

Agree with Data, that is still mining, albeit with a filtering mech in place.

Having said that, you may still well have something, you may have mined a nugget. I data mine all the time, you do find some nuggets, just not as often as you would think.

This next season where you can test it going forward will tell you a lot.

**josie88** · 02-12-09, 11:57 PM

Agreed. The season's almost upon us. I'll let you know how it goes.

I'd like to thank everyone for their input. I appreciate the straight feedback.

**Dark Horse** · 02-13-09, 12:38 AM

Originally posted by josie88

By proxy I think I've already done that.

Let me reason this out - please bear with me. I'm still a student of this.

Scenario one is that the system is created with a dataset of one season and applied to fresh data and produces profitable results.

Scenario two is that the system is created with a dataset of five seasons and applied to the same five season dataset and produces similar profitable results over the five seasons.

So, what if the technique to devise the system was scenario two, but upon further examination there was no distinction in results? That is to say, if I had devised the system from scenario one's strategy or two the results are the same?

You see, I can take the system, apply it to any one season and produce the results. My reasoning is, perhaps by coincidence, since the results can be duplicated with smaller datasets and by the whole dataset as well, then I have accomplished my task. The point is (sorry for rambling) had i never incorporated the other season's data, the results would be the same. And we still have the benefit of a nice sample size.

What do you think?

The 'proxy' part would be the problem. You can't extract data from a season and then apply your system to that same season. That's like lifting a blindfold to look at a painting, put it back on, and then provide a perfect description of the painting to your astonished self...

You want scenario one, not scenario two. If you don't make that distinction you're data mining and artificially creating a higher winning expectation (upon which to later base your overvalued bet size...; the painful part

)

**smoke a bowl** · 02-14-09, 11:18 AM

FWIW Josie, if you are not currently a winning gambler just keep at it. You have the right thought process and approach to this and will be very successful if you put the work in. You definitely see the big picture when it comes to ways to beat this stuff. As far as your original question goes, it's tough to know whether what you are doing to come up with the plays is predictive for future results w/o knowing more about how you came up with them but my guess would be that you have a good bit of +EV going forward judging by your respect for sample sizes and other thought processes you have expressed here.

**josie88** · 02-15-09, 04:54 PM

Thanks for the encouraging words. I'm going to have to go back and re-do this from data's feedback. I can't include all the year's data. After reading up a bit I realized I fell into a classic trap. Oh, well. lesson learned.

This is why I posted, though. I'm way more committed to finding a way to win than being right.

I am going with the scenario one approach now. I will strip the approach back to it's most basic premise (value dogs) and apply the methodology again. Test data will be 2004 and 2005 data and applied to fresh data in years 2006-2008.

Several thoughts. Since we can't have future unplayed game data to test the premise against, we are substituting played but unknown game results (to the 'system') in seasons 04-07. That should suffice I think.

I can't knock out any games due to injuries, etc. I expect this to marginally have a negative impact on ROI.

Why am I thinking 2 years data to develop and 3 to test? This is a gut call frankly. Any one season can be 'off' in terms of the long run, but I find it much harder to think that two seasons, never mind consecutive, could do the same. Yeah, it's still possible, but less probable.

I'm curious to see what happens. Maybe still a winner, but lower ROI?

feedback welcome! I'll post as I work my way through this.

**xyz** · 02-16-09, 03:12 AM

You may be interested in the topic of cross validation from machine learning: http://en.wikipedia.org/wiki/Cross-validation

**josie88** · 02-18-09, 01:19 PM

Update. The approach didn't make it. Two of the stats turned out to be -EV, one quite significantly.

This is starting to look like a wild goose chase, but I'm going to keep up with it for a bit more.

**u21c3f6** · 02-18-09, 03:13 PM

Originally posted by josie88

Update. The approach didn't make it. Two of the stats turned out to be -EV, one quite significantly.

This is starting to look like a wild goose chase, but I'm going to keep up with it for a bit more.

Not so fast!!! I don't know what stats you are testing, but some of my better "edges" were developed by testing stats that I thought would give me a +EV only to find out the answer was the opposite!!!!

If whatever you are testing is that significantly -EV, doing the opposite may very well be the +EV.

Joe.

**gamble4heisman** · 02-18-09, 10:01 PM

If your numbers are/were accurate (though it sounds like you sorta datamined) at those win percentages and average odds i would bet the SHIT out of it. Taking your numbers if you had a bankroll of 50K, you could bet 1K per game, and maintain a risk of ruin of .01% making over 30K assuming 300 plays per season. you may want to phantom it this year, and maybe try it in 2010 if 2009 goes well. Also are you assuming matchbook odds? if not you're in even better shape. good luck, i love the research.

**josie88** · 02-19-09, 02:21 PM

Yeah, it was datamining even though I would have argued that it wasn't. I learned pretty quick that the method needs to have fresh data to test otherwise it's not predictive of the future. That is a key learning for me that will save me big $$.

It is proving exceedingly difficult to get a workable model that is predictive. It has been fairly easy actually to come with +EV models, but the ROI is so small i.e. <2% that it is essentially unplayable. This is due to the fact (at least this is my working hypothesis until I can prove otherwise) that I am limited to Vegas odds. The database pulls from covers - seems to poll early lines and then repolls closing lines and that's what stays in the database. Often the online books give better odds and since I don't have the time to drive around town looking for the best odds I will likely experience too much slippage.

Maybe 2% in the theoretical world will prove translatable in the real world. I will have to track this with Vegas and Pinny odds at first - see if there's in fact a disparity, and then compare to the database odds. That's lot of work for what could be nothing. I should know pretty quickly if the Vegas lines have any value though.

And, I'm not done yet. Learning as I go, so to speak. One of the giants here in the forum gave me some great advice. The work I'm doing now is taking the coaching and translating the written word into query expressions. Not easy, I assure you. Imagine trying to write a mathematical expression to define a team that's hot for example. What's hot? Being in first place? Won their last three games? Scoring more than 10 runs in their last 2 games? Had more than 40 at bats in their last matchup? A basement team on a 3 game streak? A combination? Or none of the above? Ask 10 people and get 10 different answers. Both teams must be handicapped. Is the query for defining a hot team translatable into a query for a cold team? Are the two queries mutually exclusive of each other or are they dependent? One of the things I'm grappling with is that the definition of a hot team (our example) may be dynamic, i.e. the predictive stats change as the cold team query opponent changes. A hot team query could be worthless against a certain type of pitcher on a cold team for example. The one(s) we want is the one(s) that gives us a predictive edge of course. Which may not be what you or I think it should be. I'm using sabremetrics as a guide but essentially this is a brute force attack for now. Just thinking out loud.

Also, I've learned 5 years of data is really limiting. Twice that would be really helpful.

**reno cool** · 02-19-09, 06:16 PM

Are you saying you feel you can get almost 2% edge using one books lines or after shopping around? If the former is true than thats pretty good.

**gamble4heisman** · 02-19-09, 11:27 PM

Dude if the numbers in your original post are accurate that is GREAT ROI. I did the math for you in my last post, it would be a low risk tremendously profitable model, even more so at the best odds (matchbook). Of course its if your numbers hold.

**Munson15** · 02-20-09, 11:10 PM

Using Matchbook will increase your % no matter what angle you are using.

**josie88** · 02-23-09, 05:58 PM

system development data: 04-05 seasons
test data: 06-08 seasons

Results:

04-05: 1277 - 1057 ROI 1.2%

raw data 06-08: 1381-891 ROI 2.7%

The upward results on the raw data is due to a -1.3% ROI in 2004. All other seasons showed a profit.

Note that I was able to find value in some faves. Assuming laying the moneyline faves to win 100 units and betting 100 units to win the line with dogs (yes, this is not mathematically correct but that's how the database is set up - Kelly punters should do better) here's the result:

2004: -1965
2005: +5615
2006: +3465
2007: +4275
2008: +5880

Well, I think that's it. I've answered my own question. Yes, I would bet it. Even with 2004.

Thanks to all who contributed. Will post again in May with an update.

**josie88** · 05-01-09, 12:16 PM

April results

Well, so far so good. We're at +4.6% ROI. It's way too early to draw any conclusions and I expect the return to drop as 4.6% is probably not sustainable.

A couple of observations. The odds in Vegas are not as different as I thought they would be from online but they are not quite as good. I placed wagers generally at Station Casinos. The impact of this is about half as many games to wager on as the theoretical work would indicate. Not unexpected. There are better deals in town - Jerry's nugget in North Las Vegas is advertising best lines in town and the Hilton is giving a dime line on Saturdays. I just don't have the time...

In regards to the creation of a profitable system, I did some more research and have refined my 'test for predictability' procedure. When I started this, I used 2 years data tested against 3 years raw data. That seems to work well, but I have to tell you it is a slow process. There is a better way and I have adopted this as my default procedure. It's not all that different but the main advantage is it speeds up the whole process.

Instead of 2 datasets, have 3. With 2 datasets, you would have a training set of data and a test set of data. With 3 sets you have a training set, a test-while-training set of data, and the test data. This concept was described by Mark Jurik in a neural net program he wrote called braincel (I have to give credit where credit's due!). I'm sure some are already familiar - I doubt I'm the first to think of this.

Anyway, the advantage is you can now toggle back and forth between the training and test-while-training datasets to quickly explore and optimize your theory without datamining. I suppose one could argue that is perilously close to datamining, but if the results are consistent with the first two datasets in terms of ROI or win% or whatever metric makes sense for your angle, and the test data is also giving similar results you are probably looking good. Mark recommends 40% for training, 40% for test-while-training and the remaining 20% for testing. Frankly I'm not comfortable with that and err on the side of a larger test set at around 30,30,40.

I've had a number of people ask about the database. I'm not sure if that's permitted but if someone in the know would give the green light I'll post the URL.

**Wrecktangle** · 05-01-09, 07:02 PM

Not sure why the cross-validation wiki didn't say this, but two terms you need to look up are: bootstrap and jacknife in the resampling area of stat. I find jacknife works pretty well to isolate factors that will tend overfit in sports. Permutation sets of factors work also.

**josie88** · 10-05-09, 05:05 PM

Final results 2009

I ended up with a ROI of -1.2%.

I think I did OK for my level of experience. This is really the first time I've attacked handicapping with any serious intention. I've learned much along the way. Made a lot of trips to the sportsbooks. Made some embarrassingly stupid mistakes too. And, had a lot of fun. Cheap tuition for my Freshman year. Let's hope my Sophomore year is +EV.

I'm also disappointed of course that I didn't make money out of the gate but even the theoretical work didn't show a profit one season.

Maybe the biggest lesson I learned is making a living at this would be a hard way to make a living indeed. My hat's off to anyone who can do this and pay the bills. For me, this will stay a hobby.

Cheers

**roasthawg** · 10-07-09, 01:44 AM

Originally posted by josie88

I ended up with a ROI of -1.2%.

I think I did OK for my level of experience. This is really the first time I've attacked handicapping with any serious intention. I've learned much along the way. Made a lot of trips to the sportsbooks. Made some embarrassingly stupid mistakes too. And, had a lot of fun. Cheap tuition for my Freshman year. Let's hope my Sophomore year is +EV.

I'm also disappointed of course that I didn't make money out of the gate but even the theoretical work didn't show a profit one season.

Maybe the biggest lesson I learned is making a living at this would be a hard way to make a living indeed. My hat's off to anyone who can do this and pay the bills. For me, this will stay a hobby.

Cheers

Thanks for the update man. It's definitely rough trying to find a way to make money at this... but definitely entertaining if nothing else! Good luck to you in the future.