[Cheese1976]

I hear a lot of chatter regarding sample sizes being too new or insufficient - any ideas on what kind of a number is adequate/reliable?

[ForgetWallStreet]

I hear a lot of chatter regarding sample sizes being too new or insufficient - any ideas on what kind of a number is adequate/reliable?

With respect to what?

[brettd]

Hey i've got one... can i make a UFC model with detailed stats on only 110 fights?

[sharpcat]

My opinion 1,000 samples

[Wrecktangle]

For Gaussian distributed things, one can not be able to say much about anything until you get 30 samples or more. You may want to read up on z-scores, confidence intervals, etc.

Sharp, if you wait for 1000 samples, you'll never do much of anything.

[u21c3f6]

There is no one-size fits all answer for this question. For example, you would need a lot more trials to prove a profitable edge of 10% for a system that has a 10% winning % by chance compared to a system that has a 50% winning % by chance.

Here is how I look at it:

Let's assume ATS wagers which have a random chance of 50% wins.

The std dev would be sqrt(.5 * .5 * # of trials) = 1 std dev

Your results should fall at least more than 1 std dev to continue testing and eventually IMO you want your results to be greater than 2.5 (3 is better) std dev.

Joe.

[Cheese1976]

There is no one-size fits all answer for this question. For example, you would need a lot more trials to prove a profitable edge of 10% for a system that has a 10% winning % by chance compared to a system that has a 50% winning % by chance. Here is how I look at it: Let's assume ATS wagers which have a random chance of 50% wins. The std dev would be sqrt(.5 * .5 * # of trials) = 1 std dev Your results should fall at least more than 1 std dev to continue testing and eventually IMO you want your results to be greater than 2.5 (3 is better) std dev. Joe.

This is very helpful thank you! I was wondering about this, because I have certain situations that are quite rare but show extremely good promise. Thanks again for the formula

[Cheese1976]

I'm looking at this one:

40-15 (73%)

Using your formula gives me:

sqrt of(.73 * .73 * 55) = 5.41

Am I doing this right or should I be using .27 instead of .73 because the sample of games still seems too small to trust?

[u21c3f6]

I'm looking at this one:

40-15 (73%)

Using your formula gives me:

sqrt of(.73 * .73 * 55) = 5.41

Am I doing this right or should I be using .27 instead of .73 because the sample of games still seems too small to trust?

Yes, you should be using .27 * .73. The formula is the chance of losing * the chance of winning. The way I use it though is to do the calculation based on the winning % by chance. Then I compare my results to the chance results to see how many std dev I am away from chance.

Joe.

[Dark Horse]

The formula for the Z-score (not including juice) is:

(wins-losses)/square root of sample size.

So for 40-15 it is (40-15)/sqrt (40+15) = 25/7.41 = 3.37

I would suggest to have at least a 100 samples, preferably more, before trusting this type of score. Unless the sample is spread out over a longer period of time. The idea is to rule out the effect of short term fluctuations, which, of course, also occurs over a long stretch of time. There are many angles that will produce spectacularly for a season, only to drop off the chart the next season.

[Cheese1976]

The formula for the Z-score (not including juice) is: (wins-losses)/square root of sample size. So for 40-15 it is (40-15)/sqrt (40+15) = 25/7.41 = 3.37 I would suggest to have at least a 100 samples, preferably more, before trusting this type of score. Unless the sample is spread out over a longer period of time. The idea is to rule out the effect of short term fluctuations, which, of course, also occurs over a long stretch of time. There are many angles that will produce spectacularly for a season, only to drop off the chart the next season.

Totally...thanks for this. I'm trying to find several situations like this one and play the ones that are overlapping - I've noticed this overlapping almost certainly increases the win %

[Cheese1976]

btw if anyone's intersted this 40-15 set comes from the following combined criteria:

1) NBA totals that haven't moved or come back to equal the opening line within an hour of game time
2) From the "team expert" consensus page at covers.com, selcting games in which team experts are not between 45%-70%
3) Tracked since 12/1/09 - eliminating the first month of the season deliberately

"Under" record by month as follows:

December: 10-4 (71%)
January: 19-3 (86%)
February: 11-5 (69%)
March: 0-3 (0%)

[Flying Dutchman]

Cheese, interesting system. It sounds like it is datamined, i.e. you found it after searching thru a lot of data. How long have you been tracking it day-by-day after for found it? And what was the record during that time?

[Peregrine Stoop]

OP, you should really get a stats and probability book and learn about the distributions you are using.
Lots of pain to follow if you half ass it.

[Dark Horse]

When dealing with the sample size problem you can cut to the chase with action points. Action points show you by how many points an angle beats the spread. If you bet over 202 and the game ends on 212, that's 10 action points along with the W. If your angle beats the spread by 10 pts consistently you know you've got something. For greater definition I would keep track of wins and losses separately. For instance, your record may be 19-16, with action points for wins +10, and action points for losses -2.5. This suggests that the angle is stronger than the actual W/L record. Keeping track of action points also allows you to see when and to what extent an angle loses steam.

[Cheese1976]

When dealing with the sample size problem you can cut to the chase with action points. Action points show you by how many points an angle beats the spread. If you bet over 202 and the game ends on 212, that's 10 action points along with the W. If your angle beats the spread by 10 pts consistently you know you've got something. For greater definition I would keep track of wins and losses separately. For instance, your record may be 19-16, with action points for wins +10, and action points for losses -2.5. This suggests that the angle is stronger than the actual W/L record. Keeping track of action points also allows you to see when and to what extent an angle loses steam.

Love this - never really thought to consider margin of win/loss by. Adding columns to excel now, thanks!

[marcoforte]

My suggestion for sample size is at least 60 with a z value of 2.5 or preferably 3. I follow the 60/60 rule for datamining. 60% win rate in at least 60 samples. Then go to the z value.

[Cheese1976]

This is such great information! Glad I started this thread - Thank you so much everyone.

[mrmarket]

Is the sample size requirement based on your total W/L record or the types of bets you place?

[20Four7]

I think we will all know the answer once fishhead gives it to us..... I'm waiting till then.

DH, I knew you used action points but that post probably explained it the best. Nice work.

[Miz]

http://www.fourmilab.ch/rpkp/experim...sis/zCalc.html

if you want to skip the calculator and get your probability you just do this in excel:

=1-NORMSDIST(Z-Score)

This, of course, assumes a normal distribution.

[Wrecktangle]

When dealing with the sample size problem you can cut to the chase with action points. Action points show you by how many points an angle beats the spread. If you bet over 202 and the game ends on 212, that's 10 action points along with the W. If your angle beats the spread by 10 pts consistently you know you've got something. For greater definition I would keep track of wins and losses separately. For instance, your record may be 19-16, with action points for wins +10, and action points for losses -2.5. This suggests that the angle is stronger than the actual W/L record. Keeping track of action points also allows you to see when and to what extent an angle loses steam.

Instead of "action points" (a gambling term), use residuals when you google.

An examination of your residuals whenever you are fitting a "model" of any type is always a very good idea as it tells you where your biases are. It helps greatly in finding a solution knowing that you have a problem in the first place.

[Miz]

my 2 cents on residuals. I think it is important to make a diminishing returns function... maybe for m > l ... m' = l + (m-l)^0.5

or something similar. Where you don't get an artificially skewed impression for blowouts (in either direction)

m = margin of victory, l = limit before diminshing returns function is implemented

[Cheese1976]

http://www.fourmilab.ch/rpkp/experim...sis/zCalc.html if you want to skip the calculator and get your probability you just do this in excel: =1-NORMSDIST(Z-Score) This, of course, assumes a normal distribution.

Hey Miz - I'm not so well educated - I plugged this formula into excel with a random z score of "3" - it returns a value of .00135 - what is that number telling me?

[Miz]

Sorry, I was at the beach for the first time in like 4 months. We have had a record winter in FL this year...

From the link I mentioned:

One of the most common probability calculations is determining, given the measured z value from an experiment or set of experiments, the probability of the result being due to chance.

So...

The equation above performs the function of the calculator on the link, directly in excel assuming a normal distribution. I keep a running calculation of this while I track my record. So a z-score as high as 3 is very good, and tells you that there is a 0.135% chance that luck alone caused your results.

My MLB totals record from 2009 was 78-61-9 or 56.12% over 139 games (I toss out the pushes), which was a pretty good year for me honestly. Some of it was likely due to luck as the record varies year to year. Anyway, all you can do is try to quantify the likelihood that luck played a significant role in your results.

In the example, you get (139/4)^0.5 = 5.89

and you get Z = [78 - (139/2)] / 5.89

So Z in the example is 1.442... (far below your 3)

Then if you plug it into the equation Q = 1 - P, with P being (NORMSDIST(Z)).

Q represents the possibility that luck alone created your results.

Q = ~7.5% in this example.

Not exactly the warm fuzzy feeling you need to bet large sums of money is it?

Nope, so I can improve my resolution by performing more backtesting and increasing my sample size. So to make it simple, if we just went forward with it for another year instead of backtesting, and the winning % continued this way in 2010 for the record to be 156-122 over 2009 and 2010, then I could conclude that over those two years the chance that luck alone created those results would be

Q = ~2.1%

This is still not enough for me to feel good about it. To me, I want Q <0.01 (1%),

Even if you are good enough to get Q < 1%, that still means that for every 100 mopes with a system, one is going to fool himself into thinking his system is great, and probably regress rather quickly as soon as he ups his amounts.

So, I think it is best to first divise a well thought out approach, then create your model, then back test it on an out of sample set of data.

If you don't do this, you will come up with 100 systems, and dwell on the one that just happens to perform well over some limited set of data, and you won't really have accomplished anything. This is pretty much what datamining is.

I may be mis-stating some of this, and I'm sure the think tank math police will let me know if I am, but this is how I interpret it. Anyone can feel free to chime in and correct anything I have stated incorrectly.

Ganchrow was far more knowledgeable and has many posts that explain this more eloquently.

I suggest hitting wikipedia and dusting off your stats books, as probabilities are crucial to judging performance.

[Cheese1976]

Wow! Thank you so much for the explanation, I think I understand now - I already keep three columns in a running total (wins,losses,win %). I suppose I can just enter these formulas (Z and Q) in two columns to the right of "win %" to ensure I always have an accurate read on it.

[Miz]

yeah, it isn't the end all be all, but is good to be aware of. I've asked the same question before and this is probably the best approach for doing what you're trying to do. good luck

[LT Profits]

This entire thread needs to be nominated, nominating just one post inside it would be an injustice to some other posts.

[Cheese1976]

yeah, it isn't the end all be all, but is good to be aware of. I've asked the same question before and this is probably the best approach for doing what you're trying to do. good luck

Well, your insight is much appreciated. If I start producing some consistent winners I'll be happy to share them with you.

[Miz]

You're welcome. Don't just take my word for it though. I encourage you to look it up and read about it. I have mistakenly taken things like this at face value, only to find out later that the author was incorrectly parroting something he had heard somewhere else, or goofed the math, or just made an honest mistake.

The worst case scenario is if incorrect information in some post on the internet made you abandon a potentially profitable idea.

Some popular ideas I have read that just seem to be generally accepted as gospel, I simply don't agree with. I guess, you just have to convince yourself it is correct before you drink the koolaid.

[frankbettor]

I would say sample sizes of 10 years worth of games played in MLB, for example, would represent an adequate sample size.

[Cheese1976]

I would say sample sizes of 10 years worth of games played in MLB, for example, would represent an adequate sample size.

I might be dead in ten years.

[reno cool]

Cheese, interesting system. It sounds like it is datamined, i.e. you found it after searching thru a lot of data. How long have you been tracking it day-by-day after for found it? And what was the record during that time?

that's the important question. I hope the answer isn't March.

[Cheese1976]

that's the important question. I hope the answer isn't March.

started playing it small in mid december - it has fallen off as of late. That's kind of why I started this thread...

[Wrecktangle]

my 2 cents on residuals. I think it is important to make a diminishing returns function... maybe for m > l ... m' = l + (m-l)^0.5

or something similar. Where you don't get an artificially skewed impression for blowouts (in either direction)

m = margin of victory, l = limit before diminshing returns function is implemented

Well, this is the point of least squares in regression.

My point on residuals is, for example, you use a linear equation to fit non-linear data (action points used earlier in this thread would be a similar notion), residuals would show that it is not a good fit, so further research is need to get a better model (non-linear of some sort) to get a closer fit, which usually is shown thru Mean Squared Error.

I typically don't use regression, but in my world, transforms of some sort are need to get the fit closer, and residuals are the signal that tells me when and where.