[jbrent95]

I've been back testing a model that only triggers a few games per year. In fact, there have been only 26 triggers in the past 10-years, and the results are 20-6 How do I determine if this not simply a result of luck?

[Justin7]

Do you have a logical explanation of why the results should be favorable?

how many different models did you test? The more ideas you try, the greater the data mining risk.

In those 26 games, how did the lines move? With, against or neutral to your picks?

[mathdotcom]

It is simple to answer if all the games were bet at fixed odds (ie. -110), but if the lines are all different it is more complicated.

Justin, he is looking for a confidence interval, not confidence

[jbrent95]

All of the games are at fixed odds (-110). Where I find a formula and explanation?

[staf]

With only 26 results it's just luck.
You need a minimum of 1000 results before it becomes statistically relevant.

[icecapper]

Going to have to wait 10 more years to get a better sample.

[RickySteve]

Going to have to wait 10 more years to get a better sample.

I'd suggest 30 just to be safe.

[Dark Horse]

action points.

pretty good check for all smaller sample sizes.

[uva3021]

depends on the content of the model, if its entirely arbitrary, fundamentally anyway, then i'd say the model merely happened to fit the given structure of randomness at the time

[TakeIt]

I can (and have) come up with dozens of such methods which showed amazing results, few plays, and a long time span.

Don't trust any of em.

[Wrecktangle]

I've been back testing a model that only triggers a few games per year. In fact, there have been only 26 triggers in the past 10-years, and the results are 20-6 How do I determine if this not simply a result of luck?

With under 100 observations you would use something called a Student's T Test. You can Google it and also find sites that will allow you to enter the data and after you run it, will tell you if you have a "statistically valid" amount of observations at the 95% confidence level (two standard deviations = 1/20 chance) or above. Unfortunately, with League / Coach changes and trends over 10 years, I'm of the opinion that you will not get an answer you can really trust. Plus, in the last 10 years the market has significantly worsened against the bettor.

Unfortunately, nothing is static in this game for a 10 year period and it tends to render most statistical tests invalid.

[mathdotcom]

Wrecktangle, the result of each bet is not normally distributed - it follows Bernoulli. And 26 bets follows the binomial distribution.

The probability of having 20 or more successes on 26 trials (where the probability of success is 0.5) is 0.005. His result is statistically significant.

[MonkeyF0cker]

Even a two-tailed binomial test yields significance at the 1% level.

[bztips]

Statistically significant IF he didn't do any data mining to try to find the "best" model to fit the data.

[MonkeyF0cker]

Statistically significant IF he didn't do any data mining to try to find the "best" model to fit the data.

If that were the case, it isn't a result of luck though either.

[bztips]

If that were the case, it isn't a result of luck though either.

That depends on the model -- if it's a sunspot model, it's luck.

[MarketMaker]

Wrecktangle, the result of each bet is not normally distributed - it follows Bernoulli. And 26 bets follows the binomial distribution.

The probability of having 20 or more successes on 26 trials (where the probability of success is 0.5) is 0.005. His result is statistically significant.

That depends doesn't it. I would think statistical significance would greatly be affected depending on if this was his first model tested as opposed to if he had tested 1000 models prior to finding the one that produced the above results.

[Wrecktangle]

Wrecktangle, the result of each bet is not normally distributed - it follows Bernoulli. And 26 bets follows the binomial distribution.

The probability of having 20 or more successes on 26 trials (where the probability of success is 0.5) is 0.005. His result is statistically significant.

Bernoulli seems to be a better distribution. From a significance test view point any distribution is significant, but my real point is, over 10 years with so few observations and the changing nature of the league, etc. it easily can not be a real bettable phenom and time will tell. In the NFL, I had drilled out over 50 situations where my z-scores were over 3, some as high as 5; all eventually melted away.

Do not let a significant test settle things, stay on the hunt to improve.

[hutennis]

In the NFL, I had drilled out over 50 situations where my z-scores were over 3, some as high as 5; all eventually melted away.

Have you kept on following the same thing?
If yes, how can the virtual certainty melt away?

[mathdotcom]

Bernoulli seems to be (?) a better distribution. From a significance test view point any distribution is significant (???????), but my real point is, over 10 years with so few observations and the changing nature of the league, etc. it easily can not be a real bettable phenom and time will tell. In the NFL, I had drilled out over 50 situations where my z-scores were over 3, some as high as 5; all eventually melted away.

Do not let a significant test settle things, stay on the hunt to improve.

The question he asked was not "are my results significant and will it make me a profit tomorrow?". It was "is this result significant?".

You really are one of those guys that throws in a bunch of technical terms but really has no idea what any of them mean. First your ridiculous game theory posts, now this? Wrecktangle I hope you don't bet sports.

[jbrent95]

If I'm understanding binomial distribution correctly and applying the correct excel formula, for 26 trials,
Given:
CRITBINOM(trials,probability_s,alpha)
0.5 = probability of success for each trial
0.995 = Alpha criterion value

The smallest value for which the cumulative binomial distribution is greater than or equal to the criterion value is 19.



The resulting minimum

[wallnbroad]

You really do need a larger sample size...but depending what the system is based on, how logical it is, how few degrees of freedom it has....it might be OK

[Wrecktangle]

Mathdotcom. Interesting, I offer a small olive branch, and you use it to go full-on attack. Since you are allowed to come after me with impunity by your mod pal Justin7 (with his attendant modification of my posts lest I hurt your feelings), and since I really don't care much about getting into a "discussion" with a Troll with half-assed math skills who exists to haunt boards looking for fights, I'm done.

Have a nice shitty little life. IGNORE ON.

[mathdotcom]

Mathdotcom. Interesting, I offer a small olive branch, and you use it to go full-on attack. Since you are allowed to come after me with impunity by your mod pal Justin7 (with his attendant modification of my posts lest I hurt your feelings), and since I really don't care much about getting into a "discussion" with a Troll with half-assed math skills who exists to haunt boards looking for fights, I'm done.

Have a nice shitty little life. IGNORE ON.

At the bolded part.

Anyone can ask a question here, but you should know what you're talking about if you're going to give an answer. I've never heard you say anything valuable in this subforum... ever.

[Miz]

Hi Brent. This question has been asked before. There are very eloquent posts on this topic by people like Ganchrow. However, I worked through a quick and dirty example toward the bottom of the post below. (There is also a lot of good info above my post). I don't claim to be infallible, so if anyone sees an error in what I did, please speak up.

Even with a high z-score, I still wouldn't necessarily bet the largely. Like Wrecktangle mentioned above, I have seen things with high z-scores melt away over time. It may have to do with a dynamic market, like he mentioned, or maybe it is a statistical fluke. Either way, I would advise looking for trends that may be less pronounced, but offer more betting opportunities. Then calculate your z-score... and see where you are. I hope the trend is based on an idea first, instead of every thursday when it is raining, there is a full moon etc etc.

http://www.sportsbookreview.com/forum/handicapper-think-tank/366743-sample-sizes.html