I've been back testing a model that only triggers a few games per year. In fact, there have been only 26 triggers in the past 10-years, and the results are 20-6 How do I determine if this not simply a result of luck?
How to determine if my results are true or just luck!
Collapse
X
-
jbrent95SBR MVP
- 12-07-09
- 1221
#1How to determine if my results are true or just luck!Tags: None -
Justin7SBR Hall of Famer
- 07-31-06
- 8577
#2Do you have a logical explanation of why the results should be favorable?
how many different models did you test? The more ideas you try, the greater the data mining risk.
In those 26 games, how did the lines move? With, against or neutral to your picks?Comment -
mathdotcomSBR Posting Legend
- 03-24-08
- 11689
#3It is simple to answer if all the games were bet at fixed odds (ie. -110), but if the lines are all different it is more complicated.
Justin, he is looking for a confidence interval, not confidenceComment -
jbrent95SBR MVP
- 12-07-09
- 1221
#4All of the games are at fixed odds (-110). Where I find a formula and explanation?Comment -
stafSBR MVP
- 11-11-07
- 2521
#5With only 26 results it's just luck.
You need a minimum of 1000 results before it becomes statistically relevant.Comment -
icecapperSBR Wise Guy
- 09-29-09
- 788
#6Going to have to wait 10 more years to get a better sample.Comment -
Dark HorseSBR Posting Legend
- 12-14-05
- 13764
#8action points.
pretty good check for all smaller sample sizes.Comment -
uva3021SBR Wise Guy
- 03-01-07
- 537
#9depends on the content of the model, if its entirely arbitrary, fundamentally anyway, then i'd say the model merely happened to fit the given structure of randomness at the timeComment -
TakeItSBR Wise Guy
- 04-23-10
- 778
#10I can (and have) come up with dozens of such methods which showed amazing results, few plays, and a long time span.
Don't trust any of em.Comment -
WrecktangleSBR MVP
- 03-01-09
- 1524
#11
Unfortunately, nothing is static in this game for a 10 year period and it tends to render most statistical tests invalid.Comment -
mathdotcomSBR Posting Legend
- 03-24-08
- 11689
#12Wrecktangle, the result of each bet is not normally distributed - it follows Bernoulli. And 26 bets follows the binomial distribution.
The probability of having 20 or more successes on 26 trials (where the probability of success is 0.5) is 0.005. His result is statistically significant.Comment -
MonkeyF0ckerSBR Posting Legend
- 06-12-07
- 12144
#13Even a two-tailed binomial test yields significance at the 1% level.Comment -
bztipsSBR Sharp
- 06-03-10
- 283
#14Statistically significant IF he didn't do any data mining to try to find the "best" model to fit the data.Comment -
MarketMakerRestricted User
- 07-19-10
- 44
#17Wrecktangle, the result of each bet is not normally distributed - it follows Bernoulli. And 26 bets follows the binomial distribution.
The probability of having 20 or more successes on 26 trials (where the probability of success is 0.5) is 0.005. His result is statistically significant.Comment -
WrecktangleSBR MVP
- 03-01-09
- 1524
#18Wrecktangle, the result of each bet is not normally distributed - it follows Bernoulli. And 26 bets follows the binomial distribution.
The probability of having 20 or more successes on 26 trials (where the probability of success is 0.5) is 0.005. His result is statistically significant.
Do not let a significant test settle things, stay on the hunt to improve.Comment -
mathdotcomSBR Posting Legend
- 03-24-08
- 11689
#20Bernoulli seems to be (?) a better distribution. From a significance test view point any distribution is significant (???????), but my real point is, over 10 years with so few observations and the changing nature of the league, etc. it easily can not be a real bettable phenom and time will tell. In the NFL, I had drilled out over 50 situations where my z-scores were over 3, some as high as 5; all eventually melted away.
Do not let a significant test settle things, stay on the hunt to improve.
You really are one of those guys that throws in a bunch of technical terms but really has no idea what any of them mean. First your ridiculous game theory posts, now this? Wrecktangle I hope you don't bet sports.Comment -
jbrent95SBR MVP
- 12-07-09
- 1221
#21If I'm understanding binomial distribution correctly and applying the correct excel formula, for 26 trials,
Given:
CRITBINOM(trials,probability_s,alpha)
0.5 = probability of success for each trial
0.995 = Alpha criterion value
The smallest value for which the cumulative binomial distribution is greater than or equal to the criterion value is 19.
The resulting minimumComment -
wallnbroadSBR Rookie
- 07-31-10
- 4
#22You really do need a larger sample size...but depending what the system is based on, how logical it is, how few degrees of freedom it has....it might be OKComment -
WrecktangleSBR MVP
- 03-01-09
- 1524
#23Mathdotcom. Interesting, I offer a small olive branch, and you use it to go full-on attack. Since you are allowed to come after me with impunity by your mod pal Justin7 (with his attendant modification of my posts lest I hurt your feelings), and since I really don't care much about getting into a "discussion" with a Troll with half-assed math skills who exists to haunt boards looking for fights, I'm done.
Have a nice shitty little life. IGNORE ON.Comment -
mathdotcomSBR Posting Legend
- 03-24-08
- 11689
#24Mathdotcom. Interesting, I offer a small olive branch, and you use it to go full-on attack. Since you are allowed to come after me with impunity by your mod pal Justin7 (with his attendant modification of my posts lest I hurt your feelings), and since I really don't care much about getting into a "discussion" with a Troll with half-assed math skills who exists to haunt boards looking for fights, I'm done.
Have a nice shitty little life. IGNORE ON.At the bolded part.
Anyone can ask a question here, but you should know what you're talking about if you're going to give an answer. I've never heard you say anything valuable in this subforum... ever.Comment -
MizSBR Wise Guy
- 08-30-09
- 695
#25Hi Brent. This question has been asked before. There are very eloquent posts on this topic by people like Ganchrow. However, I worked through a quick and dirty example toward the bottom of the post below. (There is also a lot of good info above my post). I don't claim to be infallible, so if anyone sees an error in what I did, please speak up.
Even with a high z-score, I still wouldn't necessarily bet the largely. Like Wrecktangle mentioned above, I have seen things with high z-scores melt away over time. It may have to do with a dynamic market, like he mentioned, or maybe it is a statistical fluke. Either way, I would advise looking for trends that may be less pronounced, but offer more betting opportunities. Then calculate your z-score... and see where you are. I hope the trend is based on an idea first, instead of every thursday when it is raining, there is a full moon etc etc.
http://forum.sbrforum.com/handicapper-think-tank/366743-sample-sizes.htmlComment
SBR Contests
Collapse
Top-Rated US Sportsbooks
Collapse
#1 BetMGM
4.8/5 BetMGM Bonus Code
#2 FanDuel
4.8/5 FanDuel Promo Code
#3 Caesars
4.8/5 Caesars Promo Code
#4 DraftKings
4.7/5 DraftKings Promo Code
#5 Fanatics
#6 bet365
4.7/5 bet365 Bonus Code
#7 Hard Rock
4.1/5 Hard Rock Bet Promo Code
#8 BetRivers
4.1/5 BetRivers Bonus Code