How to determine if my results are true or just luck!

Collapse
X
 
  • Time
  • Show
Clear All
new posts
  • jbrent95
    SBR MVP
    • 12-07-09
    • 1221

    #1
    How to determine if my results are true or just luck!
    I've been back testing a model that only triggers a few games per year. In fact, there have been only 26 triggers in the past 10-years, and the results are 20-6 How do I determine if this not simply a result of luck?
  • Justin7
    SBR Hall of Famer
    • 07-31-06
    • 8577

    #2
    Do you have a logical explanation of why the results should be favorable?

    how many different models did you test? The more ideas you try, the greater the data mining risk.

    In those 26 games, how did the lines move? With, against or neutral to your picks?
    Comment
    • mathdotcom
      SBR Posting Legend
      • 03-24-08
      • 11689

      #3
      It is simple to answer if all the games were bet at fixed odds (ie. -110), but if the lines are all different it is more complicated.

      Justin, he is looking for a confidence interval, not confidence
      Comment
      • jbrent95
        SBR MVP
        • 12-07-09
        • 1221

        #4
        All of the games are at fixed odds (-110). Where I find a formula and explanation?
        Comment
        • staf
          SBR MVP
          • 11-11-07
          • 2521

          #5
          With only 26 results it's just luck.
          You need a minimum of 1000 results before it becomes statistically relevant.
          Comment
          • icecapper
            SBR Wise Guy
            • 09-29-09
            • 788

            #6
            Going to have to wait 10 more years to get a better sample.
            Comment
            • RickySteve
              Restricted User
              • 01-31-06
              • 3415

              #7
              Originally posted by icecapper
              Going to have to wait 10 more years to get a better sample.
              I'd suggest 30 just to be safe.
              Comment
              • Dark Horse
                SBR Posting Legend
                • 12-14-05
                • 13764

                #8
                action points.

                pretty good check for all smaller sample sizes.
                Comment
                • uva3021
                  SBR Wise Guy
                  • 03-01-07
                  • 537

                  #9
                  depends on the content of the model, if its entirely arbitrary, fundamentally anyway, then i'd say the model merely happened to fit the given structure of randomness at the time
                  Comment
                  • TakeIt
                    SBR Wise Guy
                    • 04-23-10
                    • 778

                    #10
                    I can (and have) come up with dozens of such methods which showed amazing results, few plays, and a long time span.

                    Don't trust any of em.
                    Comment
                    • Wrecktangle
                      SBR MVP
                      • 03-01-09
                      • 1524

                      #11
                      Originally posted by jbrent95
                      I've been back testing a model that only triggers a few games per year. In fact, there have been only 26 triggers in the past 10-years, and the results are 20-6 How do I determine if this not simply a result of luck?
                      With under 100 observations you would use something called a Student's T Test. You can Google it and also find sites that will allow you to enter the data and after you run it, will tell you if you have a "statistically valid" amount of observations at the 95% confidence level (two standard deviations = 1/20 chance) or above. Unfortunately, with League / Coach changes and trends over 10 years, I'm of the opinion that you will not get an answer you can really trust. Plus, in the last 10 years the market has significantly worsened against the bettor.

                      Unfortunately, nothing is static in this game for a 10 year period and it tends to render most statistical tests invalid.
                      Comment
                      • mathdotcom
                        SBR Posting Legend
                        • 03-24-08
                        • 11689

                        #12
                        Wrecktangle, the result of each bet is not normally distributed - it follows Bernoulli. And 26 bets follows the binomial distribution.

                        The probability of having 20 or more successes on 26 trials (where the probability of success is 0.5) is 0.005. His result is statistically significant.
                        Comment
                        • MonkeyF0cker
                          SBR Posting Legend
                          • 06-12-07
                          • 12144

                          #13
                          Even a two-tailed binomial test yields significance at the 1% level.
                          Comment
                          • bztips
                            SBR Sharp
                            • 06-03-10
                            • 283

                            #14
                            Statistically significant IF he didn't do any data mining to try to find the "best" model to fit the data.
                            Comment
                            • MonkeyF0cker
                              SBR Posting Legend
                              • 06-12-07
                              • 12144

                              #15
                              Originally posted by bztips
                              Statistically significant IF he didn't do any data mining to try to find the "best" model to fit the data.
                              If that were the case, it isn't a result of luck though either.
                              Comment
                              • bztips
                                SBR Sharp
                                • 06-03-10
                                • 283

                                #16
                                Originally posted by MonkeyF0cker
                                If that were the case, it isn't a result of luck though either.
                                That depends on the model -- if it's a sunspot model, it's luck.
                                Comment
                                • MarketMaker
                                  Restricted User
                                  • 07-19-10
                                  • 44

                                  #17
                                  Originally posted by mathdotcom
                                  Wrecktangle, the result of each bet is not normally distributed - it follows Bernoulli. And 26 bets follows the binomial distribution.

                                  The probability of having 20 or more successes on 26 trials (where the probability of success is 0.5) is 0.005. His result is statistically significant.
                                  That depends doesn't it. I would think statistical significance would greatly be affected depending on if this was his first model tested as opposed to if he had tested 1000 models prior to finding the one that produced the above results.
                                  Comment
                                  • Wrecktangle
                                    SBR MVP
                                    • 03-01-09
                                    • 1524

                                    #18
                                    Originally posted by mathdotcom
                                    Wrecktangle, the result of each bet is not normally distributed - it follows Bernoulli. And 26 bets follows the binomial distribution.

                                    The probability of having 20 or more successes on 26 trials (where the probability of success is 0.5) is 0.005. His result is statistically significant.
                                    Bernoulli seems to be a better distribution. From a significance test view point any distribution is significant, but my real point is, over 10 years with so few observations and the changing nature of the league, etc. it easily can not be a real bettable phenom and time will tell. In the NFL, I had drilled out over 50 situations where my z-scores were over 3, some as high as 5; all eventually melted away.

                                    Do not let a significant test settle things, stay on the hunt to improve.
                                    Comment
                                    • hutennis
                                      SBR Wise Guy
                                      • 07-11-10
                                      • 847

                                      #19
                                      Originally posted by Wrecktangle
                                      In the NFL, I had drilled out over 50 situations where my z-scores were over 3, some as high as 5; all eventually melted away.
                                      Have you kept on following the same thing?
                                      If yes, how can the virtual certainty melt away?
                                      Comment
                                      • mathdotcom
                                        SBR Posting Legend
                                        • 03-24-08
                                        • 11689

                                        #20
                                        Originally posted by Wrecktangle
                                        Bernoulli seems to be (?) a better distribution. From a significance test view point any distribution is significant (???????), but my real point is, over 10 years with so few observations and the changing nature of the league, etc. it easily can not be a real bettable phenom and time will tell. In the NFL, I had drilled out over 50 situations where my z-scores were over 3, some as high as 5; all eventually melted away.

                                        Do not let a significant test settle things, stay on the hunt to improve.
                                        The question he asked was not "are my results significant and will it make me a profit tomorrow?". It was "is this result significant?".

                                        You really are one of those guys that throws in a bunch of technical terms but really has no idea what any of them mean. First your ridiculous game theory posts, now this? Wrecktangle I hope you don't bet sports.
                                        Comment
                                        • jbrent95
                                          SBR MVP
                                          • 12-07-09
                                          • 1221

                                          #21
                                          If I'm understanding binomial distribution correctly and applying the correct excel formula, for 26 trials,
                                          Given:
                                          CRITBINOM(trials,probability_s,alpha)
                                          0.5 = probability of success for each trial
                                          0.995 = Alpha criterion value

                                          The smallest value for which the cumulative binomial distribution is greater than or equal to the criterion value is 19.



                                          The resulting minimum
                                          Comment
                                          • wallnbroad
                                            SBR Rookie
                                            • 07-31-10
                                            • 4

                                            #22
                                            You really do need a larger sample size...but depending what the system is based on, how logical it is, how few degrees of freedom it has....it might be OK
                                            Comment
                                            • Wrecktangle
                                              SBR MVP
                                              • 03-01-09
                                              • 1524

                                              #23
                                              Mathdotcom. Interesting, I offer a small olive branch, and you use it to go full-on attack. Since you are allowed to come after me with impunity by your mod pal Justin7 (with his attendant modification of my posts lest I hurt your feelings), and since I really don't care much about getting into a "discussion" with a Troll with half-assed math skills who exists to haunt boards looking for fights, I'm done.

                                              Have a nice shitty little life. IGNORE ON.
                                              Comment
                                              • mathdotcom
                                                SBR Posting Legend
                                                • 03-24-08
                                                • 11689

                                                #24
                                                Originally posted by Wrecktangle
                                                Mathdotcom. Interesting, I offer a small olive branch, and you use it to go full-on attack. Since you are allowed to come after me with impunity by your mod pal Justin7 (with his attendant modification of my posts lest I hurt your feelings), and since I really don't care much about getting into a "discussion" with a Troll with half-assed math skills who exists to haunt boards looking for fights, I'm done.

                                                Have a nice shitty little life. IGNORE ON.
                                                At the bolded part.

                                                Anyone can ask a question here, but you should know what you're talking about if you're going to give an answer. I've never heard you say anything valuable in this subforum... ever.
                                                Comment
                                                • Miz
                                                  SBR Wise Guy
                                                  • 08-30-09
                                                  • 695

                                                  #25
                                                  Hi Brent. This question has been asked before. There are very eloquent posts on this topic by people like Ganchrow. However, I worked through a quick and dirty example toward the bottom of the post below. (There is also a lot of good info above my post). I don't claim to be infallible, so if anyone sees an error in what I did, please speak up.

                                                  Even with a high z-score, I still wouldn't necessarily bet the largely. Like Wrecktangle mentioned above, I have seen things with high z-scores melt away over time. It may have to do with a dynamic market, like he mentioned, or maybe it is a statistical fluke. Either way, I would advise looking for trends that may be less pronounced, but offer more betting opportunities. Then calculate your z-score... and see where you are. I hope the trend is based on an idea first, instead of every thursday when it is raining, there is a full moon etc etc.

                                                  http://forum.sbrforum.com/handicapper-think-tank/366743-sample-sizes.html
                                                  Comment
                                                  SBR Contests
                                                  Collapse
                                                  Top-Rated US Sportsbooks
                                                  Collapse
                                                  Working...