Backtesting - How much is too much?

[kosti]

Working through some ATS stuff for each team in the NBA. I can get information as far back as 1990 (21 years) for everything that I need (lines, scores, etc.)...but I'm starting to wonder how much backtesting do I need?

Obviously difficult to answer that without knowing the specifics of what I'm doing but just curious as to how far back you guys tend to go when backtesting an idea or how you determine how far back you need to go? What is a meaningful sample size?

[hutennis]

Working through some ATS stuff for each team in the NBA. I can get information as far back as 1990 (21 years) for everything that I need (lines, scores, etc.)...but I'm starting to wonder how much backtesting do I need?

Obviously difficult to answer that without knowing the specifics of what I'm doing but just curious as to how far back you guys tend to go when backtesting an idea or how you determine how far back you need to go? What is a meaningful sample size?

There is no meaningful sample size for past data. No matter how much data you looked at, all you are getting from it is past patterns that every one else who is interested in the same subject has already seen.
A lot of very smart people don't see how the argument that all past information is already incorporated in the current price can be defeated, but if you don't believe it you are welcome to try to use your back-testing to predict future for the purpose of making money.

Now you will need a meaningful sample size, but not for your back testing, but for your predictions.
When your have large enough number of predictions with a statistically significant positive result, then you know your are good.

[evo34]

It's always a tradeoff of recency (more relevant and more likely to be predictive) and quantity. I think the betting markets change rapidly enough that you really wouldn't want to take too seriously data from more than ~5 years ago. I think the key is to look for decay over time, which would indicate the anomaly is getting priced in and will go away. All this is assuming you have a large enough sample (at least 500 games) and have not overfitted anything.

[evo34]

but if you don't believe it you are welcome to try to use your back-testing to predict future for the purpose of making money.

I've used backtesting to make a living trading and betting for the last 10 years. Thanks for the idea.

[hutennis]

I've used backtesting to make a living trading and betting for the last 10 years. Thanks for the idea.

We kinda have to take your word for it but regardless whether you telling the truth or not, backtesting may or may not be a reason you were able to make a living betting and trading.

People pray for rain to come for thousands of years, that does not mean praying is the reason rain drops start falling.

[VLR100]

hutennis, I think if someone can make a living solely by trading and betting over 10 years, there is a statistically significant probability (I'm guessing > .99) that this was not solely due to chance (or some other reason not related to his backtesting, etc).

Of course there is indeed a chance that a simple dart thrower could make that much money, but in a world where we're too busy to care whether this is the one outcome out of ten thousand which *may* occur, most people use a simple heuristic faced with such proof; they accept a causal relationship.

[evo34]

Rain dancing has been working for me for 10 years, apparently. No idea why I spend so much time researching, as praying is just as likely to work. Curious: have you ever made money betting in any 24-month period? Or are you too busy forward-testing your ideas on paper and waiting until they show significance...after which time of course most ideas will have entirely lost their edge. At least you can say, "look, I had something"...though your bank account will never reflect it.

[hutennis]

hutennis, I think if someone can make a living solely by trading and betting over 10 years, there is a statistically significant probability (I'm guessing > .99) that this was not solely due to chance (or some other reason not related to his backtesting, etc).

Of course there is indeed a chance that a simple dart thrower could make that much money, but in a world where we're too busy to care whether this is the one outcome out of ten thousand which *may* occur, most people use a simple heuristic faced with such proof; they accept a causal relationship.

Yeah, that's all very nice and interesting but I have another very simple heuristic for you.

The claim of making money over 10 years using back testing has a very significant probability (I'm guessing > .99) to be nothing more but an empty claim.
And since "claim without proof can be dismissed without proof" I always do just that - I ignore such claims.
That's the best course of action in general and for me personally.

Of course, nobody expect full doc posted on a public forum, but, thankfully, there is a much simpler way.

Posting picks ahead of time with odds for both sides and time stamp for verification should paint a complete picture fairly fast.
Especially if back testing model is so robust as it allows for such a consistent and meaningful results.

Anyone who is not willing to back his claims up with so little effort is simply making fool of himself making claims, b/c there is no way any one
using smallest amount of common sense can take those claims seriously.

[VLR100]

Indeed you're right, without the evidence it is nothing but an empty claim. But if he was winning as much as he said he was, he's not doing it via dumb luck.

[hutennis]

Indeed you're right, without the evidence it is nothing but an empty claim. But if he was winning as much as he said he was, he's not doing it via dumb luck.

It's a pretty big BUT, don't you say?

In this case we do have some records though.
I just clicked on "work" link in his post.

It's is not 10 year record. Just year and a half or so.
Also, if that's all he makes, it is a pretty miserable living, if you ask me, but that's beyond the point.
He does have a winning record.

The only thing is, we are missing an avg IP of his bets and without it there is no way to properly evaluate his results.
I don't want to make an assumption, but if I would assume a 50% on average then there is 26% chance to get where he is simply by luck alone.

Even if he bets dogs most of the time (and he does not) and his Avg IP is 49% then probability of luck alone (13%) is not even close to were it should be to start considering his back testing pride seriously.

Of course, 51% avg implied will get us right to a skill level of a drunk monkey.
51% is like taking -109 in -109/-101 line on avg and it looks pretty close to being the case based on what I glanced over. I'm not sure though, so I don't want to speculate. Let's call it a possibility.

[roasthawg]

The more the better with an emphasis on recent seasons. Sometimes the more sophisticated models take a long time to run... it might take a few days or more to run a season's worth of games.

[Dark Horse]

It depends on the clarity of the idea or combination of ideas that you are backtesting. If those ideas are extremely clear and accurate, you would, in theory, not have to backtest at all.

[DukeJohn]

I try and use 10 years of back testing on new data for most sports, although I only did 5 years for the NBA when I started with NBA a few years ago.

BoL,

[cyberbabble]

Some years may not be representative - NBA short season last year.

Small rules changes/playing styles probably don't have much effect over a few years, but they can add up over time. I read an academic article about NFL quarterback stats over 30+ years. Completion percent went up and interception percent went down over the period. There was a slow but steady drift over time and probably enough to invalidate the older stats. Interesting that average yards per pass didn't change much.

Take a 3(or 4) year moving average of a stat and see how stable it is. Run a regression on the stat and see if it trended up or down.

[evo34]

hutennis, where's your record? If it's so easy to track, why not show it? Also, why the silence when I asked if you have ever made money over any 24-month period? Final tip: a +6% edge per bet, betting only major US sports is far from a miserable living...

[evo34]

"Or are you too busy forward-testing your ideas on paper and waiting until they show significance...after which time of course most ideas will have entirely lost their edge."

No reply to this either? Hmm.

[evo34]

It depends on the clarity of the idea or combination of ideas that you are backtesting. If those ideas are extremely clear and accurate, you would, in theory, not have to backtest at all.

You wouldn't "have to", huh? Why on earth would you push away potentially valuable information if the cost of obtaining it was essentially zero. Say you have the "clearest" and most "accurate" idea ever (whatever the penetrate that means...christ), and then you have the opportunity to see how it would have performed in the recent past. You should just pass up that opportunity? How about taking the info. and factoring it in to your ultimate decision? That's all backtesting is -- a piece of information. Anyone who actively avoids looking at pertinent information is suspect in my book.

[hutennis]

hutennis, where's your record? If it's so easy to track, why not show it? Also, why the silence when I asked if you have ever made money over any 24-month period? Final tip: a +6% edge per bet, betting only major US sports is far from a miserable living...

Since I have no intention to provide any proof, I don't make any claims.

But why would you need it anyway?
What does it have to do with an argument?

[evo34]

Since I have no intention to provide any proof, I don't make any claims.

But why would you need it anyway?
What does it have to do with an argument?

So you haven't ever won consistently in sports betting. Sorta puts a ceiling on your credibility, no?

[Dark Horse]

You wouldn't "have to", huh? Why on earth would you push away potentially valuable information if the cost of obtaining it was essentially zero. Say you have the "clearest" and most "accurate" idea ever (whatever the penetrate that means...christ), and then you have the opportunity to see how it would have performed in the recent past. You should just pass up that opportunity? How about taking the info. and factoring it in to your ultimate decision? That's all backtesting is -- a piece of information. Anyone who actively avoids looking at pertinent information is suspect in my book.

What part of 'in theory' do you not understand?

[evo34]

How are theoretical musings relevant to practical sports betting? If it is not, why clutter the board with it?

Unrelatedly, how's your groundbreaking research on Super Bowl turnovers going? Any conclusions drawn from further study?

[hutennis]

So you haven't ever won consistently in sports betting.

Who said so?

I don't make this claim b/c I don't intend to provide any proof on public forum.
I does not mean that I never did.
You are making very common logical mistake.

Sorta puts a ceiling on your credibility, no?

This is another one.

What does anything about me personally have to do with my argument?
Deal with an argument not with a person who made it.

[MonkeyF0cker]

Since I have no intention to provide any proof, I don't make any claims.

You mean like constantly stating that sports markets are efficient and cannot be beaten through statistical analysis? Yeah. No.

[evo34]

Who said so?

I don't make this claim b/c I don't intend to provide any proof on public forum.
I does not mean that I never did.
You are making very common logical mistake.



This is another one.

What does anything about me personally have to do with my argument?
Deal with an argument not with a person who made it.

Your off-topic "argument" consists of random speculation about me. You claim that I don't even know if my success was due to backtesting or not. Umm. Wow. You claim that I have not even had success. The original argument and topic of this post was on the value of backtesting. I first provided my opinion on the topic, and then countered your baseless claim that backtesting cannot be useful with a concrete example of how it had been (still on topic here). Then, apparently unable to defend your position on the uselessness of backtesting, you decided to hijack the thread to speculate about me. So who is staying on topic and who is being personal?

The truth is you have hijacked every thread you have participated in by dogmatically trying to deny nearly every statement made by another poster. It's sad. You got banned strictly for being too illogical and stubborn about it. (Has that ever happened before or since to anyone else?) Beyond sad when you consider how obsessed you are with this topic (sports betting) and yet have not seen any success in it. I cannot imagine devoting so much time to an endeavor in which I had no talent or achievement. I would probably become quite bitter and then try to take it out on random strangers by hijacking threads with "no one can succeed because I haven't" drivel. While you are content to spend all your time describing how impossible the problem is to solve, some of us are actually making progress and money.

[hutennis]

You mean like constantly stating that sports markets are efficient and cannot be beaten through statistical analysis? Yeah. No.

This is not a claim.
Claim would be something like this:
"Using backtesting, I'm able to make a living betting sports for the past ten years"
This is a claim. Claim which is to be taken seriously requires statistically significant proof.

"Backtesting is not going to help you to get any edge over sports market" is not a claim.
It is an negative argument. And since it is a negative argument is does not need to be proven.
Burden of proof is on another side.


@evo34

Then, apparently unable to defend your position on the uselessness of backtesting...

I'm sorry you struggle with such a simple concept, or maybe you keep on ignoring it b/c it too inconvenient for you but I can do nothing about it.

Rules and methods for proper communication, discussion or dealing with argument has been with us for thousands of years.
According to them, negative proposition, like backtesting is useless (NO use) does not need to be defended.

If you stating the opposite then you are the one with a looooot of defending to do.
Again, sorry you might not like it, but that's how the world works.

[MonkeyF0cker]

This is not a claim.
Claim would be something like this:
"Using backtesting, I'm able to make a living betting sports for the past ten years"
This is a claim. Claim which is to be taken seriously requires statistically significant proof.

"Backtesting is not going to help you to get any edge over sports market" is not a claim.
It is an negative argument. And since it is a negative argument is does not need to be proven.
Burden of proof is on another side.

You're not intelligent. It apparently doesn't need to be proven even though you prove it with every post.

claim /klām/

Verb:
State or assert that something is the case, typically without providing evidence or proof.

[Dark Horse]

How are theoretical musings relevant to practical sports betting? If it is not, why clutter the board with it?

Either learn to read or stfu.

[evo34]

Either learn to read or stfu.

I read it pretty clearly, actually. You stated that "If those ideas are extremely clear and accurate, you would, in theory, not have to backtest at all." I would love to know what sort of theoretical world you're referring to (does it happen to involve playing quidditch?) and how it relates to sports betting, which is (as you may recall) the topic of this forum.

[evo34]

hutennis -- reread my post (#24). Take a look in the mirror. Delete your account here. Take courses in English. Find somewhere else to troll. Thank you.

[HeeeHAWWWW]

Yay, another thread derailed. Thanks guys.

[hutennis]

Yay, another thread derailed. Thanks guys.

I don't understand what these guy's problem is.
All these "troll" issues and shit...

I'm pointing out very well known, solidly researched stuff.
Backtesting old data does not work. Every one interested has already seen it, played with it every possible way and
every last drop of value has been already squeezed out of it. Old information is priced in. Period.

As Nate Silver put it in his book “Past performance is not indicative of future results” is there for a reason.
He has a whole chapter dedicated to this subject.
Here is another quote from him on this:

Like the historical patterns on the frequency of earthquakes, market data seems to occupy a sort of purgatory wherein it is not quite random but also not quite predictable. Here, however, matters are made worse because market data ultimately describes not some natural phenomenon but the collective actions of human beings. If you do detect a pattern, particularly an obvious-seeming one, the odds are that other investors will have found it as well, and the signal will begin to cancel out or even reverse itself.

Its like arguing with people who believe that Earth is 6 thousand years old. They just keep on rejecting the obvious.

Now, if you figured out the way to collect and analyze new information so it becomes old to you when it is still new for most, then yes, you can have an edge. But we are not talking about this, are not we?

Moreover, they want to rewrite rules on how people conduct arguments that have been around for thousands of years.
They want me to prove negative statement (like Backtesting old data does not work.)
This is laughable.
If you want to state the opposite - it is your job prove it, not another way around.

BTW, I looked at evo's data. I copied it to excel before he removed his link to it and played with it a bit.
If he thinks that he has "countered baseless claim that backtesting cannot be useful with a concrete example of how it had been"
he is better to rethink his definition of "concrete".
There is nothing there even remotely resembling anything "concrete" (statistically significant)
Yes, there is a winning record there, but it means nothing. Nothing statistically significant anyway.
And I don't even question the methodology used when putting that data together, which is a huge issue in and of itself. I gave him a full benefit of the doubt on that.

As Nate Silver put it

Very often, we fail to appreciate the limitations imposed by small sample sizes and mistake luck for skill when we look at how well someone’s predictions have done

Yeah, it is just funny. Funny and sad at the same time.

[u21c3f6]

Working through some ATS stuff for each team in the NBA. I can get information as far back as 1990 (21 years) for everything that I need (lines, scores, etc.)...but I'm starting to wonder how much backtesting do I need?

Obviously difficult to answer that without knowing the specifics of what I'm doing but just curious as to how far back you guys tend to go when backtesting an idea or how you determine how far back you need to go? What is a meaningful sample size?

To get back to the question, I do very little back testing and when I do, I never go farther back than the previous season. Going back further than last season does not work with my method of selection which focuses on how the lines (odds) are priced currently. Most people IMO confuse backtesting with data mining. I believe hutennis is really referring to data mining which for the most part is usually not very helpful.

Once I have an idea and/or concept I want to test, I may look at this and/or the last season to see if there may be any value to the idea and/or concept. If I see any value, then the real testing is the forward testing done with real money. Small at first but quickly increased if testing is proving profitable.

Joe.

[Dark Horse]

Three main areas.

Gazillions of data. Field: science, beyond sports betting. These data can organize themselves into meaningful patterns that hint at an underlying idea, not yet understood by science.

Almost no data. Field: sports betting; experience: beginner or expert. The simplicity of the beginner differs vastly from the simplicity of the expert. The latter can see through the complexity and identify the few very clear ideas that matter. These come with a clear expectation, that is quickly confirmed by data.

In between these two areas is where most of sports handicapping plays out. The rough area of several hundred to several thousand samples. This is a complex and risky area, because neither of the advantages of the above two are in play. The starting idea is not super clear, so there is a tendency to datamine. And the number of data is not nearly large enough to organize themselves into meaningful patterns. Computer power helps, but there's a vast army of nerds looking at essentially the same stuff.