Question about what counts towards ROI

[The HG]

I assume that the amount bet on a game that winds up as a push counts when calculating ROI, but what about a game that gets rained out?

The money was in use to make the bet, but since the game was never "official", that money was never actually at risk.

Do you count money wagered on a rained out game when calculating ROI?

[Data]

If I were to calculate ROI I would use neither but the amounts posted up and withdrawn.

[Arilou]

If you're calculating how much you make when you place a wager, the wagers you placed should count whether the game was played or not, or so I would think.

[Wheell]

I disagree with Arilou. If a bet pushes due to a tie (A 3 point favorite winning by 3), then yes, that counts, but a game that could never win nor lose should not count.

I'll give an example: when calculating my hold on baseball I remove games that were nullified due to rain or pitching changes.

I understand the possibility of ties does affect your return on investment (think of an arb on a .5 vs. an arb on a non-zero whole number), but I find that hard to reconcile with one's hold in a game that simply never reached the point of grading.

[MrX]

I'm with Wheel. The event has to happen for an investment to have been made.

[VideoReview]

I have never read a serious academic paper on sports gambling that includes "rained out" or games postponed or cancelled for any reason and would definitely not count these.

As far as pushes go, virtually all of the recent (last 10-15 years) papers I have read specifically say that the results are statistically more "robust" when the pushes are excluded. In other words, when a correlation is found, they found that future predicted results were more accurately made when the pushes were excluded completely from the analysis.

I have also had this method confirmed and the best alternative by a well informed moderator here at SBR who I suspect will be chiming in on this thread soon.

So, my recommendation to you is not to use either.

[Data]

I disagree with Arilou.

I could not understand neither of you. What is the meaning or significance of calculating ROI this way? From economical standpoint, you have to count all the money you post up no matter whether you bet them or not.

[Ganchrow]

I think this really begs the question "Counts towards ROI to what end?" as I'm personally aware of no Platonic Ideal of ROI.

ROI is just one possible metric of many and the way to best calculate it is going to be a function of how one plans to use it.

On the one hand, from an investment perspective I think it makes sense to include no-actioned games insofar as funds were at risk (even briefly) and could potentially have been lost. I see this as true regardless of whether the game was no actioned due to a push or due to it having been physically canceled.

The rationale for this is that an investor attempting to choose between alternative strategies would need to take into account each outcome set in its entirety in order to make a truly optimal decision. Imagine, for example, a hypothetical strategy taking advantage of the observed fact that a particular style of pitcher tends to outperform market expectations given sufficiently rainy weather. While this strategy might correspond to a conditional ROI of let's say 5%, we'd still need to consider that a relatively high proportion of games would be rained out, and as such an n-Kelly investor would never choose this strategy in preference to one with identical conditional edge but a lower push probability (although a sufficiently risk averse Markowitz investor might -- but this is only indicative of some nasty peculiarities of Markowitz.) Of course in reality an investor's choices aren't usually so stark as strategies aren't usually all-or-none propositions. Nevertheless when allocating between strategies the fact that one is more likely to be no-actioned than another needs to be a part of the allocation optimization.

Of course, whether we consider win/loss probabilities in absolute or conditional terms (i.e., conditioned on not pushing) won't impact optimal single bet n-Kelly staking. What it will impact, however, is expected n-Kelly cardinal utility (but not ordinal utility, meaning that the relative preference rankings of various stake sizes won't change regardless of how probability is construed). This means that even though single-bet n-Kelly stakes won't change, simultaneous bet n-Kelly stakes will change. In practice, however, this will generally be a relatively minor effect.

On the other hand, when back testing a strategy in isolation there's compelling practical rationale for factoring out pushes and focusing solely on conditional probabilities and edges. Specifically, when one includes pushes in one's analysis it's no longer correct to model results solely using the binomial distribution. Rather one would need to use the trinomial distribution insofar as there exist three possible outcomes for each event. This adds a not insubstantial additional layer of complexity. However, because the difference in sample Sharpe ratios (that’s the sample mean divided by the sample deviation) can be shown to be minimal for edges and push rates reasonably close to zero, most academic papers on sports betting ignore no actioned bets by dealing solely with conditional probabilities and edges. This is very much standard practice although it should be noted that this will tend to slightly overestimate the sample Sharpe ratio of events that may push. (To give an idea of scale, a bet on an event paying out at even odds is observed over 300 trials to win, lose, and push 55%, 40%, and 5% of the time respectively. The correct trinomial Sharpe ratio would be 2.6977, while the approximated binomial Sharpe ratio would be 2.6994, corresponding to Gaussian p-values of 99.651% and 99.653%, respectively.)

But yes, if you really wanted to be completely anal about it you should really factor in all no actioned bets (and hence use the trinomial distribution) when calculating significance.

So when comparing classes of strategy a practitioner is going to be best served factoring in the impact of pushes. If there also exists a substantial discrepancy between the relative frequencies of physically canceled games, then this should be taken into account as well. For example, assuming he could only bet one, an n-Kelly bettor would choose to bet an NBA game rather than an MLB game with identical conditional payout characteristics due to the higher probability of the latter game being no-actioned. (Academic? Maybe a bit.)

When backtesting single strategies, however, a bettor will find nearly identical p-values (for reasonable push frequencies) whether he looks at conditional or absolute probabilities. Because using conditional probabilities allows the bettor to use the binomial distribution, this generally represents a more tractable solution.

Lastly, Data's interpretation of ROI is probably the most theoretically sensible when looking at the overall macro success of a particular bettor, but probably isn't as useful for evaluating the statistical properties of an individual strategy or when comparing multiple strategies that generally put into use no more than a fairly small percentage of bankroll at any given time. Certainly ROI by itself as construed above tells us relatively little but that's why it's usually looked at in conjunction with odds ranges, frequency of bets, and variance.

I'd certainly agree with Data that strategy capacity/bankroll utilization should ultimately be considered, but this is more of a concern in "big picture" scenarios such as when trying to determine whether or not one should quit one's job to become a professional gambler, or how much one should allocate to one's betting portfolio versus one's stock market portfolio. (Another time this might come into play would be when trying to determine optimal allocations amongst difficult-to-transfer-between sportsbooks that house different strategies.)

Still, this interpretation of ROI+variance doesn't preclude either of the two others insofar as nothing would prevent an investor from calculating expectation and standard deviation in terms of percent of total bankroll. While doing this does circumvent the issue of how to treat pushes in the case of calculating ROI, it still leaves open the fundamentally linked issue of how to treat pushes when calculating variance.

[Data]

I am trying to understand how ROI can be used while developing, comparing and backtesting betting strategies.

Certainly ROI by itself as construed above tells us relatively little

That is my impression as well to the extent that I can only extract a bit of useful information judging by ROI being either positive or negative. What I do not see is the "bridge" between ROI and the weighting odds (linemaking), it just feels somewhat being "backwards" to use ROI for this. So, I never thought about using ROI this way but you guys certainly know more about this, so I hope you can explain.

Also, I think that at best ROI shows by how much your lines were better than the bookmakers lines in the past. It does not show how good your lines are. Because of that, its predictive value is very questionable. What do you think?

[VideoReview]

Lastly, Data's interpretation of ROI is probably the most theoretically sensible when looking at the overall macro success of a particular bettor, but probably isn't as useful for evaluating the statistical properties of an individual strategy or when comparing multiple strategies that generally put into use no more than a fairly small percentage of bankroll at any given time. Certainly ROI by itself as construed above tells us relatively little but that's why it's usually looked at in conjunction with odds ranges, frequency of bets, and variance.

BINGO.

In regard to Data's question, as a single stake Kelly bettor (I do bet multiple games but my fractional Kelly is certainly low enough to compensate and my BR is relatively low enough for me not to worry about it at the moment), I can personally confirm for you that ROI can be predictive in a substantial (not just a very small edge either, e.g. 1%) and statistically significant way in the way that Ganchrow has advised. Ask yourself "what if" the market/bookmaker wanted the ROI to be that way on purpose (i.e. true odds were known by the bookmaker/market and well informed shading has occured). Ganchrow's explanation above gives the blueprint for this.

I'd certainly agree with Data that strategy capacity/bankroll utilization should ultimately be considered, but this is more of a concern in "big picture" scenarios such as when trying to determine whether or not one should quit one's job to become a professional gambler, or how much one should allocate to one's betting portfolio versus one's stock market portfolio.

Your answer was not directed to me but it's like you read my mind. I was aware that I would have to eventually consider the push/cancel frequency and came to the conclusion that my time was best spent working on the systems rather than factoring this in, at least until my BR became large enough to have to make the determination you suggested above.

[RickySteve]

If I were to calculate ROI I would use neither but the amounts posted up and withdrawn.

So then you're willing to buy a CD that matures in 50 years and pays you a robust 20% of principal?

[Data]

In regard to Data's question, as a single stake Kelly bettor (I do bet multiple games but my fractional Kelly is certainly low enough to compensate and my BR is relatively low enough for me not to worry about it at the moment), I can personally confirm for you that ROI can be predictive in a substantial (not just a very small edge either, e.g. 1%) and statistically significant way in the way that Ganchrow has advised.

How come it can be predictive if all it shows is the advantage over arbitrary number (the line) in the past? What if the bookmakers line gets sharper due to personnel changes and technology advancements?

Ask yourself "what if" the market/bookmaker wanted the ROI to be that way on purpose (i.e. true odds were known by the bookmaker/market and well informed shading has occured).

I did not get this, please explain.

[MrX]

How come it can be predictive if all it shows is the advantage over arbitrary number (the line) in the past?

Just as an athlete's past performance is predictive of his future performance, so can the bookmakers' past behavior be predictive of future behavior.

What if the bookmakers line gets sharper due to personnel changes and technology advancements?

Of course that is going to happen, but it doesn't mean that there is no predictive value there.

[Ganchrow]

How come it can be predictive if all it shows is the advantage over arbitrary number (the line) in the past? What if the bookmakers line gets sharper due to personnel changes and technology advancements?

Whenever you work off of any any sort statistically modeled data, regime change always needs to be a consideration. This is just as true in quantitative sports betting as it is quantitative finance or weather prediction or non-gambling related sports modeling. But we don't want to be tossing the baby with the bathwater, either.

I'll agree that because of this very issue one should be exceedingly cautious when dealing with predictive models that base forecasts solely off of market microstructure without regard to concrete measures of future probability. For example earlier this year I posted to this thread regarding the performance of large NCAA football dogs ATS in early bowl games.

Over the data set, there's been a very clear statistically significant outperfomance on the part of these dogs, making them excellent bets historically. This trend continued last year (out-of-sample), to the extent that it was actually the strongest year to date.

Nevertheless, while I have no problem performing this type of analysis in my capacity on this forum, I personally abhor strategies such as this and would be exceedingly hesitant to bet into them as a professional player (and would certainly never recommend them to others without massive qualification) for the exact reason you've indicated -- even if this does represent a real phenomenon, we'd have no way of knowing ex ante if and when the bookmakers (or "the market" if you prefer) were to "catch up" to reality and modify pricing habits.

What I much prefer are strategies that first make concrete predictions of prior outcome probabilities without any regard to the current market line, and then only after that use Bayesian inference (if necessary) to temper prior beliefs, creating posterior forecasts that are allowed to change with market price.

The testing procedure initially involves examining the raw error in forecast priors and then comparing that with realized success using historical market lines.

The grave concern with the methodology would be if forecasts were biased insofar as a correlation existed between perceived forecast strength and realized forecast shortfall (due perhaps to some factor that was "hidden" from the model -- imagine, for example, if you were to cap MLB games without any knowledge of the concept of starting pitchers).

This could cause a situation where although forecasts were quite accurate "in a vacuum" the bettable lines that in reality presented opportunity were those where forecasts tended to be the least reliable. This is a phenomenon that's almost always seen when using predictive models not specifically designed for sports betting (Sabermetrics, for example).

This is one of the issues for which we should test -- do forecasts tend to be stronger on unbettable lines than on bettable lines and do they tend to be weaker still for the highest magnitude forecasts (magnitude relative to the actual market price)? In my experience in both finance and sports betting the answer to both these questions is always an unfortunate yes. The final round of Bayesian inference is supposed to reduce this -- but in reality of course it's going to be far from perfect.

So yeah, ROI is by itself a rather blunt instrument, but when taken in conjunction with other statistics it does provide a good indication of past strategy performance. How this might relate to future strategy performance is an open-ended question that's going to be as much a function of not only how the model is sampled and updated but also of what it purports to accomplish.

[Ganchrow]

Nevertheless, while I have no problem performing this type of analysis in my capacity on this forum, I personally abhor strategies such as this and would be exceedingly hesitant to bet into them as a professional player (and would certainly never recommend them to others without massive qualification) for the exact reason you've indicated -- even if this does represent a real phenomenon, we'd have no way of knowing ex ante if and when the bookmakers (or "the market" if you prefer) were to "catch up" to reality and modify pricing habits.

I should note that many professional bettors (and one in particular on this message board) have enjoyed incredible success using methods such as these.

There's nothing inherently "wrong" with betting in this manner and while it's true that many of these inefficiencies do eventually right themselves, the corrections tend to come about drift rather than appear suddenly.

"Abhor" was probably too strong a word.

[Data]

Thank you for sharing your thoughts, Ganchrow.

This is one of the issues for which we should test -- do forecasts tend to be stronger on unbettable lines than on bettable lines and do they tend to be weaker still for the highest magnitude forecasts (magnitude relative to the actual market price)? In my experience in both finance and sports betting the answer to both these questions is always an unfortunate yes. The final round of Bayesian inference is supposed to reduce this -- but in reality of course it's going to be far from perfect.

If I understand you correctly, by "bettable" you mean "the market line that is sufficiently different from the forecast". It seems you are saying here that a custom made line is always weaker than the market line. Leaving the financial markets alone and sticking to sports betting, I think, for the mutual rejoice, this cannot be true. Considering today's state of sports betting markets (lines) efficiency, the lines are not the reflections of market's equilibrium but first and foremost are just human products. There is always someone whose product is going to be better than the others, and everyone hopes its going to be his.

[Ganchrow]

It seems you are saying here that a custom made line is always weaker than the market line.

No, I wasn't saying that at all.

[Data]

No, I wasn't saying that at all.

...weaker when it differs the most. This clarification does not change my point.

[Ganchrow]

...weaker when it differs the most. This clarification does not change my point.

No, I'd say the two aren't the same at all.

Forecasts represent prior likelihood estimates. Market lines represent additional evidence (the net belief of "the market"). When Bayesian inference does is show us exactly how new evidence should be taken in conjunction with our priors in order to create posterior forecasts.

Ignoring additional evidence by not allowing it update one's priors completely discounts the nonzero power, knowledge, and efficiency (and that's efficiency-as-process, not efficiency-as-absolute-end-result) of the market. The more market lines differ from prior forecasts, then by Bayes' Theorem the more they'll generally inform these priors.

This doesn't mean that market lines are somehow "better" than what you refer to as "custom lines" (in fact one could even say the more one's forecasts differ from the market the weaker this implies the market lines to be), but rather that they contain additional information that a careful forecaster would be wise not to ignore.

[Ganchrow]

Also note that I'm bringing up two related but still distinct issues.

1. A "good model" should have low correlation between forecast strength and realized forecast shortfall.
2. In reality even the best models will tend to have some correlation. This correlation is reduced (and hopefully eliminated -- but that's just a dream) by utilizing Bayesian inference to update prior beliefs.

[VideoReview]

How come it can be predictive if all it shows is the advantage over arbitrary number (the line) in the past?

Mr. X's response to this hits the nail on the head.

What if the bookmakers line gets sharper due to personnel changes and technology advancements?

Sharper for whom? When systemic biases are permitted to exist by the books in the first place, there is no reason to assume that the books would want to move shaded lines closer to "true" values as their operations become more proficient. On the contrary, with advancements they may be able to "further" shade the lines creating even more profitable opportunities. Interestingly, some of the best angles I use appear to be getting "stronger" over recent years and not weaker. Whether this is due to macro events like economic slowdown and hence more squares gambling lunch money or is simply a statistical blip, I am unsure. However, some biases are so profound and have lasted so long (i.e. 25-30 years) and are as strong as they ever were which leads me to the conclusion that immediate market correction is not an absolute certainty. I am relieved to read that Ganchrow has lightened up a little on his "abhor" statement as I have recently started to benefit from this methodology. However, I am in agreement with Ganchrow that one must tread carefully and be aware that the market may change over an extended period of time.

I did not get this, please explain.

Please don't take this the wrong way Data but if I said any more I would be giving away my proprietary information.

[Data]

Just as an athlete's past performance is predictive of his future performance, so can the bookmakers' past behavior be predictive of future behavior.

My observation is that both, the athletes and the linesmakers, progress.

[Data]

1. A "good model" should have low correlation between forecast strength and realized forecast shortfall.

I was not aware of this. Thank you, as always.

[Data]

When systemic biases are permitted to exist by the books in the first place

How sure are you about this? To me, this sounds like a conspiracy theory. I think that shading the line is rather an infrequent practice and can be observed at sharp opinionated books which are the minority in the industry. I do not care much about recreational books that shade the line according to their clientele.

[VideoReview]

How sure are you about this?

Quite sure. Regardless of my self-calculated p-value of less than .05 based on an entire population (with nothing excluded) quickly approaching 1000 bets and running a 10 million Monte Carlo trial, Ganchrow has already confirmed someone else in this forum who has used this very profitably. Also, from Mr. X's comments, I assume he would also concur.

To me, this sounds like a conspiracy theory.

I guess that is why it keeps working for decades at a time

I think that shading the line is rather an infrequent practice and can be observed at sharp opinionated books which are the minority in the industry. I do not care much about recreational books that shade the line according to their clientele.

Neither do I. I am not talking about places like SIA that shade their lines 20 cents and then bump limits down to $50 if you try and bet the other side. All of my bets and most of my analysis are based on Pinnacle's historical lines.

[VideoReview]

What I much prefer are strategies that first make concrete predictions of prior outcome probabilities without any regard to the current market line, and then only after that use Bayesian inference (if necessary) to temper prior beliefs, creating posterior forecasts that are allowed to change with market price.

I find this pretty heady stuff and my only hope is that I can make some more money before you win all of it.

[Wheell]

This is the finest thread I have ever read at SBR or any other forum and I'm guessing it has been read by fewer than 30 people and understood by fewer than 15.

Some notes:

Ganch has a different background than most of the rest of us and he is understandably cautious regarding systems that have done all of their testing in sample and have been good in predicting the past. His point about sabermatricians not having designed their work for gambling markets is quite apt, particularly given that the market is itself aware of many of the advancements made by the sabermatricians.

I should add that the only betting lines worth discussing are the betting lines that the observer wishes to attack. For some of us only Matchbook and Pinnacle matter due to being limit capped at every other location. For others, 1 unit is $50 at Bodog. Designing a system to beat the market you are attacking is goal #1 and for some of us that is simpler than for others.

Long term market inefficiencies are obviously sustained by some facet of human or mathematical nature. There is one sport I can think of where if you bet the under on the middle 80% of over-unders (ignoring all but the highest 10% and lowest 10%) you would turn a small but steady profit for every year that wagering data exists. This inefficiency appears to me to be the result of human nature simply not understanding the nature of the sport and being unable to reconcile there vision of the game and the actual result. Or maybe people just don't enjoy betting on the under in the sport I am thinking of. Whatever the case may be broad market inefficiencies can persist unless corrected by a specific agent.

The book "Super Crunchers" discusses an air travel site that not only makes predictions but that also makes predictions about the quality of their predictions. With enough data you can become extremely confident in your aggregate success rate, particularly if you are constantly updating your database and digesting the new data.

As for books creating sharper lines, I can say with certainty that the lines are sharper than they were as recently as 5 years ago, and I can point to the factors why. As a general rule, sharp lines refers to lines that are more accurately predictive of the underlying reality. This doesn't mean the lines are unbeatable or that a 3 point favorite will rarely win by more than 20.


Past performance is not a guarantor of success, but if you are familiar with statistics, it can be a pretty good guide for how you should act in the future, and that more than anything else is what we are looking for.

[chemist]

If I were to calculate ROI I would use neither but the amounts posted up and withdrawn.

I never saw the relevance or understood the meaning of ROI in sports betting. What you're talking about sounds more like rate of return to me, eg I deposited $1000 and a year later I withdrew $10000 so my RoR is 900% pa. Doesn't seem especially useful to me either.

[chemist]

I was not aware of this. Thank you, as always.

I think he's talking about homoskedasticity.

[Wheell]

I wonder what the difference between EV, Hold, and ROI is...

[RickySteve]

This is the finest thread I have ever read at SBR or any other forum and I'm guessing it has been read by fewer than 30 people and understood by fewer than 15.

Some notes:

Ganch has a different background than most of the rest of us and he is understandably cautious regarding systems that have done all of their testing in sample and have been good in predicting the past. His point about sabermatricians not having designed their work for gambling markets is quite apt, particularly given that the market is itself aware of many of the advancements made by the sabermatricians.

I should add that the only betting lines worth discussing are the betting lines that the observer wishes to attack. For some of us only Matchbook and Pinnacle matter due to being limit capped at every other location. For others, 1 unit is $50 at Bodog. Designing a system to beat the market you are attacking is goal #1 and for some of us that is simpler than for others.

Long term market inefficiencies are obviously sustained by some facet of human or mathematical nature. There is one sport I can think of where if you bet the under on the middle 80% of over-unders (ignoring all but the highest 10% and lowest 10%) you would turn a small but steady profit for every year that wagering data exists. This inefficiency appears to me to be the result of human nature simply not understanding the nature of the sport and being unable to reconcile there vision of the game and the actual result. Or maybe people just don't enjoy betting on the under in the sport I am thinking of. Whatever the case may be broad market inefficiencies can persist unless corrected by a specific agent.

The book "Super Crunchers" discusses an air travel site that not only makes predictions but that also makes predictions about the quality of their predictions. With enough data you can become extremely confident in your aggregate success rate, particularly if you are constantly updating your database and digesting the new data.

As for books creating sharper lines, I can say with certainty that the lines are sharper than they were as recently as 5 years ago, and I can point to the factors why. As a general rule, sharp lines refers to lines that are more accurately predictive of the underlying reality. This doesn't mean the lines are unbeatable or that a 3 point favorite will rarely win by more than 20.


Past performance is not a guarantor of success, but if you are familiar with statistics, it can be a pretty good guide for how you should act in the future, and that more than anything else is what we are looking for.

I don't agree with the first sentence, but you get the blue ribbon for the thread.

[Wheell]

Ricky: I should ask: What are the best threads you have read here at SBR?

[Ganchrow]

I think he's talking about homoskedasticity.

While certainly a related concept, that's not exactly what I'm talking about.

I'm saying that in an ideal model the distribution of the error term should be independent of the deviation between the market and model prices. (Remember that because these are prior forecasts, we're assuming that market price, or any function thereof, isn't already a regressor of the model.)

Another way of putting this is that in such a model the market price should provide no additional information not already available to the model itself (because it would already be considering and efficiently processing all market-attainable information).

[Data]

1. A "good model" should have low correlation between forecast strength and realized forecast shortfall.

As a model progress the correlation lowers naturally. Can you clarify what is a "low correlation" in this context?

[RickySteve]

Ricky: I should ask: What are the best threads you have read here at SBR?

I'd consider dozens of Ganchrow threads much more insightful than this one. I suppose the discourse here may be generally more intelligent than usual.

My comment was more in reference to the content elsewhere.