How should I structure my data?

**Maverick22** · 02-11-13, 11:29 AM

Are you writing the program that will read the data? If so... it probably doesn't matter how you lay it out in your file.

If you want the absolute prettiest data file possible. Write it to xml and and/or write a conversion to a character delimited file (think csv or excel)

I would of course recommend folk to put their data in a relational database, but I've stopped suggesting such a thing

**Carl-Haakon** · 02-12-13, 12:59 PM

Thank you for your reply!

I actually am writing the data as an xml-file at the moment, but I am not sure how I should go from there; as I wrote, my favourite ML-library for Python only takes data sorted into neat lines (which I assume is what you were referring to when you said "character delimited file"). What would a conversion to such a format look like?

Actually, I wouldn't mind learning about relational databases if they could give me an edge. What are they and do you have any literature on the subject that you could recommend?

**339955** · 02-12-13, 04:43 PM

carl, just put it in sqlite3 DB. If you can use python to scrape it, it will just take you a couple days to learn how to store it and access it from sqlite3.

Maverick, how do you store you data? how much data have you saved?

**Topo** · 02-17-13, 06:05 PM

I maintain a few small databases and have found the results disappointing. The data is nice to have, but it really just reveals what I wished were not true: I need to find another edge other than statistical history in order to turn a betting profit. That said, if you look at smaller betting markets then you can acquire an advantage through data analysis. But any reasonably sized sports betting market will probably not be beatable through data analysis alone.

SQLServer Express and MySQL are fairly simple to learn and free to acquire. You can learn how to structure and edit your databases using these programs in a few days. Querying them is easy to once you read a tutorial or two on Structured Query Language. There are many free tutorials offered on the web.

**littlezola** · 02-17-13, 11:14 PM

[QUOTE=Topo;17842864] But any reasonably sized sports betting market will probably not be beatable through data analysis alone.
[QUOTE]

Why not?

**Carl-Haakon** · 02-18-13, 10:29 AM

Originally posted by littlezola

Originally posted by Topo

But any reasonably sized sports betting market will probably not be beatable through data analysis alone.

Why not?

This caught my interest as well; would you care to explain what you mean, Topo?

**Jontheman** · 02-18-13, 04:51 PM

Presumably because, if this data is freely available, then many people have already done what you have done and their betting patterns have adjusted typical lines; and so prices already accurately reflect the information you are acquiring.

**littlezola** · 02-18-13, 06:46 PM

Originally posted by Jontheman

Presumably because, if this data is freely available, then many people have already done what you have done and their betting patterns have adjusted typical lines; and so prices already accurately reflect the information you are acquiring.

Which presumes that -
A)All possible avenues of interpreting data have been exhausted
B)All markets are efficient.

Prove A or B.

**sbrhedge** · 02-19-13, 12:07 AM

Originally posted by Jontheman

Presumably because, if this data is freely available, then many people have already done what you have done and their betting patterns have adjusted typical lines; and so prices already accurately reflect the information you are acquiring.

trading and sportsbetting have one thing in common: the more data that is available, the worse the index / line / point spread gets out of line. the problem is not data itself, the problem is the number of boneheads that use the data. the advent of electronic data/trading has pretty much guaranteed a fund manager's career. sportsbetting seems to have substantially more horrible lines then say 5-10 years ago.

**Carl-Haakon** · 02-20-13, 01:07 PM

Originally posted by sbrhedge

trading and sportsbetting have one thing in common: the more data that is available, the worse the index / line / point spread gets out of line. the problem is not data itself, the problem is the number of boneheads that use the data. the advent of electronic data/trading has pretty much guaranteed a fund manager's career. sportsbetting seems to have substantially more horrible lines then say 5-10 years ago.

Everywhere else I've read (and I'm mostly talking old HTT posts) everybody seems to be of the opposite opinion, but I'll bite. Do you have data to support your assertion, and in which sports/markets?

**Carl-Haakon** · 02-20-13, 01:15 PM

Originally posted by Jontheman

Presumably because, if this data is freely available, then many people have already done what you have done and their betting patterns have adjusted typical lines; and so prices already accurately reflect the information you are acquiring.

But in order for this to work out in practice, you'd require limitless (or high limits) betting and big volumes. I didn't mention this in my first post, but the volume bet on horse racing in my home country is relatively small, and the fact that the market is pari-mutuel means it's useless to bet large amounts since you'd mostly be betting against yourself (thus creating a sort of self-limiting mechanism).

This means (based on my probably flawed understanding) the market is unlikely to be efficient in any strong sense, doesn't it?

**Blax0r** · 02-20-13, 01:24 PM

In regards to your first question, I would actually recommend using a database (I made a similar change myself over the past few months). I imagine python has libraries to make queries as painless as possible, and having data easily query-able is such a huge gain. Another side-gain is that as your data grows, under your current format, you'll be putting a lot of stuff in memory and coding probably-complicated lines to order or organize that data. With a DB, you can just query for data you need at whatever moment with powerful SQL syntax.

Just my 2 cents.

And I really hope to prove Topo wrong, but his statement definitely resonates loudly with me.

**Carl-Haakon** · 02-20-13, 01:41 PM

Originally posted by Blax0r

In regards to your first question, I would actually recommend using a database (I made a similar change myself over the past few months).

Thank you! I've been looking into databases ever since I first ran into this problem. I guess I just have to look up more internet tutorials on the subject and tinker more.

Originally posted by Blax0r

And I really hope to prove Topo wrong, but his statement definitely resonates loudly with me.

If you don't mind my asking, how far have you come in your modelling efforts?

**Blax0r** · 02-20-13, 01:50 PM

Yea it's surprisingly not too difficult (I hooked up Matlab and Postgresql), and the technical gains are astounding. But you will undergo some time-cost in order develop a solid DB schema, tune tables for better performance, and develop a way to create your INSERT scripts from your current data.

Last year was my first "live" year, finished negatively (~10%) though after being up a decent amount (~25-30%) just before the summer olympics (i bet tennis). At this point, Topo's point has been true for me, but there are quite a few things I want to try out before declaring defeat.

**Miz** · 02-20-13, 05:27 PM

It may be true for TOPO, but it isn't true for everyone.

**strixee** · 02-20-13, 08:02 PM

My guess is you might need like 6 tables : Horses, Owners, Jockeys, Tracks, Races, Results.
Start the process with ER modeling, that'll hive you the exact answer.
MySQL works well with hundreds of thousands of records on shared hostings, for 10M+ you should use VPS.
If you're running it on your own PC, you won't have any performance troubles for a long time.

**TravisVOX** · 02-22-13, 06:39 AM

FWIW -- I have maintained a "horse racing" database for a handful of years now but spent the last few months redesigning it because it wasn't scaling well (it was basically a dump of data, no real structure). I use MySQL because I'm a hack PHP programmer and they play together well. Like with any sport, the database is big. For example, six months of data represents about 25,000 races, 215,000 starters, 65,000 horses etc. So, a database is the route you want to go.