[Leverage]

What is the best free tool to scrape the SBRodds site? I'm tired of doing it by hand.

[5050]

Good question, would like to know too

[twister]

I'd like to know too.

Preferably scraping it straight into Excel, because I collect closing lines for all the sports I bet on.

[feltball]

learn some coding I guess.

[runnershane14]

use modzilla browser and get this add-on..it works pretty well actually

http://www.outwit.com/

[williams22]

use modzilla browser and get this add-on..it works pretty well actually

http://www.outwit.com/

Which download are you using? I'm very interested in this as I have yet to find an easy place to scrape lines from.

[skrtelfan]

My programming friends have been encouraging me to learn Perl to learn how to scrape data.

[roasthawg]

Tough to scrape anything from a page that uses javascript. Copying and pasting into excel works best in this case. Sucks but it's tough to get around unless you have a little more advanced knowledge.

[Leverage]

I have advanced knowledge, I'm just lazy. I want something already coded and compiled.

[jeccross]

I'd like to know this, it shouldn't be too hard to write an excel macro to reformat correctly after pasting into excel? Anyone done it?

[Indecent]

As was already mentioned, it's going to be tough to do a traditional HTML scrape due to javascript use, and I'm guessing that was their intention. It won't stop you from building an OCR tool that takes a snapshot and processes the text in the image automatically. Plenty of open source/free OCR tools out there, not too hard to modify or use them in your program.

[arwar]

I just wrote a scraper. It was indeed tricky. I considered OCR but concluded that it is not a viable option. if you are interested PM me.

[arwar]

here is an example .xls for Dec 08,

[arwar]

1 row = 1 game

[runnershane14]

thanks arwar thats awesome..is there any chance you could do nhl 2008-2009 the same way. I would really appreciate it.

[Dave Head]

I've submitted the following message to SBR:

Regarding the "Handicapper's Thinktank" thread "SBRodds scraping":

Some Posters are scraping data from the SBR website, and would like to know the best way. I propose that if we ask nicely, SBR may provide downloadable text packages, and save us all a lot of scraping. These packages could be compressed in zip format which would reduce the overhead on the SBR server.

[arwar]

thanks arwar thats awesome..is there any chance you could do nhl 2008-2009 the same way. I would really appreciate it.

no problem for NHL - I'll take a look at it when I get a chance

[arwar]

I can zip em up

[man3645]

I've submitted the following message to SBR:

Regarding the "Handicapper's Thinktank" thread "SBRodds scraping":

Some Posters are scraping data from the SBR website, and would like to know the best way. I propose that if we ask nicely, SBR may provide downloadable text packages, and save us all a lot of scraping. These packages could be compressed in zip format which would reduce the overhead on the SBR server.

Good idea.

[so im zach]

I really like jQuery + Firebug.

[bolekblues]

Thank you arwar, this is what i've been looking for. Looks awesome!!

Dave Head great idea, hopefully this will work.

[Wrecktangle]

Great. Thanks!

[arwar]

Great. Thanks!

I thought you wrote your own scrapers?

[runnershane14]

any luck on the nhl arwar? Actually maybe we could work out some deal with betpoints if you could provide me with info on other sports too? This is good stuff!

[arwar]

any luck on the nhl arwar? Actually maybe we could work out some deal with betpoints if you could provide me with info on other sports too? This is good stuff!

I just haven't had a chance to look at NHL yet. You need to realize that historical data and real time data are different animals. Both are do-able, just different. Exactly how are you planning to give up points?

[Pucks]

I've submitted the following message to SBR:

Regarding the "Handicapper's Thinktank" thread "SBRodds scraping":

Some Posters are scraping data from the SBR website, and would like to know the best way. I propose that if we ask nicely, SBR may provide downloadable text packages, and save us all a lot of scraping. These packages could be compressed in zip format which would reduce the overhead on the SBR server.

This is a great Idea.

[runnershane14]

well ive only got 6 points at the moment because i just bought a freeplay but give me a few weeks and i will have some points to pay you. I am not as interested in current seasons but basically previous seasons (all sports). You name the price, I am pretty open as far as how many points per season of information because i do realize its probably a lot of work. I do however think this information is very valuable in some of the things I am trying to do. I would also like for the files to be emailed to me or something instead of posting in this thread since I would be the only one paying points for it, unless other posters would like to collectively pay you for the files. Let me know, thanks arwar.

[arwar]

these are through 01/05/10

[rfr3sh]

how would you scrape the line movements and the final outcome of nba games to an excel spread sheet

01/06 18:46 +13½ +100 / -13½ -110
01/06 18:51 +13½ -101 / -13½ -109
01/06 18:57 +13½ +101 / -13½ -111
01/06 19:01 +13½ +105 / -13½ -115
01/06 19:06 +13½ +106 / -13½ -116

like that transfere to an excel?

[dickersonka]

I created an automated program that will scrape the odds and save the open / closing lines to a database for me. Based upon those odds I will make my picks and save them to my database. Throughout the day another program runs and will automatically grade my picks as winners or losers for me.

Odds scraping isn't rocket science, its just tedious work that needs to be put together the first time, then it will run until wherever you are scraping from decides to change their site.

As previously stated, current odds vs archive are a different story, just much more work.

Any specific questions you have about it?

[Bsims]

I created an automated program that will scrape the odds and save the open / closing lines to a database for me. Based upon those odds I will make my picks and save them to my database. Throughout the day another program runs and will automatically grade my picks as winners or losers for me.

Odds scraping isn't rocket science, its just tedious work that needs to be put together the first time, then it will run until wherever you are scraping from decides to change their site.

As previously stated, current odds vs archive are a different story, just much more work.

Any specific questions you have about it?

Two questions. What software did you use to develop the programs? What sites can you scrape from?

[arwar]

well ive only got 6 points at the moment because i just bought a freeplay but give me a few weeks and i will have some points to pay you. I am not as interested in current seasons but basically previous seasons (all sports). You name the price, I am pretty open as far as how many points per season of information because i do realize its probably a lot of work. I do however think this information is very valuable in some of the things I am trying to do. I would also like for the files to be emailed to me or something instead of posting in this thread since I would be the only one paying points for it, unless other posters would like to collectively pay you for the files. Let me know, thanks arwar.

I'm not looking for any points, I'll try to take a peek at NHL this weekend.

[arwar]

Two questions. What software did you use to develop the programs? What sites can you scrape from?

hey ole bill - I thought you have your own scraping (at least for baseball) tools. FWIW, I generally code in Delphi because it has a TWebBrowser component and a slew of other useful web based options. I imagine VB has them as well (and I make a lot of API calls, which are certainly supported there) but it is too kludgy to compile. A 5 line VB app ends up with 50 megs of dlls, etc. Delphi compiles into a neat standalone app - even smaller as a console app. Lately I have been fooling around with the VBA interface inside of EXCEL, but I still don't like VB even though I am certified in it. I did get a copy of Power Basic as you suggested, but it isn't as robust as Delphi. I guess Delphi has a lot bigger user base and thus more people developing components. Another benefit of Delphi is Kylix which allows apps to be ported to Linux which is hosting most web servers. I scrape covers (easy), nfl.com, atsdatabase, mlb, espn, of course sbr, yahoo, usatoday , et al. The first obstacle is getting on the same page with team names between sites. EG one site may have have Boston Red-Sox and the next one will be Boston Red Sox or Bosox. It's not too bad for pro sports, but NCAA baskets has something like 350 teams. I am just posting this stuff here to give people data to play with. BTW I haven't found any usuable trends (yet!!)

[investor]

hey arwar, know anyone who can code really good in flash media script?

[arwar]

hey arwar, know anyone who can code really good in flash media script?

Not really.

I use:------>> FlashWindow(Application.Handle,True);
in some of my scrapers to make the system tray icon flash while scraping multiple pages (like a whole baseball season) so when it stops flashing I know the data is ready for manipulation. When I look at the code for some of the scripts it looks a lot like JAVA.