An introduction to research

Originally Posted by Data

Hm, most sites? No way. The largest projects traffic wise are collecting several years worth of box scores and play-by-plays. Everything else is peanuts. The only site that ever temporarily blocked my scrapping was !Yahoo.

I'm hearing both sides on this. I had trouble with Covers last year in the NBA and then went to a piece of software where I could change my IP and problem went away. Then I quit changing IP for a while and didn't get blocked. Or were they just having site trouble?

I also had trouble on FoxSports and CBS as I recall.

While scraping boxscores, I make a courtesy 1 sec pause after processing each boxscore. Not sure if everybody does this but they should.

Originally Posted by Data

While scraping boxscores, I make a courtesy 1 sec pause after processing each boxscore. Not sure if everybody does this but they should.

If you don't mind me asking, how long have you been using this method?

I have my scraper pause for a random number of seconds (usually 10-25 but it will go shorter and longer) to try to simulate a human browsing the pages. If you've been using 1 second delay successfully for a while with no bans, etc, I might have to drop my delay times considerably.

I finished my last big scrapping project about a year ago. I only scap boxscores nowadays as I calculate all the stats I need myself. Well, I do import some stuff into Excel too but not that much.

where is the best play to scrape boxscores?

the site for that sports (mlb.com, nba.com...

Thank you dearly for the information!

Great information. I look forward to following this post

A great thread.

Originally Posted by Jeremy Nguyen

Hello to every One
Do anybody remember what time ? the ball kick off from 2nd between Tenn adn Jacksonvill? Please!! Thank you

One of the all time strangest first and only posts.

Originally Posted by uva3021

that's because "sht" doesn't exist as a sheet. There is something wrong with your naming conventions

I.E. this is how my range, "NFLTeams" is structured

Arizona2009
Atlanta2009
....
NY+JETS2004
....
SAN+DIEGO2000

Every team from 2009 to 2000 is named in accordance to how they are formatted in the statfox link

Copy the names from a team report page, replace all spaces with a "+", the run the code

This works great. However, I'd like to be able to get it all on a single sheet instead of having each team/year on a separate one. How would I do this?

rob peter to pay paul

Please don't rail me, but I have absolutely no knowledge of computer programming, except when I took a class in middle school, and little in the way of math skills, but I am fiercely determined to become a skilled handicapper like many of the esteemed in this forum. I didn't even know you could create programs to "scrape" data off of the Internet and program it to run simulations or whatever...absolutely fascinating.. So, where do I begin? Do I just start learning Python as suggested by the OP and go from there? Is that what I need to learn to create programs that can determine the winning probabilitiies of a game and the positive edge I have on each bet? I know that there is much more than statistics and math when it comes to betting and handicapping a game, but I think this is a vital aspect too, and I want in! If anyone has any advice whatsoever besides telling me to go work at 7-11 and to give up gambling forever, it would be greatly appreciated.

Thanks in advance.

"I know that there is much more than statistics and math when it comes to betting and handicapping" I disagree... Even things like trades, injuries, game time conditions, etc. can be mathematically represented.

I was trying to work through the example in the OP, but as I tried to download the mlb ex 1.csv file but the website it is hosted on seems to be down. Does anyone have another link to the file? or something equivalent so that I could learn on? Any help would be GREATLY appreciated.