Scraping

**Full Time Hobo** · 09-28-11, 08:15 PM

Try the sticky at the top of the Think Tank...

Intro to research

**phillybadboy** · 09-28-11, 08:35 PM

Originally posted by Full Time Hobo

Try the sticky at the top of the Think Tank... Intro to research

just follow ssk, the ats atm

**Maverick22** · 09-28-11, 09:19 PM

Pay someone to do it.

Fastest way. You ramping up to learn how will take ages.

OR... Look online for some application that takes an html table, and convert it to csv. That will same ALOT of time and headaches. But still require some time to ramp up, to understand how to use whichever application you can find

**Blax0r** · 09-29-11, 12:29 AM

Originally posted by Maverick22

Pay someone to do it.

Fastest way. You ramping up to learn how will take ages.

OR... Look online for some application that takes an html table, and convert it to csv. That will same ALOT of time and headaches. But still require some time to ramp up, to understand how to use whichever application you can find

For someone with 0 knowledge, I'd go w/ Maverick's first option. The time lost + high probability of mashing your data in a dangerous way (ex: flipping td and ints or something) will be much more expensive than paying someone.

**Cap dat 4ss** · 09-29-11, 02:44 AM

WOuld someone on here be willing to take payment via Paypal for something like this or are you recommending we go find a local to take care of this for us? I wouldn't even know where to begin to look for someone to something like this, except here.

**Wrecktangle** · 09-29-11, 08:15 AM

Originally posted by Cap dat 4ss

WOuld someone on here be willing to take payment via Paypal for something like this or are you recommending we go find a local to take care of this for us? I wouldn't even know where to begin to look for someone to something like this, except here.

Well, if you PM on paying someone for anything here at SBR, the mods will pull your PM privileges, so this isn't a decent place to advert either.

Oh BTW, if you read between the lines: PMs aren't P either.

**Emily_Haines** · 09-29-11, 08:57 AM

Originally posted by Wrecktangle

Oh BTW, if you read between the lines: PMs aren't P either.

Were the fuk SBR get this idea from. Just like the US government they think they got the right to read your PM's. Complete bullshit.

**LtDementia** · 09-29-11, 10:53 AM

I'd have a look at excel's 'Import External Data' first. It would be easy to try:

Go to page that you want in a browser (or have the url handy).
In Excel to to the menu Item 'Data -> Import External Data -> New Web Query' and Select.

A window will appear - highlight the url at the top and paste in your url. Hit the options button and disable date recognition. Press the 'Go' button and the site that you are interested in will appear.

Decide what data you want and click the yellow marks to select - the marks will turn into green checks.

Hit 'Import' and boom - the page will appear in excel all separated into columns. Save as a .csv file ....

LT

Originally posted by Cicima6709

I'v read a lot on here about people "scraping" data from a website and converting it into excel (i think?).

Now, i have like 0 knowledge of programming, however i am good at excel, so i am curious how do-able this is, and how hard it would be for me to learn to do this.

I would like to take a website such as http://sports.yahoo.com/nhl/players/...y7eJzqAEdivLYF and convert all those stats into excel.

Now i realize i can just highlight the data, copy and paste it, and clean it up, showing me the columns i'd like to see. I guess i can just keep copy and pasting values, knowing which columns theyll be in, and set up equations to only extract data from those columns.

My question is...is there a faster way to do this? Is there a way i can just simply change the player # in the yahoo sports game log link, and have excel automatically extra data the way i want it to? I guess i just want to cut down on taking the time to copy and paste the table and cleaning it up, was hoping there was some kind of code to do that for me?

**Flight** · 09-29-11, 11:41 PM

A page like the one you linked sometimes copies well into Excel straight up. Try all browsers - IE, Firefox, Chrome. Some will copy paste to excel better than others for certain tables.

For ESPN pages I copy paste straight to Excel. I can analyze player props within 5 minutes with this method. The only thing that a scraper would do would be to bring that number down under a minute. This would be nice, but I still have yet to do it... I just stick with the manual approach when it comes to player props. But if you need to update constantly (like NHL or MLB), then yah you need a sweet program to get data from the internet.

**arwar** · 09-30-11, 02:44 AM

ncaa boxscores

well i code scrapers almost exclusively. i am attaching 'scraped' boxscores for the the first 4 weeks of ncaa season. if have any questions or need explanation of columns send me a pm. as always these are free to use and distribute. change the extension to .csv and pull them into Excel. Enjoy

Attached Files

**arwar** · 09-30-11, 02:51 AM

here are co0lumn headings for above files:
1st downs,3rd dwn eff (2col), 4th dwn eff, tot yds, ps yds, com/att (2col),yds/ps,rsh yds, rsh att,yds/rsh,penal (2col),fumbles lost,ints, top (secs)

**podonne** · 10-01-11, 12:06 AM

Originally posted by Wrecktangle

Well, if you PM on paying someone for anything here at SBR, the mods will pull your PM privileges, so this isn't a decent place to advert either.

Oh BTW, if you read between the lines: PMs aren't P either.

Really? I made an offer to write scrapers for people on this forum and no one from SBR said anything. Can I get paid in points?

**arwar** · 10-02-11, 08:51 AM

week 5 boxes

a couple rows (A&M & nor ill) need a bit of edit

Attached Files

stat5-fixed.txt (9.9 KB, 189 views)

**Wrecktangle** · 10-06-11, 09:15 PM

Originally posted by podonne

Really? I made an offer to write scrapers for people on this forum and no one from SBR said anything. Can I get paid in points?

Keep it up, they'll get you eventually.

But if you'd rather, I can turn you in...your choice.

**podonne** · 10-06-11, 09:36 PM

Originally posted by Wrecktangle

Keep it up, they'll get you eventually.

But if you'd rather, I can turn you in...your choice.

Not trying to do anything wrong. I posted my own thread and no admins\mods commented there either. I never saw anywhere where it said you couldn't offer other your services to other handicappers on here. Just trying to make a few bucks during unemployment, and since I write so many scrapers myself, seemed natural.

I'll contact SBR directly as ask what their policy is.

**MonkeyF0cker** · 10-07-11, 12:08 AM

Originally posted by Wrecktangle

Keep it up, they'll get you eventually.

But if you'd rather, I can turn you in...your choice.

Whoever said you were a douchebag? They were OBVIOUSLY wrong.

**MonkeyF0cker** · 10-07-11, 12:11 AM

Originally posted by podonne

Not trying to do anything wrong. I posted my own thread and no admins\mods commented there either. I never saw anywhere where it said you couldn't offer other your services to other handicappers on here. Just trying to make a few bucks during unemployment, and since I write so many scrapers myself, seemed natural.

I'll contact SBR directly as ask what their policy is.

I'd let him tattle on you instead.

**Wrecktangle** · 10-07-11, 08:09 AM

Originally posted by MonkeyF0cker

Whoever said you were a douchebag? They were OBVIOUSLY wrong.

Monkey droppings, my one stab at your sarcastic games...

...no where near your level, obviously.

**Maverick22** · 10-11-11, 10:59 AM

The world needs more douche bags. Noone likes smelly vaginas.

**subs** · 10-11-11, 05:20 PM

**Jackson1003** · 01-26-12, 06:39 AM

good luck on your scraping

**ManBearPig** · 01-31-12, 12:55 AM

Google Excel SMF Add-In...may save time from learning to code or buy you time while learning