I'd like to scrape screens to log lines for various books. I already do it for several, but none of them require logins to see the lines. I'd like to write scrapers for some that do (I.e. 5dimes) but I'm worried what their reaction may be if they found out what I was doing, especially since it will all be linked to my account. Maybe they wouldn't care, seeing as it's all publically available info anyway. Anyone have any thoughts/experience with this?
-
-
tweek,
You probably want to ask. Send them an email with the IP address that you want to hit their site with. The reason book's don't like it is because of bandwidth. The IT guys can see quickly that it's a bot and will block the IP. They may want to give you an xml and limit you to everu 15 - 45 seconds or even 2 minutes.Comment -
Ok... I will do that. I generally only log them every hour, so bandwidth shouldn't be an issue. I will check with them, though.
Thanks!Comment