retrosheet script (UNIX)

Collapse
X
 
  • Time
  • Show
Clear All
new posts
  • uva3021
    SBR Wise Guy
    • 03-01-07
    • 537

    #1
    retrosheet script (UNIX)
    I wrote a customize-able UNIX bash retrosheet script to retrieve, parse, and place all data into a local mysql database. (One can probably adopt it for windows fairly easily, or use Cygwin)

    Was hoping somebody here would test it out to see if its working properly on other computers.

    Basically I did it just for future reference in case something happens to my database, so I don't have to go through the painstaking process of having to manually do everything again. Though it can be used to fetch the new data for a given year.

    I tested parsing the 2010 and 2011 data in a virtualbox server session running ubuntu. This was just to save resources and time. I'm assuming from there getting earlier years shouldn't be a problem. Be advised, however, the database uses up a lot of space, and the script might take a while to run, depending on how far back you want data.

    If you don't have a database, download apache and mysql (and phpmyadmin, eg.), then follow the mysql manual to create a username and password

    Open up the file in a text editor and read how to configure and run.
    Attached Files
  • uva3021
    SBR Wise Guy
    • 03-01-07
    • 537

    #2
    accidentally uploaded older version, set up a github repo so you can just grab it from there

    GitHub is where people build software. More than 150 million people use GitHub to discover, fork, and contribute to over 420 million projects.
    Comment
    • Maverick22
      SBR Wise Guy
      • 04-10-10
      • 807

      #3
      I will give this a try. What are you experiences with it so far? Are you happy with it?
      Comment
      • Indecent
        SBR Wise Guy
        • 09-08-09
        • 758

        #4
        I didn't have time to test it out but just wanted to say it's cool that you are open sourcing some of your code.

        I have plans to open source some of my database schemas and some db related code, I'll be sure to post them here when I do. If I get time this weekend I'll test this out for you if no one else has.
        Comment
        • matthew919
          SBR Sharp
          • 11-21-12
          • 421

          #5
          Thanks for sharing this. I'm dusting off the cobwebs from my database corner of my brain at the moment, and I need to write a script that will download play by play and event files mid-season. Does anyone happen to know if retrosheet updates their game log files on a daily basis? Or are all season stats just posted at the end of the year? If the latter, is it even possible to update a database in the middle of a season with the current season's stats using retrosheet? Or do I need to look to scraping the MLB.com box scores?

          Thanks,
          Matt
          Comment
          • Maverick22
            SBR Wise Guy
            • 04-10-10
            • 807

            #6
            I don't know anything about retrosheet. Look at the website, and look in their FAQ section, that has to have been asked (hopefully).

            If you are going to do custom coding for a web scraper... then you should get your data from the mlb.com xml files.
            Comment
            • bztips
              SBR Sharp
              • 06-03-10
              • 283

              #7
              All retrosheet game and event files are posted after the season, usually late December or early January. There is nothing available mid-season; you would have to do a daily scrape from mlb.com or elsewhere.
              Comment
              • matthew919
                SBR Sharp
                • 11-21-12
                • 421

                #8
                Thanks, bz. Sigh, I guess it's time to learn a new language. Does anyone have a scraping script that I can get started with? It would save me untold time and effort, I'm sure. What do I have... 79 SBR points at the moment? They're all yours for anyone kind enough to provide well-commented code with an example. (Preferably using Python or php, since I have at least some working knowledge of those languages...)

                Cheers,
                Matt
                Last edited by matthew919; 01-07-13, 02:31 PM.
                Comment
                SBR Contests
                Collapse
                Top-Rated US Sportsbooks
                Collapse
                Working...