Oddsportal opening and closing odds

Collapse
X
 
  • Time
  • Show
Clear All
new posts
  • dimitrisko
    SBR Rookie
    • 05-21-16
    • 6

    #1
    Oddsportal opening and closing odds
    Hello everyone i am new here. I see you have some skills and would like to make a question.
    Well i want to scrape oddsportal historical odds to put them in excel but the thing is i want opening and closing odds for several lines like 1X2, ht/ft over/under from several bookmakers etc. The problem is that the opening odds are not able to see if i disable javascript, also not visible when i open developer tools. It's like a pop up think that shows opening odds but cannot be seen in the code. Do anyone know how to do it?
    Last edited by dimitrisko; 05-21-16, 12:56 PM.
  • statnerds
    SBR MVP
    • 09-23-09
    • 4047

    #2
    perhaps it would be easier to begin at the end

    what is your intention here, once you have the data secured, what is it you think you can gain from back testing?
    Comment
    • dimitrisko
      SBR Rookie
      • 05-21-16
      • 6

      #3
      well i want to gather more statistics to secure my picks the best i can. I want to archive a lot of opening and closing odds from 1X2, over/under, gg/ng and ht/ft. There is some pattern in this i am sure and mostly in opening-closing odds of 1X2 moneyline from 2-3 bookmakers. The problem with oddsportal is that if you put the mouse on an odd you see a pop up thing that say all changes of odds before it start and also opening odd but if i disable javascript from the page this pop up thing doesn't work and in developer mode you cannot see where is this odd coming from or whatever. If someone know a way to do it would be great or if someone know a site that keep that statistics in easy way.
      Comment
      • benjy
        SBR MVP
        • 02-19-09
        • 2158

        #4
        Look into headless browsers.

        I use Python with the Spynner package. Had I to do it again I'd likely not use Spynner and try Selenium with PhantomJS.
        Comment
        • bullock
          SBR Hustler
          • 08-11-10
          • 60

          #5
          I wrote a scraper in PHP some 4 years ago, it wasn't a piece of cake
          Comment
          • PerfectGrape
            SBR Hall of Famer
            • 09-20-11
            • 6761

            #6
            selenium webdriver with java is much easier to use than learning php/beautiful soup in my opinion if you already can code in java. you can locate things on the page, hover your mouse over boxes, click on stuff, emulate a real person using a browser. i use the firefox driver, the headless drivers don't always work as well. this dude is also trying to scrape oddsportal like you. scraping espn, yahoo, nba.com box scores with selenium webdriver is a piece of cake, wrote an nba scraper in an afternoon. it starts to get hard when the websites become more advanced like oddsportal.

            Last edited by PerfectGrape; 05-22-16, 03:37 PM.
            Comment
            • PerfectGrape
              SBR Hall of Famer
              • 09-20-11
              • 6761

              #7
              something like this



              bookName,homeOpener,homeCloser,drawOpene r,drawCloser,awayOpener,awayCloser
              10Bet,-185,-213,+330,+370,+445,+580
              12BET,-189,-213,+300,+359,+500,+560
              188BET,-185,-208,+290,+359,+409,+560
              18bet,-196,-213,+295,+355,+440,+560
              1xBet,-149,-200,+332,+376,+450,+640
              5Dimes,-189,-204,+290,+380,+425,+618
              888sport,-167,-208,+320,+375,+525,+615
              Bestbet,-159,-179,+300,+350,+400,+500
              bet-at-home,-192,-217,+305,+338,+501,+543
              bet365,-175,-200,+280,+375,+500,+600
              Betclic,-192,-208,+309,+340,+525,+550
              Betfred,-200,missing,+300,+359,+600,missing
              BetGun,-172,-192,+305,+345,+495,+505
              BetOnline,-182,-213,+309,+350,+459,+580
              Betrally,-185,-213,+334,+375,+450,+585
              Betsafe,-182,-200,+340,+350,+525,+575
              Betsson,-182,-200,+340,+359,+550,+625
              BetVictor,-200,-227,+333,+380,+600,missing
              Betway,-189,-222,+330,+350,+500,+600
              BoyleSports,-189,-200,+320,+350,+500,+600
              bwin,-154,-222,+250,+359,+450,+625
              ComeOn,-185,-208,+330,+359,+445,+570
              Coral,-149,-200,+300,+350,+450,+600
              Dafabet,-189,-213,+300,+359,+500,+560
              DOXXbet,-192,-227,+309,+340,+476,+537
              Expekt,-192,-208,+309,+340,+525,+550
              Guts,-182,-200,+325,+359,+525,+575
              Intertops,-182,-222,+309,+340,+500,+550
              Interwetten,-222,-182,+370,+340,+550,+450
              Island Casino,-189,-204,+290,+380,+425,+618
              Jetbull,-159,-200,+300,+350,+400,+600
              Jojobet,-185,-213,+330,+370,+445,+580
              Ladbrokes,-182,-200,+309,+359,+400,+600
              Leonbets,-172,-222,+279,+342,+450,+576
              Luxbet,-208,-213,+330,+345,+550,missing
              Marathonbet,-164,-208,+345,+395,+490,+630
              mybet,-200,-222,+330,+350,+550,+600
              NetBet,-185,-200,+330,+340,+445,+560
              NordicBet,-182,-208,+340,+359,+525,+600
              Noxwin,-154,-208,+300,+350,+450,+500
              Oddsring,-167,-217,+286,+359,+462,+592
              OtoBet,-167,-213,+300,+365,+500,+620
              Pinnacle Sports,-185,-204,+351,+388,+565,+591
              RealDealBet,-185,-213,+330,+365,+445,+575
              Redbet,-185,-204,+330,+359,+560,+600
              SBOBET,-196,-213,+295,+350,+440,+580
              Sportingbet,-182,-227,+333,+359,+500,+600
              Tempobet,-189,-167,+334,+315,+600,+550
              Tipico,-154,-200,+280,+350,+450,+550
              Titanbet,-200,-208,+320,+375,+550,missing
              TonyBet,-182,-208,+355,+384,+509,+590
              Unibet,-182,-208,+350,+375,+535,+615
              Vernons,-200,-208,+333,+375,+525,+550
              VitalBet,-189,-213,+345,+388,+568,+588
              William Hill,-182,-200,+300,+320,+500,+550
              Winlinebet,-185,-204,+318,+367,+520,+587
              Winner,-200,-208,+333,+375,+525,+550
              youwin,-182,-227,+333,+359,+500,+600
              Comment
              • dimitrisko
                SBR Rookie
                • 05-21-16
                • 6

                #8
                Yes PerfectGrape excactly like this. I want to scrape in this way Date,League,Match,Bookmaker for 1X2, o/u etc did you do it?
                Comment
                • dimitrisko
                  SBR Rookie
                  • 05-21-16
                  • 6

                  #9
                  I would like to mention also that i am new to programing (begginer at python) and all this is a pain. If someone could guide me through i would appreciate it.
                  Comment
                  • PerfectGrape
                    SBR Hall of Famer
                    • 09-20-11
                    • 6761

                    #10
                    You on Windows/linux? How many hours are you willing to put in learning to code? Because it might be anywhere from 15-100 hours, or even longer. I have no idea how long it takes to learn a programming language. You might be better off hiring a programmer(me) if you're only learning python to scrape lines.

                    I don't really know python too well, http://www.learnpython.org/



                    to scrape a site like oddsportal, you are probably going to have to learn to use Selenium or some other tool. Not sure about the other tools, but I know you can acheieve everything you are looking to do through python and Selenium. I wrote the code in Java, so maybe you could learn from my code to get you a head start.

                    I use google chrome to inspect the sites to find what i need to scrape.

                    for example on this page http://www.oddsportal.com/soccer/eur...hJZn1pn/#1X2;2

                    in google chrome, right click on the first link to 10Bet, click Inspect. It should open up the Chrome DevTools. Find the highlighted section, right click, go to copy, and click Copy Xpath.

                    Discover the magic of the internet at Imgur, a community powered entertainment destination. Lift your spirits with funny jokes, trending memes, entertaining gifs, inspiring stories, viral videos, and so much more from users.


                    Look at what you are inspecting, it looks like a link to the 10Bet website.

                    {a class="name" title="Go to 10Bet website!" onclick="return !window.open(this.href)" href="/bookmaker/10bet/link/"}10Bet{/a}

                    Now paste into notepad or a word processor. You should see the double quotation marks. You need to edit them to single quotation marks as shown.

                    before editing:
                    //*[@id="odds-data-table"]/div[1]/table/tbody/tr[1]/td[1]/div/a[2]

                    after editing:
                    //*[@id='odds-data-table']/div[1]/table/tbody/tr[1]/td[1]/div/a[2]

                    This is called an XPath which is a unique way to get to that exact part of any word, image, or link on a webpage.

                    Selenium is a tool to automate web browsers. The Java code below automatically opens up a new Firefox browser, waits 10 seconds for the page to load

                    Pastebin.com is the number one paste tool since 2002. Pastebin is a website where you can store text online for a set period of time.


                    In Selenium, there is a function you can call named findElement.
                    In python it looks like this
                    driver.find_element_by_xpath( )

                    You are going to be calling this all the time. This is the workhorse of selenium. It is pretty much the only thing you need to use.

                    Now you take the Xpath from earlier and put it in the findElement or find_element_by_xpath( )

                    You also have to add the getText() part in Java to get the words in between the brackets <a> </a>. Not sure if you need to do that in python, it might automatically get the Text for you.

                    So to summarize,
                    1. Go to google chrome
                    2. right click on whatever you want to scrape and hit Inspect.
                    3. Find the highlighted section, right click, copy, copy by xpath
                    4. Change the Xpath from double quotation marks to single quotation marks.
                    5. Create a new FireFox Driver in selenium
                    6. load page in selenium and do a wait of 10 seconds for the page to load
                    7. use the find_element_by_xpath in selenium

                    Now you need to write a loop to go through all the rows. the tr[1] means row 1, etc
                    //*[@id='odds-data-table']/div[1]/table/tbody/tr[1]/td[1]/div/a[2]
                    //*[@id='odds-data-table']/div[1]/table/tbody/tr[2]/td[1]/div/a[2]
                    //*[@id='odds-data-table']/div[1]/table/tbody/tr[3]/td[1]/div/a[2]
                    ....
                    should be enough to get you started
                    Comment
                    • dimitrisko
                      SBR Rookie
                      • 05-21-16
                      • 6

                      #11
                      Thanks for your perfect response perfectgrape. Its really helpful. I will do whatever you say here and hope to make somehow. The code you wrote is good but when i inspect the odds the opening odds are nowhere the only thing it writes in page source is onmouseover and odds are nowhere here is what i meanClick image for larger version

Name:	Screenshot_1.png
Views:	1
Size:	47.9 KB
ID:	29133915Click image for larger version

Name:	Screenshot_2.png
Views:	1
Size:	50.6 KB
ID:	29133916. You think with selenium i can find those onmouse over odds?
                      Comment
                      • PerfectGrape
                        SBR Hall of Famer
                        • 09-20-11
                        • 6761

                        #12
                        it can be done with selenium, will post a solution in a few days when I'm free
                        Comment
                        • PerfectGrape
                          SBR Hall of Famer
                          • 09-20-11
                          • 6761

                          #13
                          In google chrome, right click, inspect on the odds.

                          Do the usual, right click inspect

                          Discover the magic of the internet at Imgur, a community powered entertainment destination. Lift your spirits with funny jokes, trending memes, entertaining gifs, inspiring stories, viral videos, and so much more from users.


                          Quickly scroll to the bottom of the code, you have 5-10 seconds before it disappears. Or maybe it just disappears because my mouse goes to a different part and generates new code. Look for one of the last lines, div id =tooltipdiv

                          Discover the magic of the internet at Imgur, a community powered entertainment destination. Lift your spirits with funny jokes, trending memes, entertaining gifs, inspiring stories, viral videos, and so much more from users.


                          Expand the arrows to find the opening odds inside.

                          Discover the magic of the internet at Imgur, a community powered entertainment destination. Lift your spirits with funny jokes, trending memes, entertaining gifs, inspiring stories, viral videos, and so much more from users.


                          Now if you try to copy the opening odds by Xpath, this is what it displays.

                          //*[@id="tooltiptext"]/strong[1]

                          Now if you tried to find_element_by_xpath( ) with the above XPath, it would not work. It will only work if the user's mouse is over one of those boxes to make the onMouse box pop up. There is code in Selenium to fool the browser into thinking there is a mouse cursor there so that the odds box will pop up.

                          driver = new FirefoxDriver();
                          driver.get("http://www.oddsportal.com/soccer/europe/champions-league/real-madrid-manchester-city-dhJZn1pn/#1X2;2");
                          TimeUnit.SECONDS.sleep(10);

                          WebElement imaginaryMouseLocation = driver.findElement(By.xpath("//*[@id='odds-data-table']/div[1]/table/tbody/tr[1]/td[2]/div"));

                          Actions action = new Actions(driver);
                          action.moveToElement(imaginaryMouseLocat ion);
                          action.perform();
                          TimeUnit.SECONDS.sleep(3);
                          WebElement awayOdds = driver.findElement(By.xpath("//*[@id='tooltiptext']/strong[1]"));
                          System.out.printf(awayOdds.getText());

                          Summary
                          1. Find the XPath of where you want the "imaginary mouse" in Selenium to go.
                          2. Create a new Action,
                          3. moveToElement,
                          4. Perform,
                          5. Then do a wait for a few seconds to give the box time to appear.
                          6. Now that the box is appearing on the website, you can findElement "//*[@id='tooltiptext']/strong[1]"

                          Step 6 will always be the same, you will always try to findElement "//*[@id='tooltiptext']/strong[1]". Step 1 is where you have to change the xpath for all the different boxes.

                          Also most of the time you will see
                          "//*[@id='tooltiptext']/strong[1]" = the closing odds.
                          "//*[@id='tooltiptext']/strong[2]" = the opening odds

                          If the closing odds are missing, then
                          "//*[@id='tooltiptext']/strong[1]" = the opening odds
                          "//*[@id='tooltiptext']/strong[2]" = XPath to nothing, if you tried to findElement it would give you an error

                          findElement is to only find a single element. findElements will find multiple elements and put them into a data structure.

                          If you wanted to find a single element..
                          WebElement awayOdds = driver.findElement(By.xpath("//*[@id='tooltiptext']/strong[1]"));


                          if you wanted to find all the elements which contain the "strong" tab (strong[1] and strong[2])
                          List<WebElement> awayOdds = driver.findElements(By.xpath("//*[@id='tooltiptext']/strong"));

                          You can use the findElements and then test the size of the data structure. If the size is 1, then the closing odds are missing so
                          "//*[@id='tooltiptext']/strong[1]" = the opening odds

                          If the size is 2, then
                          "//*[@id='tooltiptext']/strong[1]" = closing odds
                          "//*[@id='tooltiptext']/strong[2]" = opening odds

                          My approach is to just findElements, test the size of the data structure, and then access elements of the data structure to get the odds.

                          Here is the code in Java. The bold part is the only part you need to change, loop through it to visit rows 1, 2, 3.... columns 2(home odds), 3(draw odds), 4(away odds)

                          WebDriver driver = new FirefoxDriver();
                          driver.get("http://www.oddsportal.com/soccer/europe/champions-league/real-madrid-manchester-city-dhJZn1pn/#1X2;2");
                          TimeUnit.SECONDS.sleep(10);


                          Actions action = new Actions(driver);
                          WebElement imaginaryMouseLocation = driver.findElement(By.xpath("//*[@id='odds-data-table']/div[1]/table/tbody/tr[1]/td[2]/div"));
                          action.moveToElement(imaginaryMouseLocat ion);
                          action.perform();
                          TimeUnit.SECONDS.sleep(3);
                          List{WebElement} awayFull = driver.findElements(By.xpath("//*[@id='tooltiptext']/strong"));
                          if (awayFull.size() == 2) {
                          System.out.printf(awayFull.get(1).getTex t() + "," + awayFull.get(0).getText() + ",");
                          } else {
                          System.out.printf(awayFull.get(0).getTex t() + "," + "missing" + ",");
                          }

                          The example code might have gotten home and away confused, i forget that the home team is listed first in soccer

                          These guys write python code doing the moveToElement, maybe it will help you. Good luck

                          Comment
                          • dimitrisko
                            SBR Rookie
                            • 05-21-16
                            • 6

                            #14
                            Thank you very much for the detailed answer and for the code!! I 'll try to do it with your way in java first and if i cannot do it i will try to do similar in python with the link above.
                            Comment
                            • PerfectGrape
                              SBR Hall of Famer
                              • 09-20-11
                              • 6761

                              #15
                              Originally posted by dimitrisko
                              Thank you very much for the detailed answer and for the code!! I 'll try to do it with your way in java first and if i cannot do it i will try to do similar in python with the link above.
                              here is my code in java, it expires in a week so copy it onto your computer if you want to save it.

                              Pastebin.com is the number one paste tool since 2002. Pastebin is a website where you can store text online for a set period of time.


                              Importing Selenium is a little tricky, I use Maven to import Selenium into Eclipse. You may not need to use Maven to import Selenium into eclipse, but thats how I do it. Probably easier to avoid using Maven if possible, one less thing to learn.

                              Maven is a tool which automatically gets all the stuff you need for Selenium or any other programming library, but it takes a bit of time to learn. You might not need to use Maven, you might be able to import Selenium into Eclipse without Maven. I found some youtube video walking through it, this is with older versions of Eclipse and Selenium though.

                              Eclipse is good to use if you are learning Java, if you have bugs you can do AutoFix which will suggest the best fix and automatically fix your bugs. I would use IntelliJ or Eclipse to learn/program in Java.

                              Selenium is an umbrella project for a range of tools and libraries that enable and support the automation of web browsers. It provides extensions to emulate user interaction with browsers, a distribution server for scaling browser allocation, and the infrastructure for implementations of the W3C WebDriver specification that lets you write interchangeable code for all major web browsers. This project is made possible by volunteer contributors who have put in thousands of hours of their own time, and made the source code freely available for anyone to use, enjoy, and improve.




                              Downloads Below is where you can find the latest releases of all the Selenium components. You can also find a list of previous releases, source code, and additional information for Maven users.


                              The essential tools for any Java developer, including a Java IDE, a Git client, XML Editor, Mylyn, Maven integration and WindowBuilder




                              Comment
                              • Ezone
                                SBR Rookie
                                • 08-21-15
                                • 3

                                #16
                                Nice thread for the newbies
                                Please does this still works?
                                Comment
                                • Naokia980
                                  SBR Rookie
                                  • 08-28-16
                                  • 18

                                  #17
                                  They are changed a lot!
                                  Comment
                                  SBR Contests
                                  Collapse
                                  Top-Rated US Sportsbooks
                                  Collapse
                                  Working...