[dimitrisko]

Hello everyone i am new here. I see you have some skills and would like to make a question.
Well i want to scrape oddsportal historical odds to put them in excel but the thing is i want opening and closing odds for several lines like 1X2, ht/ft over/under from several bookmakers etc. The problem is that the opening odds are not able to see if i disable javascript, also not visible when i open developer tools. It's like a pop up think that shows opening odds but cannot be seen in the code. Do anyone know how to do it?

[statnerds]

perhaps it would be easier to begin at the end

what is your intention here, once you have the data secured, what is it you think you can gain from back testing?

[dimitrisko]

well i want to gather more statistics to secure my picks the best i can. I want to archive a lot of opening and closing odds from 1X2, over/under, gg/ng and ht/ft. There is some pattern in this i am sure and mostly in opening-closing odds of 1X2 moneyline from 2-3 bookmakers. The problem with oddsportal is that if you put the mouse on an odd you see a pop up thing that say all changes of odds before it start and also opening odd but if i disable javascript from the page this pop up thing doesn't work and in developer mode you cannot see where is this odd coming from or whatever. If someone know a way to do it would be great or if someone know a site that keep that statistics in easy way.

[benjy]

Look into headless browsers.

I use Python with the Spynner package. Had I to do it again I'd likely not use Spynner and try Selenium with PhantomJS.

[bullock]

I wrote a scraper in PHP some 4 years ago, it wasn't a piece of cake

[PerfectGrape]

selenium webdriver with java is much easier to use than learning php/beautiful soup in my opinion if you already can code in java. you can locate things on the page, hover your mouse over boxes, click on stuff, emulate a real person using a browser. i use the firefox driver, the headless drivers don't always work as well. this dude is also trying to scrape oddsportal like you. scraping espn, yahoo, nba.com box scores with selenium webdriver is a piece of cake, wrote an nba scraper in an afternoon. it starts to get hard when the websites become more advanced like oddsportal.

http://stackoverflow.com/questions/2...nium-in-python

[PerfectGrape]

something like this

http://www.oddsportal.com/soccer/eur...hJZn1pn/#1X2;2

bookName,homeOpener,homeCloser,drawOpene r,drawCloser,awayOpener,awayCloser
10Bet,-185,-213,+330,+370,+445,+580
12BET,-189,-213,+300,+359,+500,+560
188BET,-185,-208,+290,+359,+409,+560
18bet,-196,-213,+295,+355,+440,+560
1xBet,-149,-200,+332,+376,+450,+640
5Dimes,-189,-204,+290,+380,+425,+618
888sport,-167,-208,+320,+375,+525,+615
Bestbet,-159,-179,+300,+350,+400,+500
bet-at-home,-192,-217,+305,+338,+501,+543
bet365,-175,-200,+280,+375,+500,+600
Betclic,-192,-208,+309,+340,+525,+550
Betfred,-200,missing,+300,+359,+600,missing
BetGun,-172,-192,+305,+345,+495,+505
BetOnline,-182,-213,+309,+350,+459,+580
Betrally,-185,-213,+334,+375,+450,+585
Betsafe,-182,-200,+340,+350,+525,+575
Betsson,-182,-200,+340,+359,+550,+625
BetVictor,-200,-227,+333,+380,+600,missing
Betway,-189,-222,+330,+350,+500,+600
BoyleSports,-189,-200,+320,+350,+500,+600
bwin,-154,-222,+250,+359,+450,+625
ComeOn,-185,-208,+330,+359,+445,+570
Coral,-149,-200,+300,+350,+450,+600
Dafabet,-189,-213,+300,+359,+500,+560
DOXXbet,-192,-227,+309,+340,+476,+537
Expekt,-192,-208,+309,+340,+525,+550
Guts,-182,-200,+325,+359,+525,+575
Intertops,-182,-222,+309,+340,+500,+550
Interwetten,-222,-182,+370,+340,+550,+450
Island Casino,-189,-204,+290,+380,+425,+618
Jetbull,-159,-200,+300,+350,+400,+600
Jojobet,-185,-213,+330,+370,+445,+580
Ladbrokes,-182,-200,+309,+359,+400,+600
Leonbets,-172,-222,+279,+342,+450,+576
Luxbet,-208,-213,+330,+345,+550,missing
Marathonbet,-164,-208,+345,+395,+490,+630
mybet,-200,-222,+330,+350,+550,+600
NetBet,-185,-200,+330,+340,+445,+560
NordicBet,-182,-208,+340,+359,+525,+600
Noxwin,-154,-208,+300,+350,+450,+500
Oddsring,-167,-217,+286,+359,+462,+592
OtoBet,-167,-213,+300,+365,+500,+620
Pinnacle Sports,-185,-204,+351,+388,+565,+591
RealDealBet,-185,-213,+330,+365,+445,+575
Redbet,-185,-204,+330,+359,+560,+600
SBOBET,-196,-213,+295,+350,+440,+580
Sportingbet,-182,-227,+333,+359,+500,+600
Tempobet,-189,-167,+334,+315,+600,+550
Tipico,-154,-200,+280,+350,+450,+550
Titanbet,-200,-208,+320,+375,+550,missing
TonyBet,-182,-208,+355,+384,+509,+590
Unibet,-182,-208,+350,+375,+535,+615
Vernons,-200,-208,+333,+375,+525,+550
VitalBet,-189,-213,+345,+388,+568,+588
William Hill,-182,-200,+300,+320,+500,+550
Winlinebet,-185,-204,+318,+367,+520,+587
Winner,-200,-208,+333,+375,+525,+550
youwin,-182,-227,+333,+359,+500,+600

[dimitrisko]

Yes PerfectGrape excactly like this. I want to scrape in this way Date,League,Match,Bookmaker for 1X2, o/u etc did you do it?

[dimitrisko]

I would like to mention also that i am new to programing (begginer at python) and all this is a pain. If someone could guide me through i would appreciate it.

[PerfectGrape]

You on Windows/linux? How many hours are you willing to put in learning to code? Because it might be anywhere from 15-100 hours, or even longer. I have no idea how long it takes to learn a programming language. You might be better off hiring a programmer(me) if you're only learning python to scrape lines.

I don't really know python too well, http://www.learnpython.org/

http://selenium-python.readthedocs.i...g-started.html

to scrape a site like oddsportal, you are probably going to have to learn to use Selenium or some other tool. Not sure about the other tools, but I know you can acheieve everything you are looking to do through python and Selenium. I wrote the code in Java, so maybe you could learn from my code to get you a head start.

I use google chrome to inspect the sites to find what i need to scrape.

for example on this page http://www.oddsportal.com/soccer/eur...hJZn1pn/#1X2;2

in google chrome, right click on the first link to 10Bet, click Inspect. It should open up the Chrome DevTools. Find the highlighted section, right click, go to copy, and click Copy Xpath.

http://imgur.com/C5RSIl7

Look at what you are inspecting, it looks like a link to the 10Bet website.

{a class="name" title="Go to 10Bet website!" onclick="return !window.open(this.href)" href="/bookmaker/10bet/link/"}10Bet{/a}

Now paste into notepad or a word processor. You should see the double quotation marks. You need to edit them to single quotation marks as shown.

before editing:
//*[@id="odds-data-table"]/div[1]/table/tbody/tr[1]/td[1]/div/a[2]

after editing:
//*[@id='odds-data-table']/div[1]/table/tbody/tr[1]/td[1]/div/a[2]

This is called an XPath which is a unique way to get to that exact part of any word, image, or link on a webpage.

Selenium is a tool to automate web browsers. The Java code below automatically opens up a new Firefox browser, waits 10 seconds for the page to load

http://pastebin.com/QcVuDdai

In Selenium, there is a function you can call named findElement.
In python it looks like this
driver.find_element_by_xpath( )

You are going to be calling this all the time. This is the workhorse of selenium. It is pretty much the only thing you need to use.

Now you take the Xpath from earlier and put it in the findElement or find_element_by_xpath( )

You also have to add the getText() part in Java to get the words in between the brackets . Not sure if you need to do that in python, it might automatically get the Text for you.

So to summarize,
1. Go to google chrome
2. right click on whatever you want to scrape and hit Inspect.
3. Find the highlighted section, right click, copy, copy by xpath
4. Change the Xpath from double quotation marks to single quotation marks.
5. Create a new FireFox Driver in selenium
6. load page in selenium and do a wait of 10 seconds for the page to load
7. use the find_element_by_xpath in selenium

Now you need to write a loop to go through all the rows. the tr[1] means row 1, etc
//*[@id='odds-data-table']/div[1]/table/tbody/tr[1]/td[1]/div/a[2]
//*[@id='odds-data-table']/div[1]/table/tbody/tr[2]/td[1]/div/a[2]
//*[@id='odds-data-table']/div[1]/table/tbody/tr[3]/td[1]/div/a[2]
....
should be enough to get you started

[dimitrisko]

Thanks for your perfect response perfectgrape. Its really helpful. I will do whatever you say here and hope to make somehow. The code you wrote is good but when i inspect the odds the opening odds are nowhere the only thing it writes in page source is onmouseover and odds are nowhere here is what i meanScreenshot_1.pngScreenshot_2.png. You think with selenium i can find those onmouse over odds?

[PerfectGrape]

it can be done with selenium, will post a solution in a few days when I'm free

[PerfectGrape]

In google chrome, right click, inspect on the odds.

Do the usual, right click inspect

http://imgur.com/j5z33v4

Quickly scroll to the bottom of the code, you have 5-10 seconds before it disappears. Or maybe it just disappears because my mouse goes to a different part and generates new code. Look for one of the last lines, div id =tooltipdiv

http://imgur.com/uDJgw7s

Expand the arrows to find the opening odds inside.

http://imgur.com/SQMmxyG

Now if you try to copy the opening odds by Xpath, this is what it displays.

//*[@id="tooltiptext"]/strong[1]

Now if you tried to find_element_by_xpath( ) with the above XPath, it would not work. It will only work if the user's mouse is over one of those boxes to make the onMouse box pop up. There is code in Selenium to fool the browser into thinking there is a mouse cursor there so that the odds box will pop up.

driver = new FirefoxDriver();
driver.get("http://www.oddsportal.com/soccer/europe/champions-league/real-madrid-manchester-city-dhJZn1pn/#1X2;2");
TimeUnit.SECONDS.sleep(10);

WebElement imaginaryMouseLocation = driver.findElement(By.xpath("//*[@id='odds-data-table']/div[1]/table/tbody/tr[1]/td[2]/div"));

Actions action = new Actions(driver);
action.moveToElement(imaginaryMouseLocat ion);
action.perform();
TimeUnit.SECONDS.sleep(3);
WebElement awayOdds = driver.findElement(By.xpath("//*[@id='tooltiptext']/strong[1]"));
System.out.printf(awayOdds.getText());

Summary
1. Find the XPath of where you want the "imaginary mouse" in Selenium to go.
2. Create a new Action,
3. moveToElement,
4. Perform,
5. Then do a wait for a few seconds to give the box time to appear.
6. Now that the box is appearing on the website, you can findElement "//*[@id='tooltiptext']/strong[1]"

Step 6 will always be the same, you will always try to findElement "//*[@id='tooltiptext']/strong[1]". Step 1 is where you have to change the xpath for all the different boxes.

Also most of the time you will see
"//*[@id='tooltiptext']/strong[1]" = the closing odds.
"//*[@id='tooltiptext']/strong[2]" = the opening odds

If the closing odds are missing, then
"//*[@id='tooltiptext']/strong[1]" = the opening odds
"//*[@id='tooltiptext']/strong[2]" = XPath to nothing, if you tried to findElement it would give you an error

findElement is to only find a single element. findElements will find multiple elements and put them into a data structure.

If you wanted to find a single element..
WebElement awayOdds = driver.findElement(By.xpath("//*[@id='tooltiptext']/strong[1]"));


if you wanted to find all the elements which contain the "strong" tab (strong[1] and strong[2])
List awayOdds = driver.findElements(By.xpath("//*[@id='tooltiptext']/strong"));

You can use the findElements and then test the size of the data structure. If the size is 1, then the closing odds are missing so
"//*[@id='tooltiptext']/strong[1]" = the opening odds

If the size is 2, then
"//*[@id='tooltiptext']/strong[1]" = closing odds
"//*[@id='tooltiptext']/strong[2]" = opening odds

My approach is to just findElements, test the size of the data structure, and then access elements of the data structure to get the odds.

Here is the code in Java. The bold part is the only part you need to change, loop through it to visit rows 1, 2, 3.... columns 2(home odds), 3(draw odds), 4(away odds)

WebDriver driver = new FirefoxDriver();
driver.get("http://www.oddsportal.com/soccer/europe/champions-league/real-madrid-manchester-city-dhJZn1pn/#1X2;2");
TimeUnit.SECONDS.sleep(10);


Actions action = new Actions(driver);
WebElement imaginaryMouseLocation = driver.findElement(By.xpath("//*[@id='odds-data-table']/div[1]/table/tbody/tr[1]/td[2]/div"));
action.moveToElement(imaginaryMouseLocat ion);
action.perform();
TimeUnit.SECONDS.sleep(3);
List{WebElement} awayFull = driver.findElements(By.xpath("//*[@id='tooltiptext']/strong"));
if (awayFull.size() == 2) {
System.out.printf(awayFull.get(1).getTex t() + "," + awayFull.get(0).getText() + ",");
} else {
System.out.printf(awayFull.get(0).getTex t() + "," + "missing" + ",");
}

The example code might have gotten home and away confused, i forget that the home team is listed first in soccer

These guys write python code doing the moveToElement, maybe it will help you. Good luck

http://stackoverflow.com/questions/2...nium-in-python

[dimitrisko]

Thank you very much for the detailed answer and for the code!! I 'll try to do it with your way in java first and if i cannot do it i will try to do similar in python with the link above.

[PerfectGrape]

Thank you very much for the detailed answer and for the code!! I 'll try to do it with your way in java first and if i cannot do it i will try to do similar in python with the link above.

here is my code in java, it expires in a week so copy it onto your computer if you want to save it.

http://pastebin.com/MkZbELBj

Importing Selenium is a little tricky, I use Maven to import Selenium into Eclipse. You may not need to use Maven to import Selenium into eclipse, but thats how I do it. Probably easier to avoid using Maven if possible, one less thing to learn.

Maven is a tool which automatically gets all the stuff you need for Selenium or any other programming library, but it takes a bit of time to learn. You might not need to use Maven, you might be able to import Selenium into Eclipse without Maven. I found some youtube video walking through it, this is with older versions of Eclipse and Selenium though.

Eclipse is good to use if you are learning Java, if you have bugs you can do AutoFix which will suggest the best fix and automatically fix your bugs. I would use IntelliJ or Eclipse to learn/program in Java.

http://www.seleniumhq.org/docs/appen..._via_maven.jsp

https://maven.apache.org/guides/gett...e-minutes.html

http://www.seleniumhq.org/download/maven.jsp

http://www.eclipse.org/downloads/pac...velopers/mars2

https://www.youtube.com/watch?v=xaWLIBKAQps

https://www.youtube.com/watch?v=zylSll8hsPs

[Ezone]

Nice thread for the newbies
Please does this still works?

[Naokia980]

They are changed a lot!