Trying to get odds data

[elo]

Hi, i can't scrap odds data from betfair's website. I'm using PHP. I'm getting this error all the time:

Error:

response: {"faultcode":"Client","faultstring":"DSC-0008","detail":{}}

Could anyone tell me what's wrong with my code?

$url = "[URL="https://www.betfair.com/www/sports/navigation/facet/v1/search?_ak=nzIFcwyWhrlwYMrh&alt=json"]https://www.betfair.com/www/sports/n...wYMrh&alt=json[/URL]";

$ch = curl_init();

curl_setopt($ch, CURLOPT_URL, $url);
curl_setopt($ch, CURLOPT_POST, 1);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);
curl_setopt($ch, CURLOPT_SSL_VERIFYPEER, 0);
curl_setopt($ch, CURLOPT_HTTPHEADER, array(
    #'User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/80.0.3987.149 Safari/537.36',
    'Accept: application/json',
    'Content-Type: application/json',
    'Referer: [URL]https://www.betfair.com/exchange/plus/football')[/URL]);

$postData = '[{ "params": {"filter": {}} }]';


curl_setopt($ch, CURLOPT_POSTFIELDS, $postData);

# execute

$output = curl_exec($ch);
curl_close($ch);

[jabe]

When my prog sends curls, everything I want to be considered a string on receipt at the destination has double quotes round it. If you could show us the final curl string you send, that would make things massively easier.

I've never seen any code on here from anyone attempting to scrape data from Betfair. Everyone else has been using an API.

[elo]

Hi jabe, thank you for your response. Here is my final curl string:

final curl string: Array
(
[url] => https://www.betfair.com/www/sports/navigation/facet/v1/search?_ak=nzIFcwyWhrlwYMrh&alt=json
[content_type] =>
[http_code] => 0
[header_size] => 0
[request_size] => 0
[filetime] => -1
[ssl_verify_result] => 0
[redirect_count] => 0
[total_time] => 0
[namelookup_time] => 0
[connect_time] => 0
[pretransfer_time] => 0
[size_upload] => 0
[size_download] => 0
[speed_download] => 0
[speed_upload] => 0
[download_content_length] => -1
[upload_content_length] => -1
[starttransfer_time] => 0
[redirect_time] => 0
[redirect_url] =>
[primary_ip] =>
[certinfo] => Array
(
)

[primary_port] => 0
[local_ip] =>
[local_port] => 0
[http_version] => 0
[protocol] => 0
[ssl_verifyresult] => 0
[scheme] =>
[appconnect_time_us] => 0
[connect_time_us] => 0
[namelookup_time_us] => 0
[pretransfer_time_us] => 0
[redirect_time_us] => 0
[starttransfer_time_us] => 0
[total_time_us] => 0
)

http status: 400

response: {"faultcode":"Client","faultstring":"DSC-0008","detail":{}}

type of response: string
Process finished with exit code 0

I was inspired by this solution: https://www.betangel.com/forum/viewtopic.php?t=18742

[jabe]

The DSC-0008 means there's some sort of syntax error in the code you're using to request data.

It's been a long time since I used PHP. Could you show me what's in these?
$ch, CURLOPT_POSTFIELDS, $postData

[elo]

Hi jabe, all my code is in first post. Could you precise please what should i show you?

[jabe]

Ah, okay. The JSON coding isn't right. Everything that's a string needs double quotes round it.

Where you've got a string containing

Accept: application/json

It needs to be

"Accept": "application/json"

The name of the parameter has to be in quotes. The parameter value can be without quotes if it's numeric (but it'll be accepted with quotes in most cases). Always wrap marketIds in quotes; they look numeric, but they're not, and if you omit quotes, you could lose a final zero.


These might help with JSON:

https://www.w3schools.com/js/js_json_intro.asp

https://jsonlint.com/?code=

[elo]

Thank you jabe. I changed format of my request string but now I'm getting this kind of error:

response: {"faultcode":"Server","faultstring":"DSC-0002","detail":{}}

I couldn't find what that error means on the official betfair api website. I've just found that it means "FrameworkError" on some page on the github (https://github.com/betfair/cougar-do...t_Reporting.md).

Except that "DSC-0002" i'm getting HTTP 500 eror. Sorry for many edits.


Here is the modified request:

$url = "https://www.betfair.com/www/sports/navigation/facet/v1/search?_ak=nzIFcwyWhrlwYMrh&alt=json";
$ch = curl_init();
curl_setopt($ch, CURLOPT_URL, $url);
curl_setopt($ch, CURLOPT_POST, 1);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);
curl_setopt($ch, CURLOPT_SSL_VERIFYPEER, 0);
curl_setopt($ch, CURLINFO_HEADER_OUT, true);
curl_setopt($ch, CURLOPT_HTTPHEADER, array(
'"Accept":"application/json",
"Content-Type":"application/json",
"Referer":"https://www.betfair.com/exchange/plus/football"' ));

$postData = '[{"filter": {} ]';

Here is the response:

final curl string: Array
(
[url] => https://www.betfair.com/www/sports/navigation/facet/v1/search?_ak=nzIFcwyWhrlwYMrh&alt=json
[content_type] =>
[http_code] => 0
[header_size] => 0
[request_size] => 0
[filetime] => -1
[ssl_verify_result] => 0
[redirect_count] => 0
[total_time] => 0
[namelookup_time] => 0
[connect_time] => 0
[pretransfer_time] => 0
[size_upload] => 0
[size_download] => 0
[speed_download] => 0
[speed_upload] => 0
[download_content_length] => -1
[upload_content_length] => -1
[starttransfer_time] => 0
[redirect_time] => 0
[redirect_url] =>
[primary_ip] =>
[certinfo] => Array
(
)

[primary_port] => 0
[local_ip] =>
[local_port] => 0
[http_version] => 0
[protocol] => 0
[ssl_verifyresult] => 0
[scheme] =>
[appconnect_time_us] => 0
[connect_time_us] => 0
[namelookup_time_us] => 0
[pretransfer_time_us] => 0
[redirect_time_us] => 0
[starttransfer_time_us] => 0
[total_time_us] => 0
)

http status: 500

response: {"faultcode":"Server","faultstring":"DSC-0002","detail":{}}

type of response: string
Process finished with exit code 0

[jabe]

I can't find a description for DSC-0002, but an http error 500 means something is wrong at the web

[elo]

After i added "curl_setopt($ch,CURLOPT_HEADER, true);" option to my header, I'm getting additional info about error. I've used Google Chrome developer tools to check how looks request header. Maybe link to send post request is wrong? I mean:

https://www.betfair.com/www/sports/navigation/facet/v1/search?_ak=nzIFcwyWhrlwYMrh&alt=json

HTTP/1.1 500 Server Error
Date: Fri, 22 May 2020 12:49:25 GMT
Cache-Control: no-cache
Content-Type: application/json;charset=utf-8
Content-Length: 59
Content-Security-Policy-Report-Only: default-src 'unsafe-inline' 'unsafe-eval' data: https: wss:; report-uri https://shk.betfair.com/csp
X-XSS-Protection: 1; mode=block

{"faultcode":"Server","faultstring":"DSC-0002","detail":{}}

[jabe]

We do have another problem; I understand what the ultimate purpose of your code is, but I'm not familiar with the method or how it works.

[elo]

Ok, I understand. Thank you for your help jabe. Is it even possible to scrap odds data from Betfair without using API? As I mentioned above, I've used some info from this thread: https://www.betangel.com/forum/viewtopic.php?t=18742. That guy didn't use API and probably, as he wrote, he succeeded in scrap odds.

[jabe]

I don't know, I'm afraid. I think the person who posted originally on the BetAngel site made a typo - I'm pretty sure taking data from websites in this manner is called screen scraping, so he should have written scrape rather than scrap. However, from the links you provided, it appears that the software you're using was written by Betfair themselves, so it should be usable.

In the past I have "borrowed" data from websites by downloading an entire webpage and saving it as a file so that I could later write a program to read each file and extract the specific data that I wanted from it. (I used PHP, as it was able to read a webpage into a string, so I could then save it as a file. This was around 20 years ago, however!)

I've looked at the links you provided. Perhaps it's something I should be taking a closer look at myself. Unfortunately things are very busy at present and are likely to remain that way, so I don't know when I'd be able to find the time to do it.

If I find time to try it out for myself and can get it to work, I'll return to this page and add whatever information seems important. In the meantime, I wish you luck.

[elo]

Thank you again jabe! You are very helpful! I will try "to fight" with this 500 error and I will wait for your so valuable information

[bfexplorer]

Thank you again jabe! You are very helpful! I will try "to fight" with this 500 error and I will wait for your so valuable information

You are trying to access not public web api of betfair web site, what could be tolerated to some level.

You should use betfair api.

What data are you actually trying to parse from betfair web site?

[jabe]

You are trying to access not public web api of betfair web site, what could be tolerated to some level.

You should use betfair api.

What data are you actually trying to parse from betfair web site?

But using something called Cougar, which was written by Betfair.