VBNet Sample (Yes really)

[SimonN]

Confirmation as Betdynamics called it:

Hi Simon.

I've investigated this further and can confirm that although the textQuery search provided by the API uses the same underlying service as the website, the website uses a specific internal Event Based Search API which was built specifically on top of the existing functionality to optimize the websites search; hence the difference between the results. There are no plans to expose the Event Based Search API used by the website to external developers.

Kind Regards,

Neil

Betfair Developer Program

[SimonN]

Application speed - how do they do it?

Guys,

How are sites like http://www.oddsmonkey.com/OddsSearch.aspx able to search the entire BF database and retrieve the information so quickly - if I think about the amount of requests involved there- I dont understand how they can do this so fast?

Thank you.

[jabe]

Hi Simon

It looks to me like they're constantly updating a database and just sending a snapshot of it when someone opens or refreshes the page.

I don't really think it is a great deal of data, to be honest. At present there are 239 matches on 10 pages, so as long as the site's query program knows the market id for Match Odds, all it needs to do is retrieve 239 marketBooks on a cycle basis.

Some of the games are several days away so the odds won't change much at all and may only need checking once every - what? - 5, 10, 15 minutes?, or possibly even more infrequently than that.

My program accumulates and displays the total amount of data it's retrieved and the number of API calls since it started. When I start the program, it has to retrieve MarketCatalogue data for events it doesn't know about (but then I save this data and reload it from file if I restart the program). There's often a lot of data to start with, but within maybe 20 minutes or so, it's averaging one call a second.

I've just started it up. It's 04.09 Wednesday morning. It's found 136 games for today (and reloaded 181 unresolved games from file from previous days) and in the first minute has made 518 API calls and loaded 2.5MB of data. That is, admittedly, a fair bit of data, but it includes one-offs, such as getting country and competition data, as well as querying the final state of the 181 games to find which the winning runners were, on top of the Event and marketCatalogue calls for the new games.

In the second minute, it's made 74 calls and loaded 290kB. Five minutes in, it's averaging fewer than 50 calls a minute since the first minute. Eight minutes in and it's downloaded 3.9MB, so that's 200k average for the last seven minutes or 12MB per hour. I've downloaded files bigger than that. At ten minutes in, it's made 848 calls, so that's 330 in the last 9 minutes, or about 37 a minute.

Compare that 12MB in an hour with the speed for streaming a film and it's way off. I've done a search and found an article that suggests 120MB/hr is the least you could expect. The Standard Definition rate for Netflix is 700MB/hr and their Ultra HD rate is a whopping 3GB/hr. That's big.

I'm about to close my program - it's now 935 calls in 1083 seconds.

Hope that's some help.

[SimonN]

Very helpful as always thank you Jabe.

The program, to my way of thinking, needs to check every single market price on BF to find out the percentage arb it is compared with bookmakers and then display - as you say - top 10 pages of highest percent arbs. How many requests would you think that that would be or maybe there is a way of getting all at once? Even then it has to get all the rest of the higher level details like event name and time and competition etc etc?!

This free edition is updated every 30 mins, so when you go to the bookie site at the point of the 30 min up date time you have the best chance of being successful in getting the advertised bookie price before it is cut.

The paid edition though is constantly live so it has to provide up to the second live prices at the bookies and the exchange and obviously both the bookie price and the exchange price have to be there for someone to avail themselves of the advertised arb in order for the program to work.

But yes the program cannot know which the top arbs are without first fetching every single marketbook price and checking these prices with bookie prices and then ranking the results.

[SimonN]

By making a request to the Navigation Data service, they are able to obtain a list of all BF markets in one visit which obviously in that case (oddsmonkey) makes it much faster.

I have managed to get this working as described in the developers guide ( "https://api.betfair.com/exchange/betting/rest/v1/en/navigation/menu.json/" ) but have a couple of queries:

1. Is it possible to limit the Navigation Data service to certain event type? I am presuming it is not an option.

2. What is the best way to parse the Navigation Data service responseText data?

T.Y.

S.

[jabe]

Aha! There you have the advantage! I tried adding the navigation last year at some point but we didn't get on, so I left it for another day. My program still has the remnants of my useless code in it, ready for a review at some point in the future. I didn't need it, as it turns out, but may return to it.

[jabe]

By making a request to the Navigation Data service, they are able to obtain a list of all BF markets in one visit which obviously in that case (oddsmonkey) makes it much faster.

Was there more than just football? I couldn't persuade it to show me any other sports.

[SimonN]

Was there more than just football? I couldn't persuade it to show me any other sports.

Here is my code.

    Dim xhr: Set xhr = CreateObject("MSXML2.XMLHTTP")
  
    With xhr
        .Open "Get", "https://api.betfair.com/exchange/betting/rest/v1/en/navigation/menu.json/", True    
        .setRequestHeader "Connection", "keep-alive"
        .setRequestHeader "X-Application", AppKey
        .setRequestHeader "X-Authentication", Session
        '.setRequestHeader "Content-Type", "application/json"
        .setRequestHeader "Accept", "application/json"
        .setRequestHeader "Accept-Encoding", "gzip,deflate"              'Accept-Encoding: gzip,deflate
    End With
    
    
    xhr.send Data
    SendRequest = xhr.responseText

The response is coming back with no errors and contains quite a lot of markets although not as many as I would have thought for the whole of Betfair or maybe the Excel VBA Immediate window where I am picking the response up from is truncating the display of the SendRequest output to a max number of characters.

When I copied it from the Immediate window to a text editor I found just 3 event_types (tennis, volleyball, winter sports) but as I say I am assuming it is being limited by the VBA Immediate window display.

I am also assuming it is not possible to retrieve only certain event types (eg "soccer" only) using the Navigation Data Service which, although, not so ideal is still potentially better for me than countless server requests.

This is as far as I have got - I am a bit stuck on the deserialization side which you seem to be pretty hot on Jabe.

Was thinking of buying the Programming for Betfair book by James Butler but not sure whether or not from looking at the contents page it covers the Navigation Data service or show how to deserialise it.

My goal is to build a web app (using Data Navigation Service) that is quick like oddsmonkey but I don't know how to deserialize the JSON response into searchable structures and I don't know what web tools would best lend themselves to building this web app - was thinking about Python but I suppose until I have worked out how to deseriliaze the Data navigation Service, even just in any language to start with (maybe will start with VBA?) then I won't know which web technology will be best.

Feeling like have hit a bit of a wall!

Edit> And i just want football markets lol no others!
Edit> Got JSON responsetext output to file - 4.5KB! 29 Event Types

[jabe]

I'll take another look. It appears that what I tried to do was save the returned navigation data to a file for investigating, but there isn't one in the folder I aimed it at.

What we'll need to do is work out the structure of the data and create a class file (or files) that would be appropriate.

I'll resurrect the code in my program and add a button to get it.



Edit: 24Mar2016: my code to get the navigation file via VB.NET now works, so I'll add the code to the thread in the next few days.

[SimonN]

I'll take another look. It appears that what I tried to do was save the returned navigation data to a file for investigating, but there isn't one in the folder I aimed it at.

What we'll need to do is work out the structure of the data and create a class file (or files) that would be appropriate.

I'll resurrect the code in my program and add a button to get it.

Here's my code outputting the responsetext to the file. The file is not populated until the Output stream is closed.

xhr.send Data
Open "D:TempTest.txt" For Output As #1
Write #1, xhr.responseText
Close #1

'SendRequest = xhr.responseText

That's the hard part - working out the response structure - just looking at it now - can't see any obvious hierarchy although we know that this is the structure of it here

https://api.developer.betfair.com/se...ions-Structure

[jabe]

I tried using the code you posted but had to make tweaks, perhaps because I'm using the free version of VS2015. It didn't like the definition of xhr or the Set statement, so I need to look into that. I did get it to accept a version of the code, but it failed when I ran it.

It's quite a problem with borrowed bits of code!

[SimonN]

Ah, I'm using VBA! Not VS, lol!

That'll be also why your class definitions weren't working for me then lol!

[jabe]

Aha! That explains it!

I did manage to get an OK/200 return code back, but I've no idea where my data string is!

On a more positive note, I left the hierarchy problem bubbling away on the backburner last night and it came up with the answer, so I'll code the classes up later and you'll have to change them all to VBA.

It occurs to me that the navigation data really required recursive code to dig it out. It ought to be in a tree-view, and I think I might have to leave that to you to sort out, as I've never used one before.

[jabe]

My current state of play is that I still haven't got hold of a string containing the menu.json file.

I have, however, coded (but obviously not tested) classes and a VB statement to turn the menu.json string into a VB.NET object.

That's all in the next post...

[jabe]

This, I hope, is going to give current and future time and space travellers vital practical help about getting the navigation / menu data from Betfair and making it into usable data.

I am lacking one element so far, but if you have managed to excavate the menu.json file into a string, this should give you all you need to proceed.


These Class definitions should suffice:

class ClassNavRoot
   dim children() as EventType   ' EVENT_TYPE
   dim id as string              ' = "0" on ROOT
   dim name as string            ' = "ROOT"
   dim type as string            ' = "GROUP"
end class
class ClassNavEventType
   dim children() as Object      ' GROUP or EVENT or RACE
   dim id as string              ' Betfair EventTypeId
   dim name as string            ' = name of sport
   dim type as string            ' = "EVENT_TYPE"
end class
Class ClassNavGroup
   dim children() as Object      ' GROUP or EVENT
   dim id as string              ' Other Number - unique but otherwise not meaningful
   dim name as string            ' = group name
   dim type as string            ' = "GROUP"
end class
Class ClassNavEvent
   dim children() as Object      ' GROUP or MARKET or EVENT
   dim id as string              ' Betfair EventId
   dim name as string            ' = name of event
   dim countryCode as string     ' 2-character code
   dim type as string            ' = "EVENT"
end class
Class ClassNavRace               ' For Horse Racing of Greyhounds only
   dim children() as Object      ' MARKET
   dim id as string              ' Betfair RaceId
   dim name as string            ' = distance as text, probably
   dim startTime as datetime     ' (or string?) eg "2014-08-12T11:15:00.000Z"
   dim type as string            ' = "RACE"
   dim venue as string
   dim raceNumber as string      ' US specific eg "R1"
   dim countryCode as string     ' 2-character code
end Class
Class ClassNavMarket
   dim exchangeId as string      ' = "1" usually
   dim id as string              ' Betfair marketID
   dim marketStartTime as dateTime 
   dim marketType as string      ' eg WIN, PLACE, etc
   dim numberOfWinners as string ' (or numeric, but PDF example is in quotes)
   dim name as string            ' eg "Match Odds"
   dim type as string            ' eg "MARKET"
end class

This is from the documentation:

https://api.betfair.com/exchange/bet...tion/menu.json

In plain English:

A ROOT group node has one or many EVENT_TYPE nodes

An EVENT_TYPE node has zero, one or many GROUP nodes

An EVENT_TYPE node has zero, one or many EVENT nodes

A Horse Racing EVENT_TYPE node has zero, one or many RACE nodes

A RACE node has one or many MARKET nodes

A GROUP node has zero, one or many EVENT nodes

A GROUP node has zero, one or many GROUP nodes

An EVENT node has zero, one or many MARKET nodes

An EVENT node has zero, one or many GROUP nodes

An EVENT node has zero, one or many EVENT nodes

https://api.developer.betfair.com/se...r+Applications


NOTES
I've added "Nav" into the class names to denote that they relate to the navigation file data.

The data hierarchy begins with the ROOT which contains an array of EVENT_TYPEs. EVENT_TYPEs each have an array of GROUP or EVENT or RACE items.

The GROUP items are the kind that appear on the Betfair menu as SATURDAY GAMES or DIVISION 3 or EUROPEAN CUP or THE ASHES SERIES or whatever.

When navigating the data, the type will always tell you what type of item you are dealing with. This is just as well, because the Class items for EVENT_TYPE and GROUP are identical and I expect the deserialiser to mix these up, so checking the class/object type wouldn't work. It's just as well, then, that the type string is there.

Because some of the children arrays - in the EVENT_TYPE and GROUP classes - can contain different types of data, they had to be defined as the unspecific OBJECT rather than a more specific class. The ROOT class is essentially the same too, except that its array always contains EVENT_TYPE data.

I have coded the children() as arrays, but they could also be collections in VB.NET and, I expect, other data structures in other languages.

Any set of items should end in a Market item, unless the sport in question is intermittent in nature, in which case you might encounter an empty children() array. Expect this.

I would guess - and I may be wrong - that the navigation file is altered on a daily basis.

I do want to include code for retrieving the menu.json file from Betfair, but it will have to wait until I can persuade mine to behave.


DESERIALISING
If you get to the point where you have the entire menu.json file existing as a string, with the classes from above in your program, it is fairly simple to turn it into an object.

This is what I've been using; another apology - this time to whoever originally coded it as I can't give them a name check owing to the mists of time.

I have this line at the top of my project:

Imports System.Web.Script.Serialization

This is a generalisation of how to turn a JSON string into an object, assuming you have appropriate class definitions:

Dim jss As New JavaScriptSerializer() 'JSON serialiser 

yourObject = jss.Deserialize(Of ClassofYourObject)(yourJSONString)

If you want to go from an object to a JSON string, you can do this:

aJSONString = jss.Serialize(anObject)

Being specific to our current task, this will turn your menu.json string into a convenient object:

Dim jss As New JavaScriptSerializer() 'JSON serialiser
Dim navData as ClassNavRoot = jss.Deserialize(Of ClassNavRoot)(yourMENUdotJSONString)

You might want to turn it into a tree view.


* * * *


If you spot any glaring errors, please let me know and I'll correct the text in this post so that the future lost folk don't have to hunt around for corrections.