Paging markets

[Phaedos]

When I call `listEventTypes` with a filter on for Soccer, but all markets it usually gives me around 14,000 markets as marketCount.

Yet when I try and get hold of these markets with `listMarketCatalogue` I typically get more than this because maybe 2000 are duplicates. Maybe there's a problem with my paging method:

Since maxResults is 1000 for this endpoint, I usually try to page through the markets returned by having them returned date sorted, then getting the datetime of the last market and feeding it into the next call as the `fromDate` parameter (any better ways to get all these markets more than welcome). In python code, my request params look like:

    mFilter = {"eventTypeIds": ["1"]                   
                             "marketStartTime": {"from": fromDate}
                             }
    projection = ["EVENT", "COMPETITION", "RUNNER_DESCRIPTION"]
    market_catalogue_req = json.dumps({'jsonrpc': '2.0',
                                 'method': 'SportsAPING/v1.0/listMarketCatalogue',
                                 'params': {'filter': mfilter,
                                            'maxResults': '1000',
                                            'marketProjection':
 projection,
'sort': 'FIRST_TO_START',

                                            },
                                 'id': 1,
                                 })

I then just grab the new `fromDate` of the last item like

fromDate = marketCatalogue['result'][-1]['event']['openDate']

(remembering the `FIRST_TO_START` sorting), and feed that into the next call. Rince and repeat until no more markets to grab.

How can I do it without getting so many duplicates? (even using the marketType filter will not help me as for example "Match Odds" usually have near 1500 markets, so still hit the same problem)

[betdynamics]

Whilst not as efficient, you could do a listEvents first, then just cycle through each event via listMarketCatalogue, specifying the eventId in the call.

Or listEvents and then listMarketCatalogue with a batch of eventIds.

[Phaedos]

Thanks, and sorry for the delay getting back to you.

Yes this does not seem like a bad approach, and not too slow if done in batches of event ids. (The fromDate paging has even more problems than I anticipated, such as when there are 1000 events with the same datetime it loops forever. You might think adding a second to the previous datetime would help, but then you can end up missing events that you did not grab in the previous batch with the same datetime as the ones at the end of your last 1000).

Regarding the listEvents first method though. Lets say I grab 1500 events in the call, and end up with a list like eventIds = ['u'27157164', u'27157165', u'27157160', u'27157162', u'27157163',....].
Each of which has a certain number of markets associated with it (in my experiments between 1 and ~70). Now if I split the eventIds list up into batches and make the listMarketCatalogue endpoint call. I need to ensure that the sum of marketCounts for all the events represented in my eventIds subList is <1000.

How would you recommend doing that?

[betdynamics]

Just assume a max number of markets per event (let's say it is 75 for comfort)

If you can return 1000 markets in a single call (I think that depends on the data being returned but may be wrong!) then you can calculate that the max no. of events per call is 1000 / 75 = 13.

So I would just batch calls with 13 events ids - again not the most efficient but fairly straightforward.

[Phaedos]

I wrote a small function (it might not be the most elegant but it seems to work):

def eventIdBreaker(eventsList):
    '''
    events come with an Id and a marketCount.
    When we make the call to listMarketCatalogue it
    will return at most 1000 markets.
    This means we need to break the
    events list in such a way each batch is <=1000
    markets in total.
    '''
    MAX_RES = 1000
    eventIds = []
    subList = []
    mCounter = 0
    
    for event in eventsList:
        mCounter += event['marketCount']
        if mCounter < MAX_RES:
            subList.append(event['event']['id'])
           
        else:
            # Append old subList, create new subList and resent count.
            eventIds.append(subList)
            subList = [event['event']['id'], ]
            mCounter = event['marketCount']

    # Remember to append final subList
    if subList:
        eventIds.append(subList)
    return eventIds

Hopefully this way the batches will be as large as possible.

[betdynamics]

Much more elegant.

You appear to be gathering all markets for all events (perhaps not!) - are you trying to build a menu of events/markets?

If so, then I think the navigation data (https://api.developer.betfair.com/se...r+Applications) is supposed to be used for that.

[Phaedos]

Yes, well I wish to gather ultimately all soccer markets (this is for an arbitrage website application). So this Navigation data would given me everything but the prices themselves?? If that's the case it would make my life infinitely easier..

[betdynamics]

But listMarketCatalogue doesn't give you prices either.

So, if you just want to start with a list of markets that you can use with listMarketBook, then you can use the navigation data. From memory it gets updated every 15 minutes, so you could rescan it in the background and see if any additional markets have been added (new matches are added all the time).

Would be interesting to see if it takes longer to gather/parse the navigation data or to cycle through the events collecting marketIds. I can see it going either way depending on how many matches there are that day.

[Phaedos]

As far as I can tell I couldn't get the runner names from this navigation data (I normally get this with the appropriate market projection on /listMarketCatalogue, then get the prices from /listMarketBook, then match runner names with prices afterwards). Thus I'd have to call /listMarketCatalogue and /listMarketBook anyway it seems?

I'm looking to ultimately end up with data that looks like:

{                                                                                                          [11/42]
        "_id" : ObjectId("5439cc784ede56286e287031"),
        "eventName" : "Slovan Liberec v Slovacko",
        "compName" : "Synot Liga",
        "exchangeName" : "Betfair",
        "shortDateTime" : "10 18",
        "markets" : {
                "1.115841207" : {
                        "totalMatched" : "0.0",
                        "marketName" : "Match Odds",
                        "runners" : {
                                "10503" : {
                                        "runnerName" : "HOME",
                                        "availableToLay" : [
                                                {
                                                        "price" : 2.38,
                                                        "size" : 7.88
                                                },
                                                {
                                                        "price" : 3.05,
                                                        "size" : 8.27
                                                }
                                        ]
                                },
                                "1533861" : {
                                        "runnerName" : "DRAW",
                                        "availableToLay" : [
                                                {
                                                        "price" : 5.2,
                                                        "size" : 7.88
                                                }
                                        ]
                                },
                                "58805" : {
                                        "runnerName" : "AWAY",
                                        "availableToLay" : [
                                                {
                                                        "price" : 4.5,
                                                        "size" : 15.77
                                                },
                                                {
                                                        "price" : 950,
                                                        "size" : 4.18



                                                }
                                        ]
                                }
                        }
                }
        },
        "team1" : "Slovan Liberec",
        "team2" : "Slovacko",
        "dateTime" : "2014-10-18T15:00:00.000Z"
}

My current path to this is /listEvents, /listMarketCatalogue, /listMarketBook

[AliyaCynthia]

I have been getting this error lately when calling live ( non delayed ) betfair API. I dont know whats the issue, can someone please help ?
UNO Online