The Perpetual Mistake

[Lopiner]

I'm starting to look at the Exchange Streaming API ("ESA"). It's a great concept, you only get data when it changes and is relevant to you reducing the use of resource consumption on our side and on Betfair servers. It opens the door to increase advantage for being closer to the servers and the possibility for Betfair receiving money from clients who want to co-locate (i discussed this in another post).

My main problem relating the ESA is that is very error prone and i was interested in knowing in what ways people try to reduce the possibility of errors. There are well know situations in the financial markets where servers became so busy that this methodology of distributing data as it comes created major problems, one of them was that it was reporting old data like in the flash crash of 2010.
But i'm not focusing on server side errors, i'm more interested in client side errors.

When we use the request / response paradigm of the API-NG we are in fact unaware of market changes but when we query the market we always see the complete and current picture. In some sense, if we rely on invalid data in one second we can have the correct picture in the consequent call. This is very resilient.

When we use a subscribe paradigm like in the ESA if we get something wrong that mistake perpetuates until we make a new initial subscription. This is scary. I know that good coding can prevent this, but good coding is hard and sometimes only achieved after making some mistakes.

An analogy between the two paradigms is always checking your wallet for the amount of money you have (API-NG) or depending on the initial balance and the subsequent expenses and receivables you get, you just look into the wallet once (ESA).

One way i thought of reducing the possibility/impact of errors in ESA was to have a police routine routinely check the ESA image of the markets against a request from API-NG. If any discrepancy was detected then a full initial image was requested.

Another way was relying on the ESA to be informed that a market has changed but all data was then requested by an old API-NG call.

Any thoughts on this?

[Fred77]

There's many things to muse over when putting your money on the line!

Running a side by side comparison of APING and ESA should put your mind at ease as to the reliability of ESA and confirm that you're interpreting the stream correctly. Once you get started you'll find that implementing a decoder for the stream is relatively straightforward, it shouldn't be error prone at all. You could try your police method to make sure that your code is correct and doing as you expect, but in fast moving markets there will definitely be differences between APING and ESA simply due to timing.

Using ESA to be informed that a market has changed and then requesting the data again via APING does seem a bit wasteful. I suppose you could reduce that waste by only double-checking before actually placing a bet.

If you're following long slow markets then I guess there's a concern that over time your view of reality could get distorted, e.g. spurious memory corruption in your application or something. Maybe request full image once every 24 hours.

Whether it's from the APING or ESA or your own code, how do you tell if a price is real? Even if the APING and ESA give the same data can you trust it? How far should you go? You could cross-check with other exchanges and bookmakers to make sure a price isn't too far out of line. But then again, a price that is too far out of line may represent outstanding value (ie another user's error!) rather than client/platform error.

At the end of the day, the only sure fire way to protect your money is to only leave an amount in your account that you're prepared to lose. Whatever your data source your application needs safeguards that prevent run-away behaviour. I guess the lengths that you go to to check data integrity will ultimately depend on the size of your stakes and how risk averse you are.

[rwilson1982]

It does not seem that the APING is reliable either. I am stuggling to pick up jockey changes like the 2:55 at York today (18th May 2017). When I pull down runner metadata in the morning I see W. Buick on the Black Princess. I check it again 45 seconds before the race and it is still W. Buick... I check the result online after the race and it turns out L. Detorri rode the Black Princess...

And Betfair want me to pay £300 for this privilege?

Starting to feel that I am wasting my time collecting data as I now have no idea how reliable it is..

[betdynamics]

I don't think the runner metadata is designed to be dynamic data, i.e. I don't think it gets updated after it is initially populated.

[rwilson1982]

I don't think the runner metadata is designed to be dynamic data, i.e. I don't think it gets updated after it is initially populated.

Thanks betdynamics, I believe you are correct. I assumed it was dynamic and find it counterintuitive that it is not. Can you tell me if the streaming API reports on such changes or is it just for price data?

[LiamP]

No Jockey data isn't updated via the market stream, the website uses the raceCard API which I think might be updated, here is the code I use to access it:

https://github.com/liampauling/betfa...ts/racecard.py

[LiamP]

@Lopiner in response to your original question there a few bits of data betfair provide which can help reduce inaccuracy.

The first one is the 'publishTime', using this you can get a good idea of how delayed the data is, its impacted by your connection and/or how fast you are pulling data off the socket.

Betfair also provide a 'conflation' flag, I am not sure what algorithm is used for conflation but if you see this present in an update you know that the data has been delayed betfair side either due to the conflationMs supplied or a slow down their side.

Check against /listMarketBook would be nice but with betfair updating so often the chances of getting streaming too match would be difficult in itself.