Streaming API questions

[Mr Magoo]

I've been using the exchange streaming API recently, and it is a great way to get instant notifications of market and bet changes. It is so much better than having to keep polling lots of events!

However, I have two questions about the API:

1) Is there a roadmap or any release schedule for adding it to the Australian side of the API? I bet in both the UK and the AUS exchanges, so it would be a great help if Betfair could offer streaming there too. Without it, many of the streaming API benefits are lost: It's not possible to throw away older code that constantly queried/refreshed markets individually. If you have two different mechanisms to bet with, then the streaming API is only increasing the burden on users. It means more maintenance, more coding and more debugging. If customers have to keep the old systems around, and they still work, why use the new one at all?

2) Feature request: It would be very helpful if the order streaming would also notify clients about settled bets. The streaming API is meant to be a better way to keep track of goings on without having to check lots of markets and make lots of API calls. But without any way to be notified on bet settlement, users still have to frequently call getAccountStatement() or listClearedOrders() to spot updates.

Thanks!

[BetfairDeveloperProgram]

Stream API Questions

Hi Mr Magoo,

Thanks for your feedback.

1) Is there a roadmap or any release schedule for adding it to the Australian side of the API? I bet in both the UK and the AUS exchanges, so it would be a great help if Betfair could offer streaming there too. Without it, many of the streaming API benefits are lost: It's not possible to throw away older code that constantly queried/refreshed markets individually. If you have two different mechanisms to bet with, then the streaming API is only increasing the burden on users. It means more maintenance, more coding and more debugging. If customers have to keep the old systems around, and they still work, why use the new one at all?

Yes, there are plans to offer AUS/NZ markets via the Stream API. There will be a further announcement regarding this in the near future (before the end of next week)

2) Feature request: It would be very helpful if the order streaming would also notify clients about settled bets. The streaming API is meant to be a better way to keep track of goings on without having to check lots of markets and make lots of API calls. But without any way to be notified on bet settlement, users still have to frequently call getAccountStatement() or listClearedOrders() to spot updates.

The Order Stream API sends the closed flag as part of the OrderChangeMessage once a market that contains your orders is closed.

Key fields:

<as for ChangeMessage>
oc / OrderAccountChange - the modifications to account's orders (will be null on a heartbeat)
closed - indicates when the market is closed
id / Market Id - the id of the market the order is on
orc / Order Changes - a list of changes to orders on a runner
id / Selection Id - the id of the runner (selection)
uo / Unmatched Orders - orders on this runner that are unmatched

Here's an example of the message received when a market that you have orders in has closed:

{"op":"ocm","id":2,"clk":"AKSkAQD24QIAkd0CAPbSAwC54","pt":1468335315451,"oc":[{"closed":true,"id":"1.125549486"}]}

Thanks

Neil

[StefanBelo.]

Neil, Mag is right because bet streaming api does not offer all bet transitions till settled event. Already matched bet/s can be voided at the market event settled procedure.

There is quite a lot of questions according to the way your team implemented streaming api, just one use case showing that your team actually did not think about end user and how he will use your api.

Bet streaming socket receives any bet transitions for user account. Let's say you run fully automated strategy on horse racing markets on your server. Now at evening football match you take your windows tablet with your trading app to trade manually on watched match, of course all bets transitions, from running horse racing strategies are replicated to tablet app as well.

Now tell me where is common sense in such approach as base design feature of api streaming is performance as well.

Separating market streaming api and betting streaming api has one big disadvantage as well, as bet operation is no more atomic operation when we talk about bet status retrieval, actually from this point of view NG api offers better and mainly safer operation.

When I implemented streaming api, I ask you to extend ladderLevels to 10, to support compatibility with NG where you can set bestPricesDepth to 10. Please after those months try to force you team to change at least this one.

[BetfairDeveloperProgram]

Hi Stefan.

Thanks for your input.

Totally agree, the Order Stream should handle and push out all bet states including void and this was part of the original requirements. We'll review this with the development team.

When I implemented streaming api, I ask you to extend ladderLevels to 10, to support compatibility with NG where you can set bestPricesDepth to 10. Please after those months try to force you team to change at least this one.

This is on the list of stream API improvement and we expect this to be delivered by the end of August.

Thanks

Neil

[Mr Magoo]

Thanks for the detailed answer, Neil.

Sorry, it was my mistake not spotting the 'closed' flag. It's missing from the API docs that I've been using, but they are several months out of date!

However, I've been testing it out and there are strange discrepancies between the NG & Streaming APIs. If I get a notification about a closed market, and then call listMarketBook() (to see who won), the market is often not labelled as closed. (n.b. I'm not using a delayed API key)

What I was hoping for in the streaming API was details of how the bets in the (now closed) market were settled. What was the profit/loss on the bet? What odds were used? What commission was paid? The streaming API gets so close to this, but without the final few pieces of information, the API cannot replace any existing systems of recording profits & bets.

The differing market 'closed' flag between APIs sounds tiny but it's not. If the streaming API notifies my program about a closed market, but loading the market shows that it isn't shut yet, then how can I work out the results of my bet? There might be more non-runners that haven't been removed yet, for instance. Now I need to keep polling the market until the other parts of Betfair's systems decide that the market really is finished. A streaming API is meant to replace repeatedly querying the system to spot changes. And yet the shortcomings of the API mean that users have to do it.

Finally, even if both APIs were consistent with each other, there's still not enough information provided to let customers calculate the returns. Non-runners, changing reduction factors, dead heats, and so on, all will make manual profit calculations unreliable.

All the API really needs is an extra field that contains the settled amount for the bet. That's it!


Yes, there are plans to offer AUS/NZ markets via the Stream API. There will be a further announcement regarding this in the near future (before the end of next week)

Great news, thanks for the update.

[BetfairDeveloperProgram]

Stream API Questions

Hi Mr Magoo,

Please see comments below.

Sorry, it was my mistake not spotting the 'closed' flag. It's missing from the API docs that I've been using, but they are several months out of date!

The latest Exchange Stream API documentation is now published in full via http://docs.developer.betfair.com/do...nge+Stream+API

I've been testing it out and there are strange discrepancies between the NG & Streaming APIs. If I get a notification about a closed market, and then call listMarketBook() (to see who won), the market is often not labelled as closed. (n.b. I'm not using a delayed API key)

We'll investigate this, but it sounds like the streaming message is ahead of the time it takes for the result data to be written to the database.

What I was hoping for in the streaming API was details of how the bets in the (now closed) market were settled. What was the profit/loss on the bet? What odds were used? What commission was paid?

Price and order streaming was primarily designed to reduce the requirements for customers to have to repeatably poll for market definition, price and order changes. The reporting aspect for closed markets will continue be handled by listClearedOrders, although we do understand the need for customers to be able to efficiently use the message returned by the streaming API to make the requests you've mentioned i.e. listMarketBook for results and listClearedOrders for settled bet details.

Let me know if you have any further feedback.

Thanks

Neil

[Mr Magoo]

The latest Exchange Stream API documentation is now published in full via http://docs.developer.betfair.com/do...nge+Stream+API

Thanks, I found it after your earlier message about the protocol. I should have checked online earlier, my old docs were labelled as
pre-release but I didn't think to check

Price and order streaming was primarily designed to reduce the requirements for customers to have to repeatably poll for market definition, price and order changes. The reporting aspect for closed markets will continue be handled by listClearedOrders, although we do understand the need for customers to be able to efficiently use the message returned by the streaming API to make the requests you've mentioned i.e. listMarketBook for results and listClearedOrders for settled bet details.

You're right, and my last post came across as far too critical, I didn't mean it quite like that, sorry!

I've found the streaming market updates to be very useful, and incorporating them into my existing code turned out to be very simple. I think my problems with the streaming order updates have been that the API and my existing code for bets don't gel together as well, I didn't mean to take my frustrations out on you though, I appreciate your feedback!

The order streaming and the market timing is a bit annoying though. It would be great if once the streaming API told me that a market with my bets in had been closed, I could immediately call listClearedOrders to find out how they have been settled. I guess a simple workaround is just for me to delay the listClearedOrders by a few seconds or so until the market really is settled. As long as the delay is relatively constant, that would work fine. Anything to avoid having to repeatedly call the API until the change is seen - which is what code that wasn't using the streaming API would end up doing anyway.

[Mr Magoo]

Let me know if you have any further feedback.

One other streaming API question for you: Is it possible to subscribe to some order/market changes *without* getting an initial state sent to you?

In several of the places where I am using the streaming API, my code doesn't actually need to know the initial state, it only cares about being notified about updates. But I can't see any way of communicating this to the API, and as a result I get sent large numbers of initial messages that are wasting everyone's time. It would be great to be able to skip them.

I know that the API supports re-subscribing by supplying the initialClk and clk values. But in these situations, I don't have a state to resubscribe to. Could I skip the initial messages by reusing the initialClk/clk values that have been sent to a *different* streaming connection? Or would that cause confusion between the two streaming sessions (the other one might still be running, for instance!)

[BetfairDeveloperProgram]

Hi Mr Magoo,

One other streaming API question for you: Is it possible to subscribe to some order/market changes *without* getting an initial state sent to you? Could I skip the initial messages by reusing the initialClk/clk values that have been sent to a *different* streaming connection?

No, this isn't possible. How do you handles delta updates without the initial market/orders image?


Thanks

Neil

[LiamP]

Hi Neil,

What happens when a runner is removed, I assume the ladder is emptied and 'marketDefinition' is updated?

[BetfairDeveloperProgram]

Hi LIAMP,

Yes, this is correct, the marketDefinition will be updated with the status REMOVED against the withdrawn runner and the ladder will be emptied.

The Order Stream data is updated as per the guidance in the documentation via http://docs.developer.betfair.com/do...theOrderStream

Kind Regards

Neil

[Mr Magoo]

No, this isn't possible. How do you handles delta updates without the initial market/orders image?

I've got a big mess of individual programs that handle different betting strategies. For example, one program watches for horse racing markets to turn in-play so that it can place bets to take advantage of the different win/place market odds. Before the streaming API, this code had to poll each upcoming race to spot when a market turns in-play.

With the streaming API, I can improve this by watching the market updates - this way, the API notifies me exactly when a market has turned in-play. The rest of the code is still using the original NG API to discover races, so it does not need to rely on the streaming API to maintain any overall picture of market states. It just needs the pre-race => in-play transition.

Potentially, I could rewrite the whole program to use the streaming market updates entirely, but I see no point in that - my existing code works so I'm loathe to rip it out. And in this way, the program will continue to work if the streaming API fails. Having the two API sources should make the program more robust.

[BetfairDeveloperProgram]

Hi Mr Magoo,

Please try creating a single subscription that just requests market definition updates (EX_MARKET_DEF) as this should provide only new Market Definition as they occur, which includes updates for when markets are turned in-play.

Kind Regards

Neil