C# The beginnings of a bot

[Macberrypro]

Don't use date/time stamps. Moreover use an increasing integer based stamp. For enterprise solutions, this will scale both horizontally and vertically

[vossie]

In the meantime I'll go crawl back under my stone for another few months.

Please don't do that

[The Geeks Toy]

Not my field of expertiese so I have no idea of the hardware requirements & bandwidth costings. I'll shall leave that to the real geniuses to work out. I just offered up a possible solution for the change points.

In the meantime I'll go crawl back under my stone for another few months.

[vossie]

that looks good, but like I said theory is great until you stick it on an enterprise environment. How large cluster do you need to make the IS scalable and by the time you have built and configured this what benefit have you gained? a couple of gigs worth of bandwith but you have the overhead of the new clusters, software/hardware maintenance, increased complexity etc.

[The Geeks Toy]

Hi Maverick, Good to see you back here, hope all is well

Things are fine. Been a little busy in recent times with a product launch.

Re change points. You can hold the full history on the server, if you hold a timestamp against each data row. { And this is the really simplified one server , one market, one selection, 4 price, solution }

The incremental server { IS } holds a snapshot of the market data along with a timestamp against each row. EG

10:00 AM 1.01 £10
10:00 AM 1.02 £0
10:00 AM 1.03 £20
10:00 AM 1.04 £30

First client request for the market uses a timestamp of DateTime.Min. Data returned is... {NB You don't need to return the timestamp for each row to the client, just the highest one on the IS. Just included below for explanation clarity.}

10:00 AM 1.01 £10
10:00 AM 1.03 £20
10:00 AM 1.04 £30

Client now holds the above, and a timestamp of 10:00AM
================================================== =============

New data is pushed to the IS from the market server @ 10:01

10:01 AM 1.01 £10
10:01 AM 1.02 £0
10:01 AM 1.03 £20
10:01 AM 1.04 £40

When merged, the resulting IS snapshot is

10:00 AM 1.01 £10
10:00 AM 1.02 £0
10:00 AM 1.03 £20
10:01 AM 1.04 £40

Client makes incremental call with the 10:00 timestamp, and is returned

10:01 AM 1.04 £40

Client now holds

10:00 AM 1.01 £10
10:00 AM 1.03 £20
10:01 AM 1.04 £40

and a timestamp of 10:01AM
================================================== =============

New data is pushed to the IS from the market server @ 10:02

10:02 AM 1.01 £0
10:02 AM 1.02 £0
10:02 AM 1.03 £30
10:02 AM 1.04 £40

When merged, the resulting in the IS snapshot is

10:02 AM 1.01 £0
10:00 AM 1.02 £0
10:02 AM 1.03 £30
10:01 AM 1.04 £40

Client makes incremental call with the 10:01 timestamp, and is returned

10:02 AM 1.01 £0
10:02 AM 1.03 £30

Client now holds
10:02 AM 1.03 £30
10:01 AM 1.04 £40

and a timestamp of 10:02AM
================================================== ==============

Another client can make a call with a 10:00 timestamp & get

10:02 AM 1.01 £0
10:02 AM 1.03 £30
10:01 AM 1.04 £40

Yada, Yada, Yada. Any client can call with any timestamp & get just the changes. As long as you have a single consistent timestamp / changepoint from the market data source, it should scale easily to multiple API servers.

Only other thing I can think of off the top of my head is the IS data for each market needs to be locked when being updated by the Market Server, however I doubt this would be much of an issue.

Anyway, long story short, you have only a single cache object for each market which covers all changepoints, which you merge new market data into instead of replacing it.

[vossie]

I agree polling is better, but I think it needs taking one step further in that especially for GCMPC & GMTVC you should combine the calls into one & use an incremental update process between client & server instead of returning the entire market on each call.

Same timestamp thoery, but only returning data lines that have changed since the last timestamp, with the client feeding back the last returned timestamp on the next call.

It's a bit more work at both ends, however seeing as the new generation of apps are utilising these calls more, it would cut down the bandwidth requirement to a fraction of that currently used.

Hi Maverick,

Good to see you back here, hope all is well I have been looking at the possibilities of incremental updates and even thow the theory behind it is solid and undeniable you do have a certain level of head ache that comes with it.

• The first is complexity, I know the BF APIs is not what you would describe as simple but doing this would add another layer of complexity which we would prefer to move away from (guess we could ship a betfair.dal.dll for .NET clients)
• Secondly you need to store change points and then how long do you cache them for 1, 2 or 3 seconds and in the first that case we are stuck with say 5+1 cache objects rather than 1 multiply number of active markets.
• blah blah blah (tired of my own voice now)

I think the short and simple would be that yes I would like to see incremental updates for price and bet data probably cached for 2 change ticks and if you are out by more than 2 ticks you get the whole lot. Note: Changes get pushed up to the server in batches, so it would be 2 change pushes for incremental increases. On busy markets this would be sub-second.
(This is being researched by the way)

[The Geeks Toy]

I agree polling is better, but I think it needs taking one step further in that especially for GCMPC & GMTVC you should combine the calls into one & use an incremental update process between client & server instead of returning the entire market on each call.

Same timestamp thoery, but only returning data lines that have changed since the last timestamp, with the client feeding back the last returned timestamp on the next call.

It's a bit more work at both ends, however seeing as the new generation of apps are utilising these calls more, it would cut down the bandwidth requirement to a fraction of that currently used.

[vossie]

Push from a public API perspective

I agree that a push system would be advantages to some users.

Lets look at push in a real world scenario. The first problem is that the top 70% of transactional API consumers are not really as price and change sensitive as everyone else thinks they are, don't get me wrong everyone is price sensitive but the difference between 3 and 5 volatile data request in a single second is hardly noteworthy from a client perspective.

With push a client system that previously used a well thought out polling mechanism would simply chuck that away and subscribe to streams that quite possibly update at a rate of 200ms. Now this client previously polled at a rate of twice a second with no real impact on the effeciency of their system or Betfairs, now multiply those 3 aditional push responses introduced by hundreds of thousands and the scope of impact becomes clear. To enable push you also have to keep the communication channel open between server and listener and this is most definately a finite resource, in a controlled enterprise environment you can do this easily and scale just as easily but in the wild web you have all sorts of problems to contend with, most notably connection drops and latency. With push the client sessions are sticky which means you could get stuck on a single server with a heavy user impacting your performance.

The alternative and well supported polling approach can much more efficiently handle the data distribution using http protocols, admittendly the shape of the current Betfair requests and responses are questionable at best and can be greatly improved by adding a last request timestamp to the prices request that returns null if the cached prices on the server has not changed since your last request.
This type of transaction where no data is returned on no change would be a slightly heavier data object than the heartbeat you would have on the push channel but it means that you can stagger the responses even if only by fractions of a second and you can also load balance over a much larger cluster without too many coherency issues.

[MichaelRNye]

Work is currently being done to improve the API, my best guess is that the result of these efforts will not be publicly available for another 12 months and changes and upgrades to the current API will be limited to bug fixes and petrformance changes. Any large scale investment would go on delivering API7 but in the meantime everyone including Betfair first party developers has to use the existing API. The likelihood of seeing push functionality in the next version of the API is at best very slim. Current push solutions is very vendor/platform specific and extremely resource intensive in an exchange environment, untill these technology concerns is addressed and widely accepted, supported and adopted by open source and proprietary solutions provider like SOAP it just does not make commercial sense. If you want to test the statement write a simple application server capable of concistently servicing 1000 un-cached complex request/responses per second with a spike rate capability of 2000, now add push functionality (WCF, Comet etc.) to that server and see what happens.

Oh I don't doubt it would be a huge strain I often wonder how the betfair servers manage to cope with the no doubt enormous amount of requests being sent every second. Understanding that it is a huge undertaking to release a new API whilst retaining compatability with the current system, I would imagine that moving to a push style system would hopefully reduce the load on the servers. It's hard to gauge, but I think most of the current queries are mainly people poking to see if somethings changed rather than meaningful queries.

That is what I realised and decided to release the Lignite project as an unfinished work and shared the idea and reasons for the BfPDS - The BfPDS will adress your concerns, provide a lightweight language agnostic abstraction layer, shielding the consuming clients against API changes and when the new API7 becomes available the BfPDS will still serve up the data in the same format without the consuming clients even being aware of the change.
You are absolutely right the solution will not work for everyone and there will always be individuals that truly enjoy reinventing wheels, that said I can see a great number of individuals who will benefit from it [discuss....]

I also see huge benefit in a standardized trader. Even more benefit in a standardized .Net API. Something like your BfPDS is a step in the right direction, and I'd be very interested in seeing your design plans for it when you chose to make them available. I can try to offer help & insight but no guarantees! However ultimately, as long as the BfPDS is still using the current API, it will always come down to the same issues (just wrapping them up in a nicer package). I'm guessing number 1 priority will be speed of access, so whatever the solution may be it has to be as efficient as possible.

[vossie]

Hi Michael,

I think effort could be better spent in improving the API, in terms of cleaning up documentation, improving calls, etc. There are still some weird features (like the functionality of PlaceBets) that could be significantly improved IMO. An event driven API would be of huge benefit also, even if it only pushes events when market status changes, bets get matched, and simple things.

Work is currently being done to improve the API, my best guess is that the result of these efforts will not be publicly available for another 12 months and changes and upgrades to the current API will be limited to bug fixes and petrformance changes. Any large scale investment would go on delivering API7 but in the meantime everyone including Betfair first party developers has to use the existing API. The likelihood of seeing push functionality in the next version of the API is at best very slim. Current push solutions is very vendor/platform specific and extremely resource intensive in an exchange environment, untill these technology concerns is addressed and widely accepted, supported and adopted by open source and proprietary solutions provider like SOAP it just does not make commercial sense. If you want to test the statement write a simple application server capable of concistently servicing 1000 un-cached complex request/responses per second with a spike rate capability of 2000, now add push functionality (WCF, Comet etc.) to that server and see what happens.

The problem with building a "one program fits all" bot framework is there are a lot of different trading methods out there. Some will use betfair purely which is ok, but others will look at several different websites & data sources to build up a model. Integrating this I imagine would be difficult with such a framework. Although done correctly I'm sure it could be a great success.

Because of this, I feel the majority of people will still be tapping directly into the API which makes the effort put into these extra 'layers' only benefiting the people who use them. Improve the core API functionality and everyone benefits!

That is what I realised and decided to release the Lignite project as an unfinished work and shared the idea and reasons for the BfPDS - The BfPDS will adress your concerns, provide a lightweight language agnostic abstraction layer, shielding the consuming clients against API changes and when the new API7 becomes available the BfPDS will still serve up the data in the same format without the consuming clients even being aware of the change.
You are absolutely right the solution will not work for everyone and there will always be individuals that truly enjoy reinventing wheels, that said I can see a great number of individuals who will benefit from it [discuss....]

Regards
Vossie

[MichaelRNye]

I think effort could be better spent in improving the API, in terms of cleaning up documentation, improving calls, etc. There are still some weird features (like the functionality of PlaceBets) that could be significantly improved IMO. An event driven API would be of huge benefit also, even if it only pushes events when market status changes, bets get matched, and simple things.

The problem with building a "one program fits all" bot framework is there are a lot of different trading methods out there. Some will use betfair purely which is ok, but others will look at several different websites & data sources to build up a model. Integrating this I imagine would be difficult with such a framework. Although done correctly I'm sure it could be a great success.

Because of this, I feel the majority of people will still be tapping directly into the API which makes the effort put into these extra 'layers' only benefiting the people who use them. Improve the core API functionality and everyone benefits!

[vossie]

To add to that, if you want to sell or release software (even free software) you have to comply with this http://bdp.betfair.com/index.php?opt...d=76&Itemid=58 and http://bdp.betfair.com/index.php?opt...d=69&Itemid=59

Now to add to the complexity of the DRC it is not calculated on a per second basis but rather by total requests in a wall clock second based on the call weighting for all your activity including site, so if you have 2 bots that is allowed 20 requests per second (rps) both bots need to know that they can really only run at about 8 rps because an allowance has to be made for web activity but if you bet you can calculate your commission and implied commission that is offset against the DRC which would mean you can possibly crank the bots up to 11 rps each on certain days. It is also worth noting that the default 20 rps is not per account but for all your accounts as a whole. So if you have 2 accounts registered you can still only make 20 rps combined not 40 rps and this still does not include the commission offset allowance.

Surely all these rules and T&Cs has already given you guys enough of a headache?

[vossie]

Programming against the Betfair API is about as friendly as a shark attack.

Hi Drifter & BfHopsen,

By participants I mean members of the community willing to help test the alpha version moving to beta and help with ideas around the domain model and additional functionality and methods you would like to see incorporated in the BfPDS.

To address Drifters comments about it being beneficial to Betfair, well not really if Betfair does have a performance and capacity issue they would just throw some tin at it or come up with yet another charging scheme to discourage bad behaviour? The problem I am trying to solve is the ones mentioned before and my concern is really just with helping you guys.

Lets look at it from the angle of new and exsisting developers and what they have to deal with
- DRC http://content.betfair.com/aboutus/c...fair%20Charges
- Virtual Bets http://bdp.betfair.com/index.php?opt...=237&Itemid=62
- Not to mention the quirks in the actual API and just check out the charges page at http://content.betfair.com/aboutus/?...=GBR&locale=en for some added reading

The other comment made in this forum that I thought priceless was that programming against the Betfair API is about as friendly as a shark attack, so lets do something about it. The BfPDS will make data available as Binary .NET objects, SOAP, JSON etc so you wont be bound by one formatting option, secondly local push will deal with probably 22% of your develop headaches

Continued on the next page...

[Drifter]

Just to add to that, I am also very pleased you have made this (Lignite) available. Unfortunately, rather like bfhopson, I lack the skill level to contribute in any meaningful way but there is probably much I can learn by studying it.

Pardon me if this sounds like I am behind the curve a bit, but with the personal data server concept, is it your intention that in effect all connections would be served the same well-known data (sub) set, dependant on subscription? And that the individual local applications then pull the data as they see fit? I can see that might have considerable efficiency gains at your server end, eliminating a lot of API error calls, 'throttling' messaging etc.

[bfhopsen]

Unfortunately I am not skilled enough to participate, sounds too heavy even with this short draft of features and concept, but I wish you all luck. What kind of licence you plan to use? Hope it will be something free.