C# The beginnings of a bot

[vossie]

I created the following application as a proof of concept. It is by no means production ready but I thought I would make it available as it might be of use to somebody building a bot and looking for a foot up or just some crazy ideas for their interface.

The first person or team to complete this project is welcome to Free full API access for up to 5 accounts, just mail bdp@betfair.com. By complete I mean tested, bet placement functionality added and data request charge counters added. Remember the original copyright holder will continue to be Betfair and this work is released under the GNU General Public License.

If you make improvements to this library please post them here so we can include these into the project.

Also be careful this thing stores the user name & password in the config file (yes I know I should have removed this) and secondly if you configure it wrong you will get hit by the data request charge.

To run the compiled program do the following
1. Extract to your local machine
2. run ..\Lignite 1.0\Lignite.Console\bin\Debug\Lignite.Console.exe
3. Go to Edit -> Main Config File -> Betfair and set your username and password. Close config
4. Click Start
5. Click Stop
6. Go to Edit -> Main Config File -> Betfair and un-set your username and password. Close config

The idea behind this is rather simple separate everything, data view, data collection and make the strategy part a plug-in.

Happy coding


*** The Legal Stuff ***

The library is provided "as is" without warranty of any kind, either expressed or implied, including, without limitation, warranties that the library is free of defects, merchantable, fit for a particular purpose or non-infringing. the entire risk as to the quality and performance of the library is with you. Should any prove defective in any respect, you (not the initial developer or any other contributor) assume the cost of any necessary servicing, repair or correction.

[saint]

Nice example, cheers

[vossie]

Nice example, cheers

Thanks, was starting to wonder if anyone would like it

[bfhopsen]

No doubt

I just wandered what happened with your open source C# Project you started several months ago, and what happened with your participation in this forum, then you come with Lignite. Be sure that someone like it and that it is very useful especially because this forum has started to lack answers on some very important and interesting questions (I am really wandering what is going on with responsiveness on the BDP forum, it is as bad as betfair api responses on GetMarketTradedVolumeCompressed calls in the past month )
I am coding in Java, but nevertheless, in Lignite I can find some very useful answers and concepts. Just continue.
Also hope that you will have time to answer some questions if I step into some problems.

P.S Sorry for English

[vossie]

Where do we go next?

I just wandered what happened with your open source C# Project you started several months ago, and what happened with your participation in this forum, then you come with Lignite. Be sure that someone like it and that it is very useful especially because this forum has started to lack answers on some very important and interesting questions (I am really wandering what is going on with responsiveness on the BDP forum, it is as bad as betfair api responses on GetMarketTradedVolumeCompressed calls in the past month )
I am coding in Java, but nevertheless, in Lignite I can find some very useful answers and concepts. Just continue.
Also hope that you will have time to answer some questions if I step into some problems.

P.S Sorry for English

No problem mate and I will try to answer questions where possible

The work and project started previously was conceptionally sound but lacked in many respects (most notably time & funding).

This project was a proof of concept built on those ideas explored previously and helped iron out some questions and performance considerations.

I have drawn the following conclusions based on the research and work done in private and publicly

• We all redesign the wheel when we start a new Betfair API project
• We all do pretty much the same thing (except for the trading desision part)
• There is a lot of guys out there comfortable with scripted languages that does not support built in multi threading
• Local push based on a local network model would go a long way to get users going (in other words, an event driven architecture See: http://www.amazon.co.uk/Event-Driven.../dp/0321322118)
• A way of centrally dealing with the plethora of different charging schemes the guys dream up

Work will shortly be starting on what I affectionately call the "Betfair Personal Data Server" or BfPDS (maybe the community can help me come up with a better name)

Continued...

[vossie]

The idea is this:
1 - You have a single instance of the BfPDS running on your local network and all your applications, scripts notifiers etc. talks to the BfPDS.
2 - The BfPDS is built using WCF technology to allow for interop and allow communication over multiple protocols
3 - The BfPDS exposes it's own Betfair domain model isolating it to changes in the Betfair API WSDL
4 - The BfPDS has plug-in architecture and internal event streams, so you can use it in conjunction with something like PHP in a simulated push architecture. The process flow for the post plug-in will be: You send a request to the BfPDS polling method with the parameters and data request you want it to process, incrementally it makes the data request and caches the response locally, if the data changed an internal event gets fired the plug-in listens for this event and posts to the configured endpoint i.e. http://localhost/myscripts/php5/marketdataupdated.php This is only for languages that does not support the built-in 2 way communication.
5 - It also exposes the service with the option of MS http dual binding
6 - Must be mono compliant (possibly with some acceptable feature loss)
7 - Single outbound data request manager with multiple internal consumers from a network perspective.

This is a rough overview but once I have completed the design schematics and proposed screen shots I will have a project page created and assemble a merry team of participants.

Would anybody be interested.

[vossie]

BfPDS

The initial version of BfPDS will expose the following services in a unified way
- Betfair Global
- Betfair Exchange Sports
- Timeform Horse Racing (See the Timeform project I posted on this forum https://forum.bdp.betfair.com/showthread.php?t=340)
- Betfair Tote/s

And possibly at a later stage we could add
- Betfair Exchange Games

[bfhopsen]

Unfortunately I am not skilled enough to participate, sounds too heavy even with this short draft of features and concept, but I wish you all luck. What kind of licence you plan to use? Hope it will be something free.

[Drifter]

Just to add to that, I am also very pleased you have made this (Lignite) available. Unfortunately, rather like bfhopson, I lack the skill level to contribute in any meaningful way but there is probably much I can learn by studying it.

Pardon me if this sounds like I am behind the curve a bit, but with the personal data server concept, is it your intention that in effect all connections would be served the same well-known data (sub) set, dependant on subscription? And that the individual local applications then pull the data as they see fit? I can see that might have considerable efficiency gains at your server end, eliminating a lot of API error calls, 'throttling' messaging etc.

[vossie]

Programming against the Betfair API is about as friendly as a shark attack.

Hi Drifter & BfHopsen,

By participants I mean members of the community willing to help test the alpha version moving to beta and help with ideas around the domain model and additional functionality and methods you would like to see incorporated in the BfPDS.

To address Drifters comments about it being beneficial to Betfair, well not really if Betfair does have a performance and capacity issue they would just throw some tin at it or come up with yet another charging scheme to discourage bad behaviour? The problem I am trying to solve is the ones mentioned before and my concern is really just with helping you guys.

Lets look at it from the angle of new and exsisting developers and what they have to deal with
- DRC http://content.betfair.com/aboutus/c...fair%20Charges
- Virtual Bets http://bdp.betfair.com/index.php?opt...=237&Itemid=62
- Not to mention the quirks in the actual API and just check out the charges page at http://content.betfair.com/aboutus/?...=GBR&locale=en for some added reading

The other comment made in this forum that I thought priceless was that programming against the Betfair API is about as friendly as a shark attack, so lets do something about it. The BfPDS will make data available as Binary .NET objects, SOAP, JSON etc so you wont be bound by one formatting option, secondly local push will deal with probably 22% of your develop headaches

Continued on the next page...

[vossie]

To add to that, if you want to sell or release software (even free software) you have to comply with this http://bdp.betfair.com/index.php?opt...d=76&Itemid=58 and http://bdp.betfair.com/index.php?opt...d=69&Itemid=59

Now to add to the complexity of the DRC it is not calculated on a per second basis but rather by total requests in a wall clock second based on the call weighting for all your activity including site, so if you have 2 bots that is allowed 20 requests per second (rps) both bots need to know that they can really only run at about 8 rps because an allowance has to be made for web activity but if you bet you can calculate your commission and implied commission that is offset against the DRC which would mean you can possibly crank the bots up to 11 rps each on certain days. It is also worth noting that the default 20 rps is not per account but for all your accounts as a whole. So if you have 2 accounts registered you can still only make 20 rps combined not 40 rps and this still does not include the commission offset allowance.

Surely all these rules and T&Cs has already given you guys enough of a headache?

[MichaelRNye]

I think effort could be better spent in improving the API, in terms of cleaning up documentation, improving calls, etc. There are still some weird features (like the functionality of PlaceBets) that could be significantly improved IMO. An event driven API would be of huge benefit also, even if it only pushes events when market status changes, bets get matched, and simple things.

The problem with building a "one program fits all" bot framework is there are a lot of different trading methods out there. Some will use betfair purely which is ok, but others will look at several different websites & data sources to build up a model. Integrating this I imagine would be difficult with such a framework. Although done correctly I'm sure it could be a great success.

Because of this, I feel the majority of people will still be tapping directly into the API which makes the effort put into these extra 'layers' only benefiting the people who use them. Improve the core API functionality and everyone benefits!

[vossie]

Hi Michael,

I think effort could be better spent in improving the API, in terms of cleaning up documentation, improving calls, etc. There are still some weird features (like the functionality of PlaceBets) that could be significantly improved IMO. An event driven API would be of huge benefit also, even if it only pushes events when market status changes, bets get matched, and simple things.

Work is currently being done to improve the API, my best guess is that the result of these efforts will not be publicly available for another 12 months and changes and upgrades to the current API will be limited to bug fixes and petrformance changes. Any large scale investment would go on delivering API7 but in the meantime everyone including Betfair first party developers has to use the existing API. The likelihood of seeing push functionality in the next version of the API is at best very slim. Current push solutions is very vendor/platform specific and extremely resource intensive in an exchange environment, untill these technology concerns is addressed and widely accepted, supported and adopted by open source and proprietary solutions provider like SOAP it just does not make commercial sense. If you want to test the statement write a simple application server capable of concistently servicing 1000 un-cached complex request/responses per second with a spike rate capability of 2000, now add push functionality (WCF, Comet etc.) to that server and see what happens.

The problem with building a "one program fits all" bot framework is there are a lot of different trading methods out there. Some will use betfair purely which is ok, but others will look at several different websites & data sources to build up a model. Integrating this I imagine would be difficult with such a framework. Although done correctly I'm sure it could be a great success.

Because of this, I feel the majority of people will still be tapping directly into the API which makes the effort put into these extra 'layers' only benefiting the people who use them. Improve the core API functionality and everyone benefits!

That is what I realised and decided to release the Lignite project as an unfinished work and shared the idea and reasons for the BfPDS - The BfPDS will adress your concerns, provide a lightweight language agnostic abstraction layer, shielding the consuming clients against API changes and when the new API7 becomes available the BfPDS will still serve up the data in the same format without the consuming clients even being aware of the change.
You are absolutely right the solution will not work for everyone and there will always be individuals that truly enjoy reinventing wheels, that said I can see a great number of individuals who will benefit from it [discuss....]

Regards
Vossie

[MichaelRNye]

Work is currently being done to improve the API, my best guess is that the result of these efforts will not be publicly available for another 12 months and changes and upgrades to the current API will be limited to bug fixes and petrformance changes. Any large scale investment would go on delivering API7 but in the meantime everyone including Betfair first party developers has to use the existing API. The likelihood of seeing push functionality in the next version of the API is at best very slim. Current push solutions is very vendor/platform specific and extremely resource intensive in an exchange environment, untill these technology concerns is addressed and widely accepted, supported and adopted by open source and proprietary solutions provider like SOAP it just does not make commercial sense. If you want to test the statement write a simple application server capable of concistently servicing 1000 un-cached complex request/responses per second with a spike rate capability of 2000, now add push functionality (WCF, Comet etc.) to that server and see what happens.

Oh I don't doubt it would be a huge strain I often wonder how the betfair servers manage to cope with the no doubt enormous amount of requests being sent every second. Understanding that it is a huge undertaking to release a new API whilst retaining compatability with the current system, I would imagine that moving to a push style system would hopefully reduce the load on the servers. It's hard to gauge, but I think most of the current queries are mainly people poking to see if somethings changed rather than meaningful queries.

That is what I realised and decided to release the Lignite project as an unfinished work and shared the idea and reasons for the BfPDS - The BfPDS will adress your concerns, provide a lightweight language agnostic abstraction layer, shielding the consuming clients against API changes and when the new API7 becomes available the BfPDS will still serve up the data in the same format without the consuming clients even being aware of the change.
You are absolutely right the solution will not work for everyone and there will always be individuals that truly enjoy reinventing wheels, that said I can see a great number of individuals who will benefit from it [discuss....]

I also see huge benefit in a standardized trader. Even more benefit in a standardized .Net API. Something like your BfPDS is a step in the right direction, and I'd be very interested in seeing your design plans for it when you chose to make them available. I can try to offer help & insight but no guarantees! However ultimately, as long as the BfPDS is still using the current API, it will always come down to the same issues (just wrapping them up in a nicer package). I'm guessing number 1 priority will be speed of access, so whatever the solution may be it has to be as efficient as possible.

[vossie]

Push from a public API perspective

I agree that a push system would be advantages to some users.

Lets look at push in a real world scenario. The first problem is that the top 70% of transactional API consumers are not really as price and change sensitive as everyone else thinks they are, don't get me wrong everyone is price sensitive but the difference between 3 and 5 volatile data request in a single second is hardly noteworthy from a client perspective.

With push a client system that previously used a well thought out polling mechanism would simply chuck that away and subscribe to streams that quite possibly update at a rate of 200ms. Now this client previously polled at a rate of twice a second with no real impact on the effeciency of their system or Betfairs, now multiply those 3 aditional push responses introduced by hundreds of thousands and the scope of impact becomes clear. To enable push you also have to keep the communication channel open between server and listener and this is most definately a finite resource, in a controlled enterprise environment you can do this easily and scale just as easily but in the wild web you have all sorts of problems to contend with, most notably connection drops and latency. With push the client sessions are sticky which means you could get stuck on a single server with a heavy user impacting your performance.

The alternative and well supported polling approach can much more efficiently handle the data distribution using http protocols, admittendly the shape of the current Betfair requests and responses are questionable at best and can be greatly improved by adding a last request timestamp to the prices request that returns null if the cached prices on the server has not changed since your last request.
This type of transaction where no data is returned on no change would be a slightly heavier data object than the heartbeat you would have on the push channel but it means that you can stagger the responses even if only by fractions of a second and you can also load balance over a much larger cluster without too many coherency issues.