Using VB2008 to acccess the Betfair API: A tutorial

[monkeymagix]

Handling Partial Matches Safely

Hi Mumbles

I am in the process of testing my auto trading bot which aims to automatically place trade bets on selections that have been pre-determined by a backend DB. My database is complete and selecting runners perfectly and I am in the process of testing with live data and money. On a side note it is a real pain not having a test mode to simulate real betting as I am having to waste real money just to debug my system under live conditions.

Everything is working fine apart from partial matches as obviously I need the lay amount to be equal to (or more if overlaying) to the back stake.

If I place the WIN part of the trade and it is only partially matched say £2 out of £10 then I don't want to place the LAY part at £2 if the remaining £8 is then matched later on for the WIN either on the market or at SP as I could lose out big time. This has actually happened today and because the win bet was matched in two stages the lay part was set at a lower figure than it should have been which meant when the horse lost I lost instead of winning money!

Therefore I have come up with the following business logic. I would appreciate it if you could check this logic for me and tell me if it looks correct to you.


1. My back end system calculates the runners to place trades on at intervals throughout the day and sets up records in a PENDING_BETS tables that specifies all the parameters such as Selection, Market, Bet Amount, Maximum Liability, Max / Min odds, best price, current status and so on.

2. For TRADE bets there are 2 corresponding records linked by a Trade ID key and at first only the WIN part of the bet is set to PENDING whilst the LAY part is not active. This way a LAY bet is only ever placed once the WIN part has already been matched.

3. My BOT polls this table at intervals checking for new bets to be placed and to check for settled bets or changes in statuses.

4. A WIN bet is placed at the best possible price and if the bet is matched then the LAY part of the TRADE is set to PENDING and it has it MAX Price set to the WIN bets Matched Price and a BEST Price set to something lower.

5. The bot tries to match the BEST price for the LAY bet up until the race is near and if it has not matched the BEST price by then it will resort to matching up to the MAX Price. The code will also calculate overlays when the best price can be obtained to obtain profits from losing runners.

Now the tricky bit with partial matches

I am currently checking for PARTIAL MATCHES in my PlaceBet method by checking the Size Matched against my initial desired amount to see if it is under. If they are equal then the full bet amount was taken.

I have found from my tests that a PARTIAL MATCH bet will use the same BetID for all parts. Therefore once I know that at least some of the bet has been matched I take this shared BetID from the successfully matched part and call a CancelBet method straight away. In theory this will only ever cancel any unmatched parts of the bet as you cannot cancel a matched bet.

Also if the bet was just unmatched rather than partially matched then this shouldn't cause any problems as the whole bet will be cancelled and then re-attempted on the next poll loop.

If the CancelBet call is unsuccessful due to the unmatched part having been taken OR only partially cancelled due to a slice of it being taken then I call my GetBetStatus method to get the current matched size of the bet safely knowing that this will now not change as I have cancelled everything I possibly can.

I can then call my SaveBetStatus method which will update my PENDING_BETS table in the DB so that the LAY bet amount is set to the correct amount.

For the remaining amount that was cancelled I can then create a new PAIR of WIN & LAY bets in my PENDING_BETS table.

Can you see any problems with this logic?

Thanks for your help on this matter.

[Mumbles0]

Your Trading App

Monkeymagix,

I’m pleased to hear that your database is selecting runners perfectly. I wish mine would select perfect runners.

I don’t know if I’m the best person to discuss trading methods. I’m sure there are traders out there who know a lot more about it than I do. But for what it’s worth, allow me to make a few comments.

Betting strategies are personal things. If you get onto a good one then it’s probably best to keep it to yourself.

Perhaps your system is a bit idealistic. After your back bet has been matched it’s all very well setting a best and max lay price, but what do you do if your runner’s price won’t behave itself and drifts up?

It certainly would be nice to have a test facility. No doubt you have had some “dry runs”, simply observing when your bets would have been matched had they been placed.

Partial matching occurs frequently. Dealing with partially-matched bets certainly makes any betting system a lot more complicated but they must be catered for. It’s interesting that you have noticed that both portions of a partially-matched bet retain the same betId. It would also be interesting to know if the bets array returned by getMUBets lists these the portions as separate bets with the same betId.

Have I got this right? If your back bet is partially matched you immediately cancel the unmatched portion and resubmit a new back bet for the same price and amount (as the just-cancelled unmatched portion). Must you do this? It would be better to work with the unmatched portion as is (if possible) because it retains its position at the head of the queue, where as a new bet at the same price comes in at the end of the queue.

Also, what do mean by “just unmatched” ?

[monkeymagix]

Handling Partial Matches

Hi Mumbles

Cheers for getting back to me.

My logic is obviously changing all the time as I haven't been able to find any kind of logical flow diagram that describes how a system engaging the betfair API should proceed in terms of logical steps (e.g how to place a bet and handle all possible consequences). If it hadn't have been for this tutorial I don't know where I would have started as some of the other free code I have downloaded off the web has been pretty awful to say the least.

In terms of my "perfect" selection system, nothing is ever perfect and I didn't mean that all my selections were 100% accurate just that my back end logic for selecting daily runners to place bets on is working fine. It is only the front end logic that engages the Betfair API that is throwing up the odd curve ball.

Not give my game away but I have spent the best part of a year working on a racing system and I am basing my trading choices on a number of factors that reduce the likelihood of unexpected price drifts one way or another. Not to say that this doesn't happen but I am trading on runners that have a good chance of winning so that if the LAY bet cannot be matched at ideal prices I have very good odds of gaining the full profit from a win bet rather than just the difference between the WIN and LAY price. Overall (with my system) it seems that this method seems to return better profits than just always trying to LAY even if that price is higher.

At the moment I am not using getMUBets in my code as I am literally placing a bet, checking the matched size matched the desired size and if not cancelling ASAP.

For "just unmatched" I meant that the bet hadn't been matched on the exchange market and was flagged with a status of U (with no matching parts at all).

I didn't realise that my logic of cancelling ASAP would affect my pricing strategy because of the queueing system as if I cancel the unmatched portion of the bet there will be a whole "logical loop" iteration before any re-attempt is taken place which involves:

-updating the DB to tell the system that the bet has been placed along with the matched amount and price taken
-a creation of new records in the DB to take the place of any unmatched part
-a new BOT call to the API which will look to my DB for any new PENDING bets
-a new call to the market to get the best available price at that moment
-logic to decide whether or not to place the bet

So when the unmatched part is re-attempted the chances are that a new price will be given and taken anyway. I don't know if I could retain my unmatched bet AND modify the price so that it keeps its position in the queue?

Also on a side note I never seem to be able to keep my previous session ID from my last API request and always end up having to re-login. I have tested this by outputting the Session ID once it has been obtained, saved and on use.

My application is a C# console app that is currently running on a scheduled job once a minute.

The main code that handles the session storage and re-use is here (taken from your tutorial)

Can you see anything wrong with it or think of a reason why I would not be able to use re-use the session?

public class BetfairBot
{

	private BFGlobal.APIRequestHeader HeaderGL;
	private BFGlobal.BFGlobalService BetfairGL;
	private BFUK.BFExchangeService BetfairUK;

	private BetfairRaceCourseAbreviations RacecourseABV;

	// I store the path to the text file in my config file
	private string SessionPath = ConfigurationManager.AppSettings["SessionPath"].ToString();


	public BetfairBot()
        {
            HeaderGL = new BFGlobal.APIRequestHeader();
            BetfairGL = new BFGlobal.BFGlobalService();
            BetfairUK = new BFUK.BFExchangeService();

            // load in any session values
            string sessionval = File.ReadAllText(SessionPath);

	    // This is just a debug message that logs to a file so I can see later on what went wrong!
            Program.ShowMsg("Set session value to " + sessionval.ToString());

            HeaderGL.sessionToken = sessionval;
        }

	private BFUK.APIRequestHeader oHeaderUK(){
            BFUK.APIRequestHeader Header = new BFUK.APIRequestHeader();
            Header.sessionToken = HeaderGL.sessionToken;

            return Header;
        }

	private void CheckHeader(BFGlobal.APIResponseHeader Header)
        {
            HeaderGL.sessionToken = Header.sessionToken;

            // if we have broken the API request limit wait until the next minute when it gets reset
            if (Header.errorCode.ToString() == "EXCEEDED_THROTTLE")
            {
                // wait until the next minute
                BeatThrottleLimit();
            }           

        }

	public void SaveSession(){

            Program.ShowMsg("Save session value = " + HeaderGL.sessionToken.ToString());

            File.WriteAllText(SessionPath, HeaderGL.sessionToken);
        }

}

[Monairda]

How to control the Data Request Charge?

Good morning I read about this issue.

"Charges Apply to Both API and website data requests. Only market data (for example market prices) and private betting data (for example current bets, P & L) Requests Are subject to the charge. At the end of Every Day. And Any customer making More Than 20 Requests data in one second will be charged 0.1p per data requestabove 20. "

But how can we control these calls not to exceed that limit? Controlling the number of calls over statecount every second!

Private StateCount As Integer 'The unique state object

Thanks!

[Mumbles0]

Your Trading App

Monkeymagix,

Perhaps I should give a disclaimer. Note that anything I say about the way the exchange operates is purely my own (unofficial) view. As you know, some API details are not clearly documented, hence we must learn through our own observations.

Re queuing of an unmatched portion: I’m not 100% sure if this exactly what happens, but think this is the case. Indeed changing the price (with updateBets) effectively creates a new bet which joins the end of the queue at the new price.

When you place a bet it looks like you are having a “quick stab” at the market, cancelling if the bet (or any part) is not matched immediately. If you left the unmatched bet sitting in there for a while there’s a good chance of getting a match at the price you want. But to do this you would have to poll the bet’s status to see when (or if) it is matched using say, getMUBets. This would increase the complexity of your program but it might be worth it.

Saving the session token:

Your code looks OK on the surface. I’m not all that familiar with the order in which things execute. The session token should be read from the file when the program starts, and saved when the program terminates. Make sure you call CheckHeader after every API call (especially login). You’ve got a lot of debug code in there - you tell me what’s wrong.

Since you are using an application setting to hold the path to the auxiliary file, why not save the session token in an app setting instead? It would be a lot simpler.

By the way, this is a VB tutorial thread. I don’t think it’s a good idea to paste too much C# code here - it may confuse some readers.

[Mumbles0]

Data request charges

Monairda,

While it is possible to exceed the 20 calls/second limit with the Free API, the problem mainly affects users of the Full Access API. The charges are explained here if you can understand this.

The main point is you may be charged if you exceed 20 API calls per second, also if you place or edit more than 1000 bets in an hour. If you don’t like this it’s up to you to ensure that your application stays below these limits.

Let’s say you have a multi-market app and, for each market, you make calls to getMarketPricesCompressed and getMarketTradedVolumeCompressed at regular intervals from a timer Tick event, i.e. 2 API calls per market.

If your sampling time interval is 1 sec, then you can look at up to 10 markets at once
i.e. 10 markets x 2 calls each = 20 calls in 1 sec

If your sampling time interval is 2 sec, then you can look at up to 20 markets at once
i.e. 20 markets x 2 calls each = 40 calls in 2 sec
etc...

If you are operating close to the limit you should also leave a bit of headroom for other calls (placeBets, etc).

The StateCount variable is used for something else (it prevents an async call error). You could implement a simple call limiting scheme using a call counter, but I don’t think it would be very satisfactory.

[Monairda]

How to control the number of data requests ?

As always thank you very much Mumbles0

So not to exceed the limit of 20 calls per second, could put a counter between calls, and if it reaches 15 wait a second?

Dim Market As MarketForm
Dim counter As Int16 = 0

For Each Market In MyMarkets
            counter + = 1
            If counter > 15 Then
                System.Threading.Thread.Sleep(1000)
                counter= 0
            End If


            Market.CallCount = 0  'Clear the call counter

            Dim oMPCreq As New BFUK.GetMarketPricesCompressedReq
            With oMPCreq
                .header = oHeaderUK()
                .marketId = Market.MarketId
            End With
            StateCount += 1
            BetFairUk.getMarketPricesCompressedAsync(oMPCreq, StateCount)  'Call the API
            

            Dim oMTVCReq As New BFUK.GetMarketTradedVolumeCompressedReq      'The request object
            With oMTVCReq
                .header = oHeaderUK()
                .marketId = Market.MarketId
            End With
            StateCount += 1
            BetFairUk.getMarketTradedVolumeCompressedAsync(oMTVCReq, StateCount)    'Call the API
            
        Next

Sorry if this solution is not good
Thanks!

[Mumbles0]

Limiting data requests

Monairda,

That's a very brutal approach. You are freezing your program for 1 sec if you open more than 15 markets. Does this work?

I would be interested to know a bit more about your project:

• Are you using the Free or Full API?
• Is the code you've posted being called from a timer Tick? If so, what is the time interval?

[Monairda]

Monairda,

That's a very brutal approach. You are freezing your program for 1 sec if you open more than 15 markets. Does this work?

I would be interested to know a bit more about your project:

• Are you using the Free or Full API?
• Is the code you've posted being called from a timer Tick? If so, what is the time interval?

Hi Mumbles0

I use the free API, I followed all the steps you've taken in the tutorial and there is a timer that has a refresh interval of 5 seconds. I am understanding what I do is to distribute the calls in those 5 seconds, right?

Friend MyMarkets As New Collection 'The Markets of Interest

What makes the application is to cover the entire collection mymarkets every 5 seconds, but the problem is that if the collection has more than 10 markets over the limit of 20 calls per second.

Thanks!!

[Mumbles0]

Limiting data requests

Monairda,

Putting the main thread to sleep is no good. I think it’s better to do it this way.

I have made a few changes to Sub GetMarketData. Instead of updating all markets every 5 sec, try updating each market in turn every 1 sec. Change your timer’s Interval to 1000 to call Sub GetMarketData every second.

[COLOR="Gray"]Friend MyMarkets As New Collection  'The markets of interest
[COLOR="Black"]Private MarketNo As Integer    'The market index[/COLOR]

Private Sub GetMarketData()
  Dim Market As MarketForm

  [COLOR="Black"]If MyMarkets.Count > 0 Then  'There are markets selected

    MarketNo += 1   'Next market 
    If MarketNo > MyMarkets.Count Then MarketNo = 1 'All markets done, start again
    Market = MyMarkets(MarketNo)[/COLOR]

    Market.CallCount = 0  'Clear the call counter

    Dim oMPCreq As New BFUK.GetMarketPricesCompressedReq
    With oMPCreq
      .header = oHeaderUK()
      .marketId = Market.MarketId
    End With
    StateCount += 1
    BetFairUK.getMarketPricesCompressedAsync(oMPCreq, StateCount)  'Call the API

    Dim oMTVCReq As New BFUK.GetMarketTradedVolumeCompressedReq      'The request object
    With oMTVCReq
      .header = oHeaderUK()
      .marketId = Market.MarketId
    End With
    StateCount += 1
    BetFairUK.getMarketTradedVolumeCompressedAsync(oMTVCReq, StateCount)    'Call the API

  [COLOR="Black"]End If[/COLOR]

End Sub[/COLOR]

This Sub now makes the API calls for each market in turn. When all markets are done the cycle repeats. This means that if, for example, 3 markets are selected, each market is updated every 3 sec.

Doing it this way means that getMarketPricesCompressed and getMarketTradedVolumeCompressed are each called 60 time per sec, so you shouldn’t get EXCEEDED_THROTTLE. Also, only 2 API calls are made each second, well within the 20 calls/sec limit.

[Monairda]

Limiting Data Request

Thank you very much Mumbles0

If I understand correctly, what you do is select the varialbe MarketNo market position within the collection, right?

And if you want to refresh 6 markets every second?
Do not give me the solution I will try it and you wonder if the solution is correct!


Another question, how I can remove from the collection a market that has been closed?

Let me explain:
When the application is monitoring the market is removed from the collection (mymarkets) if the last matched price of any horse has reached 1.01, but sometimes the last matched price does not reach the 1.01 and remains at 1.05 or 1.03 or 1.10 etc ...

I use this code, when the market is closed, but the .marketid is equals to zero and I do not know how to identify the market to close

[COLOR="Gray"]Private Sub BetfairUK_getMarketTradedVolumeCompressedCompleted(ByVal sender As Object, ByVal e As BFUK.getMarketTradedVolumeCompressedCompletedEventArgs) Handles BetFairUk.getMarketTradedVolumeCompressedCompleted
        Try
            If Not e.Cancelled Then
                With e.Result

                    CheckHeader(.header)
                    If .errorCode = BFUK.GetMarketTradedVolumeCompressedErrorEnum.OK Then

                        Dim oTradedVolume As New UnpackMarketTradedVolumeCompressed(.tradedVolume)  'Unpack the traded volumes
                        Dim Key As String = .marketId           'The key for this market
                        If MyMarkets.Contains(Key) Then            'The MarketForm is open
                            Dim Market As MarketForm = MyMarkets(Key)   'A reference to the form for this market
                            Market.TradedVolumeReceived(oTradedVolume)  'Process the volume response
                        End If
                    [COLOR="Black"]ElseIf .errorCode = BFUK.GetMarketTradedVolumeCompressedErrorEnum.EVENT_CLOSED Then
                        If MyMarkets.Contains(.marketId) Then            'The MarketForm is open
                            MyMarkets.Remove(.marketId)
                        End If[/COLOR]
                   Else
                        Print("ErrorCode = " & .errorCode.ToString)
                    End If
                End With
            End If
        Catch ex As ApplicationException  'If problem
            Print(ex.Message)
        End Try

    End Sub[/COLOR]

Any suggestions?

Thank you very much for your help

[monkeymagix]

Market Stabs, Sessions and Security

Hi Mumbles

Sorry for posting c# code here but I have tried starting a couple of C# threads but haven't received a single reply e.g

http://forum.bdp.betfair.com/showthread.php?t=1056
http://forum.bdp.betfair.com/showthread.php?t=1038

You are correct in that I am having "stabs" at the market and maybe I don't need to when placing the LAY part of a trade bet. The reason I cancel after the WIN part is so that I set the LAY bets boundaries up correctly as there is no point in trading if both parts don't cover each over and I have been stung a couple of times because of partial matches on the WIN bet which meant the corresponding LAY bet boundaries (e.g size) was set incorrectly.

I am trying to make my front end console app into a "dumb" front end as all the business logic that handles selections, bet types and prices is done in the back end. I just want the front end to literally pick up a list of bets to either check the status of or place and then do the job correctly updating the DB once the action is complete. In this regards it would be very nice to come up with an ideal logical flow for placing, cancelling and managing bets and it does seem that Betfair is lacking slightly in this regards which is why your tutorial is really my only port of call.


Regarding the Session issue I am using the following test harness.

public void TestSession()
{

     // this will load up the session value from the text file
     BetfairBot betfair = new BetfairBot();

     // using your standard keep alive function (see code I attached last time)
     betfair.KeepAlive();

     try
     {
	 double balance = betfair.GetAccountBalance();

	 Program.ShowMsg("Got Account Balance using last session value: " + balance.ToString());
     }
     catch (Exception e)
     {
	 Program.ShowMsg("Failed to get Account Balance with last session value - login now");

	 string betfairPassword = GetPassword("myencrypedpassword");
	 string betfairLogon = DecodeFrom64("myencrypedlogin");

	 Program.ShowMsg("Login for account: " + betfairLogon + " - password: " + betfairPassword);

	 // login
	 if (betfair.Login(betfairLogon, betfairPassword))
	 {
	     Program.ShowMsg("Logged In");
	 }
	 else
	 {
	     Program.ShowMsg("Not Logged In");

	     return;
	 }
     }
     finally
     {
	 Program.ShowMsg("Save Session and Logout");

	 betfair.SaveSession();

	 betfair.Logout();
     }

     Program.ShowMsg("Completed Test");             
}

public double GetAccountBalance()
{
    double AccountBalance = 0;

    BFUK.GetAccountFundsReq AccountReq = new BFUK.GetAccountFundsReq();
    BFUK.GetAccountFundsResp AccountResp = new BFUK.GetAccountFundsResp();
	
    // check header is still valid
    AccountReq.header = oHeaderUK();

    AccountResp = BetfairUK.getAccountFunds(AccountReq);

    // check for valid response (fails without a session)
    CheckHeader(AccountResp.header);

    // this doesnt run with no session
    if (AccountResp.errorCode == BFUK.GetAccountFundsErrorEnum.OK)
    {
	AccountBalance = AccountResp.availBalance;

	Program.ShowMsg("Successfully got available balance: " + AccountBalance.ToString());
    }

    return AccountBalance;
}

And I get this debug

Start BetfairBot
Start Betfair Bot Job: TEST
Test Session
Set session value to U2q6Hx/Bvz+U8is3lu9WJL/rREdrUEDWbyuH6rrZSos= saved from last visit
KeepAlive
HeaderCode = NO_SESSION
Got Account Balance using last session value: 0
Save Session and Logout
Error saving session value to text file: Object reference not set to an instance of an object.
Logout
HeaderCode = NO_SESSION
ErrorCode = API_ERROR
Completed Test
Test Session Complete
End BetfairBot


Note that the Catch in the Try/Catch doesn't actually get run as no error is raised in the GetAccountBalance method. However as there is no session the amount returned is 0 and you can see the HeaderCode = NO_SESSION error message that is returned by the API when I try to check the AccountReq.header value.

Also notice the Object reference not set error which happens when I try to save the session value back to the text file.

I guess I am going wrong somewhere but cannot spot where at the moment.

Just thinking aloud here >> It wouldn't be some kind of internal security Betfair has implemented to prevent XSS hacks and session hijacking would it? Maybe they can spot that I am using a console application and not a browser and preventing session re-use for those reasons? If you can literally take a string and then use it to re-open a previous session then there could be serious issues regarding security as if I could obtain someone else's session value somehow I could access their betfair account without their knowledge.

Thanks for your help.

[Mumbles0]

Your Trading App

MonkeyMagix,

Looks like I’m the only one who will talk to you, but I’m still not exactly sure what you are seeking.

Bet placement is an important part of your strategy. The API provides the tools to place, modify, monitor and delete bets. It’s up to you to decide how to go about this. There is no “standard”, or “ideal” procedure. It’s your call!

Your thoughts on session token persistence seem to be a bit wide of the mark. I (and presumably other users) have no problem with this. If I don’t log out, my session token usually remains current for a day or so. It looks like you’ve got a classic bug.

[monkeymagix]

Backend Logic

Okay cheers for your help I will debug the session values some more as it must just be a bug somewhere.

Regarding the "ideal solution" I know there is never going to be a one size fits all solution and the API is so big that you can do so many actions in any order and I am not expecting you to provide me with one.

What would be nice though (and this might be pie in the sky thinking as well) but obviously betfairs own website has a certain logical order to the actions that are needed to be carried out when you say place a bet or cancel a bet and it would be nice to find out the correct steps for each somehow.

For example my bot placed a bet earlier today which was unmatched however as it got nearer to the race my logic decreed that it would cancel the bet and then reset the price so that it had more chance of being taken.

On cancelling the bet the size matched was > 0 which meant that only a partial cancellation was carried out. This is a similar issue to partial matches and it would be nice to know how to handle these types of situations as with partial matches both the Cancelled part of the bet and the Matched part have the same BetID (same with settled and cancelled).

I have found that using a GetBetStatus function and passing in the betID will return a status of Settled once the race is over but I have not managed to find the status returned with a partial cancellation before the race ( although I expect it was C as my database record had that status code stored)

I guess I am not really asking for help with the front end logic as your API tutorial does a good job of explaining all those parts but I am wondering if you know of any guides or documentation that have been written or publicly available that specifically look at the backend database structure that would be ideal for a betfair application.

Something that would detail the entity relationships between bet objects e.g ONE BET can have MANY BET PARTS and so on.

Also on another point a lot of UK races were abandoned today (like the last couple of weeks) due to frozen ground and I had a lot of voided bets.

As I am storing the selectionID and marketID in my DB on the earliest possible opportunity so that I don't have to keep unpacking markets it means I have lots of pending bets that are trying to be placed on cancelled markets. I know that once a market has been cancelled the marketID will be 0 but I don't want to have to look up selections and markets on each job as 99% of the time it will just be pointless requests that mean I keep reaching the API throttle limit.

I am thinking about having a new job just to check the market is still active that would be run once an hour or so but I was wondering if there was another quicker way to check whether a market was still available which I could use by passing in my existing market ID.

Thanks for all your help

[monkeymagix]

Fixed

Also just to let you know I have fixed the session issue.

It was purely because I was calling the Logout function when I shouldn't have been!

Thanks