Response delays [.NET]

[bigbooting]

Hello all,

For the past few weeks I have been receiving significant response times from every call I am doing (UK and AUS). i.e. listMarketBook, placeBet, cancelBet etc. It happens once to several times a minute and the responses ranges from 200 milliseconds all the way to a 3 second time out that I've configured for the web request.

This is the code I use to create the web request. ContentLength is set afterwards to the size of the JSON object.

protected static HttpWebRequest CreateWebRequest(String restEndPoint, int timeout) {
            HttpWebRequest request = (HttpWebRequest)WebRequest.Create(restEndPoint);
            request.Method = "POST";
            request.ContentType = "application/json";
            request.ContentLength = 0;
            request.Timeout = timeout;
            request.KeepAlive = true;
            request.AllowAutoRedirect = true;
            request.Accept = "application/json";
            request.Headers["X-Application"] = BetFairService.ApplicationKey;
            request.Headers["X-Authentication"] = BetFairService.SessionKey;
            request.AutomaticDecompression = DecompressionMethods.GZip | DecompressionMethods.Deflate;
            request.Headers.Add(HttpRequestHeader.AcceptCharset, "ISO-8859-1,utf-8");
            request.Headers.Add(HttpRequestHeader.AcceptLanguage, "en-US");
            request.Headers.Add(HttpRequestHeader.Pragma, "no-cache");
            request.Headers.Add(HttpRequestHeader.CacheControl, "no-cache");
            return request;
        }

Because of these slowdowns it is making my bot quite inefficient so any help would be greatly appreciated.

Thanks you

[betdynamics]

I'm not sure you can do anything about the timeouts, but you can speed up the response time by ensuring that you are not using the header “Expect: 100-continue"

Whilst you are not explicitly adding that header, the .Net Framework adds it automatically.

To get round, this you need to use the following code:

System.Net.ServicePointManager.Expect100Continue = false;

Another thing to be aware of - since Betfair suffered a Denial of Service attack, they have added some traffic filtering software to their servers. This has increased the response time for API calls.

[bigbooting]

Hi betdynamics,

Thank you for the suggestion. Unfortunately, I have configured all requests with this new property and the problem persists.

Regarding the traffic filtering do you know if Betfair have some kind of whitelist to allow specific IPs without these delays? If not, are all API customers currently getting these delays? I feel like these forums would be packed with people complaining if that was the case.

[betdynamics]

This is what I got sent recently by the BDP team:

The DDOS protection is still in place; unfortunately this has added ~15 milliseconds of additional latency to all API customers however nobody is at a disadvantage. This will be in place for the foreseeable future although we will continue to monitor to ensure we alleviate any performance issues where possible.

[jptrader]

I have also had a high number delays in these time ranges for a while. Actually I had them even before the DDOS protection was in place, see e.g. here http://forum.bdp.betfair.com/showthread.php?t=2798. Also, the delays are much higher than the mentioned 15ms, so there must be something else. As you write, the delays may appear for listMarketBook, placeOrders, cancelOrders etc. I have tried to change everything I can think of on my side, with no success.

[betdynamics]

@bigbooting

Have you used something like Fiddler to double-check that your requests are NOT being sent with the "Expect:100-Continue" header.

Again, the following information was sent to me recently from the BDP team:

The Betfair API team have been investigating issues with the API where the response time intermittently increases by several orders of magnitude.

Our assessment is that a significant cause of this is requests using the HTTP header “Expect: 100-continue”. This HTTP header is not required for the Betfair Exchange API, but has two (related) negative consequences:

1. Customers submitting requests in this fashion will experience additional latency on their requests, equal to at least the round-trip travel time between the client and the API. Clearly this is bad for the individual customer submitting the request.

2. Communication resources at the API are locked for the duration of the round-trip travel time. Clearly this reduces our resources available to service all clients, which is bad for the entire API user community.

[jptrader]

Thanks for that information betdynamics. Also in my case the "Expect:100-Continue" header is not set. I have also disabled the "Nagle algorithm", if that should matter. Below is information from Fiddler for an example of a slow reply for a listMarketBook. (I have a lot of examples similar to this one) For me it looks like the Betfair server uses 500ms to give a response to the request. This is a request for prices for about 40 markets. Any comments are welcome.

Request headers:

POST /exchange/betting/rest/v1.0/listMarketBook/ HTTP/1.1
Accept: application/json
X-Application: xxxxxxxxxxxxxxx
X-Authentication: xxxxxxxxxxxxxxxxxxxxxx
Content-Type: application/json; charset=utf-8
Host: api.betfair.com
Content-Length: 827
Accept-Encoding: gzip, deflate

Response headers:

HTTP/1.1 200 OK
Date: Wed, 03 Jun 2015 12:06:09 GMT
Server: Cougar - 4.1.3 (Cougar 2 mode)
Cache-Control: no-cache
Content-Type: application/json;charset=utf-8
Content-Encoding: gzip
Vary: Accept-Encoding
Content-Length: 7527

Further info from Fiddler about this request:

SESSION STATE: Done.
Request Entity Size: 827 bytes.
Response Entity Size: 7527 bytes.

== FLAGS ==================
BitFlags: [IsHTTPS, ClientPipeReused, ServerPipeReused] 0x19
X-CLIENTIP: 127.0.0.1
X-CLIENTPORT: 53639
X-EGRESSPORT: 53732
X-HOSTIP: 84.20.200.10
X-PROCESSINFO: betfairapp:6132
X-RESPONSEBODYTRANSFERLENGTH: 7537
X-SERVERSOCKET: REUSE ServerPipe#1191

== TIMING INFO ============
ClientConnected:	13:02:16.453
ClientBeginRequest:	13:06:04.738
GotRequestHeaders:	13:06:04.738
ClientDoneRequest:	13:06:04.739
Determine Gateway:	0ms
DNS Lookup: 		0ms
TCP/IP Connect:	0ms
HTTPS Handshake:	0ms
ServerConnected:	13:02:47.420
FiddlerBeginRequest:	13:06:04.739
ServerGotRequest:	13:06:04.739
ServerBeginResponse:	13:06:05.287
GotResponseHeaders:	13:06:05.287
ServerDoneResponse:	13:06:05.287
ClientBeginResponse:	13:06:05.287
ClientDoneResponse:	13:06:05.287

	Overall Elapsed:	0:00:00.549

The response was buffered before delivery to the client.

== WININET CACHE INFO ============
This URL is not present in the WinINET cache. [Code: 2]
* Note: Data above shows WinINET's current cache state, not the state at the time of the request.
* Note: Data above shows WinINET's Medium Integrity (non-Protected Mode) cache only.

And here is the request:

{"marketIds":["1.118971136","1.118971141","1.118971144","1.118971142","1.118971139","1.118971138","1.118971188","1.118971189","1.118971190","1.118971191","1.118971147","1.118971184","1.118971148","1.118971185","1.118971186","1.118971187","1.118971239","1.118971192","1.118971198","1.118971202","1.118971199","1.118971242","1.118971203","1.118971241","1.118971240","1.118971245","1.118971204","1.118971243","1.118971200","1.118971244","1.118971197","1.118971246","1.118971247","1.118971194","1.118971195"],"priceProjection":{"priceData":["EX_BEST_OFFERS"],"exBestOffersOverrides":{"bestPricesDepth":3,"rollupModel":"NONE","rollupLimit":30,"rollupLiabilityThreshold":0.0,"rollupLiabilityFactor":0},"virtualise":false,"rolloverStakes":false},"orderProjection":"ALL","matchProjection":"NO_ROLLUP","locale":null,"currencyCode":null}

[betdynamics]

What is the delay time if you request just a single market?

[jptrader]

Both for calling single market and calling multiple markets the delays are small for more than 95% of the calls. But both for single market and multiple markets I get these delays for a few percent of calls.

I just made a simple test application calling listMarketBook for the same single market every 200ms. For about 97% of the requests the delays were less than 30ms. For about 2-3% of the requests the delays were between 500 and 600 ms. See the attached jpg for a Fiddler screenshot.

[betdynamics]

This may just be down to the load on the Betfair servers at any given time.

Are the poor responses at random times, or are they grouped together?

I think you probably need to talk to Betfair directly so their tech guys can take a look at this. I would suggest that you email them at bdp@betfair.com