listMarketCatalogue drama

[OzPunter]

Hi guys,
This problem is driving me nuts. Its taken ages to nail down the problem so if anyone can tell me what is wrong with this request I'd very much appreciate it


This request Fails:

Param{"filter":{"marketBettingTypes":["ODDS"],"marketTypeCodes":["CORRECT_SCORE"],
"marketStartTime":{"from":"2015-01-04T22:31:00.000Z","to":"2015-01-05T02:31:00.000Z"},
"eventTypeIds":["1"]},"maxResults":1000,"marketProjection":["COMPETITION","EVENT","MARKET_START_TIME","RUNNER_ DESCRIPTION"],
"sort":"FIRST_TO_START"},"id":"1","jsonrpc":"2 .0"}

Error=HTTP/1.1 400 Bad Request



and this request succeeds

Param{"filter":{"marketBettingTypes":["ODDS"],"marketTypeCodes":["CORRECT_SCORE"],
"marketStartTime":{"from":"2015-01-04T23:07:00.000Z","to":"2015-01-05T03:07:00.000Z"},
"eventTypeIds":["1"]},"maxResults":1000,"marketProjection":["COMPETITION","EVENT","MARKET_START_TIME","RUNNER_ DESCRIPTION"],
"sort":"FIRST_TO_START"},"id":"1","jsonrpc":"2 .0"}

What exactly is wrong with the first request?

These requests are sent every five minutes,
they are both sent to ListMarketCatalogue for Soccer matches by the same program.

I have compared each line, both look identical to me (except for the times).

Anybody??

Kind Regards
OzPunter

[Franklin1]

they are identical apart from dates.

Do you get some more information about the error other than the HTTP status code? From the response you should be able to get a Betfair error code from the body.

Have you tried using Firefox-HTTP Resource Test to build and send the HTTP and see received HTTP? Configure the headers and put the body in and see what it says (betfair error codes are DSC-xxx)

[OzPunter]

they are identical apart from dates.

Do you get some more information about the error other than the HTTP status code? From the response you should be able to get a Betfair error code from the body.

Have you tried using Firefox-HTTP Resource Test to build and send the HTTP and see received HTTP? Configure the headers and put the body in and see what it says (betfair error codes are DSC-xxx)

Thanks Franklin,

I was unaware of the existance of the BetFair Error Code, I'll go hunting for it.. Usually if there's an error I get nothing back except "Bad Request" which is not very helpful.

Not sure if I can if I can use the firefox HTTP method as these requests are made every 5 minutes and the majority pass. I'd have to catch a faulty one on the fly which is a bit difficult to do.

Thanks for your suggestion

Kind Regards
OzPunter

[betdynamics]

If you get the HTTP/1.1 400 Bad Request message, why not just try resubmitting the listMarketCatalogue request again.

If the second attempt yields the same result, then it suggests something is amiss with the request.

If the second attempt works, then it suggests a temporary network/web issue.

[Franklin1]

Not sure if I can if I can use the firefox HTTP method as these requests are made every 5 minutes and the majority pass. I'd have to catch a faulty one on the fly which is a bit difficult to do.

Think you misunderstood. I was suggesting the 'HTTP resource test' method to manually build and send the erroneous request and to view the response. I could actually send the Json you provided using this method. This would definitely make the betfair error code obvious as you see it in the response. Obviously there are other ways to debug, but I found the Firefox add on to be an excellent step between the API visualiser and my code.

[OzPunter]

Think you misunderstood. I was suggesting the 'HTTP resource test' method to manually build and send the erroneous request and to view the response. I could actually send the Json you provided using this method. This would definitely make the betfair error code obvious as you see it in the response. Obviously there are other ways to debug, but I found the Firefox add on to be an excellent step between the API visualiser and my code.

Dear Franklin,
The problem is still occurring and I still haven't found a solution.

The problem I see with your HTTP solution is that the times in these requests change every five minutes, most requests pass. If I capture a bad request It will be out of date before I have a chance to set up the manual test.

The program runs 24hrs a day so the problem could happen during the night. The only common thing I can see with the failed requests is that the “From” and “To” times extend through midnight GMT. I'll change the program to capture every request and see if the problem occurs at any other times.

The program is running on a VPS so I'm not able to watch it 24/7

I thnak you for your help,
Kind Regards
OzPunter

[Franklin1]

The problem I see with your HTTP solution is that the times in these requests change every five minutes, most requests pass. If I capture a bad request It will be out of date before I have a chance to set up the manual test.

I see - that's a fair point.
I suspect if you can just save the whole response it will be obvious from the Betfair error code.

I do something like this, vb.net, not Delphi but might help:
Catch e As System.Net.WebException
If e.Response IsNot Nothing Then
Dim reader As New StreamReader(e.Response.GetResponseStream())
Dim body = reader.ReadToEnd()
apiResponse.JSONError = JsonConvert.DeserializeObject(Of JSONError)(body)
e.Response.Close()


Please do post your findings!

[OzPunter]

I see - that's a fair point.
I suspect if you can just save the whole response it will be obvious from the Betfair error code.

I do something like this, vb.net, not Delphi but might help:
Catch e As System.Net.WebException
If e.Response IsNot Nothing Then
Dim reader As New StreamReader(e.Response.GetResponseStream())
Dim body = reader.ReadToEnd()
apiResponse.JSONError = JsonConvert.DeserializeObject(Of JSONError)(body)
e.Response.Close()


Please do post your findings!

Dear Franklin,

I delved in a little deeper into the Delphi version of exception handling and I didn't find an "e.responce" but the closest I can see would be E.GetBaseException.ToString so I'm going to record that (and a few other variables) and see what it reveals.

I really can't see why this request fails though, it looks Ok to me..

Thanks for your help

Kind Regards
OzPunter

[OzPunter]

Dear Franklin,

I delved in a little deeper into the Delphi version of exception handling and I didn't find an "e.responce" but the closest I can see would be E.GetBaseException.ToString so I'm going to record that (and a few other variables) and see what it reveals.

I really can't see why this request fails though, it looks Ok to me..

Thanks for your help

Kind Regards
OzPunter

The error message that accompanies the Bad Request is

TD:09012015 22:54:00 Error= HTTP/1.1 400 Bad Request

Exception class name = EIdHTTPProtocolException

Exception message = HTTP/1.1 400 Bad Request

The TD is the time & date GMT, but I don't understand the HTTP Protocol exception class. Why is this request raising a HTTP Protocol exception and the others dont?

Thoughts?

Kind Regards
OzPunter

[OzPunter]

Actually, they "Bad Request" error may not be coming from the BetFair servers, more likely it's created by a VPS HTTP protocol failure.

Because only one of these requests out of hundreds causes this problem I'd be better off finding a way to step over the failure and not have it case my program to fail.

New train of thought...

Regards
OzPunter

[Franklin1]

Actually, they "Bad Request" error may not be coming from the BetFair servers, more likely it's created by a VPS HTTP protocol failure.

400="The request could not be understood by the server due to malformed syntax"
http://stackoverflow.com/questions/1...r-code-meaning

Can you set your code up to write all the serialized HTTP requests and responses to disk so we can diagnose? Perhaps using Fiddler...

I hate giving up debugging and coding to be robust to weirdness instead, but it usually saves a lot of time!

[OzPunter]

Thank you Franklin,

from the link in your post I found this statement

"A 400 means that the request was malformed. In other words, the data stream sent by the client to the server didn't follow the rules.

In the case of a REST API with a JSON payload, 400's are typically, and correctly I would say, used to indicate that the JSON is invalid in some way according to the API specification for the service."

which suggests that the JSON is malformed in some way.. Remember that about 90% of these requests pass with no problems, so what’s wrong with the one that failed???

If I can find a way to step over the failed request and just carry on I don't think it'll be critical but like yourself I'd love to find out why.. These things have a way of coming back to haunt you further down the track..

I'll did a little deeper into the HTTP and see what I can find.

Kind Regards
OzPunter

[OzPunter]

More on this problem

I captured every request and response over a 24 hour period and every request up to the failure worked fine then after the failure every subsequent request fails..

*** These request & response are OK
{"filter":{"marketBettingTypes":["ODDS"],"marketTypeCodes":["CORRECT_SCORE"],"marketStartTime":{"from":"2015-01-12T14:32:00.000Z","to":"2015-01-12T18:32:00.000Z"},"eventTypeIds":["1"]},"maxResults":1000,"marketProjection":["COMPETITION","EVENT","MARKET_START_TIME","RUNNER_ DESCRIPTION"],"sort":"FIRST_TO_START"},"id":"1","jsonrpc":"2.0" }

Response = [{"marketId":"1.116929758","marketName":"Correct Score","marketStartTime":"2015-01-12T17:00:00.000Z","totalMatched":291.46,"runners":[{"selectionId":1,"runnerName":"0 - 0","handicap":0.0,"sortPriority":1},{"selectionId" :4,"runnerName":"0 - 1","handicap":0.0,"sortPriority":2},{"selectionId" :9,"runnerName":"0 - 2","handicap":0.0,"sortPriority":3},{"selectionId" :16,"runnerName":"0 - 3","handicap":0.0,"sortPriority":4},{"selectionId" :2,"runnerName":"1 - 0","handicap":0.0,"sortPriority":5},{"selectionId" :3,"runnerName":"1 - 1","handicap":0.0,"sortPriority":6},{"selectionId" :8,"runnerName":"1 - 2","handicap":0.0,"sortPriority":7},{"selectionId" :15,"runnerName":"1 - 3","handicap":0.0,"sortPriority":8},{"selectionId" :5,"runnerName":"2 - 0","handicap":0.0,"sortPriority":9},{"selectionId" :6,"runnerName":"2 - 1","handicap":0.0,"sortPriority":10},{"selectionId ":7,"runnerName":"2 - 2","handicap":0.0,"sortPriority":11},{"selectionId ":14,"runnerName":"2 - 3","handicap":0.0,"sortPriority":12},{"selectionId ":10,"runnerName":"3 - 0","handicap":0.0,"sortPriority":13},{"selectionId ":11,"runnerName":"3 - 1","handicap":0.0,"sortPriority":14},{"selectionId ":12,"runnerName":"3 - 2","handicap":0.0,"sortPriority":15},{"selectionId ":13,"runnerName":"3 - 3","handicap":0.0,"sortPriority":16},{"selectionId ":9063254,"runnerName":"Any Other Home Win","handicap":0.0,"sortPriority":17},{"selection Id":9063255,"runnerName":"Any Other Away Win","handicap":0.0,"sortPriority":18},{"selection Id":9063256,"runnerName":"Any Other Draw","handicap":0.0,"sortPriority":19}],"competition":{"id":"804044","name":"Israeli Liga Leumit"},"event":{"id":"27336005","name":"Ironi Tiberias v Hapoel Ramat Gan","countryCode":"IL","timezone":"Europe/London","openDate":"2015-01-12T17:00:00.000Z"}}]

{"filter":{"marketBettingTypes":["ODDS"],"marketTypeCodes":["CORRECT_SCORE"],"marketStartTime":{"from":"2015-01-12T14:37:00.000Z","to":"2015-01-12T18:37:00.000Z"},"eventTypeIds":["1"]},"maxResults":1000,"marketProjection":["COMPETITION","EVENT","MARKET_START_TIME","RUNNER_ DESCRIPTION"],"sort":"FIRST_TO_START"},"id":"1","jsonrpc":"2.0" }

Response = [{"marketId":"1.116929758","marketName":"Correct Score","marketStartTime":"2015-01-12T17:00:00.000Z","totalMatched":323.56,"runners":[{"selectionId":1,"runnerName":"0 - 0","handicap":0.0,"sortPriority":1},{"selectionId" :4,"runnerName":"0 - 1","handicap":0.0,"sortPriority":2},{"selectionId" :9,"runnerName":"0 - 2","handicap":0.0,"sortPriority":3},{"selectionId" :16,"runnerName":"0 - 3","handicap":0.0,"sortPriority":4},{"selectionId" :2,"runnerName":"1 - 0","handicap":0.0,"sortPriority":5},{"selectionId" :3,"runnerName":"1 - 1","handicap":0.0,"sortPriority":6},{"selectionId" :8,"runnerName":"1 - 2","handicap":0.0,"sortPriority":7},{"selectionId" :15,"runnerName":"1 - 3","handicap":0.0,"sortPriority":8},{"selectionId" :5,"runnerName":"2 - 0","handicap":0.0,"sortPriority":9},{"selectionId" :6,"runnerName":"2 - 1","handicap":0.0,"sortPriority":10},{"selectionId ":7,"runnerName":"2 - 2","handicap":0.0,"sortPriority":11},{"selectionId ":14,"runnerName":"2 - 3","handicap":0.0,"sortPriority":12},{"selectionId ":10,"runnerName":"3 - 0","handicap":0.0,"sortPriority":13},{"selectionId ":11,"runnerName":"3 - 1","handicap":0.0,"sortPriority":14},{"selectionId ":12,"runnerName":"3 - 2","handicap":0.0,"sortPriority":15},{"selectionId ":13,"runnerName":"3 - 3","handicap":0.0,"sortPriority":16},{"selectionId ":9063254,"runnerName":"Any Other Home Win","handicap":0.0,"sortPriority":17},{"selection Id":9063255,"runnerName":"Any Other Away Win","handicap":0.0,"sortPriority":18},{"selection Id":9063256,"runnerName":"Any Other Draw","handicap":0.0,"sortPriority":19}],"competition":{"id":"804044","name":"Israeli Liga Leumit"},"event":{"id":"27336005","name":"Ironi Tiberias v Hapoel Ramat Gan","countryCode":"IL","timezone":"Europe/London","openDate":"2015-01-12T17:00:00.000Z"}}]

************ This request fails and all requests after this point fail till the program is stopped and restarted.

{"filter":{"marketBettingTypes":["ODDS"],"marketTypeCodes":["CORRECT_SCORE"],"marketStartTime":{"from":"2015-01-12T14:42:00.000Z","to":"2015-01-12T18:42:00.000Z"},"eventTypeIds":["1"]},"maxResults":1000,"marketProjection":["COMPETITION","EVENT","MARKET_START_TIME","RUNNER_ DESCRIPTION"],"sort":"FIRST_TO_START"},"id":"1","jsonrpc":"2.0" }

Error = :HTTP/1.1 400 Bad Request

{"filter":{"marketBettingTypes":["ODDS"],"marketTypeCodes":["CORRECT_SCORE"],"marketStartTime":{"from":"2015-01-12T14:47:00.000Z","to":"2015-01-12T18:47:00.000Z"},"eventTypeIds":["1"]},"maxResults":1000,"marketProjection":["COMPETITION","EVENT","MARKET_START_TIME","RUNNER_ DESCRIPTION"],"sort":"FIRST_TO_START"},"id":"1","jsonrpc":"2.0" }

Error = :HTTP/1.1 400 Bad Request

{"filter":{"marketBettingTypes":["ODDS"],"marketTypeCodes":["CORRECT_SCORE"],"marketStartTime":{"from":"2015-01-12T14:52:00.000Z","to":"2015-01-12T18:52:00.000Z"},"eventTypeIds":["1"]},"maxResults":1000,"marketProjection":["COMPETITION","EVENT","MARKET_START_TIME","RUNNER_ DESCRIPTION"],"sort":"FIRST_TO_START"},"id":"1","jsonrpc":"2.0" }

Error = :HTTP/1.1 400 Bad Request


Except for the times, everything else is identical, so why do these requests fail???

I tried to step over the error, but once the error occurs all further requests fail...

Looking at any of the failed requests what is wrong with them???


Kind Regards
OzPunter

[betdynamics]

Have you considered other possibilities?

For example, are you using the KeepAlive function to keep the session active? If not, Betfair may terminate your session which will result in all calls failing until you log in again and re-establish a session.

However, calls to the API normally return an error code which would indicate an invalid session (I'm not talking about HTTP errors).

[Franklin1]

I captured every request and response over a 24 hour period and every request up to the failure worked fine then after the failure every subsequent request fails..
...
{"filter":{"marketBettingTypes":["ODDS"],"marketTypeCodes":["CORRECT_SCORE"],"marketStartTime":{"from":"2015-01-12T14:42:00.000Z","to":"2015-01-12T18:42:00.000Z"},"eventTypeIds":["1"]},"maxResults":1000,"marketProjection":["COMPETITION","EVENT","MARKET_START_TIME","RUNNER_ DESCRIPTION"],"sort":"FIRST_TO_START"},"id":"1","jsonrpc":"2.0" }

Error = :HTTP/1.1 400 Bad Request

We still need the HTTP response! We know the status (400), but there should be some more information in there. Can you save it to file before your error is thrown?