Non-runners mislabelled in results

[Mr Magoo]

Some horse racing markets are inconsistent wrt non-runners. I've been seeing several runners where listMarketBook() gives a 'removalDate' for a horse (implying it is a non-runner), but the runner status is not 'REMOVED'.

For example, market 1.122541107 (Penn (US) 9th Jan - 00:22 R4 1m Mdn Claim), has the runners 'Awesome Bond' (10336504) and 'First On A Roll' (10205379) both with status 'LOSER', but both have a 'removalDate' set. In this case, both horses should be treated as non-runners.

However, just to confuse matters, at other times I see runners with a removalDate (and again, without status 'REMOVED') where they actually *are* still valid runners. (Sorry, can't find a recent example of this...) So there's no sensible way to handle these runners - you can't be sure to treat them as a confirmed non-runner or not.

Could a Betfair developer clarify what's going on, please? Is this an API bug, a case of mis-settled markets, or am I doing something wrong?

[betdynamics]

I would guess that it would help to rule out that you are doing anything wrong if you could show the actual JSON responses received where the data was incorrect.

I would be minded to output the JSON request/response to a file if you detect an inconsistency (along with the UTC date and time that the call was made/response was received) so that the Betfair development team can then trace the issue.

[Mr Magoo]

Here's the request:

{
   "jsonrpc" : "2.0",
   "id" : "1234",
   "method" : "SportsAPING/v1.0/listMarketBook",
   "params" : {
      "marketIds" : [
         "1.122541107"
      ],
      "priceProjection" : {
         "priceData" : [
            "EX_BEST_OFFERS",
            "SP_AVAILABLE"
         ],
         "rolloverStakes" : 0,
         "virtualise" : 0
      }
   }
}

and here's the response: (The two problematic runners are fortunately the first two listed)

{
   "jsonrpc" : "2.0",
   "result" : [
      {
         "complete" : true,
         "crossMatching" : false,
         "status" : "CLOSED",
         "inplay" : false,
         "betDelay" : 0,
         "bspReconciled" : true,
         "version" : 1209433620,
         "numberOfRunners" : 12,
         "numberOfWinners" : 1,
         "isMarketDataDelayed" : false,
         "runners" : [
            {
               "handicap" : 0,
               "adjustmentFactor" : 10.317,
               "status" : "LOSER",
               "removalDate" : "2016-01-10T01:16:55.000Z",
               "ex" : {
                  "tradedVolume" : [],
                  "availableToLay" : [],
                  "availableToBack" : []
               },
               "sp" : {
                  "actualSP" : "NaN",
                  "backStakeTaken" : [],
                  "layLiabilityTaken" : []
               },
               "selectionId" : 10336504
            },
            {
               "sp" : {
                  "backStakeTaken" : [],
                  "actualSP" : "NaN",
                  "layLiabilityTaken" : []
               },
               "selectionId" : 10205379,
               "status" : "LOSER",
               "adjustmentFactor" : 4.511,
               "handicap" : 0,
               "removalDate" : "2016-01-10T01:16:46.000Z",
               "ex" : {
                  "availableToBack" : [],
                  "availableToLay" : [],
                  "tradedVolume" : []
               }
            },
            {
               "sp" : {
                  "actualSP" : "NaN",
                  "backStakeTaken" : [],
                  "layLiabilityTaken" : []
               },
               "selectionId" : 1377840,
               "adjustmentFactor" : 32.393,
               "status" : "REMOVED",
               "handicap" : 0,
               "removalDate" : "2016-01-10T09:49:56.000Z",
               "ex" : {
                  "availableToBack" : [],
                  "availableToLay" : [],
                  "tradedVolume" : []
               }
            },
            {
               "ex" : {
                  "tradedVolume" : [],
                  "availableToLay" : [],
                  "availableToBack" : []
               },
               "handicap" : 0,
               "adjustmentFactor" : 5.867,
               "status" : "LOSER",
               "selectionId" : 10262882,
               "sp" : {
                  "actualSP" : 16.22,
                  "backStakeTaken" : [],
                  "layLiabilityTaken" : []
               }
            },
            {
               "adjustmentFactor" : 8.765,
               "status" : "LOSER",
               "handicap" : 0,
               "ex" : {
                  "availableToLay" : [],
                  "availableToBack" : [],
                  "tradedVolume" : []
               },
               "sp" : {
                  "actualSP" : 4.45,
                  "backStakeTaken" : [],
                  "layLiabilityTaken" : []
               },
               "selectionId" : 10416017
            },
            {
               "handicap" : 0,
               "adjustmentFactor" : 14.458,
               "status" : "LOSER",
               "ex" : {
                  "availableToLay" : [],
                  "availableToBack" : [],
                  "tradedVolume" : []
               },
               "sp" : {
                  "actualSP" : 5.5,
                  "backStakeTaken" : [],
                  "layLiabilityTaken" : []
               },
               "selectionId" : 10040266
            },
            {
               "sp" : {
                  "backStakeTaken" : [],
                  "actualSP" : 9.79,
                  "layLiabilityTaken" : []
               },
               "selectionId" : 10080777,
               "handicap" : 0,
               "adjustmentFactor" : 5.267,
               "status" : "LOSER",
               "ex" : {
                  "availableToBack" : [],
                  "availableToLay" : [],
                  "tradedVolume" : []
               }
            },
            {
               "adjustmentFactor" : 4.169,
               "status" : "LOSER",
               "handicap" : 0,
               "ex" : {
                  "tradedVolume" : [],
                  "availableToBack" : [],
                  "availableToLay" : []
               },
               "sp" : {
                  "layLiabilityTaken" : [],
                  "backStakeTaken" : [],
                  "actualSP" : 62.94
               },
               "selectionId" : 10007310
            },
            {
               "handicap" : 0,
               "adjustmentFactor" : 4.169,
               "status" : "LOSER",
               "ex" : {
                  "availableToLay" : [],
                  "availableToBack" : [],
                  "tradedVolume" : []
               },
               "sp" : {
                  "layLiabilityTaken" : [],
                  "backStakeTaken" : [],
                  "actualSP" : 18.34
               },
               "selectionId" : 10108504
            },
            {
               "selectionId" : 10146954,
               "sp" : {
                  "layLiabilityTaken" : [],
                  "actualSP" : 3.29,
                  "backStakeTaken" : []
               },
               "ex" : {
                  "tradedVolume" : [],
                  "availableToLay" : [],
                  "availableToBack" : []
               },
               "adjustmentFactor" : 16.563,
               "status" : "LOSER",
               "handicap" : 0
            },
            {
               "sp" : {
                  "layLiabilityTaken" : [],
                  "actualSP" : 28.61,
                  "backStakeTaken" : []
               },
               "selectionId" : 4890141,
               "adjustmentFactor" : 5.267,
               "handicap" : 0,
               "status" : "WINNER",
               "ex" : {
                  "tradedVolume" : [],
                  "availableToLay" : [],
                  "availableToBack" : []
               }
            },
            {
               "adjustmentFactor" : 3.083,
               "handicap" : 0,
               "status" : "LOSER",
               "ex" : {
                  "tradedVolume" : [],
                  "availableToBack" : [],
                  "availableToLay" : []
               },
               "sp" : {
                  "layLiabilityTaken" : [],
                  "backStakeTaken" : [],
                  "actualSP" : 49.11
               },
               "selectionId" : 10393928
            }
         ],
         "totalMatched" : 0,
         "totalAvailable" : 0,
         "numberOfActiveRunners" : 0,
         "runnersVoidable" : false,
         "marketId" : "1.122541107"
      }
   ],
   "id" : "1234"
}

(The JSON response has actually been decoded & then re-encoded in order to pretty-print it and make it readable, however I don't think we're dealing with a JSON parsing error here so it shouldn't matter.)

[betdynamics]

Definitely worth passing this over to bdp@betfair.com and asking for an explanation.

Personally, I am more worried by the scenario of active runners being shown as removed! [I am assuming that your example does not show that]

[Nick JD]

My test environment relies on reconciling the prices of all the runners after a non-runner has been removed.

I can also attest to this scenario - there is inconsistency with all aspects of removed runners in GB racing, but it is rare. Perhaps one in a thousand scratched horses is handled badly.

My assumption is that it's human (inputting) error as there is no pattern to it. It crashes my bot on very rare occasions as I also have no way to handle it (all assumptions would be risky) without ignoring the entire race.

Sometimes, it's as simple as a noticeable time difference between the price dropping to zero, and the removal date.

Could it be that this is just a mucky/human side to racing where the whole issue of scratching horses is difficult - especially when horses can get "unscratched" occasionally?

Can this aspect of the API never be 100% reliable?

[Mr Magoo]

Yeah, I think this most likely is a case of human error when dealing with withdrawn horses. To be fair, it's not an easy issue, as in some countries, sometimes horses are declared non-runners after the race is over.

But it is worrying, because if the API shows mistakes in non-runners for a settled race, doesn't that imply that the market was settled wrongly, and so people's P&L could be wrong?

I was hoping that the NG API would make my life simpler with gathering horse racing results around the world, previously I would search lots of websites to get this data, and the API should be an authoritative source for the results of every race. But it's proven to be equally unreliable (see also my other thread about the API changing the number of winners in settled markets)

Since spotting this issue, I've changed my programs to log when a runner has a 'removalDate' but isn't status 'REMOVED'. For live markets, I've only since seen this in cases where a runner is just about to be withdrawn. So I don't think it's a big risk for getting runners & non-runners mixed up in live races. You can also check for other things, like whether or not there are still odds on offer for the horse.

Oh, and I found an example where a settled market has a runner with a removalDate, status LOSER, and the horse actually did take part in the race: 2.101353912 (Pinj (AUS) 14th Dec - 04:49 R1 1684m Pace M). If you call listMarketBook() on this market now, horse 'Summer Moon' (10618302) has a removalDate of '2015-12-14T01:16:28.000Z'. But the horse did take part (Results here). So, at the moment, there's no way to reliably read the API results.

[Mr Magoo]

Definitely worth passing this over to bdp@betfair.com and asking for an explanation.

Personally, I am more worried by the scenario of active runners being shown as removed! [I am assuming that your example does not show that]

I've sent them an email. In live races, I've only seen active runners get a removalDate very shortly before being marked as non-runners, so I think we are safe in that scenario. Perhaps the removalDate is kept around if a horse is wrongly declared a NR and then gets re-added to the market again?

[Mr Magoo]

Betfair told me that these mislabelled non-runners were caused by a mistake made when originally settling the market. A non-runner was left in by accident and they had to re-settle it. As a result, the non-runners in the historic data are mixed up.

At least it's a one-off event, rather than some underlying problem with the API.