Stream API-

[newbie99]

I've managed to get the stream API up and running with node.js.

I am receiving messages, as expected and with socket.io am able to send the data from the node.js server to a javascript client (html page).

So far so good....

However, I'm a bit stuck with regards to the formatting, if I leave the messages completely formatted they just look like this (which makes no sense to me at all as on the node.js side they look fine):

1. ArrayBuffer(82) {}
   1. [[Int8Array]]: Int8Array(82) [123, 34, 111, 112, 34, 58, 34, 109, 99, 109, 34, 44, 34, 105, 100, 34, 58, 50, 44, 34, 99, 108, 107, 34, 58, 34, 65, 80, 79, 84, 70, 65, 67, 70, 49, 81, 52, 65, 108, 76, 65, 78, 34, 44, 34, 112, 116, 34, 58, 49, 53, 52, 56, 52, 49, 54, 50, 56, 49, 57, 48, 57, 44, 34, 99, 116, 34, 58, 34, 72, 69, 65, 82, 84, 66, 69, 65, 84, 34, 125, 13, 10]
   2. [[Int16Array]]: Int16Array(41) [8827, 28783, 14882, 27938, 28003, 11298, 26914, 8804, 12858, 8748, 27747, 8811, 8762, 20545, 21583, 16710, 17987, 20785, 16692, 19564, 20033, 11298, 28706, 8820, 12602, 13365, 13368, 13873, 14386, 14641, 14640, 8748, 29795, 14882, 18466, 16709, 21586, 17730, 21569, 32034, 2573]
   3. [[Uint8Array]]: Uint8Array(82) [123, 34, 111, 112, 34, 58, 34, 109, 99, 109, 34, 44, 34, 105, 100, 34, 58, 50, 44, 34, 99, 108, 107, 34, 58, 34, 65, 80, 79, 84, 70, 65, 67, 70, 49, 81, 52, 65, 108, 76, 65, 78, 34, 44, 34, 112, 116, 34, 58, 49, 53, 52, 56, 52, 49, 54, 50, 56, 49, 57, 48, 57, 44, 34, 99, 116, 34, 58, 34, 72, 69, 65, 82, 84, 66, 69, 65, 84, 34, 125, 13, 10]
   4. byteLength: (...)
   5. __proto__: ArrayBuffer
      1. byteLength: (...)
      2. constructor: ƒ ArrayBuffer()
      3. slice: ƒ slice()
      4. Symbol(Symbol.toStringTag): "ArrayBuffer"
      5. get byteLength: ƒ byteLength()
      6. __proto__: Object

The above is take from the console.debug of my html page.

Google suggested I needed to do the below:

var enc = new TextDecoder("utf-8");
var encoded = enc.decode(data);
console.debug(encoded);

Which worked to some extent as the messages now look like this in console.debug:

{"op":"mcm","id":2,"clk":"AMWZCQCNngcA9KAH","pt":1 548415126818,"ct":"HEARTBEAT"}

{"op":"mcm","id":2,"clk":"AMCCCwD1mggA9IcI","pt":1 548415131580,"mc":[{"id":"1.152390692","rc":[{"atb":[[1.1,30283.09],[1.01,72396.26],[1.16,12349.93],[1.12,17337]],"atl":[[1.17,20804.36]],"trd":[[1.16,265474.31]],"batb":[[6,1.1,30283.09],[0,1.16,12349.93],[4,1.12,17337]],"batl":[[0,1.17,20804.36]],"bdatb":[[6,1.1,30289.89],[0,1.16,18689.01],[4,1.12,17375.13]],"bdatl":[[0,1.17,21165.21]],"tv":1.010452096E7,"id":235},{"atb":[[30,6.96]],"id":60443},{"bdatb":[[1,7.2,4077.44],[3,6.8,3518.58],[4,6.6,217.78]],"bdatl":[[4,8.4,387.62],[6,8.8,56.16]],"id":10301}],"con":true,"tv":2.241855563E7}]}

However, as I receive multiple types of messages (i.e. the 2 above, plus a 3rd type once I include orders), I need to be able to write an if statement (or case/switch to determine what actions to take), easy in theory...but now comes the problem.

In order to look at the field values and write a simple if statement (i.e. something along the lines of if exists ct then .... else ....) I tried to use the following to ensure the format is correct as a JSON:

var enc = new TextDecoder("utf-8");
var encoded = JSON.parse(enc.decode(data));
console.debug(encoded);

This results in the Heartbeat message working:

{"op":"mcm","id":2,"clk":"AMWZCQCNngcA9KAH","pt":1 548415126818,"ct":"HEARTBEAT"}

The console formats this as expected, however for the pricing message I just get a load of errors, as it says there are unexpected characters.

Now when I copy paste the above pricing message into a JSON formatter it is able to read it perfectly with no errors.

So, what am I missing here, what can I not decipher the second message (and I presume I'd have problems with orders too as well as prices)?

I suspect its something really obvious, but people on here, hopefully have encountered similar as I'm guessing anyone using the streaming API would need to perform a similar action?

Any help would be greatly appreciated!

[newbie99]

I've managed to get the stream API up and running with node.js.

I am receiving messages, as expected and with socket.io am able to send the data from the node.js server to a javascript client (html page).

So far so good....

However, I'm a bit stuck with regards to the formatting, if I leave the messages completely formatted they just look like this (which makes no sense to me at all as on the node.js side they look fine):

1. ArrayBuffer(82) {}
   1. [[Int8Array]]: Int8Array(82) [123, 34, 111, 112, 34, 58, 34, 109, 99, 109, 34, 44, 34, 105, 100, 34, 58, 50, 44, 34, 99, 108, 107, 34, 58, 34, 65, 80, 79, 84, 70, 65, 67, 70, 49, 81, 52, 65, 108, 76, 65, 78, 34, 44, 34, 112, 116, 34, 58, 49, 53, 52, 56, 52, 49, 54, 50, 56, 49, 57, 48, 57, 44, 34, 99, 116, 34, 58, 34, 72, 69, 65, 82, 84, 66, 69, 65, 84, 34, 125, 13, 10]
   2. [[Int16Array]]: Int16Array(41) [8827, 28783, 14882, 27938, 28003, 11298, 26914, 8804, 12858, 8748, 27747, 8811, 8762, 20545, 21583, 16710, 17987, 20785, 16692, 19564, 20033, 11298, 28706, 8820, 12602, 13365, 13368, 13873, 14386, 14641, 14640, 8748, 29795, 14882, 18466, 16709, 21586, 17730, 21569, 32034, 2573]
   3. [[Uint8Array]]: Uint8Array(82) [123, 34, 111, 112, 34, 58, 34, 109, 99, 109, 34, 44, 34, 105, 100, 34, 58, 50, 44, 34, 99, 108, 107, 34, 58, 34, 65, 80, 79, 84, 70, 65, 67, 70, 49, 81, 52, 65, 108, 76, 65, 78, 34, 44, 34, 112, 116, 34, 58, 49, 53, 52, 56, 52, 49, 54, 50, 56, 49, 57, 48, 57, 44, 34, 99, 116, 34, 58, 34, 72, 69, 65, 82, 84, 66, 69, 65, 84, 34, 125, 13, 10]
   4. byteLength: (...)
   5. __proto__: ArrayBuffer
      1. byteLength: (...)
      2. constructor: ƒ ArrayBuffer()
      3. slice: ƒ slice()
      4. Symbol(Symbol.toStringTag): "ArrayBuffer"
      5. get byteLength: ƒ byteLength()
      6. __proto__: Object

The above is take from the console.debug of my html page.

Google suggested I needed to do the below:

var enc = new TextDecoder("utf-8");
var encoded = enc.decode(data);
console.debug(encoded);

Which worked to some extent as the messages now look like this in console.debug:

{"op":"mcm","id":2,"clk":"AMWZCQCNngcA9KAH","pt":1 548415126818,"ct":"HEARTBEAT"}

{"op":"mcm","id":2,"clk":"AMCCCwD1mggA9IcI","pt":1 548415131580,"mc":[{"id":"1.152390692","rc":[{"atb":[[1.1,30283.09],[1.01,72396.26],[1.16,12349.93],[1.12,17337]],"atl":[[1.17,20804.36]],"trd":[[1.16,265474.31]],"batb":[[6,1.1,30283.09],[0,1.16,12349.93],[4,1.12,17337]],"batl":[[0,1.17,20804.36]],"bdatb":[[6,1.1,30289.89],[0,1.16,18689.01],[4,1.12,17375.13]],"bdatl":[[0,1.17,21165.21]],"tv":1.010452096E7,"id":235},{"atb":[[30,6.96]],"id":60443},{"bdatb":[[1,7.2,4077.44],[3,6.8,3518.58],[4,6.6,217.78]],"bdatl":[[4,8.4,387.62],[6,8.8,56.16]],"id":10301}],"con":true,"tv":2.241855563E7}]}

However, as I receive multiple types of messages (i.e. the 2 above, plus a 3rd type once I include orders), I need to be able to write an if statement (or case/switch to determine what actions to take), easy in theory...but now comes the problem.

In order to look at the field values and write a simple if statement (i.e. something along the lines of if exists ct then .... else ....) I tried to use the following to ensure the format is correct as a JSON:

var enc = new TextDecoder("utf-8");
var encoded = JSON.parse(enc.decode(data));
console.debug(encoded);

This results in the Heartbeat message working:

{"op":"mcm","id":2,"clk":"AMWZCQCNngcA9KAH","pt":1 548415126818,"ct":"HEARTBEAT"}

The console formats this as expected, however for the pricing message I just get a load of errors, as it says there are unexpected characters.

Now when I copy paste the above pricing message into a JSON formatter it is able to read it perfectly with no errors.

So, what am I missing here, what can I not decipher the second message (and I presume I'd have problems with orders too as well as prices)?

I suspect its something really obvious, but people on here, hopefully have encountered similar as I'm guessing anyone using the streaming API would need to perform a similar action?

Any help would be greatly appreciated!

Am I missing the obvious here, are these messages not actually intended to be sent in JSON format and what I should be doing is treating this as a standard array (and if so, as its quite difficult to read the data, does anyone have a guide on how to format it so its readable)?

[jabe]

I haven't used the streaming API, but with the regular API the JSON strings can be converted to and from objects. This is from memory somewhat, and I've been using Visual Basic.NET, and I haven't done anything related to objects in Javascript.

A JSON string can be read into an object using JSON's own Serialize/Deserialize commands, but there is also a third party option available in .NET languages (and maybe others) from Newtonsoft. The command for dealing with received messages looks like this:

Dim evtType = Newtonsoft.Json.JsonConvert.DeserializeObject(Of ClassEventTypeResult)(API_sendBetReq("listEventTyp es", """filter"":{ }"))

In this case, it converts the received data from a call to my API_sendBetReq routine for a listEventTypes call with no parameters and converts it to an object called evtType which is defined as a ClassEventTypeResult object, as defined in the documentation.

If there is an unpopulated or unmentioned piece of data within a received JSON string, the object it is read into will treat is as a null or will disregard it.

Creating JSON strings for data requests is done differently.

[newbie99]

I haven't used the streaming API, but with the regular API the JSON strings can be converted to and from objects. This is from memory somewhat, and I've been using Visual Basic.NET, and I haven't done anything related to objects in Javascript.

A JSON string can be read into an object using JSON's own Serialize/Deserialize commands, but there is also a third party option available in .NET languages (and maybe others) from Newtonsoft. The command for dealing with received messages looks like this:

Dim evtType = Newtonsoft.Json.JsonConvert.DeserializeObject(Of ClassEventTypeResult)(API_sendBetReq("listEventTyp es", """filter"":{ }"))

In this case, it converts the received data from a call to my API_sendBetReq routine for a listEventTypes call with no parameters and converts it to an object called evtType which is defined as a ClassEventTypeResult object, as defined in the documentation.

If there is an unpopulated or unmentioned piece of data within a received JSON string, the object it is read into will treat is as a null or will disregard it.

Creating JSON strings for data requests is done differently.

So, still plugging away with this and I'm sort of closer to understanding things.

Essentially the messages are sent from Betfair to the listening Node.js server in pieces, so for example a single JSON could be split up into multiple parts.

The problem is this isn't always the case and they are cut off at different points!

I thought maybe I'd cracked it and then I got a couple through in the following order:

JSON message A (part 1)

JSON message B (complete)

JSON message A (part 2)

The problem with the above, was that part 2, was simply the last couple of characters of message A, something along the lines of: ,34} so it was impossible to match it with an ID.

The other issue, is that as each message is incomplete (but not always), you can't simply ignore incomplete messages.

All in all the Streaming API is very confusing and I'm wondering if I'm going down the right path.

I've never used Python before (other than a quick beginners tutorial), but could that be better than Node.js, or would the same issues arise?

I'm trying to re-join the broken JSON messages on the client side in Javascript, rather than on the Node.js server side. Is that a sensible approach?

I'm aware of Betfair Lightweight for Python, which I have started looking at, but I do want to build a web gui so would presumably still have the same issue with the broken messages, unless Python somehow reads them differently to Node.js.

All in all, I'm pretty stuck!

[jabe]

I'm assuming you got hold of the sample code from GitHub? I didn't, so I can't help much further.

[LiamP]

The messages are sent through the stream in order, not sure how much you know about sockets but the only way a message would be split into parts is because whatever you are using to read from socket isn’t waiting for the CRLF.

edit: Ah I can see you are using socket.io, you can only use this for socket.io servers, Betfair Streaming is not a socket.io server hence your problems.

[newbie99]

The messages are sent through the stream in order, not sure how much you know about sockets but the only way a message would be split into parts is because whatever you are using to read from socket isn’t waiting for the CRLF.

edit: Ah I can see you are using socket.io, you can only use this for socket.io servers, Betfair Streaming is not a socket.io server hence your problems.

Ah, that could be the problem then. Any advice on what I should / could be using instead (sockets in general are new to me)?

[newbie99]

I'm assuming you got hold of the sample code from GitHub? I didn't, so I can't help much further.

Yeah, thats helped me get this far, but I'm conscious that unless I understand where things are going wrong here, then I could spend a lot of time going down the wrong route.

[LiamP]

Had a quick look on github and it doesn’t look like anyone has attempted it yet using node. I wrote the streaming code for betfairlightweight a few years ago and it took me a few weeks to write and a few months before all the bugs were ironed out / optimised.

You have the complication of working with the socket and building a cache which is kept up to date, if you enjoy a challenge and want to learn about a different aspect of CS go for it! However note that the website doesn’t use the streaming API which probably highlights how tricky it is to use with JS, not sure why Betfair didn’t use socket.io or a normal websocket for the transfer protocol tbh.

[newbie99]

Had a quick look on github and it doesn’t look like anyone has attempted it yet using node. I wrote the streaming code for betfairlightweight a few years ago and it took me a few weeks to write and a few months before all the bugs were ironed out / optimised.

You have the complication of working with the socket and building a cache which is kept up to date, if you enjoy a challenge and want to learn about a different aspect of CS go for it! However note that the website doesn’t use the streaming API which probably highlights how tricky it is to use with JS, not sure why Betfair didn’t use socket.io or a normal websocket for the transfer protocol tbh.

Yep, I just got the connection code for node.js here and the rest I've tried to come up with myself (with mixed results):

https://github.com/betfair/stream-ap...node.js/app.js

So....with that in mind, I'm not averse to switching to your Python code, it will definitely save me a lot of time, although I really wanted to connect to a web browser, which I assumed I'd do with websockets or sockets.io. Using betfairlightweight, would that approach work, or is it the whole idea of a GUI just daydreaming?

I may pursue the node.js project too, but it can drop down the priority list, if I'm able to use the Python code to get closer to what I was hoping more quickly!

Oh...one final question...where is this mystery certs folder??? I have created the cert (and uploaded to Betfair etc.), but I can't find where the default folder is, it doesn't seem to exist?

[newbie99]

Yep, I just got the connection code for node.js here and the rest I've tried to come up with myself (with mixed results):

https://github.com/betfair/stream-ap...node.js/app.js

So....with that in mind, I'm not averse to switching to your Python code, it will definitely save me a lot of time, although I really wanted to connect to a web browser, which I assumed I'd do with websockets or sockets.io. Using betfairlightweight, would that approach work, or is it the whole idea of a GUI just daydreaming?

I may pursue the node.js project too, but it can drop down the priority list, if I'm able to use the Python code to get closer to what I was hoping more quickly!

Oh...one final question...where is this mystery certs folder??? I have created the cert (and uploaded to Betfair etc.), but I can't find where the default folder is, it doesn't seem to exist?

I could never find the certs folder (I presume its meant to be in some kind of root directory for Python or the project, but I couldn't work that out), however I changed the path in both the baseclient.py and exceptions.py to a valid local path where my certificate is stored. However I keep getting the following error:

Certificate folder not found in D:/Python37/Lib/site-packages/betfairlightweight/certs

Which is where the .crt file is located (I assume for Python / is necessary instead of \ as with the latter I just got a load of errors).

So...worrying about the correct way to connect from Python to a web-browser aside...I still can't actually get the login to work :-(

[LiamP]

It is looking for the crt and key file, you got both in there? You shouldn’t update the underlying library as you can pass the location into the APIClient wither certs= and the slashes will depend on your OS.

[newbie99]

It is looking for the crt and key file, you got both in there? You shouldn’t update the underlying library as you can pass the location into the APIClient wither certs= and the slashes will depend on your OS.

I created the key / cert using XCA, so its a .crt and a .pem file, both are there, but don't seem to be picked up (I downloaded a clean instance of the wrapper, just in case my messing around changed anything).

[newbie99]

I created the key / cert using XCA, so its a .crt and a .pem file, both are there, but don't seem to be picked up (I downloaded a clean instance of the wrapper, just in case my messing around changed anything).

Ok, I have made some progress, but perhaps I'm not understanding the syntax for your login.

Using the sample code on the Betfair website, although for some reason I can't get the curl command to work from command prompt (I get an HTTP Error Code 400), however I have got the sample Python login working, by using:

import requests

payload = 'username=USERNAME&password=PASSWORD'
headers = {'X-Application': 'APPKEY', 'Content-Type': 'application/x-www-form-urlencoded'}

resp = requests.post('https://identitysso-cert.betfair.com/api/certlogin', data=payload, cert=('FILEPATH.crt', 'FILEPATH.pem'), headers=headers)

if resp.status_code == 200:
  resp_json = resp.json()
  print(resp_json['loginStatus'])
  print(resp_json['sessionToken'])
else:
  print("Request failed.")

However, I still can't work out how that translates to the Betfairlightweight login, as when I try to login, forcing an over-ride of certs, as follows:

import betfairlightweight
certs=('FILEPATH.crt', 'FILEPATH.pem')
trading = betfairlightweight.APIClient('USERNAME','PASSWORD','APPKEY')
trading.login()

I just get the same FileNotFoundError: [WinError 3] The system cannot find the path specified '/certs/' which suggests that my syntax is wrong, as its still looking in '/certs/' rather than the correct folder (I still have no idea where this mystery folder is meant to be)!

[LiamP]

Let’s say my certs where both in this dir:

certs_dir = r’”C:/windows/sucks/python/certs’”””’

trading = APIClient(‘username’, ‘password’, ‘appKey’, certs=certs_dir)

edit: its stripping out the quotes, see slack