More than 1000 orders

[Neophyte]

I have one market where I have more than 1000 orders so I need to ListCurrentOrders more than once to find my position. I have some code that works but after an hour or so of doing this once every two seconds it always crashes with

SyntaxError: Unexpected token { // the token varies each time.

where I am trying to parse the resultant string. I am wondering if I even have the right approach to reading more than 1000 orders where Betfair’s limit is 1000 orders at a time so I am going to describe my method and ask for suggestions.

I am going to listCurrentOrders and if order length = 1000 looping round and doing it again adding the second string from a starting length of 1000 to the string generated first time round.

I am using Javascript and node with my code based on the sample code provided. This reads the current orders in chunks which it adds to a string called str. When there is no more data it parses the string to identify the fields in it putting the parsed string into a variable called response.

To make my loop work I had to make str globally declared rather than locally declared. Because Javascript is asynchronous this meant it would start the next market before the last market had finished and I ended up with more than one market in the same string. I learned that way why str was declared locally. So I gave my string name for listCurrentOrders only a different name with global declaration but a counter which changed for each market so each market had its own global string variable.

Global declaration
var str = [];

The trouble with looping round and adding a second string to the first string is that it has footer and header data which needs to be stripped out of the middle of the resultant string so that it can be parsed. I do this with the following line of code.

str[counter1] = str[counter1].replace(/\],"moreAvailable":true\},"id"\:1\}\{"jsonrpc"\:"2\. 0","result":\{"currentOrders"\:\[/,",");

This then works for a thousand or so iterations and then crashes at this point in the code:
var response = JSON.parse(str[counter1]);

The full code is not all that much and is set out below. It never crashes with markets with less than 1000 orders. I would like to know if there is a completely different way of doing this that is generally recognised as being better as I have just worked this way out from scratch. If I am going about it the right way can anyone see a problem with my detailed approach or know from experience what the reason for the crashes might be.

function checkOrders(options, marketId, selection,counter1,counter2) {

// processed for each market in turn
var recordCount = 1000;
console.log("MarketId = "+marketId);
orderlength = (orderlength === recordCount ? recordCount : 0);
str[counter1] = (orderlength === recordCount ? str[counter1] :'');
var requestFilters = '{"marketIds":["'+ marketId+'"],"fromRecord":"'+orderlength+'","recordCount":"'+r ecordCount+'"}';
var jsonRequest = constructJsonRpcRequest('listCurrentOrders', requestFilters );
var req = https.request(options,function (res){
res.setEncoding(DEFAULT_ENCODING);
res.on('data', function (chunk) {
str[counter1] += chunk;
});
res.on('end', function (chunk) {

// delete joiner parts of string here
str[counter1] = str[counter1].replace(/\],"moreAvailable":true\},"id"\:1\}\{"jsonrpc"\:"2\. 0","result":\{"currentOrders"\:\[/,",");

// program crashes here
var response = JSON.parse(str[counter1]);
handleError(response);
console.log("order response length = "+Object.keys(response.result.currentOrders).lengt h);
orderlength = Object.keys(response.result.currentOrders).length;


// start again if orderlength = 1000 or multiple thereof
if (orderlength % recordCount === 0) {

checkOrders(options, marketId, selection,counter1,counter2);
}

There are 1050 orders in the market so when there is a remainder from dividing orderlength by 1000 it just gets on with the rest of the program.

[jabe]

1050 orders on one market, you said; is that single marketId? Seems a huge number. No matter; it's your design, though you did mention being uncertain.

I should mention that as far as Javascript goes, I've only ever dabbled, but as a rule, when hoping to solve a problem like this, I always want to see an example of the data.

[mmenzies]

I am not sure why you're passing JSON as a string. Wouldn't it make more sense to use json.parse() to handle the JSON and then loop through the "results" array and append to an existing array. You can then handle them as needed as a normal array vs processing a bulk string?

[betdynamics]

I assume that the code never works for a market with more than 1000 orders (rather than it working for a while and then crashing).

I suspect that this may be down to your string manipulation, i.e.

str[counter1] = str[counter1].replace(/\],"moreAvailable":true\},"id"\:1\}\{"jsonrpc"\:" 2\. 0","result":\{"currentOrders"\:\[/,",");

Why don't you write out the json string to the console if you are processing a market with more than 1000 orders, so you can take a look at the data that your application is attempting to process.

Like MMenzies, I think you would be better off JSON parsing each return message rather than trying to concatenate them into a single big message.

[Neophyte]

Thanks Guys.

It is a long running political market where my program was faulty at one stage hence too many orders losing money mostly slowly and yes there really are 1050 orders.

The program really does seem to work and produce the right results 99.99% of the time. I think the problem may actually be the fact that Javascript runs asynchronously, ie concurrently, and does not wait for one process to start before the next completes. Right now it has been running for over 3 hours restarting every 3 seconds.

My latest console.log suggests that the results of the previous iteration are mixed in with the results of the final iteration and where I should get order response length = 1050 I am getting order response length = 50. I am only running this against 3 fixed markets with no fresh bets until I solve the problem so I do know exactly what the results should be.

The suggestion that I loop round after parsing is exactly what I am looking for and I will try that. I think it means I will have to accumulate my position and any other data I need from the orders before moving to the next loop.

I may still run into concurrency issues. I am using setTimeout to delay 3 seconds before starting the whole program again and normally that is enough but possibly occasional delays at reading some data explain why it works most but not all of the time. I may need another Thread on issues of how setTimeout works.

If I console.log the string it takes up all the space Windows allocates to console.log making it difficult to be sure exactly where in the process it crashes and beyond showing me what I needed to delete where the strings joined was not giving me useful information. I will try it again reducing the chunks of orders to 25 to see if that is more informative.