Best way to get historical data in database?

[JohnFromCairnsAustralia]

Hello there,

I am researching how I can best access historical data. My understanding is that upon download, the format of data is in json files. (the same as that of the stream api service)

I currently use c# and the API-NG sample code for my current trading activities. I do not currently use the stream service. (time constraints and the older code works well enough for me in the short term)

Would you guys suggest the best avenue would be to move on from API-NG sample code and implement the streaming service?

Is anyone familiar with code available open or commercial that can help with this process? or alternately, whats the quickest simplest way to get historical data into a database?

Thank you kindly for any ideas
John

[bfexplorer]

I experimented with historical data, and as they are saved in json streaming format, and my app implements streaming as well, I created this app for my personal use:

http://bfexplorer.net/Articles/Content/302

It allows me to use historical data, and test/analyze my bots on such data. I would suggest you to go the similar way. Parsing json data and transform them to your app domain, for analyzing or saving to database.

[bfexplorer]

And here it is just how your code could look like:

https://youtu.be/DEOoLElQ-1k

In my case I built service BetfairHistoricData.Load allowing me to load betfair historical json file, parse it and transform data to my app model used by my app.

Now in your case, after loading data, the data are grouped by market, you can save market data to database.

In my sample code, the app filter just WIN markets, and then shows all traded prices for winners only.

[JohnFromCairnsAustralia]

Thank you for confirming my understanding bfexplorer...seems I will have to implement the stream api...

As an aside, I see you mostly write your stuff in f#? I have seen around here people who have implemented the stream api on top of the aping sample code, I already started reinventing the wheel in a few ways, gotta stop doing that...im a c# guy so like to go that way ...but your stuff looks very interesting....like your bfexplorer ladder interface too....will check it all out..

Thanks again
John

[Alesia]

Fastest way is to download the .bz2 data from the historical data section, write a small script to handle the bz2 to json to sql. I mocked up a quick version in ~20 mins. Feel free to shoot me a PM and I could help you put a script together depending on your programming experience.

[bfexplorer]

John, F# is .net programming language, so you can consume F# assemblies/code from C# and vice versa:

https://github.com/StefanBelo/Bfexpl...ole/Program.cs

[JohnFromCairnsAustralia]

Fastest way is to download the .bz2 data from the historical data section, write a small script to handle the bz2 to json to sql. I mocked up a quick version in ~20 mins. Feel free to shoot me a PM and I could help you put a script together depending on your programming experience.

Thank you kindly Alesia, that would still only get me price delta's? correct? I still have to convert them to price snapshots, once they are viewed from the database? or your script does that as well?

All my current code works off price snapshots because I have been working with the older style GetMarketBook api... I think I am going to have to rewrite my flow to convert deltas to snapshots...