Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

L2 subsystem(s) #173

Open
5 tasks
goodboy opened this issue May 17, 2021 · 0 comments · May be fixed by #174
Open
5 tasks

L2 subsystem(s) #173

goodboy opened this issue May 17, 2021 · 0 comments · May be fixed by #174
Assignees
Labels
data-layer real-time and historical data processing and storage integration external stack and/or lib augmentations (sub-)systems general sw design and eng tsdb time series db stuff

Comments

@goodboy
Copy link
Contributor

goodboy commented May 17, 2021

Depth of book, otherwise known as L2 data. is something that's been on the list todo for quite a while.
We now not only have the data processing and streaming infra for it but also the UI components to display it efficiently in real-time on charts.

I originally had though about collecting and storing L2 using marketstore tick records as per #93 and #142. When you look at our tick schema it makes you think, ok so we need to store info at each level and I guess this goes all in one time series array?, well basically except there's already a project with a superior technique.

The forward gist is that techtonicdb is a much better fit with it's super compact DFT format:
(timestamp, seq, is_trade, is_bid, price, size).

Component design, subsystem todos

  • we need an ability to spawn a local instance using the docker container
  • a client can talk to techtonic over TCP as per the messaging spec
    • we probably want a trio-fied client to do this (which should be trivial using trio.SocketStream).
    • implement a client in our repo first, get it working with everything else, and then we can break it out later (todo make issue for that if we want)
  • we'll want to extend the existing L1 feed UI to show bars and/or lines at each book-price level (this could also get switched later to a curve like in cryptowatch if peeps prefer that)

Further resources

  • the set of python scripts in the techtonicdb repo
  • example L2 plotting code in python giving an idea of how to display levels using pandas and matplotlib
    • pretty sure it originally came from this blog post (Gradient Trader Part 2) from the author
      • notice the note at the end that will possibly be worth follow up:

      In the future I plan on porting such visualization to react-stockcharts so I can interactively explore data. The algorithms are already here, what is left to be done is changing matplotlib calls to d3. If you would like to collaborate with me on this, please email me.

  • an example trading bot
  • a small monitoring script also in python
  • a short blog post by the author on microstructure backtesting which gives some intuition for what real-time charting might look like
  • the original post by the author that was pre-cursor to the creation of the techtonicdb rust stack
  • an ffi.py script for reading raw data from the db's storage file
@goodboy goodboy added data-layer real-time and historical data processing and storage integration external stack and/or lib augmentations (sub-)systems general sw design and eng tsdb time series db stuff labels May 17, 2021
@guilledk guilledk linked a pull request May 17, 2021 that will close this issue
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
data-layer real-time and historical data processing and storage integration external stack and/or lib augmentations (sub-)systems general sw design and eng tsdb time series db stuff
Projects
None yet
Development

Successfully merging a pull request may close this issue.

2 participants