Position accounting refinements, bug fixes, src-asset-pnls, ledgerd daemon

[goodboy]

The major summary follow up to the .accounting subsys introduced in
#489 this is the list of outstanding but needed features and bugfixes
that should be resolved asap!

The existing #478 was the older not-as-specific issue for this stuff..

The new set of top level issue-tasks are:

• new toml libs: Better TOML libs -> tomlkit for style, tomli(-w) for speed #496
• accounting schema changes: Change Position.clears -> .events #510
• ledger CLI and offline audit system: Ledger (syncing) CLI and offline toolset #512
• paper engine clearing models: Paper-clearing: realistic market engine models #476
• dark and alert testing: Dark order and alert persistence #463
• dark / alert offline persistence: Dark order and alert persistence #463
• order mode UI outstanding bugs: order mode pane UI bugs.. #479
  • Order mode allocator limit field should not allow value smaller then current pp #267
  • Scale order indicators on low dpi #248
• missing order mode features:
  • pp indicators in sidepanel UX #224
  • order_mode further UX refinement ideas needing feedback #220

The high level sections here are:

• missing accounting calcs and documentation

  • (eg. ppu, bep)
  • cross asset pnl valuations

• UI components to display the above ^

• service daemon ledgerd or accountd to do bookkeeping out of thread

---

Missing critical calcs and accounting:

• as per Change Position.clears -> .events #510, make Position.clears -> Position.events as per
  Change Position.clears -> .events #510 to handle market specific accounting calc modifying occurrences
  (like stock splits)!

• use transfers/withdrawals to modify the Position.ppu

  • particularly important for cexes such that ppus are calcuated
    properly and graphically correct on chart!

• add a Position.bep:

  • TODO: white board math is as follows:

Pertaining notes and tasks from legacy issues:

• kraken position tracking gotchas #263, the original kraken position tracking way back before we had
  anything 😂

  • how spot is junko compared to their margin trading..
    • they do offer pp tracking on margin trading positions but
      not spot?
    • https://support.kraken.com/hc/en-us/articles/203325683-Differences-between-spot-exchange-and-margin-trading
  • Things that are either confusing af or of note in their api:
    • they have "trade balance" endpoints but only for margin:
      https://docs.kraken.com/rest/#operation/getTradeBalance
      • also wth don't they explain what all the result keys are 😂
    • here's where you can pull past spot fills/trades, but they limit to
      50: https://docs.kraken.com/rest/#operation/getTradeHistory

• kraken: balance <-> position disparities with crypto/crypto pairs #373, more kraken accounting todos from the comments:

  • for wallet transfers out of an account we get
    a Transaction-like entry in our account.<>.<>.toml of the form
    (note this is a non-real example):

    [xmr/eur.kraken]
    size = 78.31125778
    ppu = 0.3222278400000044
    bsuid = "xmr"
    clears = [
     { dt = "2022-12-21T07:20:16.316061+00:00", ppu = 0.3222278400000044, accum_size = 778.33125778, price = 0.3212, size = 778.33125778, cost = 0.4, tid = "TJKLRL-7UE2L-BX2UE5" },
     { dt = "2022-12-21T07:26:56+00:00", ppu = 0.3222278400000044, accum_size = 78.31125778, price = "NaN", size = -700.02, cost = 0, tid = "1503A92E24BAA1E8B8E7B78E79A78AAF3E6C3F3E35227EF7D2C13754876B37D5" },
    ]

    • ^ Notice the price field has a Nan value
    • this transaction record should contribute to the rolling ppu
      calc in the sense that on the next clear the weighting of the
      position should be more right?
      • well that would be the case where you presume the withdrawn
        funds take no losses when (eventually) used, which seems like
        the sanest way to copy with tracking this account's pp right?
      • what happens if you make further trades in this market-pair and
        then bring back the transferred out funds and begin to sell
        them?
    • how do we handle buying more of xmr from say another pair in
      terms of the current next ppu weighting in this position?
      • For eg. if you bought xmr/btc how do we project that
        contribution to this position?
        • say you wanted to project it back to some "main fiat" like EUR
          in this case?
        • the EUR value would come from the original (recursive)
          projection of btc/eur, which would provide the source/base
          pnl from preceding trades as they contribute to the projection
          to EUR; seems like the calculation needs to be
          recursive and rely heavily on time stamps?

  • we aren't tabulating nor using the widthdrawal fee
    schedule in our cost field in each ledger entry, obviously this should
    resolve any accounting discrepancies a transfer-as-sell clear
    should have a cost (that kraken takes) as part of the position calc.

---

UI stuff we'll probably be innovators of..

How about historical position line graphics on charts which show
breakeven price changes overlayed with actual underlyings, aka pp
liftetime "lines" more or less as super simple FSP style graphics that
show the beginning, changes, and end to a given pp such that users can
easily see past trades and their state changes on top of historical data
🏄

• as per order_mode further UX refinement ideas needing feedback #220 and much idea-ing during Rekt pps? problem? => piker.accounting #489
• bep and ppu historical lines over every chart with
  real-time update with flat lines.
• on order staging dotted-line project both the bep and ppu
  given the current level line value B)
  • idea is that for any order you're planning to make you see how it
    will change both your ppu and bep!
  • would change in rt based on user's cursor price placement
  • what to do about submitted live order?
    • some kinda line projection kept up to date like the ppu / bep
      lines?
• will likely require that we also compute and store arrays of
  breakeven or lifo prices per clear in the account<broker>.<account>.toml
• might be handy to show actual LIFO style "contextual" pricing if
  configured which could make it easy to enter contextual short term
  trades despite an overall breakeven price, egs:
  • enters long but bep-pnl is red but there's a dump and you want to
    play a bounce so you complete a bounce trade and make a profit and
    then your be price goes back to what it was before the scalp trade
  • same but say you have a green pnl (in this case entering above
    your be would bring it's mean price up) and want to show your
    original longer term be_price after the trade completes
• adding "ghost data" for paper engine fills which we can
  display as a curve overlay to the actual price history?
  • for example filling in (lesser colored) line/bar to show
    the theoretical paper engine clearing history for something like
    this:
    

---

Add a ledgerd actor-daemon?

Should we consider a separate actor-service: ledgerd which can do all
ledger record monitoring (via connections to emsd) and writing
out-of-process from brokerd/datad such that all other actors only
deal with position data via msgs?

Currently the fs IO is directly serving latency for broker backends
which call this code in thread during startup and possibly on clearing
/ updates.

Premises:

• it's way more handy to have real-time updates as a user, also
  obvsly for testing..

Solutions:

• a singleton-style actor that listens to msgs from the ems and
  does all the work of computing positions and recording them to
  whatever backing storage sys we have (for now
  account.broker.account.toml files) and thus can operate as the
  writer in a readers-writer style lock around file access.

• we should add support for immediate config file
  (ledger or pps.toml) writing

  • initial support for this will land in Rekt pps? problem? => piker.accounting #489 via commit 2f31e40
  • maybe/eventually via the trio async file IO
    apis
    which use a bg thread to avoid main thread latency.

• can we make a better incremental update system;

  • right now we have lots of file writes going on in both the paper
    engine, per clear, and in various brokers. backends inside the
    trades_dialogue() endpoint.
  • using a timestamp scanning technique in TOML and/or a different
    filesystem format (like parquet? something binary?) we can probably
    accomplish this rather easily.
  • only do ledger file writes on teardown/api-block exit by default