Release 1.206.0

.github/workflows/build.yml

@@ -85,7 +85,7 @@ jobs:
           version: ${{ env.POETRY_VERSION }}
 
       - name: Install build dependencies
-        run: python -m pip install --upgrade pip setuptools wheel pre-commit msgspec pytest-codspeed
+        run: python -m pip install --upgrade pip setuptools wheel pre-commit msgspec
 
       # ta-lib Python install currently broken
       # https://github.com/TA-Lib/ta-lib-python/issues/655
@@ -165,12 +165,12 @@ jobs:
           make test-examples
 
       # Run codspeed for latest python-version only
-      # - name: Run benchmarks (Linux)
-      #   if: ${{ matrix.python-version == '3.12' }}
-      #   uses: CodSpeedHQ/action@v3
-      #   with:
-      #     token: ${{ secrets.CODSPEED_TOKEN }}
-      #     run: make test-performance
+      - name: Run benchmarks (Linux)
+        if: ${{ matrix.python-version == '3.12' }}
+        uses: CodSpeedHQ/action@v3
+        with:
+          token: ${{ secrets.CODSPEED_TOKEN }}
+          run: make test-performance
 
   build-windows:
     strategy:

.github/workflows/coverage.yml

@@ -131,6 +131,6 @@ jobs:
       - name: Upload coverage report
         uses: codecov/codecov-action@v4
         with:
+          # fail_ci_if_error: true  # leave commented until flakiness improves
           token: ${{ secrets.CODECOV_TOKEN }}
-          files: ./coverage.xml
           verbose: true

.gitignore

@@ -26,6 +26,9 @@ logs/
 *.tar.gz*
 *.zip
 
+# Data
+*.parquet
+
 # IntelliJ and VSCode project
 *.iml
 .idea/
@@ -43,6 +46,7 @@ logs/
 *.pyd
 .history*
 .cache/
+.codspeed/
 .mypy_cache/
 .profile/
 .pytest_cache/

.pre-commit-config.yaml

@@ -83,7 +83,7 @@ repos:
         exclude: "docs/_pygments/monokai.py"
 
   - repo: https://github.com/astral-sh/ruff-pre-commit
-    rev: v0.7.2
+    rev: v0.7.4
     hooks:
       - id: ruff
         args: ["--fix"]
@@ -129,6 +129,6 @@ repos:
         ]
 
   - repo: https://github.com/kynan/nbstripout
-    rev: 0.7.1
+    rev: 0.8.0
     hooks:
       - id: nbstripout

CONTRIBUTING.md

@@ -10,17 +10,18 @@ To contribute, follow these steps:
 
 2. Once everyone is aligned, fork the `develop` branch and ensure your fork is up-to-date by regularly merging any upstream changes.
 
-3. Install and configure [pre-commit](https://pre-commit.com/) on your local machine to automatically run code checks, formatters, and linters before each commit. You can install pre-commit via:
+3. Install and configure [pre-commit](https://pre-commit.com/) on your local machine to automatically run code checks, formatters, and linters before each commit.
+   You can install pre-commit with:
     ```bash
     pip install pre-commit
     pre-commit install
     ```
 
 4. Open a pull request (PR) on the `develop` branch with a summary comment and reference to any relevant GitHub issue(s).
 
-5. The CI system will run the full test suite over your code including all unit and integration tests, so include appropriate tests with the PR.
+5. The CI system will run the full test suite on your code including all unit and integration tests, so include appropriate tests with the PR.
 
-6. [Deepsource](https://deepsource.io) will perform an automated code review. Fix any issues which cause a failed check, and add the commit to your PR.
+6. [Deepsource](https://deepsource.io) will perform an automated code review. Fix any issues that cause a failed check, and add the commit to your PR.
 
 7. Read and understand the Contributor License Agreement (CLA), available at https://github.com/nautechsystems/nautilus_trader/blob/develop/CLA.md.
 

Makefile

@@ -14,7 +14,7 @@ install-debug:
 
 .PHONY: install-docs
 install-docs:
-	BUILD_MODE=debug poetry install --with docs
+	BUILD_MODE=debug poetry install --with docs --all-extras
 
 .PHONY: install-just-deps
 install-just-deps:

README.md

@@ -46,7 +46,7 @@ including FX, Equities, Futures, Options, CFDs, Crypto and Betting - across mult
 ## Features
 
 - **Fast**: Core is written in Rust with asynchronous networking using [tokio](https://crates.io/crates/tokio).
-- **Reliable**: Type safety and thread safety through Rust. Redis-backed performant state persistence.
+- **Reliable**: Type safety and thread safety through Rust. Redis-backed performant state persistence (optional).
 - **Portable**: OS independent, runs on Linux, macOS, and Windows. Deploy using Docker.
 - **Flexible**: Modular adapters mean any REST, WebSocket, or FIX API can be integrated.
 - **Advanced**: Time in force `IOC`, `FOK`, `GTD`, `AT_THE_OPEN`, `AT_THE_CLOSE`, advanced order types and conditional triggers. Execution instructions `post-only`, `reduce-only`, and icebergs. Contingency order lists including `OCO`, `OTO`.
@@ -153,9 +153,9 @@ See the [Integrations](https://nautilustrader.io/docs/latest/integrations/index.
 
 ### From PyPI
 
-We recommend running the platform with the latest stable version of Python, and in a virtual environment to isolate the dependencies.
+We recommend using the latest supported version of Python and setting up [nautilus_trader](https://pypi.org/project/nautilus_trader/) in a virtual environment to isolate dependencies
 
-To install the latest binary wheel from PyPI:
+To install the latest binary wheel (or sdist package) from PyPI using Pythons pip package manager:
 
     pip install -U nautilus_trader
 
@@ -168,9 +168,9 @@ For MacBook Pro M1/M2, make sure your Python installed using pyenv is configured
 
     PYTHON_CONFIGURE_OPTS="--enable-shared" pyenv install <python_version>
 
-See https://pyo3.rs/latest/getting_started#virtualenvs.
+See the [PyO3 user guide](https://pyo3.rs/latest/getting-started#virtualenvs) for more details.
 
-It's possible to install from source using `pip` if you first install the build dependencies
+It's possible to install from source using pip if you first install the build dependencies
 as specified in the `pyproject.toml`. We highly recommend installing using [poetry](https://python-poetry.org/) as below.
 
 1. Install [rustup](https://rustup.rs/) (the Rust toolchain installer):
@@ -217,7 +217,12 @@ as specified in the `pyproject.toml`. We highly recommend installing using [poet
        cd nautilus_trader
        poetry install --only main --all-extras
 
-See the [Installation Guide](https://nautilustrader.io/docs/latest/getting_started/installation.html) for other options and further details.
+See the [Installation Guide](https://nautilustrader.io/docs/latest/getting_started/installation) for other options and further details.
+
+## Redis
+
+Using Redis with NautilusTrader is **optional** and only required if configured as the backend for a cache database or [message bus](https://nautilustrader.io/docs/latest/concepts/message_bus).
+See the Redis section of the [Installation Guide](https://nautilustrader.io/docs/latest/getting_started/installation#redis) for further details.
 
 ## Versioning and releases
 

RELEASES.md

@@ -1,3 +1,60 @@
+# NautilusTrader 1.206.0 Beta
+
+Released on 17th November 2024 (UTC).
+
+### Enhancements
+- Added `Portfolio.realized_pnl(...)` method for per instrument realized PnL (based on positions)
+- Added `Portfolio.realized_pnls(...)` method for per venue realized PnL (based on positions)
+- Added `TardisInstrumentProvider`
+- Added `PolymarketExecClientConfig.generate_order_history_from_trades` config setting (default False and not currently recommended)
+- Added configuration warning for `InstrumentProvider` (to warn when node starts with no instrument loading)
+- Implemented Tardis optional [symbol normalization](https://nautilustrader.io/docs/nightly/integrations/tardis/#symbology-and-normalization)
+- Implemented `WebSocketClient` reconnection retries (#2044), thanks @davidsblom
+- Implemented `OrderCancelRejected` event generation for Binance and Bybit
+- Implemented `OrderModifyRejected` event generation for Binance and Bybit
+- Improved `OrderRejected` handling of `reason` string (`None` is now allowed which will become the string `'None'`)
+- Improved `OrderCancelRejected` handling of `reason` string (`None` is now allowed which will become the string `'None'`)
+- Improved `OrderModifyRejected` handling of `reason` string (`None` is now allowed which will become the string `'None'`)
+
+### Internal Improvements
+- Ported `RiskEngine` to Rust (#2035), thanks @Pushkarm029 and @twitu
+- Ported `ExecutionEngine` to Rust (#2048), thanks @twitu
+- Added globally shared data channels to send events from engines to Runner in Rust (#2042), thanks @twitu
+- Added LRU caching for dYdX HTTP client (#2049), thanks @davidsblom
+- Improved identifier constructors to take `AsRef<str>` for a cleaner more flexible API
+- Refined identifiers `From` trait impls
+- Refined `InstrumentProvider` initialization behavior and logging
+- Refined `LiveTimer` cancel and performance testing
+- Simplified `LiveTimer` cancellation model (#2046), thanks @twitu
+- Refined Bybit HMAC authentication signatures (now using Rust implemented function)
+- Refined Tardis instrument ID parsing
+- Removed Bybit `msgspec` redundant import alias (#2050), thanks @sunlei
+- Upgraded `databento` crate to v0.16.0
+
+### Breaking Changes
+None
+
+### Fixes
+- Fixed loading specific instrument IDs for `InstrumentProviderConfig`
+- Fixed PyO3 instrument conversions for `raw_symbol` (was incorrectly using the normalized symbol)
+- Fixed reconcile open orders and account websocket message for dYdX (#2039), thanks @davidsblom
+- Fixed market order `avg_px` for Polymarket trade reports
+- Fixed Betfair clients keepalive (#2040), thanks @limx0
+- Fixed Betfair reconciliation (#2041), thanks @limx0
+- Fixed Betfair customer order ref limit to 32 chars
+- Fixed Bybit handling of `PARTIALLY_FILLED_CANCELED` status orders
+- Fixed Polymarket size precision for `BinaryOption` instruments (precision 6 to match USDC.e)
+- Fixed adapter instrument reloading (providers were not reloading instruments at the configured interval due to internal state flags)
+- Fixed static time logging for `BacktestEngine` when running with `use_pyo3` logging config
+- Fixed in-flight orders check and improve error handling (#2053), thanks @davidsblom
+- Fixed dYdX handling for liquidated fills (#2052), thanks @davidsblom
+- Fixed `BybitResponse.time` field as optional `int` (#2051), thanks @sunlei
+- Fixed single instrument requests for `DatabentoDataClient` (was incorrectly calling `_handle_instruments` instead of `_handle_instrument`), thanks for reporting @Emsu
+- Fixed `fsspec` recursive globbing behavior to ensure only file paths are included, and bumped dependency to version 2024.10.0
+- Fixed jupyterlab url typo (#2057), thanks @Alsheh
+
+---
+
 # NautilusTrader 1.205.0 Beta
 
 Released on 3rd November 2024 (UTC).
@@ -22,15 +79,15 @@ Released on 3rd November 2024 (UTC).
 - Refined Arrow serialization (record batch functions now also available in Rust)
 - Refined core `Bar` API to remove unnecessary unwraps
 - Standardized network client logging
-- Fixed all pyo3 deprecations for API breaking changes
+- Fixed all PyO3 deprecations for API breaking changes
 - Fixed all clippy warning lints for PyO3 changes (#2030), thanks @Pushkarm029
 - PyO3 upgrade refactor and repair catalog tests (#2032), thanks @twitu
 - Upgraded `pyo3` crate to v0.22.5
 - Upgraded `pyo3-async-runtimes` crate to v0.22.0
 - Upgraded `tokio` crate to v1.41.0
 
 ### Breaking Changes
-- Removed pyo3 `DataTransformer` (was being used for namespacing, so refactored to separate functions)
+- Removed PyO3 `DataTransformer` (was being used for namespacing, so refactored to separate functions)
 - Moved `TEST_DATA_DIR` constant from `tests` to `nautilus_trader` package (#2020), thanks @faysou
 
 ### Fixes
@@ -42,7 +99,7 @@ Released on 3rd November 2024 (UTC).
 - Fixed catalog query mem leak test (#2031), thanks @Pushkarm029
 - Fixed `OrderInitialized.to_dict()` `tags` value type to `list[str]` (was a concatenated `str`)
 - Fixed `OrderInitialized.to_dict()` `linked_order_ids` value type to `list[str]` (was a concatenated `str`)
-- Fix Betfair clients shutdown (#2037), thanks @limx0
+- Fixed Betfair clients shutdown (#2037), thanks @limx0
 
 ---
 
@@ -341,7 +398,7 @@ Released on 2nd August 2024 (UTC).
 - Fixed `DataEngine` unsubscribing from order book deltas (#1814), thanks @davidsblom
 - Fixed `LiveExecutionEngine` handling of adapter client execution report causing `None` mass status (#1789), thanks for reporting @faysou
 - Fixed `InteractiveBrokersExecutionClient` handling of instruments not found when generating execution reports (#1789), thanks for reporting @faysou
-- Fixed Bybit parsing of trade and quote ticks for websocket messages (#1794), thanks @davidsblom
+- Fixed Bybit parsing of trade and quotes for websocket messages (#1794), thanks @davidsblom
 
 ---
 
@@ -930,7 +987,7 @@ This will be the final release with support for Python 3.9.
 - Added `Cache.is_order_pending_cancel_local(...)` (tracks local orders in cancel transition)
 - Added `BinanceTimeInForce.GTD` enum member (futures only)
 - Added Binance Futures support for GTD orders
-- Added Binance internal bar aggregation inference from aggregated trade ticks or 1-MINUTE bars (depending on lookback window)
+- Added Binance internal bar aggregation inference from aggregated trades or 1-MINUTE bars (depending on lookback window)
 - Added `BinanceExecClientConfig.use_gtd` option (to remap to GTC and locally manage GTD orders)
 - Added package version check for `nautilus_ibapi`, thanks @rsmb7z
 - Added `RiskEngine` min/max instrument notional limit checks
@@ -1075,7 +1132,7 @@ Released on 31st July 2023 (UTC).
 - Fixed Binance reconciliation which was requesting reports for the same symbol multiple times
 - Fixed Binance Futures native symbol parsing (was actually Nautilus symbol values)
 - Fixed Binance Futures `PositionStatusReport` parsing of position side
-- Fixed Binance Futures `TradeReport` assignment of position ID (was hardcoded to hedging mode)
+- Fixed Binance Futures `TradeReport` assignment of position ID (was hard-coded to hedging mode)
 - Fixed Binance execution submitting of order lists
 - Fixed Binance commission rates requests for `InstrumentProvider`
 - Fixed Binance `TriggerType` parsing #1154, thanks for reporting @davidblom603

build.py

@@ -247,7 +247,7 @@ def _copy_build_dir_to_project(cmd: build_ext) -> None:
 
 
 def _copy_rust_dylibs_to_project() -> None:
-    # https://pyo3.rs/latest/building_and_distribution#manual-builds
+    # https://pyo3.rs/latest/building-and-distribution#manual-builds
     ext_suffix = sysconfig.get_config_var("EXT_SUFFIX")
     src = Path(TARGET_DIR) / f"{RUST_LIB_PFX}nautilus_pyo3.{RUST_DYLIB_EXT}"
     dst = Path("nautilus_trader/core") / f"nautilus_pyo3{ext_suffix}"

docs/api_reference/adapters/tardis.md

@@ -17,3 +17,43 @@
    :members:
    :member-order: bysource
 ```
+
+## Config
+
+```{eval-rst}
+.. automodule:: nautilus_trader.adapters.tardis.config
+   :show-inheritance:
+   :inherited-members:
+   :members:
+   :member-order: bysource
+```
+
+## Providers
+
+```{eval-rst}
+.. automodule:: nautilus_trader.adapters.tardis.providers
+   :show-inheritance:
+   :inherited-members:
+   :members:
+   :member-order: bysource
+```
+
+## Factories
+
+```{eval-rst}
+.. automodule:: nautilus_trader.adapters.tardis.factories
+   :show-inheritance:
+   :inherited-members:
+   :members:
+   :member-order: bysource
+```
+
+## Data
+
+```{eval-rst}
+.. automodule:: nautilus_trader.adapters.tardis.data
+   :show-inheritance:
+   :inherited-members:
+   :members:
+   :member-order: bysource
+```

docs/concepts/advanced/emulated_orders.md

@@ -14,8 +14,8 @@ The only requirement to emulate an order is to pass a `TriggerType` to the `emul
 parameter of an `Order` constructor, or `OrderFactory` creation method. The following
 emulation trigger types are currently supported:
 - `DEFAULT` (which is the same as `BID_ASK`)
-- `BID_ASK` (emulated using quote ticks)
-- `LAST` (emulated using trade ticks)
+- `BID_ASK` (emulated using quotes)
+- `LAST` (emulated using trades)
 
 Emulated orders are subject to the same risk controls as 'regular' orders, and can be
 modified and canceled by a trading strategy in the normal way. They will also be included

docs/concepts/advanced/synthetic_instruments.md

@@ -1,7 +1,7 @@
 # Synthetic Instruments
 
 The platform supports creating customized synthetic instruments, which can generate synthetic quote 
-and trade ticks. These are useful for:
+and trades. These are useful for:
 
 - Enabling `Actor` and `Strategy` components to subscribe to quote or trade feeds
 - Triggering emulated orders
@@ -62,7 +62,7 @@ self._synthetic_id = synthetic.id
 # Add the synthetic instrument for use by other components
 self.add_synthetic(synthetic)
 
-# Subscribe to quote ticks for the synthetic instrument
+# Subscribe to quotes for the synthetic instrument
 self.subscribe_quote_ticks(self._synthetic_id)
 ```
 
@@ -89,7 +89,7 @@ self.update_synthetic(synthetic)
 ## Trigger instrument IDs
 The platform allows for emulated orders to be triggered based on synthetic instrument prices. In 
 the following example, we build upon the previous one to submit a new emulated order. 
-This order will be retained in the emulator until a trigger from synthetic quote ticks releases it. 
+This order will be retained in the emulator until a trigger from synthetic quotes releases it. 
 It will then be submitted to Binance as a MARKET order:
 
 ```

docs/concepts/backtesting.md

@@ -6,7 +6,7 @@ We are currently working on this guide.
 
 Backtesting with NautilusTrader is a methodical simulation process that replicates trading
 activities using a specific system implementation. This system is composed of various components
-including the built-in engines, `Cache`, `MessageBus`, `Portfolio`, [Actors](advanced/actors.md), [Strategies](strategies.md), [Execution Algorithms](execution.md),
+including the built-in engines, `Cache`, [MessageBus](message_bus.md), `Portfolio`, [Actors](advanced/actors.md), [Strategies](strategies.md), [Execution Algorithms](execution.md),
 and other user-defined modules. The entire trading simulation is predicated on a stream of historical data processed by a
 `BacktestEngine`. Once this data stream is exhausted, the engine concludes its operation, producing 
 detailed results and performance metrics for in-depth analysis.

docs/concepts/data.md

@@ -95,18 +95,18 @@ NautilusTrader defines a unique *bar type* (`BarType`) based on the following co
 - **Instrument ID** (`InstrumentId`): Specifies the particular instrument for the bar.
 - **Bar Specification** (`BarSpecification`):
   - `step`: Defines the interval or frequency of each bar.
-  - `aggregation`: Specifies the method used for data aggregation (see the list above).
+  - `aggregation`: Specifies the method used for data aggregation (see the above table).
   - `price_type`: Indicates the price basis of the bar (e.g., bid, ask, mid, last).
 - **Aggregation Source** (`AggregationSource`): Indicates whether the bar was aggregated internally (within Nautilus) or externally (by a trading venue or data provider).
 
-Bar data can be aggregated either internally or externally:
+Bar data aggregation can be either *internal* or *external*:
 
 - `INTERNAL`: The bar is aggregated inside the local Nautilus system boundary.
 - `EXTERNAL`: The bar is aggregated outside the local Nautilus system boundary (typically by a trading venue or data provider).
 
 Bar types can also be classified as either *standard* or *composite*:
 
-- **Standard**: Generated from granular market data, such as quotes or trade ticks.
+- **Standard**: Generated from granular market data, such as quotes or trades.
 - **Composite**: Derived from a higher-granularity bar type through subsampling.
 
 ### Defining standard bars

docs/concepts/execution.md

@@ -244,7 +244,7 @@ know as "primary" (original) orders when being handled by an execution algorithm
 ```python
 from nautilus_trader.model.orders.base import Order
 
-def on_order(self, order: Order) -> None:  # noqa (too complex)
+def on_order(self, order: Order) -> None:
     # Handle the order here
 ```