v1.5-beta-2

LibreQoS V1.5-BETA-2

Download: https://libreqos.io/#download

Changelog Since 1.5-beta-1

New User Interface (UI2)

• UI2 is a complete overhaul of the LibreQoS Web UI - featuring a customizable dasboard with widgets displaying various network performance metric including TCP RTT and TCP Retransmits.
• Quick LibreQoS v1.5-BETA-2 deep dive with Herbert Wolverson

Changelog Since 1.4

LibreQoS v1.5 Features Overview:

• 30%+ Performance Improvement. Large improvements in code efficiency. LibreQoS can now push 70 Gbit/s on a $1500 AMD Ryzen 9 7950X box with a Mellanox MCX516A card.
• Intelligent Binpacking. Dynamically redistribute load (based on long-term statistics) to get the most out of all of your CPU cores.
• Flow-Based Analysis. Flows are tracked individually, for TCP retransmits, round-trip time, and performance. Completed flows can be analyzed by ASN, geolocated, or exported via Netflow.
• Unified Configuration System and GUI. No more separate ispConfig.py and lqos.conf files - all configuration is managed in one place. The web user interface now lets you manage the whole configuration, including devices and network lists.
• Support for Newer Linux Enhancements. LibreQoS can take advantage of eBPF improvements in the 6.x kernel tree to further improve your performance - but remains compatible with later 5.x kernels.
• Improved CLI tools. lqtop, a new support tool and more.

Unified Configuration System

• Replace ispConfig.py with a singular /etc/lqos.conf
• Automatically migrate previous configuration
• In-memory cache system (load the config once and share it, detect changes)
• Shared configuration between Python and Rust

Dynamic Binpacking

• It was a common problem for the CPU-queue assignment (in tree mode) to allocate too many resources
  to a single CPU.
• Each circuit is assigned a "weight". If you have Long-Term Stats, then the weight is calculated based on
  past usage AND assigned speed plan. Without LTS, it's just assigned speed plan.
• The total weight - at this time of day - is then calculated for each top-level entry.
• A "binpacking" algorithm then attempts to equalize load between CPUs.
• Significant performance improvement on many networks.

Per-Flow Tracking System

• "Flows" are detected as TCP connections, UDP connections that reuse a source/destination and ICMP between a source/destination.
• Rather than just track per-host statistics, statistics are attached to a flow.
• Flows maintain a rate estimation at all times, in-kernel.
• Flows calculate Round-Trip Time (RTT) continuously.
• Flows spot timestamp duplications indicating TCP Retry (or duplicate).
• Much of the kernel code moved from the TC part of eBPF to the XDP part, giving a modest speed-up and improvement in
  overall throughput.

Per-Flow Userland/Kernel Kernel-Userland System

• Rather than reporting RTT via a giant data structure, individual reports are fed to the kernel through a userspace callback
  system.
• Flows "closing" (clean closure) results in a kernel-userspace notify.
• Flows also expire on a periodic tick if no data has arrived in a given time period.
• This decreased kernel side overhead significantly (eBPF kernel to userspace is non-blocking send).
• This increased userspace CPU usage very slightly, but removed the processing overhead from the packet-flow execution path.

Per-Flow Reporting System

• RTT is compiled per-flow into a ringbuffer. Results from very-low (mostly idle) flows are ignored. RTT is calculated from a median of the last
  hundred reports. Significant accuracy improvement.
• Per-flow TCP retries are recorded.
• When flows "close", they are submitted for additional analysis.
• Simple protocol naming system maps ethertype/port to known protocols.

Export Flow Data in netflow version 5 and 9 (IPFIX)

Closed Flow Reporting System

• Created "geo.bin", a compiled list of by-ASN and by IP geolocations.
• lqosd will download a refreshed geo.bin periodically.
• Closed flows are mapped to an ASN, giving per-ASN performance reports.
• Closed flows are mapped to a geolocation, giving geographic performance reports.
• Closed flows are mapped to ethertype and protocol.
• User interface expanded in lqos_node_manager to display all of this.

Preflight checks for lqosd

• Prior to startup, common configuration and hardware support issues are checked.
• Single-queue NICs now get a proper error message.
• If the user tries to run both a Linux bridge and an XDP bridge on the same interface pair,
  the XDP bridge is disabled and a warning emitted.

XDP "Hot Cache"

• Much CPU time was spent running a longest-prefix match check on every ISP-facing IP address.
• Added a least-recently-used cache that matches IP adddresses to circuits with a much less
  expensive fast lookup.
• Added a "negative cache" entry to speed up "this IP still isn't mapped"
• Added cache invalidation code to handle the IP mappings changing
• This resulted in a 20-30% CPU usage reduction under heavy load.

Config UI

• lqos_node_manager is now aware of the entire configuration system.
• All configuration items may be edited.
• ShapedDevices.csv can be edited from the web UI.
• network.json can be edited from the web UI.
• Heavy validation, ensuring that devices have matching network.json entries, IPs aren't duplicated, etc.

LQTop

• New lqtop CLI tool with much prettier text UI and support for flows.

UISP Integration 2

• An all-new, massively faster UISP Integration system.
• Includes much better network map traversal.

Support Tools

• CLI tool for running a "sanity check" on common issues.
• Gather configuration into a bundle for sending.
• View the bundle.
• Submit the bundle to LibreQoS for analysis.
• A web UI (lqos_node_manager) version of the same thing, using shared code.

Misc

• Improvements and fixes to all integrations, especially Spylnx.
• Update back-end code to latest versions.

v1.5-Beta-1

LibreQoS V1.5-BETA-1

After 6 months of development, LibreQoS is proud to announce the first beta for LibreQoS version 1.5. Our V1.4 (released January, 2024) is now in use by over 150 ISPs in 22 US states and in 42 countries, for over a million subscribers - reducing network latency via CAKE, and providing extensive diagnostics and statistics to ISP support staff. We hope that v1.5 will cover the world!

LibreQoS would like to thank our subscribers, donors, and Equinix Metal, NLnet, and Zulip for their support during this process. We wouldn't be able to keep fixing the internet without you! Also a huge thanks to Ubuntu for such a solid OS, and the libbpf team for all their improvements to the underlying network code we leverage.

Get v1.5-BETA-1 here: https://libreqos.io/#download

LibreQoS v1.5 Features Overview:

• 30%+ Performance Improvement. Large improvements in code efficiency. LibreQoS can now push 70 Gbit/s on a $1500 AMD Ryzen 9 7950X box with a Mellanox MCX516A card.
• Intelligent Binpacking. Dynamically redistribute load (based on long-term statistics) to get the most out of all of your CPU cores.
• Flow-Based Analysis. Flows are tracked individually, for TCP retransmits, round-trip time, and performance. Completed flows can be analyzed by ASN, geolocated, or exported via Netflow.
• Unified Configuration System and GUI. No more separate ispConfig.py and lqos.conf files - all configuration is managed in one place. The web user interface now lets you manage the whole configuration, including devices and network lists.
• Support for Newer Linux Enhancements. LibreQoS can take advantage of eBPF improvements in the 6.x kernel tree to further improve your performance - but remains compatible with later 5.x kernels.
• Improved CLI tools. lqtop, a new support tool and more.

Changelog Since 1.4

Unified Configuration System

• Replace ispConfig.py with a singular /etc/lqos.conf
• Automatically migrate previous configuration
• In-memory cache system (load the config once and share it, detect changes)
• Shared configuration between Python and Rust

Dynamic Binpacking

• It was a common problem for the CPU-queue assignment (in tree mode) to allocate too many resources
  to a single CPU.
• Each circuit is assigned a "weight". If you have Long-Term Stats, then the weight is calculated based on
  past usage AND assigned speed plan. Without LTS, it's just assigned speed plan.
• The total weight - at this time of day - is then calculated for each top-level entry.
• A "binpacking" algorithm then attempts to equalize load between CPUs.
• Significant performance improvement on many networks.

Per-Flow Tracking System

• "Flows" are detected as TCP connections, UDP connections that reuse a source/destination and ICMP between a source/destination.
• Rather than just track per-host statistics, statistics are attached to a flow.
• Flows maintain a rate estimation at all times, in-kernel.
• Flows calculate Round-Trip Time (RTT) continuously.
• Flows spot timestamp duplications indicating TCP Retry (or duplicate).
• Much of the kernel code moved from the TC part of eBPF to the XDP part, giving a modest speed-up and improvement in
  overall throughput.

Per-Flow Userland/Kernel Kernel-Userland System

• Rather than reporting RTT via a giant data structure, individual reports are fed to the kernel through a userspace callback
  system.
• Flows "closing" (clean closure) results in a kernel-userspace notify.
• Flows also expire on a periodic tick if no data has arrived in a given time period.
• This decreased kernel side overhead significantly (eBPF kernel to userspace is non-blocking send).
• This increased userspace CPU usage very slightly, but removed the processing overhead from the packet-flow execution path.

Per-Flow Reporting System

• RTT is compiled per-flow into a ringbuffer. Results from very-low (mostly idle) flows are ignored. RTT is calculated from a median of the last
  hundred reports. Significant accuracy improvement.
• Per-flow TCP retries are recorded.
• When flows "close", they are submitted for additional analysis.
• Simple protocol naming system maps ethertype/port to known protocols.

Export Flow Data in netflow version 5 and 9 (IPFIX)

Closed Flow Reporting System

• Created "geo.bin", a compiled list of by-ASN and by IP geolocations.
• lqosd will download a refreshed geo.bin periodically.
• Closed flows are mapped to an ASN, giving per-ASN performance reports.
• Closed flows are mapped to a geolocation, giving geographic performance reports.
• Closed flows are mapped to ethertype and protocol.
• User interface expanded in lqos_node_manager to display all of this.

Preflight checks for lqosd

• Prior to startup, common configuration and hardware support issues are checked.
• Single-queue NICs now get a proper error message.
• If the user tries to run both a Linux bridge and an XDP bridge on the same interface pair,
  the XDP bridge is disabled and a warning emitted.

XDP "Hot Cache"

• Much CPU time was spent running a longest-prefix match check on every ISP-facing IP address.
• Added a least-recently-used cache that matches IP adddresses to circuits with a much less
  expensive fast lookup.
• Added a "negative cache" entry to speed up "this IP still isn't mapped"
• Added cache invalidation code to handle the IP mappings changing
• This resulted in a 20-30% CPU usage reduction under heavy load.

Config UI

• lqos_node_manager is now aware of the entire configuration system.
• All configuration items may be edited.
• ShapedDevices.csv can be edited from the web UI.
• network.json can be edited from the web UI.
• Heavy validation, ensuring that devices have matching network.json entries, IPs aren't duplicated, etc.

LQTop

• New lqtop CLI tool with much prettier text UI and support for flows.

UISP Integration 2

• An all-new, massively faster UISP Integration system.
• Includes much better network map traversal.

Support Tools

• CLI tool for running a "sanity check" on common issues.
• Gather configuration into a bundle for sending.
• View the bundle.
• Submit the bundle to LibreQoS for analysis.
• A web UI (lqos_node_manager) version of the same thing, using shared code.

Misc

• Improvements and fixes to all integrations, especially Spylnx.
• Update back-end code to latest versions.

Our lines-of-code counts are approximately:

Language	code
Rust	18251
Python	5859

Herbert: And I forgot one: if you're running the 6.x kernel line, XDP metadata kicks in and you get another decent performance boost - but it still works with the older kernels (just without the boost).
Herbert: Also... with all of these improvements stacked, we've cracked the 10gbps single-flow barrier on Payne!

Beta-2 (or 3) will contain an updated user interface (UI2). The final release of v1.5 is presently targeted for August, 2024.

v1.4

LibreQoS v1.4 Change Summary

Version 1.4 is a huge milestone. A whole new back-end, new GUI, 30%+ performance improvements, support for single-interface mode.

Some Statistics

• 1,097 Commits since 1.3.1
• 43,282 Lines of Code
  • 16,917 lines of Rust
  • 16,508 lines of HTML & JavaScript
  • 7,087 lines of Python
  • 2,770 lines of C

Peak Performance (So Far)

• Tested single-stream performance of just under 10 gbit/s on a 16-core Xeon Gold (single interface architecture, using 8 cores for each direction). The flow was shaped with Cake, and retained good (<10 ms RTT latency) performance.
• Tested 25 gbit/s total throughput on the same system. CPU was not saturated---we didn't have a bigger network to test!
• Running live at ISPs with 11 gbit/s of real customer performance and plenty of room to grow.

New Architecture

• Rust-based back-end provides:
  • lqosd - a daemon that manages:
    • Loading/setup/unloading eBPF programs.
    • Gathers statistics directly from eBPF.
    • Provides a local "bus" for transporting data between components.
    • Sets "tunables", replacing the need for a separate offloading service.
  • lqtop - a console-based utility for viewing current activity.
  • lqos_node_manager - a web-based GUI that:
    • Monitors current activity.
    • Monitors system status.
    • Provides "best/worst" summaries of RTT.
    • Provides visibility into the working of queues.
    • Categorizes traffic to match your network hierarchy, letting you quickly find the bottlenecks.
    • Let's you browse and search your shaped devices.
    • Lists "unknown IP addresses" that are passing through the shaper but do not have a rule associated.
    • Allows you to view and edit the LibreQoS configuration.
  • lqos_python - provides Python access to the bus system.
  • lqos_setup - builds enough configuration files to get you started.
  • lqos_users - authentication for the GUIs.
• High-performance Python script:
  • Batches TC commands for fast execution.
  • Batches bus-transactions to associate IP subnets with users for fast execution.
  • Improved scheduler for InfluxDB graphing.

High Performance Bridge (Bifrost)

• Optionally replace the Linux bridge system with an XDP-based bridge accelerator.
• Throughput is 30% higher in this mode.

Packet and Flow Analysis (Heimdall)

• Viewing a circuit in the web UI displays a summary of IP traffic flows for that circuit.
• A "capture" button will capture packet headers, and allow nanosecond-level analysis of traffic data.
• You can download the packet captures in libpcap format, for analysis in Wireshark and similar tools.
• Configure the capture delay in /etc/lqos.conf

Single-interface Mode

• Operate with a single network interface and VLANs for "in" and "out".

Graphs

• Graph current throughput, shaped and unshaped.
• Graph CPU and RAM performance.
• Graph individual Cake shaper tins, backlog, delays.
• TCP "round trip time" histogram showing overall network latency performance.
• Per-network node traffic graph.
• Per-network node RTT latency histogram, to let you zero-in on troublespots.

Integration Improvements

• UISP integration now uses a full cost-based spanning tree (Dijkstra) to determine node placement in topology. #328

Miscellaneous

• build_rust.sh builds the entire package from a Git update, with minimal (<1 second) downtime.
• build_dpkg.sh assembles the entire system into an Ubuntu/Debian .deb installer.
• Sample .service files for systemd integration.
• Real-time adjustment to tunables.
• Redact text into Klingon to allow screenshots without sharing customer data.
• Preliminary support for reading IP data inside MPLS packets, as long as they are ordered "VLAN->MPLS->VPLS" and not the other way around.
• Automatically trim network trees that exceed 9 levels deep.
• Very accurate timing functions for better statistics.
• Greatly improved documentation.
• Improved rejection of TCP round-trip-time outliers (from long-polled connections).
• Improved Spylnx and UISP integrations.

Better Distribution

This is in alpha testing. It has worked on some test setups, but needs production testing.

Installation via apt-get and LibreQoS's own repo. Add the libreqos repo, and you can use apt-get to install/update the traffic shaper. This doesn't get you the development toolchain.

sudo echo "deb http://stats.libreqos.io/ubuntu jammy main" > /etc/apt/sources.list.d/libreqos.list
wget -O - -q http://stats.libreqos.io/repo.asc | apt-key add -
apt-get update
apt-get install libreqos

You will be asked some questions about your configuration, and the management daemon and webserver will automatically start. Go to http://<your_ip>:9123/ to finish installation.

Gallery

Node Manager - Dashboard

The node manager displays current activity on your network

Node Manager - Circuit View

Find out exactly what's going on inside each circuit, monitoring all of the queue stats - you can even view the details of each category tin

Node Manager - Flow Analysis

Analyze what's going on for a specific client, viewing real-time traffic flow data. No need to run torch or equivalent on their router. Ideal for finding connectivity problems.

Node Manager - Packet Capture

Capture traffic and analyze per-packet in an intuitive, zoomable traffic analysis system. You can view down to nanosecond granularity to find performance problems, and see what's really going on with your traffic. Click "Download PCAP Dump" to analyze the same data in Wireshark.

Long Term Statistics (LTS)

LibreQoS' new Long Term Stats (LTS) system aggregates and graphs LibreQoS statistics for fast, efficient troubleshooting and network analysis. There is a free 30-day trial, after which the rate is $0.30 USD per shaped subscriber. You can enroll in the 30-day free trial by clicking "Start Stats Free Trial" in the top-right corner of the local LibreQoS GUI.

Long Term Statistics - Dashboard

Long Term Statistics - Report - All APs

Long Term Statistics - Report - All Sites

Long Term Statistics - Shaper Node

v1.3.1

v1.3.1 (IPv4 + IPv6)

Features

Fast TCP Latency Tracking

@thebracket has created cpumap-pping which merges the functionality of the xdp-cpumap-tc and ePPing projects, while keeping CPU use within ~1% of xdp-cpumap-tc.

Integrations

• Added Splynx integration
• UISP integration overhaul by @thebracket
• LMS integation for Polish ISPs by @interduo

Partial Queue Reload

In v1.2 and prior, the the entire queue structure had to be reloaded to make any changes. This led to a few milliseconds of packet loss for some clients each time that reload happened. The scheduled.py was set to reload all queues each morning at 4AM to avoid any potential disruptions that could theoretically cause.

Starting with v1.3 - LibreQoS tracks the state of the queues, and can do incremental changes without a full reload of all queues. Every 30 minutes - scheduler.py runs the CRM import, and runs a partial reload affecting just the queues that have changed. It still runs a full reload at 4AM.

v1.3 Improvements to help scale

HTB major:minor handle

HTB uses a hex handle for classes. It is two 16-bit hex values joined by a colon - major:minor (:). In LibreQoS, each CPU core uses a different major handle.

In v1.2 and prior, the minor handle was unique across all CPUs, meaning only 30k subscribers could be added total.

Starting with LibreQoS v1.3 - minor handles are counted independently by CPU core. With this change, the maximum possible subscriber qdiscs/classes goes from a hard limit of 30k to instead be 30k x CPU core count. So for a higher end system with a 64 core processor such as the AMD EPYC™ 7713P, that would mean ~1.9 million possible subscriber classes. Of course CPU use will be the bottleneck well before class handles are in that scenario. But at least we have that arbitrary 30k limit out of the way.

"Circuit ID" Unique Identifier

In order to improve queue reload time in v1.3, it was necessary to use a unique identifier for each circuit. We went with Circuit ID. It can be a number or string, it just needs to be unique between circuits, and the same for multiple devices in the same circuit. This allows us to avoid costly lookups when sorting through the queue structure.

If you have your own script creating ShapedDevices.csv - you could use your CRM's unique identifier for customer services / circuits to serve as this Circuit ID. The UISP and Splynx integrations already do this automatically.

Compatability Notes

The most major changes are the renaming of the fqorCake variable to "sqm",
and the addition of the Circuit identifier field.

Also after upgrading to LibreQos v1.3, a reboot is required to clear out the
old ebpf code.