NEWS

#
## Copyright (c) NVIDIA CORPORATION & AFFILIATES, 2001-2021. ALL RIGHTS RESERVED.
## Copyright (C) UT-Battelle, LLC. 2014-2019. ALL RIGHTS RESERVED.
## Copyright (C) ARM Ltd. 2017-2021.  ALL RIGHTS RESERVED.
##
## See file LICENSE for terms.
##
#

## Current
### Features:
#### RDMA CORE (IB, ROCE, etc.)
 * UCX_IB_PCI_RELAXED_ORDERING=yes now requires all memory keys to use relaxed ordering and omits strict-order companion keys; the former "only" value is removed
### Bugfixes:

## 1.22.0 (August 2, 2026)
### Features:
#### UCP
 * Added SGL datatype support for non-blocking PUT operations
 * Added GET and PUT rendezvous protocols for RMA operations
 * Added fault-tolerance recovery foundation
 * Added AM, zcopy, multi, and PSN protocol failover support
 * Added endpoint flush failover and first-fragment retransmission support
 * Added multi-lane support for device operations
 * Added memory registration flags to UCP/UCT/UCS APIs
 * Enabled v2 operations on proxy and wireup endpoints
 * Added transport and wireup debugging table dumps during context and endpoint creation
 * Set four rails by default for Vera platforms
#### UCT
 * Added UCT v2 capability flags for SGL zcopy operations
 * Enhanced endpoint connectivity checks
#### RDMA CORE (IB, ROCE, etc.)
 * Added plugin infrastructure for IB token query and plugin-provided RC extended operations
 * Added per-endpoint RC transmit queue reservation for resource checks
 * Added IB/MLX5 initiator small-fence WQE control flag
 * Applied ECE returned by ibv_query_ece by default
 * Enabled relaxed ordering by default for Vera CPU platforms
#### CUDA
 * Added GDRCopy PCIe BAR1 export option
 * Added POSIX file descriptor handle support for same-machine CUDA VMM IPC
 * Added support for CUDA device aliasing with MPS MLOPart
#### UCS
 * Added table builder debug API
 * Added ucs_string_buffer_vappendf utility
 * Added numeric range support for device configuration
 * Added Vera CPU support
#### UCM
 * Propagated memory attributes in allocation events
#### Tools
 * Added software GET responder progress support in perftest
#### Build
 * Added PR build modes
 * Added Ubuntu 26.04, TencentOS 4.4, and GCC 15.2 build coverage
 * Added GB300/aarch64 DL cluster test pipeline coverage
 * Added parallel static checks
 * Enabled on-demand DRP testing for release pipelines
 * Added configure option to append a SONAME suffix
### Bugfixes:
#### UCP
 * Fixed endpoint fault-tolerance discard and failover flows
 * Fixed fallback handling when no more lanes are available
 * Fixed AM reply endpoint lookup during initialization
 * Fixed memory release crash in active-message path
 * Fixed rcache validity check before memory handle invalidation
 * Fixed context creation when only accelerator devices are available
 * Fixed error reporting when the transport/device resource limit is exceeded
 * Fixed RMA rendezvous rkey size assertion
 * Fixed bandwidth formatting macros
 * Fixed single network-device filtering to use minimum distance
 * Fixed protocol assertions and short selection diagnostics
 * Fixed zero-length memory handle packing in trace logging
 * Fixed rendezvous remote-memory flag estimation from rkey memory-domain maps
 * Disabled RMA rendezvous protocols on non-Vera platforms
 * Fixed active-message lane selection to preserve latency priority
 * Fixed rendezvous multi-lane protocol selection to preserve combined minimum sizes
 * Fixed unnecessary rendezvous PUT fences on ordered transports
#### ZE
 * Fixed ZE driver enumeration, PCI fallback, IOV bounds, and DMA-BUF memory query semantics
 * Fixed ZE memory query system-device detection
#### CUDA
 * Fixed CUDA managed-memory Valgrind failures
 * Fixed CUDA IPC remote-cache destruction during endpoint destroy
 * Fixed CUDA cleanup for destroyed contexts
 * Fixed CUDA IPC single-process unmap checks
 * Fixed CUDA NVLink detection
 * Fixed CUDA IPC remote-cache handling for GET and PUT paths
 * Fixed CUDA IPC reachability for same-process interfaces
 * Fixed CUDA copy memory-flag detection with active CUDA contexts
#### RDMA CORE (IB, ROCE, etc.)
 * Fixed GGA GET zcopy purge completion
 * Fixed GDA DMA-BUF offsets and on-demand DMA-BUF checks
 * Fixed GDAKI Direct NIC matrix selection
 * Fixed IB DEVX handling for UARs without WC support
 * Fixed RoCE reachability for RTN_LOCAL routes
 * Reverted ARM Neon BlueFlame writes in IB transport
 * Fixed MLX5 DDP PUT fencing
 * Fixed IB port-speed query logging for unsupported devices
 * Fixed Direct NIC sibling matching when HCAs have no active ports
 * Fixed DONT_FORK handling for kernels that support RDMA copy-on-fork
 * Fixed GGA MMO fence ordering
 * Restricted EMR relaxed ordering to device memory
#### Shared Memory
 * Fixed POSIX reachability failure logging in containers
#### TCP
 * Fixed TCP negative connect-test timeout
#### Packaging
 * Fixed GDA RPM/DEB packaging and RPM development package builds
#### UCS
 * Fixed stale async file descriptor event filtering
 * Fixed IPv6 scope formatting
 * Fixed time initialization to be multi-thread safe
 * Fixed namespace identifier sizing for portability
 * Fixed portability issues for non-glibc and clang environments
#### Tools
 * Fixed perftest ZE allocator thread safety and device routing
 * Fixed I/O demo data-size range parsing
#### Build
 * Fixed GPUNetIO update failure reporting
 * Fixed CMake policy version for examples tests
 * Replaced ofed_info usage with native package-manager queries
 * Fixed CI build result reporting on exceptions
 * Excluded .ci/ changes from PR pipeline triggers
### Known issues:
 * JUCX package for this release was renamed to
   `1.22.0-1`, please avoid using `1.22.0`.

## 1.21.0 (June 24, 2026)
### Features:
#### UCP
 * Added UCX_PROTO_EMULATION_ENABLE option to force zero-copy RMA protocol selection
 * Added UCX_MAX_HCA_PER_GPU policy to limit GPU memory registrations to nearest HCAs
 * Added device lanes that can access host memory for GPU transfer fallback
 * Enabled gdr_copy for memtype endpoint transport
#### UCT
 * Added device channel pool support
 * Added CPU memory usage as AMO local buffer for device operations
#### RDMA CORE (IB, ROCE, etc.)
 * Added UCX_IB_GDA_RETAIN_INACTIVE_CTX option to control inactive CUDA context retention in GDAKI
 * Added configure option to enable or disable GGA transport
#### Build
 * Added --without-gda configure option
 * Made cuRAND an optional dependency for perftest CUDA kernels
#### CI/Testing
 * Added dry-run package installation checks to the release package build
### Bugfixes:
#### Build
 * Fixed support for -Og by disabling always-inline attributes
#### UCP
 * Fixed progress counter to return the actual operation status
 * Fixed multi protocol minimum size handling for 1-byte operations
 * Fixed endpoint finalization when no P2P or connection-manager lane is available
#### UCT
 * Fixed notify callback handling by adding a NULL check
#### CUDA
 * Fixed CUDA IPC accessibility cache separation for local and remote rkeys
 * Fixed CUDA IPC cache/LRU invariant for referenced regions
 * Fixed DMA-BUF offsets for interior CUDA addresses
#### ROCM
 * Fixed hangs in HIP MPI and OMB tests
 * Fixed endpoint flush for in-flight ROCm operations
#### RDMA CORE (IB, ROCE, etc.)
 * Fixed GDA DMA-BUF offset handling
 * Fixed GDA WQE ordering by using CAS-based readiness marking
 * Fixed GDAKI CUDA context handling during endpoint creation
 * Fixed GDAKI NIC/GPU mapping when CUDA_VISIBLE_DEVICES hides physical GPUs
### TCP
 * Fixed interface selection by skipping IPv4 link-local addresses
#### UCS
 * Reverted dynamically loaded external module/plugin support
#### Packaging
 * Fixed Debian maintainer field
 * Fixed GDA RPM build
 * Fixed GDA RPM/devel package layout for CUDA/GDA subpackages
 * Fixed RPM/DEB handling when GDA is disabled

## 1.20.1 (May 6, 2026)
### Features:
#### RDMA CORE (IB, ROCE, etc.)
 * Added 'auto' option for UCX_IB_MLX5_DEVX_OBJECTS which disables DevX when ODP is available (for Grace)
 * Prioritize routes with longer subnet masks for improved reachability check accuracy
#### Documentation
 * Clarified that user buffer can be modified after calling ucp_atomic_op_nbx
### Bugfixes:
#### Build
* Fixed IB configuration const correctness for strchr() to allow compilation with GCC 15.2.1
#### UCP
 * Increased TLS info buffer size in transport selection to prevent potential truncation
 * Fixed incorrect warning about valid environment variable names
 * Fixed ucp_config_modify not reporting an error when no matching modifiable configuration exists.
#### RDMA CORE (IB, ROCE, etc.)
 * Fixed DevX objects flag handling
 * Fixed device memory allocation alignment in MLX5 DevX
 * Fixed IB memory handle flags enum order
 * Disabled indirect atomic registration for Direct NIC
 * Fixed stale destination endpoint ID and acks from before connection reset in UD transport
 * Fix RoCE reachable route check when node_guuid is not unique among HCAs
#### CUDA
 * Fixed CUDA context handling for system device during rkey unpack
#### ROCM
 * Fixed HSA memory type check for newer ROCm releases
#### UCS
 * Fixed rcache locking for GDR copy
#### Packaging
 * Fix libnvidia-compute removal from ucx-cuda debian package dependencies, breaking existing installation
 * Obsoleted KNEM sub-package
 * Fix maintainer field in debian packaging

## 1.20.0 (December 2, 2025)
### Features:
#### UCP
 * Added new GPU device API for direct GPU-to-GPU communication
 * Added host API for GPU device management
 * Added device signaling API with cooperation levels and flags
 * Added API for working with offsets and channel id in device operations
 * Added method to write to local counter in device operations
 * Added local and remote address fields to memory list element in device API
 * Added device lane selection and allocated handle population
 * Added support for Direct NIC (DPU) data path with CUDA
 * Added rkey packing support for Direct NIC
 * Added sender flush mechanism when memory sys_dev differs from remote lane sys_dev
 * Added option to use single network device per protocol
 * Added MIN_RMA_CHUNK_SIZE configuration parameter
 * Decreased default value for MIN_RMA_CHUNK_SIZE from 16k to 8k
 * Improved protocol lane selection with find_lanes callback to minimize overhead
 * Improved send-zcopy latency factor for fast-completion cases
 * Improved multi-ppn performance estimation
 * Removed deprecated ucp_mem functions
 * Deprecated ucp_request_alloc API
#### UCT
 * Added new device API for GPU communication (rc_gda transport)
 * Added GDAKI transport with endpoint export to GPU
 * Added DEVX QP/CQ support on foreign memory
 * Added device API implementation for CUDA_IPC transport
 * Added device put multi, put partial, and atomic operations for CUDA_IPC
 * Added peer failure error handling capability for GDAKI
 * Added check for nvidia_peermem driver when using GDA transport
 * Enabled Direct NIC by default for IB transport
 * Added XDR performance recognition
 * Added support for mapping DMA_BUF handle via PCIe for Direct NIC
 * Improved GDR_COPY performance with fast-path cache lookup
#### RDMA CORE (IB, ROCE, etc.)
 * Added ConnectX-9 device support
 * Split dp_ordering flag for DV/DevX transports
 * Added VRF tables support for RoCE reachability check
 * Added EFA-specific GPUDirect support detection
#### TCP
 * Added routing table check during reachability verification
#### UCS
 * Introduced lightweight rwlock data structure
 * Added built-in atomics for rcache rwlock
 * Improved VFS symlink paths and duplicate object handling
 * Disabled error signal interception by default
#### CUDA
 * Added wrappers for NVML functions
 * Added hook for cuLibraryGetGlobal
 * Improved CUDA call logging
 * Improved source/destination memory type detection for lane performance estimation
 * Removed unsafe usage of cuCtxGetId
 * Added support for cuCtxCreate_v4 for newer CUDA versions
 * Improved context management for CUDA_IPC operations
#### UCM
 * Changed module info print to debug level by default
#### Tools
 * Added GDAKI kernel option to perftest
 * Added UCP cuda device tests to perftest
 * Added MPI+CUDA example
 * Differentiated wakeup feature and extra info options in perftest
#### Build
 * Added ability to build CUDA device code for supported architectures
 * Added ucx.spec into tarball for Universal Build System support
 * Added CUDA 13 support
 * Added GDA build failure when gpunetio not found
#### Packaging
 * Moved driver level dependencies under Recommends section in Debian packages
 * Added Provides field for upstream packages in Debian
 * Migrated JUCX publish from OSSRH to Central Portal
 * Added ib-mlx5-gda separate package
#### CI/Testing
 * Added Rocky OS support to release pipeline
 * Added RHEL 10 containers to build matrices
 * Added Debian 13 to CI build stage
 * Added ARM build testing
 * Switched to MOFED 25.07
 * Switched GPU tests to Ubuntu 24.04 DOCA 3.1 (GPUNetIO) image
 * Added support for nvidia_peermem module in testing
 * Disabled Valgrind in CI Tests stage
 * Disabled tag matching offload tests
#### GO Bindings
 * Made go bindings thread safe
#### Documentation
 * Added note about reachability check mode in README
 * Mentioned nvlink as supported transport
 * Documented return status for device APIs
#### AWS EFA
 * Added RMA WRITE operations support
 * Added flush and fence operations for SRD
 * Enabled EFA SRD support in tests
### Bugfixes:
#### UCP
 * Fixed fallback to blocking registration for network device only
 * Fixed flush_state validity check before using it
 * Fixed single net dev filtering for single proto
 * Fixed rkey size estimation for rendezvous
 * Fixed memory invalidation without RNDV
 * Fixed gather_pending_requests to execute only when reconfig occurs
#### UCT
 * Fixed CUDA_IPC protocol selection for cuda_ipc
 * Fixed GDA compilation issues
 * Fixed GDAKI wqe_idx overflow
 * Fixed MM FIFO room calculation for tail > head case
 * Fixed CUDA_IPC indices handling in put partial
 * Removed DOCA runtime dependency from GDAKI
 * Fixed GDA log spam by reducing DOCA log level
 * Fixed UAR support check when querying resources for GDA/MLX5
 * Fixed crash in GGA transport when EXPORTED_MKEY flag is missing
#### CUDA
 * Fixed stack overflow bug when calling cuPointerGetAttribute
 * Fixed mapping of DMA_BUF handle for Direct NIC
 * Returned object to mpool in case of failure in CUDA_COPY
 * Reduced log level of rkey unpacking failures
 * Handled cuMemRelease error status properly
 * Fixed context setting for local buffer in CUDA_IPC
 * Fixed host unregister error message (changed to diagnostic)
 * Fixed CUDA_IPC header installation
#### RDMA CORE (IB, ROCE, etc.)
 * Fixed RoCE network device name reading
 * Fixed Direct NIC related issues
 * Reverted RC EP address size adaptation without flush_rkey
#### UCS
 * Fixed ARCH header inclusion when building with nvcc (arm_neon.h)
 * Fixed VFS symlink path handling
 * Fixed netlink message receiving to continue until 'done' flag is set
#### Build
 * Fixed NVCC search with explicit --with-cuda
 * Fixed ZE transport build failures
 * Fixed ucs_arch_get_cpu_flag compilation
 * Fixed CUDA device code build for supported architectures
#### Testing
 * Fixed test_jenkins CI issues
 * Decreased rwlock test duration
 * Fixed error counting in gtest
 * Enabled retries for test_arch.memcpy
 * Fixed test_cuda_nvml condition relaxation
 * Skipped build when generating packages
 * Fixed CUDA device restoration in tests
 * Improved error detection in UCP device tests
 * Fixed global topo state cleanup during gtest
#### Tools
 * Fixed perftest CUDA kernel issues
#### GO Bindings
 * Fixed go bindings compilation with CUDA
#### IB/EFA
 * Fixed error message when FLID is not available
#### Packaging
 * Fixed RPM SPEC debug_package macro execution on SLES16

## 1.19.1 (Sep 18, 2025)
### Features:
#### UCP
* Do not require transport memory support if rendezvous protocol is not used
#### Build
* Added CUDA 13 support to the release pipeline
* Added Rocky OS support to the release pipeline
### Bugfixes:
#### UCS
* Fixed Netlink fetch mechanism

## 1.19.0 (June 18, 2025)
### Features:
#### UCP
* Enabled multi-GPU support within a single process
* Added dynamic selection between strong and weak fences in RMA flush operations
* Improved endpoint reconfiguration capabilities
* Added All2All lane selection for multi-NIC-GPU systems
* Improved rkey debug info when config cache limit is reached
* Improved UCP protocol selection based on available memory types
* Removed dummy memory key from irrelevant transports (TCP, CMA and CUDA)
* Improved RNDV performance with device-local staging buffers
* Enabled error handling for RMA get_offload protocols
#### UCT
* Defined uct_rkey_unpack_v2 API to support passing sys-dev
#### RDMA CORE (IB, ROCE, etc.)
* Added SRD transport support in EFA with reordering, AM, and control operations
* Removed XGVMI BF2 support (umem)
* Removed device memory indirect key
* Fixed VFS objects for DCIs and pools
* Added routing table cache to the reachability check
* Fixed strict order usage in IB auxiliary rkeys
* Improved various init logging messages
#### CUDA
* Added multi-context support for remote key unpacking to CUDA IPC
* Added context switching aware resource management to CUDA IPC
* Use buffer ID to detect VA recycling in CUDA IPC
* Added support for allocating CUDA memory on specific system devices
* Added multi-device support in CUDA copy
* Improved protocol lane selection for GPU memory operations
* Relaxed CUDA context requirements in CUDA copy
* Added deadlock prevention in CUDA copy
* Added support for address range detection for VMM
* Enabled memory attributes query after switching CUDA GPU
* Added multi-GPU send tests for CUDA transports
* Removed host-to-host performance estimation from CUDA copy transport
* Replaced cuCtxCreate by cuDevicePrimaryCtxRetain
* Improved various init logging messages
#### ROCM
* Added control parameters for IPC handle cache and signal pool size
* Optimized ROCm memory type detection with caching
#### UCS
* Removed compilation warnings
#### Tools
* Added name filter option (-F 'str') to ucx_info for config and feature dumps
* Improved ucx_info input validation
### Bugfixes:
#### UCP
* Made UCX_TLS=^ib disable all transports including auxiliary
* Fixed send request status handling
* Fixed performance degradation in RNDV by optimizing md cache updates
* Fixed protocol selection when first lane is filtered out by fragment size
* Fixed rkey selection by using memory registration flag
#### UCT
#### RDMA CORE (IB, ROCE, etc.)
* Improved reliability of DC transport by adding DCI validation and separating connection logic
* Fixed segfault in DC fence operation
#### GPU (CUDA, ROCM)
* Updated ROCm configuration for ROCm 6.3 compatibility
* Fixed system device detection for CUDA async memory operations
* Fixed legacy type detection during CUDA IPC mpack
* Fixed CUDA IPC RMA operations by using correct context for local buffers
#### UCS
* Use UCS function for counting leading zeros on x86 architecture
* Fixed a compilation warning
#### Shared Memory
* Fixed FIFO availability check for sm transport
#### Documentation
* Fixed open-mpi clone instruction
#### Build
* Fixed enum-int-mismatch warnings with GCC 15

## 1.18.0 (January 17, 2025)
### Features:
#### UCP
 * Enabled using CUDA staging buffers for pipeline protocols by default
 * Added endpoint reconfiguration support for non-reused p2p scenarios
 * Enabled non-cacheable memory domains, activated for gdr_copy
 * Added user_data parameter to ucp_ep_query
 * Added support for host memory pipeline through CUDA buffers for rendezvous protocol
 * Added global VA infrastructure and memory region in absence of error handling
 * Made protocol performance node names more informative
 * Enforced always running on the same thread in single thread mode
 * Multiple improvements in protocols selection infrastructure
 * Added UCP_MEM_MAP_LOCK API flag to enforce locked memory mapping
 * Allowed up-to 64 endpoint lanes for systems with many transports or devices
 * Added usage tracker to worker
 * Improved various logging messages
#### RDMA CORE (IB, ROCE, etc.)
 * Added environment variable to manage DC initiator capacity
 * Added DC dcs_hybrid policy
 * Reduced MLX5/DV stack size consumption
 * Added ODP support for verbs and mlx5dv
 * Added support of CUDA managed memory on IB when ODP is available
 * Added support of Adaptive Routing on RoCE
 * Enabled use of implicit ODP with relaxed ordering
 * Improved GPU-Direct detection in IB transport
 * Increased DC initiator default count to 32 for performance optimization
 * Added ConnectX-8 device support with DDP
 * Added support for subnet filter list for RoCE interfaces
 * Enhanced the error message to provide more details when a connection cannot be established due to unreachable transports
 * Added IB MLX5 as a separate UCX module with separate RPM sub-package
 * Added initial support for GGA transport, for fast DPU memory access
 * Set IB DevX atomic mode based on device capabilities
 * Removed DC keepalive mechanism, since the keepalive is done on UCP layer
 * Optimized cross-gVMI memory registration using indirect memory keys cache
 * Improved various logging messages
#### CUDA
 * Added multi-node NVlink support
 * Added CUDA Fabric memory support with detection and allocation
 * Improved gdr_copy latency estimations on AMD Milan systems
 * Added check for gdr_copy runtime/build version mismatch
 * Added handling missing IPC capability when unpacking keys
 * Added caching for CUDA IPC memory pool import operation
 * Added gdr_copy variables to optimize performance on Grace Hopper systems
 * Improved CUDA IPC concurrency for a larger count of reachable peers
#### UCS
 * Added support for wildcards in configuration parameter names
 * Added ASAN protection to several internal data structures
 * Reduced stack usage in topology detection code
 * Improved bitmaps configuration parsing with wider bitfield
 * Added options to set topology distance between devices
 * Optimized VFS unix socket watch by using user private folder
 * Added general IP subnet matching infrastructure
 * Extend array data structure to support user-provided array copy routine
 * Improved time units description
#### UCM
 * Extend CUDA memory hooks to include memory mapping APIs
#### Tools
 * Improved performance by increasing window size for put_bw and add get_bw in ucx_perftest
 * Added multi-send flag for receive operations in bandwidth benchmarks in ucx_perftest
 * Improved ucx_perftest uni-directional test with added fence
 * Detailed ucx_perftest batch section of command-line documentation
#### Documentation
 * Added a section regarding adaptive routing on RoCE
#### Architecture
 * Added CPU Model for MI300A
 * Added Fujitsu ARM specific values to ucx.conf
 * Added AMD Turin support
 * Added an optimized non-temporal memory copy implementation for AMD CPU
#### Build
 * Improved compiler error reporting with added flag
 * Improved coverity script to allow faster turnaround time
 * Improved Intel Compiler detection and support
#### GO
 * Added multi-send flag and user memh support in request params
#### Packaging
 * Improved dpkg-buildpackage sample command by explicitly adding mlx5 related arguments
### Bugfixes:
#### UCP
 * Fixed stack overflow in exported rkey unpack
 * Removed extra remote-cpu overhead from protocol estimation for zcopy
 * Fixed performance estimation for rndv pipeline protocols
 * Fixed ATP sending by picking the correct lane
 * Fixed missing reg_id on memh creation
 * Fixed repeated invalidations by retaining existing access flags
 * Fixed abort reason propagation for rendezvous RTR mtype
 * Do not check transport availability if it is disabled by UCX_TLS environment variable
 * Fixed wrong flag being used for checking BCOPY capability
 * Fixed sending too many ATPs for small messages
 * Enforced 16 bits size for Active Messages identifiers
 * Fixed unnecessary status check for emulated AMO
 * Fixed more than one fragment sending in rendezvous pipeline
 * Fixed crash by using biggest max frag across all lanes
 * Fixed missing memory handle flags by copying from parent to child
 * Fixed worker interface activate count
 * Fixed flush requests by replacing ATP/flush lane map with lane indexes
 * Fixed lost uct_flags when merging memory regions
#### UCT
 * Fixed memory domain UCT flags description
#### RDMA CORE (IB, ROCE, etc.)
 * Fixed FETCH_ADD remote access error for ODP/KSM case
 * Fixed missing conditional compilation checks for DM
 * Fixed IB MD allocation naming typo
 * Fixed invalid GIDs filter in IB
 * Fixed flags usage in MLX5 zcopy_post
 * Do not limit ODP registration retries
 * Fixed JUCX failures by considering the number of supported completion vectors
#### CUDA
 * Fixed async memory handling using CUDA memory type on Grace
 * Added rcache overhead in performance estimation
 * Fixed gdr_copy performance regression by providing maximum estimation between get and put
 * Fixed CUDA IPC reachability check
 * Fixed crash in MPI_Finalize when CUDA context is destroyed
 * Always require rcache by default for gdr_copy
 * Fixed crash in gdr_copy cleanup when registration cache is disabled
 * Fixed CUDA copy memory domain allocations
 * Fixed multiple tests for gdr_copy transport
 * Fixed race condition in CUDA IPC peer accessible cache
#### UCS
 * Fixed a crash by using heap allocation to process expired timers in batch
 * Fixed allocation issue on memtrack dump
 * Fixed deletion of the monitored folder in VFS
 * Fixed unsafe resize for DC initiator array
 * Fixed function macro invocation to match C standard
 * Fixed calling async handler on already released resource
 * Fixed performance by setting higher bandwidth for different NUMA nodes on Grace
 * Fixed undeclared value error in timer conversion routine
 * Fixed uninitialized value access in registration cache
#### UCM
 * Fixed race condition in parsing proc maps
 * Fixed mremap failure while parsing /proc/self/maps
#### ROCM
 * Fixed ROCM interface reachability test
 * Fixed memory domain fork test
#### TCP
 * Always bind endpoint to interface
#### Tools
 * Fixed buffer size potential overflow in ucx_perftest
 * Fixed missing address when packing memory keys on ucx_perftest
 * Fixed memory leak for endpoint report in ucx_info
 * Fixed build without openmp in ucx_perftest
 * Fixed UCT device override on server side on ucx_perftest
#### Build
 * Fixed using correct ASAN version for running tests
#### Configuration
 * Used POSIX bourne syntax to check equality
 * Fixed build failure by using proper flags in compiler.m4
 * Fixed perftest MAD support default guessing
#### GO
 * Added serialized thread mode to avoid subtle races between threads
 * Fixed make distcheck

## 1.17.0 (June 13, 2024)
### Features:
#### UCP
* Improved the accuracy of rendezvous protocol performance estimation
* Enabled short protocol for non-host memory types on empty messages
* Improved the accuracy of performance estimation for empty messages by removing non-relevant overheads
* Added RMA_ZCOPY_MAX_SEG_SIZE configuration parameter to allow modifying segment size for RMA-ZCOPY protocols
* Added support for separate intra/inter-node rendezvous thresholds
* Added support for minimal fragment size in rendezvous protocol
* Added support for resetting request during send operation
* Added UCX_PROTO_OVERHEAD configuration variable to allow setting protocol overheads
* Improved performance for combined Active Message/RMA scenarios by separating them to different lanes
* Added support for device staging buffers in pipeline protocols
* Enabled on-demand paging for Nvidia's Grace platforms by default 
#### RDMA CORE (IB, ROCE, etc.)
* Introduced the UCX_REVERSE_SL environment variable to configure reverse SL for DC transport. By default, it uses UCX_IB_SL.
* Added support for GID auto-detection in Floating LID based routing
* Added support for multithreading KSM registration of unaligned buffers
* Added IB_SEND_OVERHEAD and MM_[SEND|RECV]_OVERHEAD configuration variables
#### GPU (CUDA, ROCM)
* Added support for oneAPI Level-Zero library for Intel GPUs
#### UCS
* Added support for rcache dynamic region alignment
* Added dynamic bitmap data structure
* Added support for advanced key-value parsing for UCX configuration
* Added piecewise linear function data structure
* Added support for allocating dynamic arrays on stack 
#### Tools
* Added support for device memory allocation in UCX perftest
* Added a script to use for squashing commits after PR approval
* Added support for DPU cross-gvmi daemon in UCX perftest
#### Java
* Added support for EP local socket address API in JUCX
#### Build
* Added address sanitizer support
* Added a helper shell script to run static checks
#### AZP
* Replaced Valgrind tests with address sanitizer tool
* Added Ubuntu 22.04 docker image testing
#### Configuration
* Added support for filtering configuration sections by platform type
* Added configuration file with section for Grace Hopper
### Bugfixes:
#### UCP
* Fixed crash due to incorrect lane selection when active message is disabled
* Fixed RMA lane selection issue due to wrong bandwidth calculation 
* Fixed rendezvous protocol information in protocol details table
* Fixed endpoint reconfiguration issue due to wrong bandwidth calculation
* Fixed Active Message handlers issue due to out of order registration
* Fixed registration of memh evens for imported memory key
* Fixed sockaddr unreachable destination error handling
* Fixed uninitialized memory issue in new protocols infrastructure
* Fixed race condition when using strong fence by flushing all endpoints
* Fixed incorrect RMA message size on immediate completion with no datatype
* Fixed incorrect performance estimation due to fp8 pack/unpack issue
* Fixed remote access error when rcache memory is not registered with atomic access
* Fixed assertion failure when rcache fails during memh allocation
* Fixed atomic device selection issue
* Fixed worker interface deactivation while still in use by endpoints
* Fixed wire compatibility issue due to mismatched lane selection
#### RDMA CORE (IB, ROCE, etc.)
* Disabled device memory if atomics are not available
* Fixed indirect keys creation for MT registered memory
* Fixed KSM start address value when creating export key 
* Fixed DCI pool index to support maximum of 16 pools
* Fixed atomic rkey issue when using imported memory
* Fixed crash due to unsupported SRQ capability
#### GPU (CUDA, ROCM)
* Removed unused environment variable RCACHE_ADDR_ALIGN from ROCm transport
* Fixed usage of cuda device 0 when no context is active
* Removed error handling support from CUDA IPC transport
* Fixed allocation of unaligned CUDA memory
#### Shared Memory
* Fixed occasional crash when shm_unlink fails during interface initialization
#### UCS
* Fixed system device distance calculation for devices on different PCIe root
* Fixed support for large size arrays in ucs_array
* Fixed synchronization issue in rcache
* Fixed uninitialized variable access in rcache
#### Tests
* Fixed test failures when GPU is present but disabled
* Fixed Active Message hanging issue in ucp_client_server
* Fixed potential crash due to redundant munmap call in ucp mmap tests
* Fixed a crash when running CUDA gtest under valgrind
* Fixed UD endpoint timeout issue under Valgrind
#### Java
* Fixed failures in Java tests by waiting for send requests completion
* Fixed JVM segfault in Java tests when gdrcopy driver is not loaded
* Fixed go build and go tests failures
#### Packaging
* Disabled Go bindings in Debian package

## 1.16.0 (April 15, 2024)
### Features:
#### UCP
* Added tag offload rendezvous protocol in new infrastructure
* Added rcache to old protocols infrastructure
* Added multi-fragment protocols for stream API in new infrastructure
* Enabled new protocols infrastructure by default
* Removed context param from ucp_memh_put
* Added assertion if trying to register unsupported memory type
* Adjusted rendezvous latency to improve scalability
* Improved endpoint configuration logging information
* Added check for max length of user defined Active Message header
* Added rcache support for mem type memory registration
* Enabled error handling for rndv/put_zcopy protocol
* Enabled v2 as default client/server connection establishment packet version
* Enabled rendezvous protocol selection for reachable MDs only
* Added ucp_rkey_compare API to enable rkey comparison
* Added release version to worker address to enable wire compatibility
* Added support for memory invalidation for rendezvous through DC transport
* Enabled the use of strong fence with new protocols infrastructure
#### UCT
* Added UCS_MEMORY_TYPE_RDMA memory type for better latency on supported devices
* Implemented is_reachable_v2 API for IB transport
* Added ep_is_conntected API
#### RDMA CORE (IB, ROCE, etc.)
* Added Floating LID(FLID) based routing support
* Added latency and min_zcopy configuration variables to ROCm-IPC
* Added support for indirect MR for cross-gvmi mkey instead of direct MR with DEVX UMEM
#### TCP
* Added filter for eliminate bridge devices from lane selection
#### GPU (CUDA, ROCM)
* Added support for handling memh with multiple registrations
* Added performance estimation BW based on GPU type
* Adjusted rocm/ipc latency and zcopy threshold parameters
* Improved error message when libnvidia-ml not installed
* Added profiling to Cuda runtime API calls
* Adjusted gdr_copy estimated BW to improve protocol selection
#### Shared Memory
* Adjusted FIFO_SIZE to improve scalability
* Removed redundant rcahce implementation in knem transport
* Added support for symmetric rkey to improve memory usage 
#### UCS
* Improved scalability of connection establishment flow
* Improved memtype cache performance by replacing ptrhead_lock to spinlock
* Added support for VLAN over channel bonding interface
* Added LRU cache and Usage Tracker datastructures
* Improved cross-NUMA device detection
* Added support for PCIe gen5 bandwidth detection
#### Build
* Added LCOV coverage report as a build option
* Added binutils 2.40 library dependencies
* Added development modulefile
#### Tools
* Added information about sizes of ucp_request_t fields in ucx_info
* Added ucx env to profiling output
* Added MAD RTE in ucx_perftest to support setups without IPoIB
#### Tests
* Added GTEST_LOG_LEVEL env var to set log level just before test run
* Disabled protov1 and ud_verbs tests for valgrind mode
* Reduced gtest execution time
#### Documentation
* Added a few details to coding style
### Bugfixes:
#### UCP
* Reverted wireup latency calculation which caused lanes selection issue
* Fixed strong fence to always ensure ordering
* Fixed registration of memh for RNDV protocol
* Fixed rndv_put and rkey_ptr assertion failure
* Fixed performance estimation for multi-fragment protocols
* Fixed memory registration error handling
* Fixed buffer overflow of large log messages
* Fixed progress enabling for selected lanes
* Fixed atomic lanes progress enabling
* Added missing rendezvous schemes to environment variable documentation
* Fixed bcopy BW estimation for AMD 
* Fixed lanes information printing for new protocols infrastructure
* Fixed rndv_am protocol thresholds
* Fixed fp8 packing issue
* Fixed Intel OneAPI compilation error
* Fixed CM address packing on server side
* Fixed endpoint reconfiguration issue due to asymmetrical selection
* Fixed asymmetrical selection due to wire compatibility issue
* Fixed potential deadlock with cuda_copy and RTR protocol
* Fixed tag_recv return value on immediate completion  
* Fixed memory corruption by proper memh handling in tag offload rendezvous
* Changed default allocator to not use reserved huge pages
* Fixed rndv put protocol to avoid early completion
* Fixed rndv_put transport selection for device to device scenario
* Disabled rendezvous pipeline protocol selection when using non-contiguous buffer
* Fixed crash in rendezvous protocol rkey pack after failed memory registration
#### RDMA CORE (IB, ROCE, etc.)
* Fixed compilation failure when DevX is explicitly disabled 
* Fixed crash when using PCIe relaxed ordering
* Fixed remote access error with rc_verbs transport
* Fixed endpoint address management in unified mode
* Fixed assertion failure when configured with UCX_IB_ADDR_TYPE=ib_global
* Fixed overwritten MD attribute capabilities when querying a device
* Fixed ibv_reg_mr error by registering memory in rcache callback
* Disabled MR multithreading registration
* Fixed mlx5 WQE posting error due to compiler memory copy optimizations
#### TCP
* Fixed asymmetric lanes selection issue due to inconsistent device listing 
#### GPU (CUDA, ROCM)
* Fixed compilation flags to support ROCm 6.0
* Fixed values of D2H_THRESH and latencey params
* Fixed Cuda memory support for iov datatype
* Increased max number of agents in ROCm
* Fixed cuda_ipc transport being disabled if a CUDA device is not set during initialization
#### Shared Memoey
* Fixed posix and cma transport selection by enhancing reachability checks
* Fixed UGNI build failure
* Fixed latency overhead for knem and cma transports
* Fixed possible out-of-order issue in mm_iface
#### UCS
* Fixed a deadlock when forked debugger is attached during an error in rcache operation
* Fixed crash due to passing null pointer to log function
* Fixed crash due to incorrect hashing method
* Fixed crash in configuration parser cleanup by moving it after profiler cleanup
* Fixed floating point division by zero during protocols initialization
#### UCM
* Fixed occasional crash in bisto hooks by adding a lock before hooking
* Fixed compilation error when building on PPC64
#### Java
* Fixed go tests by setting CUDA device before allocating CUDA memory
* Fixed perftest error detection and hanging issue
#### Tools
* Fixed cpu model type for AMD Genoa in ucx_info
* Enhanced multi-thread test output
#### Build
* Fixed JUCX package publishing, so it will include support for ARM
* Fixed ROCm building and testing
* Removed libnvidia-compute version dependency
* Removed libibmad/libumad from default build configuration to avoid runtime dependency
#### Packaging
* Fixed already existing target error when using cmake find_package(ucx) twice

## 1.15.0 (September 28, 2023)
### Features:
#### UCP
* Added 2-stage pipeline protocol in the new protocol infrastructure
* Added reset and abort functionality of rendezvous protocols in the new infrastructure
* Added zero-copy rendezvous data send protocol in the new infrastructure
* Added support for user memory handle in the new protocol infrastructure
* Added option to force ODP registration for certain memory types
* Enabled lock free memory region deregistration
* Updated allow/deny transport list feature to control auxiliary transport selection
* Multiple performance improvements of the new protocol infrastructure
* Multiple improvements in error and debug messages
#### UCT
* Split UCT_MD_MKEY_PACK_FLAG_INVALIDATE into two flags for RMA and AMO
* Added put_zcopy and get_zcopy scheme support for self transport
* Added base implementation of is_reachable_v2 API using intra/inter flag
* Introduced MD capability for non-blocking registration memory types
#### RDMA CORE (IB, ROCE, etc.)
* Added implementation of is_reachable_v2 routine to IB interface
* Added option to control CQE zipping per CQ RX/TX direction
* Added option to specify how DCI selects port under RoCE LAG
* Added hw_dcs to the list of policies to select DCI by an endpoint
* Removed implicit on-demand paging
* Added option to set RoCE lag dct port for response under queue affinity mode
* Improved IB memlock limit logging
#### UCS
* Added ucs_string_buffer_rbrk() to split token
#### GPU (CUDA, ROCM)
* Added support for atomic reply_buffer on GPU memory
* Added system device information for AMD GPUs
* Improved performance estimation of gdr_copy transport
* Added a simplistic implementation of performance estimation of cuda_ipc transport
* Improved performance estimation of cuda_ipc on Hopper architecture
* Added rcache parameters for rocm transports
* Introduced dmabuf support for rocm transports
* Implemented asynchronous progress for the zcopy operations in the rocm_copy transport
* Added option to enable using cross-device dmabuf file descriptor for rocm
#### Java
* Added Java bindings for exported memh feature
#### Tests
* Added a rocm docker container for testing
* Added option to send client_id in iodemo test
* Added support for multiple connections to the same server in iodemo test
* Added synchronization before exit to hello world examples
#### Tools
* Added user-side memcpy option for AM benchmarks in ucx_perftest
* Added wireshark LUA dissectors for some UCX protocols
#### Build
* Added support for binutils 2.40
* Added versioned dependency to switch between packages with the same names
* Added a separate xpmem deb subpackage
* Added aarch64 support to the binary distribution pipeline
* Removed dependency on libnuma
### Bugfixes:
#### UCP
* Fixed assertion when sending from non-contiguous GPU buffer to managed buffer
* Fixed the race condition on endpoint configurations
* Fixed endpoint reconfiguration issues due to asymmetrical selection
* Fixed endpoint reconfiguration error due to wrong locality detection
* Fixed crash during connection manager cleanup
* Fixed rkey index calculation for rendezvous protocol
* Fixed rcache dump function
* Removed logging from rkey unpack in release mode
* Fixed dobule free of rkey in rendezvous protocol
* Fixed rendezvous pipeline protocol error flow
* Fixed error handling in rendezvous get zcopy protocol
* Replay pending requests of wireup EP CM during connection establishment to prevent potential ordering issues and wrong configuration
* Pass user-provided memory type to the function that checks whether the buffer can be sent inline or not
* Avoid memory registration during UCP context initialization
* Fixed CPU/device atomics selection in the new protocol infrastructure
* Multiple fixes in the new protocol infrastructure information output
#### UCT
* Added check for dmabuf kernel support in ROCm memory domain
* Fixed exported memh packing
* Fixed an error in checking return status of multi-threaded memory registration function
#### RDMA CORE (IB, ROCE, etc.)
* Fixed dma-buf based memory region registration
* Fixed memory handle data corruption when PCIe relaxed ordering is enabled
* Fixed performance degradation when indirect atomic key is not supported by the hardware
* Fixed remote access error to strict-order keys because of wrong offset
* Added check for UAR support to memory domain opening
* Fixed updating port counters for devx qp
* Fixed ibv_create_cq error message on node without Infiniband
* Fixed performance degradation due to using 2 paths on NDR400 by default
* Removed unnecessary async lock which otherwise would block UD progress
#### GPU (CUDA, ROCM)
* Fixed CUDA IPC performance degradation due to libnuma removal
#### UCS
* Fixed lane selection and added bandwidth estimation for Sapphire Rapids family
* Fixed displaying wrong environment variable suggestions
* Fixed VFS warning output
* Fixed SEGV in ucs_debug_backtrace_next(), upon previous SEGV handling, due to ENOMEM situation
* Fixed memory corruption when using UCX_MPOOL_FIFO=y
#### UCM
* Fixed conditional jump patching
* Fixed mremap() override
#### GPU (CUDA, ROCM)
* Fixed usage of dmabuf when the buffer is not page-aligned
* Removed async_cb from cuda_copy to avoid the issue with UCP worker async lock
#### Java
* Fixed leakage of jucx_request global references
#### Documentation
* Updated ucp_worker_release_address description
#### Tests
* Fixed wrong usage of ep_close in examples
#### Tools
* Fixed memory access flags in perftest
* Removed support for librte from perf
* Fixed worker flush deadlock when using multiple workers in ucx_perftest
#### Build
* Changed 'unsupported option' ICC command line warning to error
* Removed never used fault-injection configuration option
* Fixed obsolete macro warnings in new autoconf/libtool
* Fixed building UCX with GCC 13
* Fixed UCX RPM build on machines that have libxpmem-devel rpm from MLNX_OFED installation
* Fixed ucx-rdmacm package requirements
* Fixed compilation errors with armcc-22.1
* Fixed passing port number to goperftest

## 1.14.1 (May 22, 2023)
### Bugfixes:
* Fixed ROCm to prevent the locking of host pinned memory
* Added CUDA 12 based UCX builds to the release flow
* Increased the maximal number of endpoint configurations
* Fixed filter for a slow-lanes in selection logic
* Fixed TCP transport bandwidth calculation
* Fixed device detection for ROCM
* Fixed compatibility with CUDA 12
* Fixed rendezvous threshold for multi-path configurations
* Fixed error message in case of static link
* Fixed BlueField-3 detection
* Multiple fixes for Azure CI pipeline

## 1.14.0 (March 13, 2023)
### Features:
#### Core
#### UCP
* Added API for querying transport and device names on endpoint
* Added API for querying datatype object
* Added API for exporting and importing memory keys (no implementation yet)
* Added support for non-persistent active message header
* Added infrastructure to print protocols v2 performance
* Multiple performance improvements for protocols v2
* Added support for non-contiguous datatypes for rendezvous protocols v2
* Added support for reset and abort request in protocols v2
* Added support for user memory handles in RMA API
* Added multi-rail support for RMA API in protocols v2
* Added support for up to 16 different lanes per endpoint
* Added support for dmabuf memory registration in protocols v2
* Added strong fence mode for ucp_worker_fence() API
#### UCT
* Added new uct_md_mem_attach() API to support exported memory handles
* Added remote completion mode for endpoint flush (via new flag)
* Added support for dmabuf registration
* Added new uct_ep_connect_to_ep_v2() API
* Added new uct_mem_reg_v2() API
* Added new uct_md_query_v2() API
* Added support for IPv6 loopback address in TCP transport
#### RDMA CORE (IB, ROCE, etc.)
* Added ECE (enhanced connection establishment) support for RC and DC transports
* Added support for hardware DCS in DC transport
* Added UD interface and endpoint resource information to VFS
* Added CQ creation via DEVX API
* Removed support for accelerated IB transports over legacy experimental verbs
#### UCS
* Added support for auto-correction of user environment variables
#### UCM
* Implemented CUDA bistro hooks for aarch64 (to enable memory cache on this platform)
* Added support for CUDA virtual/stream-ordered memory with cudaMallocAsync
#### GPU (CUDA, ROCM)
* Implemented uct_iface_estimate_perf() function for ROCM
* Removed obsoleted ROCM gdr transport
* Added support for hsa async_copy for short operations in ROCM
* Added memory allocation functions in ROCM
#### Java
* Added methods for ucp_worker_arm() and ucp_worker_get_efd()
#### Documentation
* Added FAQ for using pkg-config tool to build applications with UCX
#### Tests
* Added prints of latency per connection in io_demo
#### Tools
* Added runtime library version to the 'ucx_info -v' output
* Added support for memory types in ucx_info

### Bugfixes
#### UCP
* Multiple fixes in keepalive protocol
* Multiple fixes and improvements in UCP rcache flows
* Fixed endpoints leak by disabling resolving remote endpoints in certain cases
* Multiple fixes and cleanups in wireup protocol and lanes selection flows
* Multiple fixes in protocols v2 infrastructure
* Fixed worker interface initialization taking atomic caps into account
* Fixed UCP AM max payload value calculation for protocols v2
* Fixed deadlock in rcache when UCX_LOG_LEVEL set to debug
* Fixed lanes weight calculation in rendezvous protocol v2
* Fixed user memory handle support in rendezvous protocol
* Fixed message split in rendezvous protocol to avoid having very small chunks
* Improved performance estimations for protocols v2
* Fixed receive descriptors leak in UCP AM rendezvous
#### UCT
* Fixed double free of server endpoint in TCP sockcm
* Updated KNEM bandwidth to be dedicated resource rather than shared
* Fixed race in CM when listener is destroyed during conn_req_cb invocation
* Updated default bandwidth value for memory mapper transports
* Disqualify posix transport if /dev/shm size is too small
* Disqualify KNEM transport if memory registration fails with it
* Fixed cuda detection (when cuda headers are not present, but nvml headers are)
#### RDMA CORE (IB, ROCE, etc.)
* Fixed device error handling (prevent coredump when iface is down/up)
* Multiple fixes in DC transport (error flows, flow control, etc)
* Multiple fixes and cleanups in UD transport
* Fixed MR registration (avoid atomic offset breaking region alignment)
* Fixed indirect key registration (avoid creating atomic KSM on top of relaxed-order key)
* Fixed thread domain usage for accelerated verbs transports
* Added print of a particular syndrome on DEVX function failures
* Fixed DEVX QP creation by setting proper ts_format attribute
* Decreased size of DC endpoint
* Fixed bandwidth calculation for RoCE LAGs
* Fixed port counters setting for DEVX QPs
* Fixed compile errors on SLES sp3
* Removed errors during md open in case of strict memlock limit
#### UCS
* Removed async_max_events limit (e.g. to support many concurrent TCP connections)
* Updated memory wc flush using DGH hint for ARM platform
* Fixed deprecation warnings because of <sys/fcntl.h> includes
* Added default bandwidth value for ZHAOXIN CPU
#### UCM
* Fixed segfault in malloc when compiled with -flto
#### GPU (CUDA, ROCM)
* Updated cuda_copy transport to use event fd instead of async callback
* Fixed ROCM IPC transport (use remote agent if available)
* Fixed clang compilation errors in CUDA copy transport
* Fixed ROCM memtype detection
* Improved performance estimation of CUDA copy transport
* Fixed send to self flows in ROCM
#### Documentation
* Updated GPU memory support section in FAQ
#### Tests
* Multiple fixes and improvements in unit tests
#### Tools
* Fixed MPI RTE send deadlock in ucx_perftest
#### Build
* Build Debian package with multi-thread support
* Fixed configure warning by using POSIX compliant sh syntax
* Multiple fixes for Debian package build
* Dropped support for Ubuntu16

## 1.13.1 (September 7, 2022)
#### Bugfixes
* Fixed flow control protocol in DC transport
* Fixed reordering of pending operations in DC transport
* Fixed relaxed order detection in IB transports
* Fixed build configuration and IB ops references
* Fixed bandwidth calculation during wireup phase
* Fixed TCP transport server port selection
* Minor fixes in CI testing

## 1.13.0 (July 7, 2022)
#### Features
##### Core
* Added new objects to VFS: local and remote address of endpoint, statistics of ucp_ep_create success/failure, failed/destroyed endpoints
* Added support for UCX static libraries
* Added profiling for rkey management routines
* PCIe relaxed order enabled by default for AMD CPUs
#### UCP
* Added API to pass pre-registered memory handle to UCP operations
* Added implementation of AM rendezvous protocol
* Added 2-stage pipeline rendezvous protocol for GPU
* Added support for fragment mem_type for v1 pipeline proto, disabled by default
* Added active message support for proto v2
* Added UCP memory registration cache
* Improved adaptive progress - deactivate iface when all p2p lanes are destroyed
* Added support for user memh in proto_v1
* Added support for selecting local address when creating a client endpoint
* Added option to limit GPUDirectRDMA size in rendezvous protocol, UCX_RNDV_MEMTYPE_DIRECT_SIZE
* Deprecated UCX_SOCKADDR_AUX_TLS configuration parameter
#### UCT
* Introduced API uct_md_mkey_pack_v2
* Introduced UCT iface features API
* Introduced max_inflight_eps parameter in perf_attr API
* Introduced UCT_SEND_FLAG_PEER_CHECK flag that forces checking connectivity to a peer
* Introduced UCX_RCACHE_PURGE_ON_FORK to enable/disable cleaning regions when application is forking
#### RDMA CORE (IB, ROCE, etc.)
* Introduced NDR autorecognition
* Introduced CQE zipping support
* Set the default MAX_RD_ATOMIC to maximum value supported by the hardware
#### ROCM
* Increased maximum number of HSA agents
#### UCS
* Added topo module infrastructure
* Added memtrack and rcache information to VFS
#### Tools
* Added support for pre-registered memory in ucx_perftest
* Added loopback transport support for UCT perf tests
### Bugfixes
#### Core
* Fixed not deallocating memory from ucp_mem_unmap if no rcache
* Fixed versioning infrastructure
* Multiple code improvements: refactoring, debug prints and assertions, etc.
* Multiple improvements in build, test and docs infrastructure
#### UCP
* Resolving remote EP ID when creating local EP disabled by default
* Multiple fixes in keepalive protocol
* Fixed initialization request send state if software RMA/AMO in use
* Fixed error handling in RMA and BW lanes selection logic
* Fixed CM wireup fallback
* Fixed occasional crash in finalize
* Fixed AM proto flags
* Fixed single zcopy proto initialization for AM
* Fixed proto v2 selection, take into account user header length
* Fixed selecting auxiliary transports when creating EP for sending EP_REMOVED
* Fixed printing invalid configuration
* Fixed allocation of indirect remote ID for internal EP if connected EP supports PEER_FAILURE
* Fixed memh allocation when no rcache
* Fixed protocol selection logic for UCP AM send
* Fixed error handling flow for EP discard requests from pending queue
* Fixed EP destroy flow
* Fixed rsc_index for prereg_md_map
* Fixed wireup error handling flow Create EP which send WIREUP_MSG/EP_REMOVED with AM lane only
* Fixed probe for multi-fragment eager
* Fixed alignment for AM rdesc init
* Fixed perf estimation for proto v2
* Fixed CM wireup with proto v2
* Fixed EP discard flow during fast-forward
* Fixed datatype issue in TAG send
* Fixed EP refcount overflow
* Fixed EP error handling flow
* Fixed wire compatibility in address unpacking
* Fixed ucp_ep_close_nb for failed endpoint when related requests have registered memory that should be invalidated
* Fixed fragmented proto v2
* Fixed UCP address v2 packing/unpacking and usage of seg_size
* Fixed purge requests on failed endpoint
* Fixed error handling of connecting p2p lanes during WIREUP phase
* Fixed UCP endpoint use after free
#### UCT
* Fixed ABI break of uct_ep_params_t
* Fixed common intra-node keepalive protocol
* Fixed a typo UCT_PERF_ATTR_FIELD_REMOTE_SYS_DEIVCE -> UCT_PERF_ATTR_FIELD_REMOTE_SYS_DEVICE
* Fixed potential crash on MD mem alloc
* Disabled PEER_FAILURE capability for XPMEM
* Updated TCP iface bandwidth calculation taking into account PCI bandwidth
#### RDMA CORE (IB, ROCE, etc.)
* Fixed 2G aligned MR registration
* Fixed FC_HARD_REQ resending
* Fixed remote access to invalidated MR
* Fixed max_rd_atomic_dc value for DV
* Fixed DC handshake logic
* Fixed error handling flows
* Fixed flush(CANCEL) with UD and DC transports
* Fixed multi-path handling for passive endpoint with UD transport
* Fixed attributes for DV QP creation
* Fixed device query
* Fixed memory leak in case of disabling RDMA transport
* Fixed dci->pool_index initialization
* Fixed fallback if port speed not detected
* Fixed tag offload recv for inlined data
* Fixed PKEY index initialization
* Disabled mlx5 ifaces on verbs MD
#### TCP
* Fixed flush(CANCEL)
* Fixed close protocol when UCT EP pairs have only RX capability
* Fixed query local/remote saddr
#### GPU (CUDA, ROCM)
* Fixed a bug in invalidating address range in CUDA_IPC
* Fixed CUDA context caching and cleanup
* Fixed ROCM initialization
* Fixed ROCM components compilation
* Fixed IPC tls reachability check
* Fixed ROCM memory type detection
* Use ROCM remote_agent if available
* Fixed CUDA module compilation with clang 13
* Fixes in ROCm memory detection and performance estimation
#### KNEM
* Fixed memory registration cost
#### UCM
* Fixed potential hang on init
#### UCS
* Fixed name shadow problem in CentOS6.x
#### Tools
* Print stream API limits and handle stream feature in ucx_info
* Replaced ucp_ep_close_nb by ucp_ep_close_nbx in examples
* Replaced completed field by checking UCS status in io_demo
#### JAVA
* Throw exception if ucp_mem_query failed
#### GO
* Disabled go bindings in rpmbuild
* Fixed configure behavior if can't find go compiler
* Standalone performance benchmark
* Increased port range + make it dependent on agent_id
* Check compiler minimum version
* Set GOCACHE to a local directory that is cleared for each job in CI
* Disabled module for goperftest
* Fixed OOS build

## 1.12.1 (March 21, 2022)
#### Bugfixes
* Fixed memory hooks for Cuda 11.5
* Fixed memory type cache merge
* Fixed continuously triggering wakeup fd when keepalive is used
* Fixed memtype cache fallback when memory hooks are not installed
* Fixed parsing header flags of worker address
* Fixed pipeline protocol when sending from host memory to GPU memory
* Fixed transport progress not deactivated when all transport's connections are closed
* Fixed progress loop in io_demo application
* Fixed ROCm segfault when using internal_ops functions
* Fixed ROCm memory hooks
* Fixed performance regression on A64FX
* Fixed DCT create failure with rdma-core v22
* Fixed golang bindings build
* Fixed .deb package build on Ubuntu 22.04
* Fixed build on archlinux

#### Important changes
* If Cuda memory hooks on driver API cannot be installed, memory type cache and
  memory registration cache will be disabled. This may lead to lower performance
  of some applications on setups with NVIDIA GPUs, even if Cuda memory is not
  being used. Prior to this change, failing to install driver API hooks could
  lead to runtime errors or data corruption when Cuda memory is used and linked
  statically with cuda runtime.
  In order to revert to previous behavior (when the application is linked
  dynamically with cuda runtime), the user can set UCX_MEM_CUDA_HOOK_MODE=reloc.
  See more info in https://github.com/openucx/ucx/pull/7865.

## 1.12.0 (January 12, 2022)
### Features:
#### Core
* Added beta-level support for Go language bindings
* Added new objects to VFS (md, component, log_level, etc.)
* Added configuration variable to specify which loadable modules are allowed
* Added build-time configuration to disable sigaction overriding
#### UCP
* Added client_id to ucp_worker_create() and ucp_conn_request_query() APIs
* Added ucp_worker_address_query() API
* Updated ucp_ep_query() API for getting local and remote addresses
* Added address versioning to correctly preserve wire compatibility starting from version 1.11.0
* Added new client/server connection establishment packet header format
* Enabled rendezvous and tag sync protocols when error handling is enabled on the endpoint
* Added iov zcopy support to RMA operations
* Reduced memory usage of unexpected messages by fitting receive buffer size to packet size
* Added support for modifying UCT and UCS configs by ucp_config_modify() API
* Optimized unpacked rkeys memory consumption
* Added request flag to influence latency vs. bandwidth protocol
* Reduced memory management overhead with new protocols
* Improved performance calculations for new protocols
* Added AMO support with GPU memory target using new protocols
* Added put_zcopy, get_zcopy and pipeline based rendezvous in new protocols
* Added support for user-defined alignment in Active Messages
* Added support for offload tag sync in new protocols
* Updated ucp_atomic_post() to use NBX flow
#### UCT
* Added API - uct_iface_is_reachable_v2()
* Added IPv6 address support in TCP
* Added latency estimation to uct_iface_estimate_perf()
* Adjusted knem and cma overhead cost
* Increased built-in TCP keep-alive interval to 2 seconds
#### RDMA CORE (IB, ROCE, etc.)
* Added detection of IB NDR devices
* Added check for CQ overrun in assert mode
* Added bitmap usage for releasing detached DCIs
* Added configuration for requests ack frequency with DevX
* Added remote QP info to tx error CQE traces
#### UCS
* Added API for a per-process aggregate-sum statistics report
* Added memory pool set data structure
* Added new ptr_array API for bulk allocation
* Added ucs_string_buffer_append_flags() for string buffer
* Added ucs_ffs32()
* Added ucs_vsnprintf_safe() which always adds '\0'
* Added thread-safe put to ptr_map
* Improved accuracy of the topology distance estimation
* Added prints of leaked callbacks from the callback queue
* Removed a diagnostic message when fuse thread is stopped
* Added configurable limit for the memory consumed by rcache
* Added configuration for VFS(FUSE) thread affinity
* Added memory limit support to memtrack
#### CUDA
* Added global memtype cache to allow UCT transports to query memory attributes
* Auto-register CUDA whole allocations to avoid repeated registration costs
* Added capability to select CUDA stream based on source and destination memory type
  (required for device memory based pipelining)
* Added selection of CUDA-IPC capabilities based on NVLINK topology
  (to prefer writes vs. reads for specific platforms using NVML)
* Added option to set cuda_copy bandwidth
* Added profiling of CUDA runtime function calls
* Added option to limit GPUDirectRDMA size in rendezvous protocol
#### Java
* Added ucp_listener_reject functionality
* Added support for setting worker id and querying it from the connection request
* Added support to bind on a free port in UcpListener
#### Packaging
* Added cmake config files for better integration with external cmake based projects
#### Tests
* Removed memcpy from AM eager flow in io_demo
* Added check_qps.sh script to detected stuck QPs
* Improved diagnostic in test_init_mt
* Added iov support in ucp_client_server
* Added option to use epoll in io_demo
* Added registration of memory allocated by io_demo in memtrack
* Extended statistics in io_demo
* Improved logging in io_demo
* Replaced rand by urand in io_demo
* More improvements in io_demo
* Generalized median calculation to support any percentile in ucx_perftest
#### Tools
* Added loop-back transport support in ucx_perftest
* Split ucx_perftest into separate modules
* Added process placement option for ucx_info
* Extended parameters correctness check in ucx_perftest
* Added support for GPU memory RMA and atomics in ucx_perftest
#### CI
* Updated gtest 1.7 to 1.10
* Increased uptime in network corrupter (used for io_demo)
* Enabled set of gtests for new protocols
* Added running CI in docker containers
* Increased thresholds for test_ucp_wait_mem
* Added test for ucx binary compatibility between OS versions
* Increased test job timeout to 6 hours
* Reduced testing time under valgrind
* Added suppressions for glibc and libnl leaks
* Relaxed performance requirements in perf test

### Bugfixes
#### Core
* Fixed invalid remote memory access after connection error
* Fixed creating more than 64K endpoints between the same peers
* Fixed simultaneous endpoint close with ucp_hello_world
#### UCP
* Fixes and improvements in new protocols infrastructure
* Fixes in AM flows
* Fixed tag short threshold selection
* Multiple fixes in keep-alive protocol
* Multiple fixes in wire-up protocol
* Fixes in error flow during rendezvous protocol
* Multiple fixes in general error flow
* Fixed fallback to PUT pipeline in rendezvous protocol
* Reduced default value of keep-alive interval to 20 seconds
* Fixes in tag_send datatype processing
#### UCT
* Fixed keep-alive protocol for intra-node transports (sm, cuda)
* Fixed deadlock in TCP
* Suppressed EHOSTUNREACH error in TCP sockcm
* Restricted connecting loop-back to other devices in TCP
#### RDMA CORE (IB, ROCE, etc.)
* Fixed pkey_index initialization when creating RC QP with DEVX
* Disabled MP_SRQ by default
* Fixed TX WQ overflow check
* Fixed dci->pool_index initialization when HAVE_DC_DV is false
* Fixed syndrome value for creating rdmacm reserved qpn
* Fixed error code on rdma_establish failure
* Fixed uct_ep_am_short_iov for UD verbs
* Fixed handling of error CQE after rc_ep is destroyed
* Fixes in flow control when error CQE is polled
* Multiple fixes in RC and DC error flows
* Fixed deadlock between DCIs and RDMA_READ credits
* Removed AM handler invocation for PURE_GRANT messages
* Fixed endpoint arbiter_group leak in DC
* Fixed resource check in flush for DC
#### UCS
* Fixed segmentation fault for ucs_stats_parser
* Fixed potential crash on cleanup when use UCX profiling
* Fixed read_profile print of new request
* Fixed uninitialized variable access in VFS
* Changed log level of inotify_init failure to diag
* Fixed integer overflow in mpool chunk allocation
#### Packaging
* Fixed with-fuse arg for RPM build
#### Documentation
* Fixes in UCP, UCT, UCS, FAQ and README documentation
#### Tests
* Multiple fixes in io_demo
#### CI
* Fixed snapshot docker name
* Fixed hipMallocManaged hook gtest
* Fixes in Azure release pipeline
* Fixes in Coverity CI
* Fixed test_uct_query gtest for ROCm
* Fixes in jenkins test script
* Fixed release commit title check

## 1.11.2 (September 30, 2021)
### Bugfixes
* Fixes in Java release pipeline
* Fixes in handling large number of devices
* Fixes in UD out-of-order processing
* Fixes in switching transports during client/server connection setup
* Fixes in transport-level error reporting

## 1.11.1 (August 31, 2021)
### Features:
#### UCS
* Added API to read boot ID value or use machine_guid

### Bugfixes:
* Fixes in Cuda memory hooks
* Fixes in setting traffic class for DCT RoCE transport
* Fixes in TCP endpoint flush
* Fixes in TCP pending operations progress
* Fixes in release pipelines
* Fixes in error handling flow
* Fixes in multi-threaded tag probe
* Fixes in TCP disconnect flow
* Fixes in RPM post-install script
* Fixes in UCT common keepalive

## 1.11.0 (July 26, 2021)
### Features:
#### Core
* Added support for UCX monitoring using virtual file system (VFS)/FUSE
* Added support for applications with static CUDA runtime linking
* Added support for a configuration file
* Updated clang format configuration
#### UCP
* Added rendezvous API for active messages
* Added user-defined name to context, worker, and endpoint objects
* Added flag to silence request leak check
* Added API for endpoint performance evaluation
* Added API - ucp_request_query
* Added API - ucp_lib_query
* Ported connection manager to a new UCT API
* Added bandwidth optimizations for new protocols multi-lane
* Added support for multi-rail over lanes with BW ratio >= 1/4
* Added support for tracking outstanding requests and aborting those in case of connection failure
* Refactored keep-alive protocol
* Added device id to wireup protocol
* Added support up to 128 transport layer resources in UCP context
* Added support CUDA memory allocations with ucp_mem_map
* Increased UCP_WORKER_MAX_EP_CONFIG to 64
* Adjusted memory type zcopy threshold when UCX_ZCOPY_THRESH set
* Refactored wireup protocols, rendezvous, get, zcopy protocols
* Added put zcopy multi-rail
* Improved logging for new protocols
* Added system topology information
* Added new protocols for eager offload protocols
#### UCT
* Extended connection establishment API
* Added active message AM alignment in iface params
* Added active message short IOV API.
* Added support for interface query by operation and memory type
* Added API to get allocation base address and length
* Added md_dereg_v2 API
#### UCS
* Added log filter by source file name.
* Added checking for last element in fraglist queue
* Added a method to get IP address from sockaddr.
* Added memory usage limits to registration cache
#### UCM
* Improved x86 parser to recognize some mov flavors
#### CUDA
* Added registration for whole CUDA allocations
* Added CUDA-IPC keepalive
* Adjusted performance estimations
* Added Improve logging
* Added allocation methods for CUDA pinned/managed memory
* Added support for a global cuda_ipc cache
#### RDMA CORE (IB, ROCE, etc.)
* Added report of QP info in case of completion with error
* Refactored of FC send operations
* Added support for  DevX unique QPN allocation
* Optimized endpoint lookup for  DCI
* Added support for RDMA sub-function (SF)
* Added support for DCI via DEVX
* Added DCI pool per LAG port
* Added support  for RoCE IP reachability check using a subnet mask
* Added active message short IOV for UD/DC/RC mlx, UD/RC verbs
* Added endpoint keep alive check for UD
* Suppressed warning if device can't be opened
* Added support for multiple flush cancel without completion
* Added ignore for devices with invalid GID
* Added support for SRQ linked list reordering
* Added flush by flow control on old devices
* Added support for configurable rdma_resolve_addr/route timeout
#### Shared memory
* Added active message short IOV support for posix, sysv, and self transports
#### TCP
* Added support for peer failure in case of CONNECT_TO_EP
* Added support for active message short IOV
#### Java
* Added full support for UCP Java API
#### Tests
* Added length/mem_type for UCP client server example
* Added port sockaddr tests for a new API
* Added test send-recv between client/server with diff UCX_IB_NUM_PATHS
* Added support for CUDA and CUDA managed memory in io_demoo
* Added support for a custom watchdog timeout from command line
* Extended  memtype hook tests
#### Tools
* Added UCP active message support to perftest
* Added error handling option to perftest
* Added wakeup option
* Added performance tests for am short iov
#### CI
* Added RHEL 7.6 with MOFED 4.7
* Added Fedora 34, RHEL 7.2, 7.4
* Added PGI support from HPC-SDK module
* Added docker image with CUDA 11.2
* Added IODEMO test
* Added Ubuntu 20.4
* Added test for connection manager fallback in client-server testing
* Added loopback interface for tcp testing

### Bugfixes:
#### Build
* Fixes in libnuma detection macro
* Fixes for cross compilation support
* Fixes for --without-dc compilation
#### Continues Integration
* Fixes in Azure pipeline build system
* Fixes in Coverity CI
* Fixes in Azure release pipeline
#### Packaging
* Fixed in DEB package - added essential system dependencies
#### Documentation
* Fixes in UCP, UCT, Readme, FAQ, and Read-the-docs documentation
#### Tests
* Fixes in CMA peer failure test
* Fixes in SRQ tests
* Fixes in the usage requests_wait
* Fixes in test_uct_query
* Fixes addressing race conditions on client user data in test_uct_sockaddr
* Fixes in IODEMO app
* Fixes in error handling flow for perftest
* Fixes in perftest batch tests
* Fixes addressing hang issues for rendezvous protocol in UCP client server example
#### UCP
* Fixes in endpoint error handling
* Fixes in error reporting failed CM lanes
* Fixes in progress worker flush
* Fixes in rendezvous pipeline flow
* Fixes in recursive protocol selection
* Fixes in error handling for AM_ZCOPY
* Fixes in length check condition in RMA PUT short
* Fixes in failure handling rendezvous offload send
* Fixes in offload completion with inlined data
* Fixes in statistics calculations for rendezvous protocol
* Fixes in ucp_worker_query() thread mode for SERIALIZED
* Fixes preventing leaks of UCP requests
#### ROCM
* Fixes in device memory registration and de-registration
* Fixes in missing mem_query definition for rocm_copy
* Fixes addressing build failure due to const violation
* Fixes in sockaddr_accessibility test for rocm_copy and rocm_ipc
* Fixes in bandwidth estimation for rocm_ipc
#### RDMA CORE (IB, ROCE, etc.)
* Fixes addressing deadlock between DCI resources and RDMA_READ credits
* Fixes in DSCP for RoCE DCT
* Fixes in flush(cancel) flow
* Fixes preventing segfault in uct_rdmacm_cm_ep_str
* Fixes in scatter-gather entries logging
* Fixes for compilation with experimental verbs
* Fixes in UD dgid filtering
* Fixes in domain resources destroying
* Fixes in PCIe bandwidth calculation
* Fixes addressing CQ creation failure using legacy ibv API
* Fixes in iov2sge converter
* Fixes in port width check on HDR100
* Fixes in SL selection
* Fixes in hardware tag matching compilation
* Fixes in uct_rdmacm_cm_cqs hash key
* Fixes for compilation with rdma-core 20
#### Java
* Fixes in tag sender mask
#### UCT
* Fixes in reachability of loopback ifaces
* Fixes addressing  possible uninitialized memory accesses
* Fixes in error flow for endpoints created upon receiving connection request
* Fixes in TCP keepalive to avoid false-positive error detection
#### UCM
* Fixes addressing heap corruption caused by ucp_set_event_handler()
* Fixes in mmap events test

## 1.10.1 (May 12, 2021)
### Bugfixes:
* Fixes in Infiniband port speed detection for HDR100
* Fixes in building gtest-all.cc and sock.c with GCC11
* Fixes addressing performance degradation with cuda memory on a self endpoint
* Fixes in JUCX listener connection handler
* Fixed in configuration of loopback TCP transport (disable by default)
* Fixes in RPM dependency on libibverbs
* Fixes in ABI backward compatibility for active message protocol
* Fixes in the DC transport - adding support for full-handshake mode (off by default)
* Fixes in Active Messages short reply protocol
* Fixes for segmentation fault while listening for connections

## 1.10.0 (March 9, 2021)
### Features:
#### Core
* Added support for Nvidia HPC SDK
* Added support for latest PGI and Clang
* Added support for ROCM-3.7+ (warning generated if older version detected)
* Added support for GCC11
#### Architecture
* Added Arm SVE memcpy()
* Redesigned Arm WFE support
* Improved clear_cache performance for Arm
* Added architecture detection for Zhaoxin CPU
#### CI
* Added release builds on CUDA 11
* Enabled performance validation in gtest
* Added new OS for release CI
#### UCP
* Added locality awareness to the transport selection logic for GPU devices
* Added put/offload/short and put/offload/zcopy protocols
* Added receive message nbx routine
* Reworked AM implementation and API, which adds support for RNDV semantics
* Added support for multi-lane connection manager over TCP
* Added support for printing AM tls with info log level
* Implement flush and destroy for UCT EPs on UCP worker
* Reduced UCP request size
* Added support for keepalive protocol
* Added support for multi-fragment protocol
* Added implementation for protocol progress for eager, bcopy, and multicopy
* Improved selection logic for protocol selection
* Added new protocols for UCP get operation
* Added bcopy protocols with support for GPU memory
* Added RNDV protocol implementation for GPU devices (CUDA, ROCm)
* Set SOCKADDR_CM_ENABLE=y by default
* Added support for fast-path short with new tag protocols
* Added a new parameter to control the CM listener's backlog
* Added support sending AM RTS over short message protocol
* Added support for shared memory multi-lane when CM is used
* Added missing async locks
#### UCT
* Added API for keepalive_timeout value
* Added add uct_completion.status
* Allowed transports to access multiple mem_types
* Removed status arg from uct_completion_callback_t
* Restructured  uct_mem_alloc/uct_md_mem_alloc to use mem_type
* Updated documentation for uct_listener_params
* Lowered the log level for certain network errors
* Added cuda_copy wakeup feature
* Added wakeup support for shared memory
#### UCS
* Added "inf" and "auto" values to time units
* Added on-stack constructors for array and string buffer
* Added ucs_ptr_map_t data structure
* Added bool CSWAP
* Improved logging
* Added optimization for namespace processing
* Fixes for connection matching functionality
#### CUDA
* Added support for global IPC cache
#### RDMA CORE (IB, ROCE, etc.)
* Added support for auto detection of adaptive routing settings
* Added an option to poll TX CQ every progress iteration
* Added local and remote addresses to the reject error message
* Added support for UAR allocation with non-cacheable memory type
* Added support for multiple flush cancel without completion
* Added async events callback support
* Added detection for  ConnectX-6, ConnectX-7 and BlueField-1/2 devices
* Added support for connection matching for UD
* Added a check for AM ordering
* Added better support for non-4K MTU values
#### Java (preview)
* Added support for a different javadoc executable path for different java versions
* Added UCS memory type constants
* Added support build on Java10+
* Added support for io-vector datatype.
* Removed libjucx from packages.
#### Tests
* Added CI for CUDA 11
* Added test_ucp_sockaddr_protocols.stream_short
* Reimplemented tests using NBX API
* Added flush(cancel) test
* Added memory_wait mode to perftest
* Added support for clang 10
* Refactored RMA and atomic tests, add memtype support
* Added test for uct_md_mem_query()
* Added request interrupt support
* Added support for connection manager fallbacks
* Added new ucp request test checking for leaks from the ptr_map
#### Documentation
* Added glossaries

### Bugfixes:
#### Portability
* Fixes in print functions to use format string like PRIx64, etc.
* Fixes for Arm v8 cross compilation support
#### Continues Integration:
* Fixes in Github release flow
* Fixes in docker image
#### Packaging
* Removed deb package dependencies
* Fixes in SPEC to make the RPM relocatable
#### Documentation
* Fixes in documentation for ucp_am_recv_data_nbx
* Fixes in quick start example
* Fixes in installation instruction
* Fixes in updates in author list
#### Tests
* Fixes for failures under valgrind runtime
* Fixes in mmap tests for 0-length RMA
* Fixes in definition of LAST_WQE wait timeout
* Fixes in ROCm for mem_buffer test
* Fixes in test name printing format
* Fixes in tcp_sockcm test
#### UCP
* Fixes in worker cleanup flow
* Fixes in RNDV RTS flow
* Fix in length check condition for RMA PUT short
* Fixes in handling failures from AM Bcopy
* Fix in a release flow of deferred data
* Fixes for invalid ID and handling of status in RNDV
* Fixes in short active message reply protocol
#### CUDA
* Fixes in managed memory support
* Fixes in topology detection
#### RDMA CORE (IB, ROCE, etc.)
* Fixes in assert definitions
* Fixes in printing an error about invalid AM Bcopy length for UD
* Fixes for thread safety support
* Fixes to get ROCE device name according to GID
* Fixes for SL selection
* Fixes in create STRICT_ORDER key
* Fixes addressing performance degradation in UD transport due to excess async events
* Fixes in QP destroy
* Fixes for CQ creation failure using old Verbs API
#### UGNI
* Fixing disable logic in config
* Fixing clang 11 warnings
#### Java
* Fixes in build dependencies
* Fixes in constructing UcpRequest object on error
* Fixes in exception handling on endpoint closure request
* Fixes for segfault in UcpErrorHandler
#### UCP
* Fixes in datatype support for get_zcopy RNDV
* Fixes in connection manager disconnect
* Fixes in assert definitions
* Fixes in completion flow for failed EP
* Fixes in flush error handling flow
* Fixes in latency calculations for wireup protocol
* Fixes in offload completion with inlined data
* Fixes in unpacking flow
* Fixes in error handling for various protocols
#### UCT
* Fixes in flush TX
* Fixes in checks for enabling GPU Direct RDMA
#### UCS
* Fixes for crashes on incorrect value set in config
* Fixes in ptr_array
* Fixes in maximal size for ucs_snprintf_safe()
* Fixes in compilation warning
* Fixes in ucs_aarch64_dsb(_op) definition
#### TCP
* Fixes in default route interface confirmation flow
* Fixes in PUT protocol
* Fixes in max connection limit and improved error reporting
#### UCM
* Fixing crash on prevent unload
* Fixes in libucm_rocm
* Fixes for few racing conditions

## 1.9.0 (September 19, 2020)
### Features:
#### UCX Core
- Added a new class of communication routines '*_nbx' that enable API extendability while
  preserving ABI backward compatibility
- Added asynchronous event support to UCT/IB/DEVX
- Added support for latest CUDA library version
- Added NAK-based reliability protocol for UCT/IB/UD to optimize resends
- Added new tests for ROCm
- Added new configuration parameters for protocol selection
- Added performance optimization for Fujitsu A64FX with InfiniBand
- Added performance optimization for clear cache code aarch64
- Added support for relaxed-order PCIe access in IB RDMA transports
- Added new TCP connection manager
- Added support for UCT/IB PKey with partial membership in IB transports
- Added support for RoCE LAG
- Added support for ROCm 3.7 and above
- Added flow control for RDMA read operations
- Improved endpoint flush implementation for UCT/IB
- Improved UD timer to avoid interrupting the main thread when not in use
- Improved latency estimation for network path with CUDA
- Improved error reporting messages
- Improved performance in active message flow (removed malloc call)
- Improved performance in ptr_array flow
- Improved performance in UCT/SM progress engine flow
- Improved I/O demo code
- Improved rendezvous protocol for CUDA
- Updated examples code

#### UCX Java (API Preview)
- Added support for UCX shared library loading from both classpath and LD_LIBRARY_PATH
- Added configuration map to ucp_params to be able to set UCX properties programmatically

### Bugfixes:
- Fixes for most recent versions of GCC, CLANG, ARMCLANG, PGI
- Fixes in UCT/IB for strict order keys
- Fixes in memory barrier code for aarch64
- Fixes in UCT/IB/DEVX for fork system call
- Fixes in UCT/IB for rand() call in rdma-core
- Fixed in group rescheduling for UCT/IB/DC
- Fixes in UCT/CUDA bandwidth reporting
- Fixes in rkey_ptr protocol
- Fixes in lane selection for rendezvous protocol based on get-zero-copy flow
- Fixes for ROCm build
- Fixes for XPMEM transport
- Fixes in closing endpoint code
- Fixes in RDMACM code
- Fixes in memcpy selection for AMD
- Fixed in UCT/UD endpoint flush functionality
- Fixes in rendezvous staging protocol
- Fixes in ROCEv1 mlx5 UDP source port configuration
- Multiple fixes in RPM spec file
- Multiple fixes in UCP documentation
- Multiple fixes in socket connection manager
- Multiple fixes in gtest
- Multiple fixes in JAVA API implementation

## 1.8.1 (July 10, 2020)
### Features:
- Added binary release pipeline in Azure CI

### Bugfixes:
- Multiple fixes in testing environment
- Fixes in InfiniBand DEVX transport
- Fixes in memory management for CUDA IPC transport
- Fixes for binutils 2.34+
- Fixes in RPM SPEC file and package generation
- Fixes for AMD ROCM build environment

## 1.8.0 (April 3, 2020)
### Features:
#### UCX Core
- Improved detection for DEVX support
- Improved TCP scalability
- Added support for ROCM to perftest
- Added support for different source and target memory types to perftest
- Added optimized memcpy for ROCM devices
- Added hardware tag-matching for CUDA buffers
- Added support for CUDA and ROCM managed memories
- Added support for client/server disconnect protocol over rdma connection manager
- Added support for striding receive queue for hardware tag-matching
- Added XPMEM-based rendezvous protocol for shared memory
- Added support shared memory communication between containers on same machine
- Added support for multi-threaded RDMA memory registration for large regions
- Added new test cases to Azure CI

#### UCX Java (API Preview)
- Added APIs for stream send/recv, tag probe, and connect request handle
- Added Java package (automatically published) to Maven central

### Bugfixes:
- Multiple fixes in JUCX
- Fixes in UCP thread safety
- Fixes for most recent versions GCC, PGI, and ICC
- Fixes for CPU affinity on Azure instances
- Fixes in XPMEM support on PPC64
- Performance fixes in CUDA IPC
- Fixes in RDMA CM flows
- Multiple fixes in TCP transport
- Multiple fixes in documentation
- Fixes in transport lane selection logic
- Fixes in Java jar build
- Fixes in socket connection manager for Nvidia DGX-2 platform

## 1.7.0 (January 19, 2020)
### Features:
- Added support for multiple listening transports
- Added UCT socket-based connection manager transport
- Updated API for UCT component management
- Added API to retrieve the listening port
- Added UCP active message API
- Removed deprecated API for querying UCT memory domains
- Refactored server/client examples
- Added support for dlopen interception in UCM
- Added support for PCIe atomics
- Updated Java API: added support for most of UCP layer operations
- Updated support for Mellanox DevX API
- Added multiple UCT/TCP transport performance optimizations
- Optimized memcpy() for Intel platforms
- Added protection from non-UCX socket based app connections
- Improved search time for PKEY object
- Enable gtest over IPv6 interfaces
- Updated Mellanox and Bull device IDs
- Added support for CUDA_VISIBLE_DEVICES
- Increased limits for CUDA IPC registration

### Bugfixes:
- Multiple fixes in UCP, UCT, UCM libraries
- Multiple fixes for BSD and Mac OS systems
- Fixes for Clang compiler
- Fixes for CUDA IPC
- Fix CPU optimization configuration options
- Fix JUCX build on GPU nodes
- Fix in Azure release pipeline flow
- Fix in CUDA memory hooks management
- Fix in GPU memory peer direct gtest
- Fix in TCP connection establishment flow
- Fix in GPU IPC check
- Fix in CUDA Jenkins test flow
- Multiple fixes in CUDA IPC flow
- Fix adding missing header files
- Fix to prevent failures in presence of VPN enabled Ethernet interfaces

## 1.6.1 (September 23, 2019)
### Features:
- Added Bull Atos HCA device IDs
- Added Azure Pipelines testing

### Bugfixes:
- Multiple static checker fixes
- Remove pkg.m4 dependency
- Multiple clang static checker fixes
- Fix mem type support with generic datatype

## 1.6.0 (July 17, 2019)
### Features:
- Modular architecture for UCT transports
- ROCm transport re-design: support for managed memory, direct copy, ROCm GDR
- Random scheduling policy for DC transport
- Optimized out-of-box settings for multi-rail
- Added support for OmniPath (using Verbs)
- Support for PCI atomics with IB transports
- Reduced UCP address size for homogeneous environments

### Bugfixes:
- Multiple stability and performance improvements in TCP transport
- Multiple stability fixes in Verbs and MLX5 transports
- Multiple stability fixes in UCM memory hooks
- Multiple stability fixes in UGNI transport
- RPM Spec file cleanup
- Fixing compilation issues with most recent clang and gcc compilers
- Fixing the wrong name of aliases
- Fix data race in UCP wireup
- Fix segfault when libuct.so is reloaded - issue #3558
- Include Java sources in distribution
- Handle EADDRNOTAVAIL in rdma_cm connection manager
- Disable ibcm on RHEL7+ by default
- Fix data race in UCP proxy endpoint
- Static checker fixes
- Fallback to ibv_create_cq() if ibv_create_cq_ex() returns ENOSYS
- Fix malloc hooks test
- Fix checking return status in ucp_client_server example
- Fix gdrcopy libdir config value
- Fix printing atomic capabilities in ucx_info
- Fix perftest warmup iterations to be non-zero
- Fixing default values for configure logic
- Fix race condition updating fired_events from multiple threads
- Fix madvise() hook

### Tested configurations:
- RDMA: MLNX_OFED 4.5, distribution inbox drivers, rdma-core 22.1
- CUDA: gdrcopy 1.3.2, cuda 9.2, ROCm 2.2
- XPMEM: 2.6.2
- KNEM: 1.1.3

## 1.5.1 (April 1, 2019)
### Bugfixes:
- Fix dc_mlx5 transport support check for inbox libmlx5 drivers - issue #3301
- Fix compilation warnings with gcc9 and clang
- ROCm - reduce log level of device-not-found message

## 1.5.0 (February 14, 2019)
### Features:
- New emulation mode enabling full UCX functionality (Atomic, Put, Get)
  over TCP and RDMA-CORE interconnects that don't implement full RDMA semantics
- Non-blocking API for all one-sided operations. All blocking communication APIs marked
  as deprecated
- New client/server connection establishment API, which allows connected handover between workers
- Support for rdma-core direct-verbs (DEVX) and DC with mlx5 transports
- GPU - Support for stream API and receive side pipelining
- Malloc hooks using binary instrumentation instead of symbol override
- Statistics for UCT tag API
- GPU-to-Infiniband HCA affinity support based on locality/distance (PCIe)

### Bugfixes:
- Fix overflow in RC/DC flush operations
- Update description in SPEC file and README
- Fix RoCE source port for dc_mlx5 flow control
- Improve ucx_info help message
- Fix segfault in UCP, due to int truncation in count_one_bits()
- Multiple other bugfixes (full list on github)

### Tested configurations:
- InfiniBand: MLNX_OFED 4.4-4.5, distribution inbox drivers, rdma-core
- CUDA: gdrcopy 1.2, cuda 9.1.85
- XPMEM: 2.6.2
- KNEM: 1.1.2

## 1.4.0-rc2 (October 23, 2018)
### Features:
- Improved support for installation with latest ROCm
- Improved support for latest rdma-core
- Added support for CUDA IPC for intra-node GPU
- Added support for CUDA memory allocation cache for mem-type detection
- Added support for latest Mellanox devices
- Added support for Nvidia GPU managed memory
- Added support for multiple connections between the same pair of workers
- Added support large worker address for client/server connection establishment
  and INADDR_ANY
- Added support for bitwise atomics operations

### Bugfixes:
- Performance fixes for rendezvous protocol
- Memory hook fixes
- Clang support fixes
- Self tl multi-rail fix
- Thread safety fixes in IB/RDMA transport
- Compilation fixes with upstream rdma-core
- Multiple minor bugfixes (full list on github)
- Segfault fix for a code generated by armclang compiler
- UCP memory-domain index fix for zero-copy active messages

### Tested configurations:
- InfiniBand: MLNX_OFED 4.2-4.4, distribution inbox drivers, rdma-core
- CUDA: gdrcopy 1.2, cuda 9.1.85
- XPMEM: 2.6.2
- KNEM: 1.1.2
- Multiple bugfixes (full list on github)

### Known issues:
#2919 - Segfault in CUDA support when KNEM not present and CMA is active
intra-node RMA transport. As a workaround user can disable CMA support at
compile time: --disable-cma. Alternatively user can remove CMA from UCX_TLS
list, for example: UCX_TLS=mm,rc,cuda_copy,cuda_ipc,gdr_copy.

## 1.3.1 (August 20, 2018)
### Bugfixes:
- Prevent potential out-of-order sending in shared memory active messages
- CUDA: Include cudamem.h in source tarball, pass cudaFree memory size
- Registration cache: fix large range lookup, handle shmat(REMAP)/mmap(FIXED)
- Limit IB CQE size for specific ARM boards
- RPM: explicitly set gcc-c++ as requirement
- Multiple bugfixes (full list on github)

### Tested configurations:
- InfiniBand: MLNX_OFED 4.2, inbox OFED drivers.
- CUDA: gdrcopy 1.2, cuda 9.1.85
- XPMEM: 2.6.2
- KNEM: 1.1.2

## 1.3.0 (February 15, 2018)
### Features:
- Added stream-based communication API to UCP
- Added support for GPU platforms: Nvidia CUDA and AMD ROCm software stacks
- Added API for client/server based connection establishment
- Added support for TCP transport
- Support for InfiniBand tag-matching offload for DC and accelerated transports
- Multi-rail support for eager and rendezvous protocols
- Added support for tag-matching communications with CUDA buffers
- Added ucp_rkey_ptr() to obtain pointer for shared memory region
- Avoid progress overhead on unused transports
- Improved scalability of software tag-matching by using a hash table
- Added transparent huge-pages allocator
- Added non-blocking flush and disconnect for UCP
- Support fixed-address memory allocation via ucp_mem_map()
- Added ucp_tag_send_nbr() API to avoid send request allocation
- Support global addressing in all IB transports
- Add support for external epoll fd and edge-triggered events
- Added registration cache for knem
- Initial support for Java bindings

### Bugfixes:
- Multiple bugfixes (full list on github)

### Tested configurations:
- InfiniBand: MLNX_OFED 4.2, inbox OFED drivers.
- CUDA: gdrcopy 1.2, cuda 9.1.85
- XPMEM: 2.6.2
- KNEM: 1.1.2

### Known issues:
#2047 - UCP: ucp_do_am_bcopy_multi drops data on UCS_ERROR_NO_RESOURCE
#2047 - failure in ud/uct_flush_test.am_zcopy_flush_ep_nb/1
#1977 - failure in shm/test_ucp_rma.blocking_small/0
#1926 - Timeout in mpi_test_suite with HW TM
#1920 - transport retry count exceeded in many-to-one tests
#1689 - Segmentation fault on memory hooks test in jenkins

## 1.2.2 (January 4, 2018)
### Main:
- Support including UCX API headers from C++ code
- UD transport to handle unicast flood on RoCE fabric
- Compilation fixes for gcc 7.1.1, clang 3.6, clang 5

### Details:
- When UD transport is used with RoCE, packets intended for other peers may
  arrive on different adapters (as a result of unicast flooding).
- This change adds packet filtering based on destination GIDs. Now the packet
  is silently dropped, if its destination GID does not match the local GID.
- Added a new device ID for InfiniBand HCA
- [packaging] Move `examples/` and `perftest/` into doc
- [packaging] Update spec to work on old distros while complaint with Fedora
  guidelines
- [cleanup] Removed unused ptmalloc version (2.83)
- [cleanup] Fixup license headers

## 1.2.1 (August 28, 2017)
### Bugfixes:
- Compilation fixes for gcc 7.1
- Spec file cleanups
- Versioning cleanups

## 1.2.0 (June 15, 2017)
### Supported platforms
- Shared memory: KNEM, CMA, XPMEM, SYSV, Posix
- VERBs over InfiniBand and RoCE.
  VERBS over other RDMA interconnects (iWarp, OmniPath, etc.) is available
  for community evaluation and has not been tested in context of this release
- Cray Gemini and Aries
- Architectures: x86_64, ARMv8 (64bit), Power64

### Features:
- Added support for InfiniBand DC and UD transports, including accelerated verbs for Mellanox devices
- Full support for PGAS/SHMEM interfaces, blocking and non-blocking APIs
- Support for MPI tag matching, both in software and offload mode
- Zero copy protocols and rendezvous, registration cache
- Handling transport errors
- Flow control for DC/RC
- Dataypes support: contiguous, IOV, generic
- Multi-threading support
- Support for ARMv8 64bit architecture
- A new API for efficient memory polling
- Support for malloc-hooks and memory registration caching

### Bugfixes:
 - Multiple bugfixes improving overall stability of the library

### Known issues:
#1604 - Failure in ud/test_ud_slow_timer.retransmit1/1 with valgrind bug
#1588 - Fix reading cpuinfo timebase for ppc bug portability training
#1579 - Ud/test_ud.ca_md test takes too long too complete bug
#1576 - Failure in ud/test_ud_slow_timer.retransmit1/0 with valgrind bug
#1569 - Send completion with error with dc_verbs bug
#1566 - Segfault in malloc_hook.fork on arm bug
#1565 - Hang in udrc/test_ucp_rma.nonblocking_stream_get_nbi_flush_worker bug
#1534 - Wireup.c:473 Fatal: endpoint reconfiguration not supported yet bug
#1533 - Stack overflow under Valgrind 'rc_mlx5/uct_p2p_err_test.local_access_error/0' bug
#1513 - Hang in MPI_Finalize with UCX_TLS=rc[_x],sm on the bsend2 test bug
#1504 - Failure in cm/uct_p2p_am_test.am_bcopy/1 bug
#1492 - Hang when using polling fd bug
#1489 - Hang on the osu_fop_latency test with RoCE bug
#1005 - ROcE problem with OMPI direct modex - UD assertion

## 1.1.0 (September 1, 2015)
### Workarounds:
### Features:
- Added support for AM based on FIFO in `mm` shared memory transport
- Added support for UCT `knem` shared memory transport (http://knem.gforge.inria.fr)
- Added support for UCT `mm/xpmem` shared memory transport (https://github.com/hjelmn/xpmem)

## 1.0.0 (July 22, 2015)
### Features:
- Added support for UCT `cma` shared memory transport (Cross-Memory Attach)
- Added support for UCT `mm` shared memory transport with mmap/sysv APIs
- Added support for UCT `rc` transport based on Infiniband/RC with verbs
- Added support for UCT `mlx5_rc` transport based on Infiniband/RC with accelerated verbs
- Added support for UCT `cm` transport based on Infiniband/SIDR (Service ID Resolution)
- Added support for UCT `ugni` transport based on Cray/UGNI
- Added support for Doxygen based documentation generation
- Added support for UCP basic protocol layer to fit PGAS paradigm (RMA, AMO)
- Added ucx_perftest utility to exercise major UCX flows and provide performance metrics
- Added test script for jenkins (contrib/test_jenkins.sh)
- Added packaging for RPM/DEB based linux distributions (see contrib/buildrpm.sh)
- Added Unit-tests infractucture for UCX functionality based on Google Test framework (see test/gtest/)
- Added initial integration for OpenMPI with UCX for PGAS/SHMEM API
  (see: https://github.com/openucx/ompi-mirror/pull/1)
- Added end-to-end testing infrastructure based on MTT (see contrib/mtt/README_MTT)