Release Notes v2024.04.19

Bug Fixes

• PR #3091 - Fixes Parquet double reads to properly account for null values
• Issue #3087 - Fixes bug when reading non-float parquet columns with null values
• Issue #3088 and PR #3090 - Fix an off by 1 bug in sparse_sum_helper

Major Updates

• Issue #3083 - Optimizes Parquet Strings read
• Issues #3033, #3054 - Optimize CSV write
• Issues #3020, #3040 - Adds nan functions to DataFrame and Series
  • isna, notna, dropna, ...
• Issues #3071, #3084 - Add permutation and shuffle to random number generators
• Issue #3030 - Creates numpy subdirectory as part of the alignment effort
• PRs #3056, #3093, #3070, #3072 - Improves and adds Array API functionality including manipulation and set functions

Minor Updates

• PR #3076 - Adds support for large string Parquet type
• Issue #3092 - Adds support for TLS token authentication
• Issue #3045 - Adds map method to Index
• Issue #3065 - Adds count to DataFrame
• Issue #2913 - Adds isdecimal to Strings
• Issue #3002 - Adds clip to pdarray
• Issue #3062 - Enhances arkouda metrics capability

Auto-Generated Release Notes

• Closes #3030 numpy alignment directory structure by @ajpotts in #3038
• Closes #3040 isna and notna for series by @ajpotts in #3048
• Closes #3033: Optimize CSV write by @stress-tess in #3053
• Closes #3063: Fix deprecation warnings by @stress-tess in #3064
• Closes #2913: add isdecimal by @jaketrookman in #3015
• Closes #3045 index.map by @ajpotts in #3057
• Array API Set functions by @jeremiah-corrado in #3070
• Fix --print-used-modules for functions registered with @arkouda.registerND by @jeremiah-corrado in #3072
• Closes #3054: Dynamically switch to batching for larger csv writes by @stress-tess in #3061
• Closes #3002: Add ak.clip functionality by @drculhane in #3043
• enhance arkouda metrics capability by @hokiegeek2 in #3067
• Partially addresses issue #3050, updating PROTO tests. by @drculhane in #3075
• Add support for large string Parquet type by @bmcdonald3 in #3076
• Closes #3071: Add permutation to our generators by @stress-tess in #3078
• Closes #3065 DataFrame.count by @ajpotts in #3081
• Part of #3088: Generate seed for sparse sum test by @stress-tess in #3090
• Fix Parquet double reads to properly account for null values by @bmcdonald3 in #3091
• Array API Manipulation Function Improvements by @jeremiah-corrado in #3056
• Closes #3084: Add shuffle to random number generators by @stress-tess in #3085
• Pdarray indexing by @jeremiah-corrado in #3093
• Fixes #3087: Failure reading non-float parquet columns with null values by @stress-tess in #3094
• Closes #3083: Optimize Parquet string read code by @bmcdonald3 in #3082
• add support for TLS token authentication by @hokiegeek2 in #3096
• Fixes #3088: sparse sum nightly failures by @stress-tess in #3098
• Closes #3020 dataframe.dropna by @ajpotts in #3101

New Contributors

• @drculhane made their first contribution in #3043

Full Changelog: v2024.03.18...v2024.04.19

Release Notes v2024.03.18

Bug Fixes

• Issue #3035 - Fixes inconsistent results when broadcasting with empty segments
• Issue #2939 - Fixes TypeError in DataFrame.reset_index
• Issue #2966 - Fixes error when pip installing from a tar file
• Issue #2897 - Fixes bug where DataFrame.corr returns DataFrame without index
• PR #3021 - Adds SegArray optimization and benchmark bug fix

Major Updates

• Issue #2958 - Renames akstats to akscipy
• Issue #2942 - Removes DataFrame.sorted
• Issue #3024 and PR #2976 - Add sparse sum helper to util with merge based and sort based workflows
• Issues #2993, #3008, #3017 - Add a random subfolder and stateful Generator objects
• Issue #2974 - Adds Series.map
• Issue #3019 - Adds outer join option to DataFrame merge
• PRs #2936, #2967, #3014, #3027 - Improve Array API functionality specifically adding stats and manipulation functions

Minor Updates

• Issue #2929 - Updates DataFrame.size to match pandas
• Issues #2906, #2945 - Add shift operators between 2 bool pdarrays and between a combination bool and int64 pdarrays
• Issues #2916, #2919 - Add isspace and capitalize to Strings
• Issue #3023 - Adds to_markdown to DataFrame and Series
• Issue #2957 - Adds Dot Function
• Issue #2960 - Adds memory_usage functions
• Issue #2924 - Updates DataFrame documentation
• Issue #2896 - Updates DataFrame columns to return an Index
• Issue #2952 - Makes Chapel 1.33 release default for CI testing
• Issue #2985 - Updates libzmq version in Makefile
• Issue #2981 - adds LICENSES folder including the licenses for numpy, pandas, and scipy
• Issues #2969, #2971, #2977, #2989 - Update failing proto_tests

Auto-Generated Release Notes

• Add sort compat modules for new sorting algorithm by @bmcdonald3 in #2941
• Closes #2906 shift operator for boolean vectors by @ajpotts in #2944
• Closes #2916 add isspace for pdarrays by @ajpotts in #2946
• Closes #2949: Add compat modules for 1.34 by @bmcdonald3 in #2950
• Closes #2952: Make Chapel 1.33 release default for CI testing by @bmcdonald3 in #2951
• Remove deprecation warnings about domain(?) vs. domain by @jeremiah-corrado in #2953
• Closes #2919 add capitalize to pdarrays by @ajpotts in #2948
• Closes #2945: add Shift Operators for Boolean and Int64 by @jaketrookman in #2954
• Closes #2924 update pydoc strings for arkouda dataframe by @ajpotts in #2943
• Closes #2942 bug in sorted by @ajpotts in #2955
• Closes #2958 rename akstats to akscipy by @ajpotts in #2959
• Closes #2929-dataframe-size-to-match-pandas by @ajpotts in #2961
• Closes #2963-PROTO_tests-tests-akscipy-unit-tests-failing by @ajpotts in #2964
• Fixes #2966: pip install from tar error by @stress-tess in #2968
• Array API manipulation functions by @jeremiah-corrado in #2936
• Add sparse sum helper to util by @stress-tess in #2976
• Closes #2969 PROTO_tests/tests/client_test.py unit tests failing by @ajpotts in #2970
• Closes #2971 PROTO_tests/tests/dtypes_test.py unit tests failing by @ajpotts in #2972
• Closes #2977 PROTO_tests/tests/setops_test.py unit tests failing by @ajpotts in #2979
• Address Random.choice deprecation by @jeremiah-corrado in #2983
• Closes #2985: Update libzmq version in Makefile by @stress-tess in #2986
• Resolve formatting issue in NumpyDType server docs by @jeremiah-corrado in #2992
• Closes #2981 add licenses by @ajpotts in #2988
• Closes #2989 PROTO_tests/tests/pdarray_creation_test.py has failing test by @ajpotts in #2990
• Remove deprecation messages for reader/writer locking default change by @jeremiah-corrado in #2987
• Closes #2994: Remove upper bound on pandas version by @stress-tess in #2995
• Closes #2896 DataFrame columns should return an Index by @ajpotts in #2962
• Closes #2993: Create random subfolder and foundation for generator by @stress-tess in #2997
• Closes #2957: Dot Function by @jaketrookman in #2996
• Closes #2934 DataFrame.unregister_dataframe_by_name string return typ… by @ajpotts in #3011
• Closes #2897 DataFrame.corr returns dataframe without index by @ajpotts in #3012
• Closes #2939 DataFrame.reset_index by @ajpotts in #3013
• Array API stats functions by @jeremiah-corrado in #2967
• Support for creating Array API objects from numpy arrays by @jeremiah-corrado in #3014
• Closes #2960 int version of memory_usage by @ajpotts in #3018
• SegArray optimization & bug fix by @brandon-neth in #3021
• Closes #3019 Add outer join option for dataframe merge by @ajpotts in #3022
• Closes #3031: Update Arkouda for upcoming Chapel 2.0 release by @bmcdonald3 in #3032
• Closes #3008: Add generator sym entry and stateful uniform distribution by @stress-tess in #3016
• Closes #2974 Series.map by @ajpotts in #3010
• Closes #3024: Add merge based workflow and update sort workflow for sparse sum helper by @stress-tess in #3025
• Remove I/O-locking and randomStream.skipTo deprecation messages by @jeremiah-corrado in #3037
• Closes #3023 to_markdown by @ajpotts in #3026
• Array API improvements by @jeremiah-corrado in #3027
• Fixes #3035 - Inconsistent results when broadcasting with empty segments by @stress-tess in #3039
• Closes #3017: Add documentation for our random number generation by @stress-tess in #3044

Full Changelog: v2024.02.02...v2024.03.18

Release Notes v2024.02.02

Bug Fixes

• Issues #2647, #2650, #2661, #2666 - Fix bugs in filter, remove_repeats, append_single and get_jth for segarrays with empty segments
• Issue #2937 - Fixes string read bug for large Parquet files

Major Updates

• Issue #2853 - Adds df.merge on a mix of String and integer columns
• Issues #2862, #2863, #2871 - Implement histogram2d and histogramdd
• Issue #2927 - Adds power divergence statistic, chisquare, and xlogy
• Issues #2905, #2912, #2914, #2918 - Add isalpha, isalnum, isdigit, and isempty for Strings
• Issues #2888, #2878 - Add float support to in1d and groupby

Minor Updates

• Issues #2873, #2886 - Enable short strings optimization for multi-column groupby
• Issue #2894 - Adds dropna option to dataframe groupby
• Issue #2461 - Adds where argument to trig functions
• Issue #2430 - Adds shift equals for pdarray
• Issue #2831 - Aligns dataframe.groupby().size() and dataframe.groupby().sum() with pandas
• PRs #2829, #2865, #2876 - Add infrastructure and partial implementation of array API
• PR #2870 - Expands DataFrame initializer
• PR #2931 - Aligns Series indexing methods with pandas
• Issue #2909 - Renames is_upper/lower/title and to_upper/lower/title to match numpy

Auto-Generated Release Notes

• Scaffolding for multi-dimensional arrays by @jeremiah-corrado in #2829
• Closes #2836: Remove computeOnSegmentsWithoutAggregation by @pierce314159 in #2854
• Closes #2857: Move init methods to compat modules for 132 by @bmcdonald3 in #2858
• Closes #2855: Read batches for Parquet strings by @bmcdonald3 in #2856
• Closes #2853: df.merge on a mix of String and integer columns by @pierce314159 in #2859
• Closes #2862: Implement histogram2d by @pierce314159 in #2864
• Closes #2866: Fix empty domain creation for block arrays by @bmcdonald3 in #2867
• Close #2871: Fix numeric precision issue in testHistogram by @pierce314159 in #2872
• DataFrame initializer improvements by @brandon-neth in #2870
• Fixes #2666: Segarray.filter bug with empty segments by @pierce314159 in #2860
• Fixes #2661: segarray.remove_repeats bug with empty segments by @pierce314159 in #2861
• Closes #2863: Implement histogramdd by @pierce314159 in #2868
• Array creation perf patch by @jeremiah-corrado in #2869
• Closes #2875: Clean up deprecations in advance of 1.33 release by @bmcdonald3 in #2874
• Closes #2873: Short strings optimization for multicol groupby by @pierce314159 in #2877
• Only use tryCreateArray interface to create 1D arrays by @jeremiah-corrado in #2879
• Fix typo in #2879 by @jeremiah-corrado in #2882
• Closes #2878: Add groupby for floats by @pierce314159 in #2884
• Fixes #2883: Fix ambiguity in makeDistArray where clauses by @jeremiah-corrado in #2885
• Closes #2886: Cap small strs groupby optimization by @pierce314159 in #2887
• Closes #2888: Add in1d float support by @pierce314159 in #2889
• Fixes #2650: Fixes segarray.append_single with empty segments by @pierce314159 in #2891
• Closes #2461: Implementation of where feature for trig functions by @jaketrookman in #2851
• Fixes #2647: Segarray.get_jth bug with bool values by @pierce314159 in #2895
• Array API infrastructure and partial implementation by @jeremiah-corrado in #2865
• Closes ticket #2831 to align dataframe.groupby().size(), dataframe.groupby().sum() to pandas by @ajpotts in #2892
• Adjustments to remove some chpl unstable warnings by @jeremiah-corrado in #2890
• Fix CommAggregation build error on 1.32 by @jeremiah-corrado in #2900
• Closes #2901: Fix performance regression due to setting ndim field in sym entry by @bmcdonald3 in #2902
• Replace use of binary 'or' with logical 'or' by @DanilaFe in #2907
• Closes #2894 add dropna option to dataframe groupby by @ajpotts in #2904
• closes #2430 add shift equals for pdarray by @ajpotts in #2908
• Closes #2905 add strings isalnum by @ajpotts in #2910
• Closes ticket #2909 rename is_upper and similar ops to match numpy by @ajpotts in #2920
• Closes #2922: Pin pandas<2.2.0 to avoid CI failures by @pierce314159 in #2923
• Closes #2912 add isalpha for pdarray by @ajpotts in #2921
• Apply black formatter to array_api frontend code by @jeremiah-corrado in #2928
• Modify fix to performance regression in #2902 by @jeremiah-corrado in #2930
• Closes #2914-add-isdigit-for-pdarray by @ajpotts in #2925
• Make arkouda server configurable w.r.t. supported types by @jeremiah-corrado in #2876
• Indexing methods for Series by @brandon-neth in #2931
• Closes #2937 : Fix string read bug for large Parquet files by @bmcdonald3 in #2938
• Closes #2918 add isempty for pdarray by @ajpotts in #2933
• closes #2927 power divergence statistic by @ajpotts in #2932

New Contributors

• @ajpotts made their first contribution in #2892

Full Changelog: v2023.11.15...v2024.02.02

Release Notes v2023.11.15

Bug Fixes

• Issue #2816 - Resolves nil config file bug
• Issue #2804 - Fixes missing values in CSV bug
• Issue #2825 - Fixes inconsistent Categorical print
• Issue #2849 - Fixes bug in getEnv

Major Updates

• PR #2844 - Drops support for Chapel 1.30
• Issue #2838 - Expands dataframe merge functions to accept multiple columns
• Issue #2810 - Expands inner_join to accept a list of pdarrays
• Issues #1882, #2833, #2843 - Update dependencies to support python 3.x/3.12

Minor Updates

• Issue #2823 - Adds casting between Strings and Categorical
• Issue #2830 - Implements division and floor division for int64 and uint64 dtypes
• PR #2821 - Adds support for reading Decimal128 Parquet columns
• Issue #2819 - Adds error when reading a Parquet type that isn't supported

Auto-Generated Release Notes

• Fixes #2816: Resolve nil config file bug by @pierce314159 in #2817
• Fixes #2804: Missing values in CSV bug by @pierce314159 in #2813
• fixed typo in README.md by @daulatojha17 in #2818
• Closes #2819 - Throw error when reading a Parquet type that isn't supported by @bmcdonald3 in #2820
• Closes #2815: Update install docs by @pierce314159 in #2822
• Fixes #2825: Inconsistent Categorical print by @pierce314159 in #2826
• Closes #2823: Casting between Strings and Categorical by @pierce314159 in #2827
• Add support for reading Decimal128 Parquet columns by @bmcdonald3 in #2821
• Closes #1882, #2833: CI failures due to python 3.x/3.12 by @pierce314159 in #2834
• Add directory with files for Parquet C++ comparison by @bmcdonald3 in #2832
• Closes #2810: Expand inner_join to accept a list of pdarrays by @pierce314159 in #2837
• Fix a Makefile conditional to test for empty string instead of 'none' by @bradcray in #2841
• Closes #2843: Add python 3.12/3.x check to CI by @pierce314159 in #2840
• Drop support for Chapel 1.30 by @brandon-neth in #2844
• Updates to the macOS build instructions + additional contributor guide by @brandon-neth in #2845
• Fixes #2849: bug in getEnv by @pierce314159 in #2850
• Closes #2830 Implement division and floor division for int64 and uint64 dtypes. by @jaketrookman in #2847
• Closes #2838: Expand dataframe merge functions to accept multiple columns by @pierce314159 in #2848

Full Changelog: v2023.10.06...v2023.11.15

Release Notes v2023.10.06

Bug Fixes

• Issue #2802 - Fixes concatenate and arange mishandling max_bits
• Issues #2472, #2800 - Fix bug in bigint indexing with max_bits set
• Issues #2675, #2773 - Fix argsort and coargsort on boolean arrays

Major Updates

• Issues #2724, #2730 - Add HDF5 support for Index and MultiIndex
• Issues #2679, #2767, #2774, #2783, #2798 - Resolve chpl 1.32 deprecation warnings
• Issue #2805 - Upgrades arrow to 11.0.0
• Issues #2759, #2757, #2762, #2764, #2790, #2794 - Work to improve pdarray creation performance and out of memory error handling
• Issue #2405 - Adds server <-> server transfers of pdarrays
• Issue #2716 - Adds dataframe merge functionality
• Issue #2712 - Adds stridable slicing to Strings
• Issue #2744 - Implements skew and hist_all

Minor Updates

• Issue #2778 - Adds convert_categoricals flag to dataframe.to_parquet
• Issue #2771 - Adds pda.value_counts
• Issue #2521 - Adds uint support to histogram
• Issue #2474 - Updates ConcatenateMsg to use aggregation for bigint

Auto-Generated Release Notes

• Closes #2757: Add overload of makeDistArray that takes an array by @bmcdonald3 in #2758
• Closes #2759: Fix tryCreateArray call in 1.32 compat module by @bmcdonald3 in #2760
• Closes #2762: Fix last outstanding error regarding OOM handling compat modules by @bmcdonald3 in #2763
• Closes #2675: Argsort for bool arrays by @pierce314159 in #2761
• Closes #2405: Add server <-> server transfers of pdarrays by @bmcdonald3 in #2406
• Closes #2744 - Implement skew and hist_all by @egelberg in #2756
• Closes #2764: Add throws to new makeDistArray compat function by @bmcdonald3 in #2765
• Closes #2767: Resolve deprecation warnings introduced by array transfer PR by @bmcdonald3 in #2766
• Closes #2679: Convert check for re2 in the compiler from a .compile() method to new regex() by @bradcray in #2777
• Closes #2712: Adds stridable slicing to Strings by @pierce314159 in #2770
• Closes #2771: Add pda.value_counts by @pierce314159 in #2772
• Fixes #2773: Fixes bug in multi-locale coargsort with bool arrays by @pierce314159 in #2776
• Closes #2768: Verify that all test files pass flake8 by @jaketrookman in #2769
• Closes #2778: Add convert_categoricals flag to dataframe.to_parquet by @pierce314159 in #2780
• Closes #2521: Add uint support to histogram by @TheOddCzar789 in #2782
• Closes #2774: Resolve deprecation warnings in advance of 1.32 release by @bmcdonald3 in #2775
• Closes #2783: Add ref this intents to comm aggregation code by @bmcdonald3 in #2784
• Closes #2724 & #2730 - Adds HDF5 support for Index and MultiIndex by @Ethan-DeBandi99 in #2731
• Closes #2790: Convert non sym entry array creations to new throwing interface by @bmcdonald3 in #2791
• Closes #2789: Updates to HDF5 for Index by @pierce314159 in #2792
• Closes #2794: Fix distributed makeDistArray call for Chapel 1.30 and 1.31 by @bmcdonald3 in #2795
• Closes #2796: Reduce Parquet IO benchmark default size by @bmcdonald3 in #2797
• Cloases #2798: Resolve chpl 1.32 deprecations for make check-deps by @pierce314159 in #2799
• Closes #2474: Update ConcatenateMsg to use aggregation for bigint by @jaketrookman in #2801
• Fixes #2802: concatenate and arange mishandling max_bits by @pierce314159 in #2803
• Closes #2807: Pin tables<3.9.0 to avoid CI failures by @pierce314159 in #2808
• Closes #2805: Upgrade arrow to 11 and add python 3.x compat by @pierce314159 in #2806
• Closes #2472, #2800: Update IndexingMsg to use & instead of mod for setting bigint pdarrays by @jaketrookman in #2793
• Closes #2716: Add dataframe merge functionality by @egelberg in #2781

New Contributors

• @egelberg made their first contribution in #2756

Full Changelog: v2023.09.06...v2023.10.06

Release Notes v2023.09.06

Bug Fixes

• Issue #2596 - Fixes datetime columns in dataframe display bug
• Issue #2612 - Fixes an oob error with multilocale read_hdf of a segarray with string values and empty segments
• Issues #2560, #2268, #2566 - Fix missing empty segments for parquet read of segarray with string values
• Issue #2567 - Fixes error when reading SegArray containing nan with parquet
• Issue #2581 - Fixes Strings.get_null_indicies incorrect results
• Issues #2681, #1493 - Fix Series equality bug
• Issues #2644, #2645 - Fix uint cast to str/float and add str cast to str
• Issue #2703 - Fixes sort bug with nans
• Issue #2617 - Fixes bug in comparison of segarrays containing empty segments
• Issue #2711 - Fixes bug in Multindex indexing with bigint values
• Issue #2579 - Fixes compression when writing bools with Parquet
• Issues #2508, #2519, #2505 - Fix uint scalar binops handling and overhauls mod/fmod
• Issue #2635 - Fixes Strings double delete

Major Updates

• Issues #2548, #2550 - Drop support for Chapel 1.29 and recommend 1.31
• Issues #2395, #2723, #2726, #2737 - Rework Register/Attach
• Issues #2512, #2541, #2614 - Adds Snapshotting via HDF5
• Issue #2493 - Adds parquet support for multi-column SegArray with String values
• Issues #2749, #1166 - Add missing aggregations for Dataframe groupby and ability to aggregate on a list column names
• Issue #2708 - Adds support for preserving DateTime, TimeDelta, and IPV4 when reading/writing with HDF5

Added New Testing Architecture

• Issue #2504 - numeric_tests.py conversion for new test framework
• Issue #2538 - client_test.py conversion for new test framework
• Issue #2542 - message_test.py Conversion for new test framework
• Issues #2547, #2553, #2554, #2555 - setops_tests.py conversion for new test framework
• Issue #2526 - Add arc/hyperbolic Tests to new test framework
• Issue #2570 - bigint_agg_test.py Conversion for new test framework
• Issue #2536 - alignment_test.py Conversion for new test framework
• Issue #2559 - dataframe_test.py conversion for new framework
• Issue #2585 - parquet_test.py conversion for new framework
• Issue #2537 - array_view_test conversion for new framework
• Issue #2607 - groupby_test.py conversion to new framework
• Issue #2616 - index_test.py conversion for new framework
• Issue #2625 - coargsort_test.py conversion for new test framework
• Issue #2602 - io_test.py conversion for new framework
• Issue #2640 - Convert security_test.py to new test framework
• Issue #2648 - logger_test.py conversion for new framework
• Issue #2583 - client_dtypes_test conversion for new framework
• Issue #2624 - import_export_test.py conversion for new test framework
• Issue #2572 - bit_ops.py Conversion for new test framework
• Issue #2605 - dtypes_test.py Conversion for new test framework
• Issue #2659 - regex_test.py conversion to new test framework
• Issue #2626 - join_test.py Conversion for new test framework
• Issue #2668 - io_util_test.py refactor for new framework
• Issue #2642 - segarray_test.py refactor for new test framework
• Issue #2694 - extrema_test.py conversion for new test framework
• Issue #2688 - operator_test.py conversion for new test framework
• Issue #2684 - symbol_table_test.py refactor for new test framework
• Issue #2705 - where_test.py refactor to new test framework
• Issue #2700 - string_test.py conversion for new test framework
• Issue #2654 - sort_test.py conversion for new test framework
• Issue #2620 - indexing_test.py conversion for new framework
• Issue #2656 - stats_test.py conversion for new test framework
• Issue #2686 - categorical_test.py conversion for new test framework
• Issue #2573 - pdarray_creation_test.py Conversion for new test framework
• Issue #2697 - datetime_test.py conversion for new test framework
• Issue #2651 - series_test.py conversion for new framework
• Issue #2591 - Add dtype testing to test_multi_array_search_interval

Minor Updates

• Issue #2053 - Adds Strings property accessors
• Issue #2568 - Provides client access to arkouda locale memory information
• Issue #2419 - Adds ability to log arkouda commands to a file
• Issue #2702 - Capture only user-specified arkouda commands on client
• Issues #2524, #2721, #2746 - Extend client to enable gRPC proxy server channel implementations
• Issues #2477, #2728 - Provide dynamic available memory calculation for all locales
• Issue #2400 - Adds arc and hyperbolic trig functions
• Issue #2471 - Adds ak.full() for Strings
• Issue #2603 - Reworks arange handling of uint and bigint arguments
• Issue #2714 - Adds regex argument to categorical substring search
• Issue #2695 - Adds uint support to extrema methods
• Issues #2658, #2575 - Implement ServerStatusDaemon
• Issue #2690 - Updates ak.load_all glob expression

Auto-Generated Release Notes

• Replace c_memcpy with OS.POSIX.memcpy in CommAggregation module by @jeremiah-corrado in #2517
• Closes #2504 - Adds New Testing Architecture by @Ethan-DeBandi99 in #2511
• Closes #2400: Add arc and hyperbolic trig functions by @jaketrookman in #2447
• 2524 extend arkouda client to enable alternative channel implementations by @hokiegeek2 in #2525
• update readme by @pierce314159 in #2530
• Closes #2529: Adjust getUserMetrics to return borrowed by @mppf in #2527
• Fixes #2508 and #2519: Fixes uint scalar handling and overhauls mod/fmod for floats by @pierce314159 in #2520
• Closes #2505: Update resolve_scalar_dtype to handle uint and bigint by @pierce314159 in #2532
• dynamic available memory calculation for all locales by @hokiegeek2 in #2533
• Closes #2493 - Multi-Column SegArray with String Values by @Ethan-DeBandi99 in #2523
• Closes #2538 - client_test.py conversion for new test framework by @Ethan-DeBandi99 in #2540
• Closes #2268 - Fixes Sporadic failures of Parquet segarray_write test with gasnet by @Ethan-DeBandi99 in #2534
• Closes #2542 - message_test.py Conversion for new test framework by @Ethan-DeBandi99 in #2544
• Closes #2545: Update MemoryMgmt to use compat modules by @pierce314159 in #2546
• Closes #2547, #2553, #2554, #2555 - setops_tests.py conversion for new test framework by @Ethan-DeBandi99 in #2556
• Closes #2548: Recommend Chapel 1.31.0 and use it for CI testing by @bmcdonald3 in #2552
• Closes #2560 - Fixes Missing empty segments for parquet read of segarray with string values by @Ethan-DeBandi99 in #2562
• Closes #2471: ak.full() for Strings by @TheOddCzar789 in #2557
• Fixes #2564 - SetOps Test Sporadic Failures by @Ethan-DeBandi99 in #2565
• Closes #2561: Add casting directly from bool to bigint by @pierce314159 in #2563
• Fixes #2566 - multi-locale failures for test_segarray_string by @Ethan-DeBandi99 in #2574
• Closes #2581 - Strings.get_null_indicies Not Reporting Correct Results in Parquet by @Ethan-DeBandi99 in #2582
• Fixes #2567 - Error when reading SegArray containing nan with parquet by @Ethan-DeBandi99 in #2587
• Closes #2588: Remove BigInt casting from 132 compat and add new compat dir by @bmcdonald3 in #2589
• Closes #2550: Drop support for Chapel 1.29 by @bmcdonald3 in #2551
• provide client access to arkouda locale memory information by @hokiegeek2 in #2595
• Closes #2053: Strings Property Accessors by @jaketrookman in #2498
• Closes #2526: Add arc/hyperbolic Tests to new test framework by @jaketrookman in #2531
• Closes #2570: bigint_agg_test.py Conversion for new test framework by @TheOddCzar789 in #2578
• Closes #2597: Remove leftover e-129 compat folder by @pierce314159 in #2598
• Closes #2536: alignment_test.py Conversion for new test framework by @TheOddCzar789 in #2569
• Closes #2559 - dataframe_test.py conversion for new framework by @Ethan-DeBandi99 in #2580
• Closes #2593: Add compat version of c_void_ptr for new c_ptr(void) by @riftEmber in #2594
• Closes #2585: parquet_test.py conversion for new framework by @pierce314159 in #2586
• Closes #2537: array_view_test conversion for new framework by @jaketrookman in #2590
• Fixes #2579 - Compression when writing bools with Parquet by @Ethan-DeBandi99 in #2601
• Fixes #2596: datetime columns in dataframe display bug by @pierce314159 in #2600
• Closes #2541 - DataFra...

Release Notes v2023.06.20

Bug Fixes

• Issue #2513 - Fixes bug where channel connection doesn't account for slurm changing the server host between connections. Refactored to create a new Channel for each client.connect execution.

Minor Updates

• Issue #2515 - Updates file I/O documentation to include SegArrays containing string values

Auto-Generated Release Notes

* refactored to create a new Channel for each client.connect execution by @hokiegeek2 in https://github.com//pull/2514 * Closes #2515 - File I/O Doc Updates by @Ethan-DeBandi99 in https://github.com//pull/2516

Full Changelog: v2023.06.16...v2023.06.20

v2023.06.16

Bug Fixes

• Issue #2481 - Fixes Multi-Column Parquet not handling Empty Files Properly
• Issue #2506 -Fixes Categorical Optional Components Required Bug
• Issue #2414 - Fixes overMemLimit calc error

Major Updates

• Issues #2424 and #2432 - Adds Strings value support for SegArray
• Issue #2443 - Read/Write SegArray of Strings for HDF5
• Issue #2444 - Adds SegArray with Strings Values Parquet Support (Does not include Multi-Colmn)
• Issue #2386 - Read/Write support for GroupBy objects in HDF5
• Issues #2434, #2459, #2462, and #2463 - Adds hashing support for Segarray, Strings, Categorical, BigInt
• Issues #2006, #2032, #2416, and #2431 - BigInt Support Improvements
• Issue #2304 - Adds inner_join on Strings and Categorical
• Issue #2417 - Filename_Codes match Categorical.codes
• Issue #2425 - Import/Export lists from/to pandas

Minor Updates

• Issue #2454 - Updates SegArray.__getitem__ to Always Return pdarray
• Issue #2418 - Adds instructions to set max per-locale CPU cores and memory
• Issue #2433 - Updates GroupBy Object to only be client side

Auto-Generated Release Notes

* Adjust these modules to avoid deprecation warnings from non-default Math symbols by @lydia-duncan in https://github.com//pull/2415 * Closes #2412: Update quickstart to v2023.05.05 by @pierce314159 in https://github.com//pull/2413 * Fix overMemLimit calc error by @hokiegeek2 in https://github.com//pull/2421 * Closes #2427 - Deprecation updates related to Memory and Memory.Diagnostics by @jabraham17 in https://github.com//pull/2422 * Closes #2425 - Import/Export lists from/to pandas by @Ethan-DeBandi99 in https://github.com//pull/2428 * Closes #2386 - `GroupBy.to_hdf` & `GroupBy.update_hdf` by @Ethan-DeBandi99 in https://github.com//pull/2426 * add instructions to set max per-locale CPU cores and memory by @hokiegeek2 in https://github.com//pull/2429 * Closes #2416 and #2006: bigint shift performance by @pierce314159 in https://github.com//pull/2423 * Closes #2431: Add bigint broadcast by @pierce314159 in https://github.com//pull/2437 * Closes #2441: Adds missing `use Biginteger` in gt-130 bigint compat by @pierce314159 in https://github.com//pull/2442 * fixed typo by @hokiegeek2 in https://github.com//pull/2448 * Closes #2417 - `Filename_Codes` match `Categorical.codes` by @Ethan-DeBandi99 in https://github.com//pull/2440 * Closes #2445 - Deprecation updates related to string and byte factory functions by @jabraham17 in https://github.com//pull/2446 * Closes #2451: Remove pragma no doc instances in Chapel code by @bmcdonald3 in https://github.com//pull/2452 * Updates for Chapel `list.append` deprecation by @jeremiah-corrado in https://github.com//pull/2450 * Closes #2432 - Revert SegArray to Client Side by @Ethan-DeBandi99 in https://github.com//pull/2439 * Closes #2304: `inner_join` on `Strings` and `Categorical` by @pierce314159 in https://github.com//pull/2453 * Closes #2467: Arrow compilation can fail with clang 15 upgrade changes default PIE by @bmcdonald3 in https://github.com//pull/2468 * Closes #2032 - BigInt Support for HDF5 by @Ethan-DeBandi99 in https://github.com//pull/2460 * Closes #2454 - Update `SegArray.__getitem__` to Always Return `pdarray` by @Ethan-DeBandi99 in https://github.com//pull/2466 * Closes #2424 - Adds `SegArray` support for `Strings` Values by @Ethan-DeBandi99 in https://github.com//pull/2469 * Closes #1211 - Remove TaskErrors Workaround by @Ethan-DeBandi99 in https://github.com//pull/2470 * Closes #2433: GroupBy back to client only by @pierce314159 in https://github.com//pull/2456 * Changes for Chapel `c_memcpy` replacement with `OS.POSIX.memcpy` by @jeremiah-corrado in https://github.com//pull/2479 * Closes #2482 - Fix capitalization of POSIX compatibility module by @jeremiah-corrado in https://github.com//pull/2483 * Closes #2443 - Read/Write SegArray of Strings HDF5 by @Ethan-DeBandi99 in https://github.com//pull/2478 * Closes #2459 and #2434: `ak.hash` for `Segarray` and `Strings` by @pierce314159 in https://github.com//pull/2475 * Fixes #2481 - Multi-Column Parquet does not handle Empty Files Properly by @Ethan-DeBandi99 in https://github.com//pull/2484 * Closes #2436 - Updates `_buildReadAllJSON` to use `ObjType` Enum by @Ethan-DeBandi99 in https://github.com//pull/2486 * Closes #2490: Change `checkInstall` path to be relative to script, not Arkouda by @bmcdonald3 in https://github.com//pull/2491 * Closes #2462: Categorical hashing by @pierce314159 in https://github.com//pull/2487 * Closes #2476: Updates Chapel Tutorial by @pierce314159 in https://github.com//pull/2494 * Design and implement client Channel class hierarchy by @hokiegeek2 in https://github.com//pull/2496 * Closes #2463: Hashing for bigint pdarrays by @pierce314159 in https://github.com//pull/2497 * Closes #2500 - Remove Old Test Prototype by @Ethan-DeBandi99 in https://github.com//pull/2501 * Closes #2502 - Remove ArkoudaWeeklyCall References by @Ethan-DeBandi99 in https://github.com//pull/2503 * Closes #2488: Quiet deprecation warnings in prep for Chapel 1.31 by @bmcdonald3 in https://github.com//pull/2489 * Fixes #2506 - Categorical Optional Components Required Bug by @Ethan-DeBandi99 in https://github.com//pull/2507 * Closes #2444 - SegArray with String Values Parquet Support by @Ethan-DeBandi99 in https://github.com//pull/2492

Full Changelog: v2023.05.05...v2023.06.17

Release Notes v2023.05.05

Bug Fixes

• Issue #2398 - Fixes parquet error on list columns containing nested lists
• Issue #2380 - Fixes SegArray register bug
• Issue #2396 - Fixes server crash caused by nested parquet fields
• Issue #2300 - Improves Strings strip performance

New Features

• Issue #2296 - Adds bitops support for bigint pdarays
• Issue #474 - Adds HDF5 overwrite dataset
• Issues #2372 and #2373 - Update Categorical HDF5 format and add update_hdf method
• Issue #2377 - Adds groupby aggregations that require min/max on bigint
• Issue #1855 - Adds divmod support

Minor Updates

• Issue #2355 - Drops support for 1.28
• Issue #2138 - Updates messaging overview docs
• Issue #2368 - Cleans up Strings references in SegArray

Auto-Generated Release Notes

* Closes #2370 - Fixes Deprecation Warnings during `make` by @Ethan-DeBandi99 in https://github.com//pull/2371 * Closes #2368 - Cleans up Strings references in SegArray by @Ethan-DeBandi99 in https://github.com//pull/2369 * Closes #1855: Implement `divmod` by @jaketrookman in https://github.com//pull/2356 * Closes #2374: Pin hdf5 version to 1.12.2 by @pierce314159 in https://github.com//pull/2375 * Closes #2355: Drop support for 1.28 by @bmcdonald3 in https://github.com//pull/2383 * Closes #2377: Add groupby aggregations that require min/max on bigint by @pierce314159 in https://github.com//pull/2378 * Closes #474 - HDF5 Overwrite Dataset by @Ethan-DeBandi99 in https://github.com//pull/2382 * Closes #2300 - `segString` Strip performance issue by @joshmarshall1 in https://github.com//pull/2379 * Fixes #2380: Segarray register bug by @pierce314159 in https://github.com//pull/2392 * Closes #2341: Quiet 131 deprecations by @bmcdonald3 in https://github.com//pull/2342 * Closes #2372 & #2373 - Categorical HDF5 Format Update & `update_hdf` method by @Ethan-DeBandi99 in https://github.com//pull/2394 * Closes #2396 - Fixes Nested parquet fields causing server crash by @Ethan-DeBandi99 in https://github.com//pull/2399 * Closes #2398 - Fixes Parquet List Columns with Nested Lists Error by @Ethan-DeBandi99 in https://github.com//pull/2401 * Closes #2403: Update c_getDatasetNames calls to match new behavior by @bmcdonald3 in https://github.com//pull/2404 * Add a use of the Math module to avoid the anticipated `pi` deprecation warning by @lydia-duncan in https://github.com//pull/2407 * Deprecation updates for `BitOps.popcount` and `bigint.mod` by @jeremiah-corrado in https://github.com//pull/2409 * Closes #2296: bitops support for bigint by @pierce314159 in https://github.com//pull/2408 * Closes #2138 Update messaging overview docs by @jaketrookman in https://github.com//pull/2391 * Deprecation updates for `datetime` and `fromTimestamp` by @bmcdonald3 in https://github.com//pull/2410

Full Changelog: v2023.04.07...v2023.05.05

Release Notes v2023.04.07

Bug Fixes

• Issue #2329 - Fixes continued SegArray read performance issues
• Issue #2299 - Fixes file writes reporting success when directory does not exist
• Issue #2297 - Fixes HDF5 single file write stall
• Issue #2327 - Fixes issue loading 16-bit and 32-bit from Parquet
• Issue #2337 - Fixes ak.DataFrame.to_parquet with IPv4 columns
• Issue #2348 - Fixes IPv4 removal from DataFrame
• Issue #2350 - Fixes DataFrame column subset access
• Issues #2306, #2317 and PR #2354 - Fix Datetime component scaling
• Issue #2328 - Fixes bug when printing Dataframes containing bigint
• Issue #2307 - Fixes reported memory usage exceeding 100 percent
• Issue #2309 - Fixes error in Groupby.unique on Categorical or Strings
• Issue #2240 - Fixes ak.coargsort empty String and Categorical bug
• Issue #2347 - Fixes broken links in README.md

New Features

• Issue #2050 - Adds File of Origin when loading data from Parquet or HDF5. Use ak.read_tagged_data to return the file origin information. More information on this function can be found here.
• Issue #2295 - Adds SegArray filters, ak.SegArray.filter() allows values to be removed from SegArray.
• Issue #2293- Adds ak.where support Strings and Categoricals
• Issue #2324 - Adds benchmark documentation
• Issue #2209 - Enhances Arkouda metrics

Minor Updates

• Issue #2015 - Updates Parquet NaN detection on float/double columns to be more efficient
• Issue #2280 - Improves max_bits handling
• Issues #2319, #2320, #2339 - Update to test framework to pytest configuration

Auto-Generated Release Notes

• Recommend Chapel 1.30.0 and use it for CI testing by @ronawho in #2291
• Closes #2299 - Fixes file writes reporting success when directory does not exist by @Ethan-DeBandi99 in #2305
• Closes #2297 - Fixes HDF5 Single File Write Stall by @Ethan-DeBandi99 in #2310
• Closes #2280: Improve max_bits handling by @pierce314159 in #2303
• Closes #2311: Reenable bulk transfer for developer builds by @bmcdonald3 in #2312
• Closes #2240: ak.coargsort empty String and Categorical bug fix by @jaketrookman in #2308
• Closes #2309: Error in Groupby.unique on Categorical or Strings by @pierce314159 in #2315
• Fixes #2306: Datetime.date scaling by @pierce314159 in #2316
• Closes #2023 - Added all compression types to legacy benchmarks by @joshmarshall1 in #2302
• Bump bigint groupby benchmark problem size by @ronawho in #2326
• Closes #2295 - SegArray Filters by @Ethan-DeBandi99 in #2325
• Closes #2329 - Fixes Continued SegArray Read Performance Issues by @Ethan-DeBandi99 in #2330
• Closes #2335 - Update Quickstart to point to v2023.03.24 by @Ethan-DeBandi99 in #2336
• Closes #2050 - File of Origin on Read by @Ethan-DeBandi99 in #2323
• Closes #2327 - Fixes Issue Loading 16-bit and 32-bit from Parquet by @Ethan-DeBandi99 in #2333
• Fixes #2328: Bug when printing Dataframes containing bigint by @pierce314159 in #2331
• Closes #2319 - PyTest Test Suite Restructure by @joshmarshall1 in #2314
• Closes #2320 - Adding Pytest structure to array_view_test.py by @joshmarshall1 in #2338
• Closes #2339 - Adding PyTest Structure to bigint_agg_test.py by @joshmarshall1 in #2340
• 2307 reported mem used greater than 100 percent by @hokiegeek2 in #2346
• Eliminate some communication from datetime methods by @ronawho in #2354
• Closes #2317: Add ArkoudaTimeShim modules to improve DateTimeMsg performance by @bmcdonald3 in #2353
• Closes #2015 - Efficient NaN read from Parquet by @Ethan-DeBandi99 in #2351
• Closes #2337 - Fixes ak.DataFrame.to_parquet with IPv4 Columns by @Ethan-DeBandi99 in #2352
• Closes #2348 - Fixes IPv4 Removal from DataFrame by @Ethan-DeBandi99 in #2358
• Closes #2293: ak.where support Strings and Categoricals by @pierce314159 in #2349
• Closes #2362: Start using Chapel 1.30 for building gh-pages by @bmcdonald3 in #2363
• Closes #2347 - Fixes broken links in README.md by @Ethan-DeBandi99 in #2360
• Closes #2350 - Fixes DataFrame Column Subset by @Ethan-DeBandi99 in #2359
• Setting pages to v3 checkout by @Ethan-DeBandi99 in #2364
• Updated to deployment v4 by @Ethan-DeBandi99 in #2365
• Closes #2366: Update graph files after IO benchmark changes by @bmcdonald3 in #2367
• Closes #2324 - Benchmark documentation by @joshmarshall1 in #2361
• 2209 enhance Arkouda metrics by @hokiegeek2 in #2332

Full Changelog: v2023.03.24...v2023.04.07