v3.00.55rc2

What's Changed

• Issue1214 another scheduled refactor by @reidsunderland in #1223
• fix #1219 enough to convert ops by @petersilva in #1224

Full Changelog: v3.00.55rc1...v3.00.55rc2

Packages Available

Ubuntu PPA: https://launchpad.net/~ssc-hpc-chp-spc/+archive/ubuntu/metpx-pre-release

PyPI: https://pypi.org/project/metpx-sr3/3.0.55rc2/

• RPM package for Redhat Version 8 (rh8) and 9 (rh9) attached here, install with rpm -ivh ... note:

  • Some dependencies are not readily available. May have to obtain packages
    from pip to get a fully functional installation. Verify with sr3 features after installation, to determine if everything needed is available.

• self-extracting windows installer also attached.

v3.00.55rc1

Briefly:

• Working through deployment issues on core operational data pumps.
• fixing more timers expiring during transfers. many reliability improvements.
• many more polls deployed, and some improvements for them.
• management UI improvements (sr3 status display of lag, queue sr3 stop works better.)
• fix for citypage download issues (and other datasets that rapidly change upstream.)

What's Changed

• Add new BC Trans API to sr3 by @andreleblanc11 in #1150
• Fix #1151 by @andreleblanc11 in #1152
• Fix #953 re-enable and fix passive queue declare by @reidsunderland in #1160
• AM adaptation for U18.04 servers by @andreleblanc11 in #1164
• The Hearty eXpert Issue1138 - documenting messageAgeMax by @petersilva in #1147
• Issue1159 by @petersilva in #1162
• Later pip needs convincing by @petersilva in #1171
• Issue1155 AMQP ack refactor by @reidsunderland in #1174
• #1167 fix clash in class names with a directory prefix. inelegant, bu… by @petersilva in #1172
• Fix #1173: problems stopping instances by @reidsunderland in #1178
• fix #1166 improved download decision criteria by @petersilva in #1175
• fix Ubuntu 20.04 flow tests by @reidsunderland in #1186
• Implement HTTP(S) stat using urllib, and HTTPS transfer class minor refactor by @reidsunderland in #1202
• Change stop output to show the number of processes still running (like v2) by @reidsunderland in #1204
• fix rejectPercent maybe other stuff? by @petersilva in #1192
• maybe fix #1195 more log fields by default by @petersilva in #1197
• Fix badly expiring timers by @petersilva in #1208
• Add scheduled_time for #1209 by @andreleblanc11 in #1210
• Issue1198 - charset option for ftp. by @petersilva in #1200
• Add more wiggle room in scheduled_flow for accumulated wait overhead by @andreleblanc11 in #1217
• Fix for Issue1218 - Permission problems in directory creations on FTP/SFTP transfers by @andreleblanc11 in #1220

Full Changelog: v3.00.54rc3...v3.00.55rc1

Packages Available

Ubuntu PPA: https://launchpad.net/~ssc-hpc-chp-spc/+archive/ubuntu/metpx-pre-release

PyPI: https://pypi.org/project/metpx-sr3/3.0.55rc1/

• RPM package for Redhat Version 8 (rh8) and 9 (rh9) attached here, install with rpm -ivh ... note:

  • Some dependencies are not readily available. May have to obtain packages
    from pip to get a fully functional installation. Verify with sr3 features after installation, to determine if everything needed is available.

• self-extracting windows installer also attached.

v2.24.08post2

• Version 2 is now in maintenance mode. Only install if you already have a version 2 configuration to maintain.
• There is nothing in v2 that is not working as well or better in sr3.
• It is recommended that users of v2 upgrade their configurations and begin transition to sr3. (although v2 will remain available for the foreseeable future.)
• Sarracenia version 3 is recommended for new installations and is a super-set of version 2.
• known issues with v2 that we do not plan to address ( https://github.com/MetPX/sarracenia/issues?q=is%3Aopen+is%3Aissue+label%3Av2only+label%3Awontfix )
  • does not support MQTT.

This release includes cleanups of issues encountered upgrading main data pumps with legacy v2 stream. This release has good compatibility with sr3 as well as older v2 configurations, and crashes less that older v2 versions.

What's Changed

• revert ftp charset change.
• V2 fix the problems found during v2 upgrade of main pump. by @petersilva in #1182

Full Changelog: v2.24.08...v2.24.08post2

Installation

Guide: https://github.com/MetPX/sarracenia/blob/v2_stable/doc/Install.rst

packages available:

Ubuntu PPA: https://launchpad.net/~ssc-hpc-chp-spc/+archive/ubuntu/metpx
python: https://pypi.org/project/metpx-sarracenia/

v2.24.08post1 possibly last ever v2 release.

• Version 2 is now in maintenance mode. Only install if you already have a version 2 configuration to maintain.
• There is nothing in v2 that is not working as well or better in sr3.
• It is recommended that users of v2 upgrade their configurations and begin transition to sr3. (although v2 will remain available for the foreseeable future.)
• Sarracenia version 3 is recommended for new installations and is a super-set of version 2.
• known issues with v2 that we do not plan to address ( https://github.com/MetPX/sarracenia/issues?q=is%3Aopen+is%3Aissue+label%3Av2only+label%3Awontfix )
  • does not support MQTT.

This release includes cleanups of issues encountered upgrading main data pumps with legacy v2 stream. This release has good compatibility with sr3 as well as older v2 configurations, and crashes less that older v2 versions.

This release is going to pre-release repositories for a little while, will eventually re-generate for the
stable one (after some pre-release testing.)

What's Changed

• Fix for #1144 - don't crash when the config doesn't exist for a missing instance by @reidsunderland in #1145
• V2 fix the problems found during v2 upgrade of main pump. by @petersilva in #1182

Full Changelog: v2.24.08...v2.24.08post1

Installation

Guide: https://github.com/MetPX/sarracenia/blob/v2_stable/doc/Install.rst

Publishing only to launchpad pre-release repository for (tentatively) one week. Once the pre-release phase
is done, will upload to stable repo, and pypi.

packages available:

Ubuntu PPA: https://launchpad.net/~ssc-hpc-chp-spc/+archive/ubuntu/metpx-pre-release
python: https://pypi.org/project/metpx-sarracenia/

v3.00.54p1 (aka post1 aka post2) - fileAgeMin fix

What's Changed

• 3.00.54 fixed typo in fileAgeMin implementation

Full Changelog: v3.00.54...v3.00.54p1

Packages Available:

Ubuntu PPA: https://launchpad.net/~ssc-hpc-chp-spc/+archive/ubuntu/metpx

• NOTE on ubuntu >= 22.04, this version is called 3.00.54post2 rather than p1 (due to technical mishaps during publication.)

PyPI: https://pypi.org/project/metpx-sr3/3.0.54post1/

• NOTE on pypi, this version has the post1 suffix, for similar, but different mishaps.

• (TODO) RPM package for Redhat Version 8 (rh8) and 9 (rh9) attached here, install with rpm -ivh ... note:

  • Some dependencies are not readily available. May have to obtain packages
    from pip to get a fully functional installation. Verify with sr3 features after installation, to determine if everything needed is available.

• (TODO) self-extracting windows installer also attached.

v3.00.54 - Wololo!, Voluble Status and Partitioned Flows

What's new

• GeoJSON style filtering support
• Implicit FTPS support
• PAHO MQTT V2 support
• Default queueName change
• Partitioned transfers now working and included in CI/CD testing (sr3 is now a super-set of v2, no remaining functionality regressions)
• The --wololo option speeds up the v2 to sr3 converter substantially.
• In place of just running or stopped in sr3 status, there is now: idle, lagging, slow, retry, rejecting, cpuSlow indicating various ways flows could be in difficulty or require tuning.
• HTTP scheduled flows can now query the resource using metadata calls for more effective polling.
• Many fixes for issues people ran into during deployments.
• Also fixes for problems with statehost setting (important for HPC deployments.)

What's Changed

• Updates to NASA Auth plugins and examples by @reidsunderland in #1042
• Issue1036 - Add unit tests for AM code by @andreleblanc11 in #1044
• Fix sanity not starting missing instances by @reidsunderland in #1053
• Build Unit test boilerplate by @gcglinton in #1052
• multi-instances will be partial, not missing by @petersilva in #1059
• Dev doc refactor release instructions by @petersilva in #1057
• to convert overwriting, one must Wololo by @petersilva in #1055
• Fix #1008 and update documentation by @mshak2 in #1056
• Fix bug caused by different variable names by @reidsunderland in #1060
• Unit Test Engineering wmo00 by @petersilva in #1048
• Unit Test Engineering of the sarracenia/config.py module. by @petersilva in #1058
• Fixes search function and logo, and adds favicon for doc website by @mshak2 in #1062
• Fix #1064 by adding try/except by @reidsunderland in #1065
• #980 Expand scope of isProblem. by @andreleblanc11 in #1046
• #1036 Add more AM unit tests for latest bug findings by @andreleblanc11 in #1068
• Issue824 morestates and issue966 and... other stuff by @petersilva in #1067
• please stop infinite loop by calling self by @petersilva in #1070
• disabled configs were showing as "unknown" by @petersilva in #1073
• Fix #1074 by @andreleblanc11 in #1075
• Bug fix for AM sender by @andreleblanc11 in #1078
• Doc updates by @petersilva in #1072
• Fix mkdir failures on hpc by @petersilva in #1076
• Fix #1079 by @andreleblanc11 in #1081
• remove plugins that are useless and will not run. by @petersilva in #1082
• Add post_baseUrl to delete on post, changes to logging by @reidsunderland in #1050
• adding new_pathWrite() as per Reid suggestion by @petersilva in #1049
• Issue1080 2 by @petersilva in #1083
• Add scheduled flow with http metadata by @reidsunderland in #1084
• Issue1035 retry too much, reduce cpu to some extent... by @petersilva in #1086
• fix typo rel_relPath -> new_relPath by @reidsunderland in #1092
• Scheduled flow with HTTP metadata: add variable expansion for the path by @reidsunderland in #1088
• honour statehost in one mkdir case by @petersilva in #1087
• fix #1091 download and/or sender plugin error crashes by @petersilva in #1093
• Doc feedback by @petersilva in #1094
• issue #982 detail in cleanup was not working when statehost set by @petersilva in #1096
• more robust validation of return value from destfnscript by @petersilva in #1085
• Add directory to send, etc. errors by @reidsunderland in #1099
• only run after_post after we actually post by @petersilva in #1101
• fix #1095 crash in sender when size missing from message by @petersilva in #1102
• Issue1089 fileAgeMin and fileAgeMax should be honoured by all components by @petersilva in #1100
• Finalize Azure Transfer tests by @gcglinton in #1051
• fix crash where 'dictionary changed size during iteration' by @reidsunderland in #1109
• Change chmod -> utime, i think it was a typo? by @reidsunderland in #1108
• Issue1104 msg rate cpu by @petersilva in #1105
• Random fixes by @petersilva in #1103
• Restoring blockmanifest transfers by @petersilva in #1110
• Issue1097 fractional rates by @petersilva in #1106
• #1115 stopped flows were marked cpuS. fix by @petersilva in #1116
• Adjust column widths, shorten units. by @petersilva in #1117
• Enable blocked tests by @petersilva in #1112
• fix #1002 putAccellerated when file names have colons by @petersilva in #1118
• Implement support for implicit FTPS on port 990 by @reidsunderland in #1122
• Fix the NASA CMR plugin, bug when using the other data source by @reidsunderland in #1123
• fix test for NASA CMR rename by @reidsunderland in #1124
• Fix logging for default poll by @reidsunderland in #1125
• Paho mqtt v2 support by @petersilva in #1119
• Issue33 - default queue name change. by @petersilva in #1128
• Issue1043 geojson by @Jeanc16888 in #1130
• Release doc touchups by @petersilva in #1136
• fix #1089 - created getIDStr function for Sarracenia.Message by @petersilva in #1135
• Fix #1133 by @andreleblanc11 in #1134
• Adding copyright explanation by @petersilva in #1139
• Authentication plugin refactor by @reidsunderland in #1142
• Fix for #1141 by @mshak2 in #1146

New Contributors

• @Jeanc16888 made their first contribution in #1130

Full Changelog: v3.0.53...v3.00.54

Packages Available:

Ubuntu PPA: https://launchpad.net/~ssc-hpc-chp-spc/+archive/ubuntu/metpx

PyPI: https://pypi.org/project/metpx-sr3/3.0.54/

• RPM package for Redhat Version 8 (rh8) and 9 (rh9) attached here, install with rpm -ivh ... note:

  • Some dependencies are not readily available. May have to obtain packages
    from pip to get a fully functional installation. Verify with sr3 features after installation, to determine if everything needed is available.

• self-extracting windows installer also attached.

v2.24.08 - Persistent messages

Status

• Version 2 is now in maintenance mode. Only install if you already have a version 2 configuration to maintain.
• Sarracenia version 3 is recommended for newer installations and is a super-set of version 2.

What's Changed

• V2 issue639 by @petersilva in #838
• Set delivery_mode=2 (persistent) for v2 by @reidsunderland in #1006
• Make persistent a configurable option (v2) by @reidsunderland in #1012
• Fix for #1144 - don't crash when the config doesn't exist for a missing instance by @reidsunderland in #1145

Full Changelog: v2.23.07...v2.24.08

Installation

Guide: https://github.com/MetPX/sarracenia/blob/v2_stable/doc/Install.rst

Publishing only to launchpad pre-release repository for (tentatively) one week. Once the pre-release phase
is done, will upload to stable repo, and pypi.

packages available:

Ubuntu PPA: https://launchpad.net/~ssc-hpc-chp-spc/+archive/ubuntu/metpx
python: https://pypi.org/project/metpx-sarracenia/

v3.00.54rc3

What's Changed

• Issue1043 geojson by @Jeanc16888 in #1130
• Release doc touchups by @petersilva in #1136
• fix #1089 - created getIDStr function for Sarracenia.Message by @petersilva in #1135
• Fix #1133 by @andreleblanc11 in #1134
• Adding copyright explanation by @petersilva in #1139
• Authentication plugin refactor by @reidsunderland in #1142
• Fix for #1141 by @mshak2 in #1146

New Contributors

• @Jeanc16888 made their first contribution in #1130

Full Changelog: v3.00.54rc2...v3.00.54rc3

Packages Available:

Ubuntu PPA: https://launchpad.net/~ssc-hpc-chp-spc/+archive/ubuntu/metpx-pre-release

PyPI: https://pypi.org/project/metpx-sr3/3.0.54rc3/

• RPM package for Redhat Version 8 (rh8) and 9 (rh9) attached here, install with rpm -ivh ... note:

  • Some dependencies are not readily available. May have to obtain packages
    from pip to get a fully functional installation. Verify with sr3 features after installation, to determine if everything needed is available.

• self-extracting windows installer also attached.

v3.00.54rc2

What's Changed

• #1115 stopped flows were marked cpuS. fix by @petersilva in #1116
• Adjust column widths, shorten units. by @petersilva in #1117
• Enable blocked tests by @petersilva in #1112
• fix #1002 putAccellerated when file names have colons by @petersilva in #1118
• Implement support for implicit FTPS on port 990 by @reidsunderland in #1122
• Fix the NASA CMR plugin, bug when using the other data source by @reidsunderland in #1123
• fix test for NASA CMR rename by @reidsunderland in #1124
• Fix logging for default poll by @reidsunderland in #1125
• Paho mqtt v2 support by @petersilva in #1119
• Issue33 - default queue name change. by @petersilva in #1128
• 3 00 54 rc2 prep by @petersilva in #1129

Full Changelog: v3.00.54rc1...v3.00.54rc2

Packages Available:

Ubuntu PPA: https://launchpad.net/~ssc-hpc-chp-spc/+archive/ubuntu/metpx-pre-release

PyPI: https://pypi.org/project/metpx-sr3/3.0.54rc2/

• RPM package for Redhat Version 8 (rh8) and 9 (rh9) attached here, install with rpm -ivh ... note:

  • Some dependencies are not readily available. May have to obtain packages
    from pip to get a fully functional installation. Verify with sr3 features after installation, to determine if everything needed is available.

• self-extracting windows installer also attached.

v3.00.54rc1 Wololo! and Voluble Status

The Wololo! release speeds up the v2 to sr3 converter substantially, and added much more helpful flow states
to the sr3 status display. In place of just running or stopped, there is now: idle, lagging, slow, retry, rejecting, cpuSlow
indicating various ways flows could be in difficulty or require tuning. HTTP scheduled flows can now
query the resource using metadata calls for more effective polling. Many fixes for issues people ran into during deployments.
Also fixes for problems with statehost setting (important for HPC deployments.)

BUGS

• #1115 .. so there will be an rc2.

What's Changed

• Updates to NASA Auth plugins and examples by @reidsunderland in #1042
• Issue1036 - Add unit tests for AM code by @andreleblanc11 in #1044
• Fix sanity not starting missing instances by @reidsunderland in #1053
• Build Unit test boilerplate by @gcglinton in #1052
• multi-instances will be partial, not missing by @petersilva in #1059
• Dev doc refactor release instructions by @petersilva in #1057
• to convert overwriting, one must Wololo by @petersilva in #1055
• Fix #1008 and update documentation by @mshak2 in #1056
• Fix bug caused by different variable names by @reidsunderland in #1060
• Unit Test Engineering wmo00 by @petersilva in #1048
• Unit Test Engineering of the sarracenia/config.py module. by @petersilva in #1058
• Fixes search function and logo, and adds favicon for doc website by @mshak2 in #1062
• Fix #1064 by adding try/except by @reidsunderland in #1065
• #980 Expand scope of isProblem. by @andreleblanc11 in #1046
• #1036 Add more AM unit tests for latest bug findings by @andreleblanc11 in #1068
• Issue824 morestates and issue966 and... other stuff by @petersilva in #1067
• please stop infinite loop by calling self by @petersilva in #1070
• disabled configs were showing as "unknown" by @petersilva in #1073
• Fix #1074 by @andreleblanc11 in #1075
• Bug fix for AM sender by @andreleblanc11 in #1078
• Doc updates by @petersilva in #1072
• Fix mkdir failures on hpc by @petersilva in #1076
• Fix #1079 by @andreleblanc11 in #1081
• remove plugins that are useless and will not run. by @petersilva in #1082
• Add post_baseUrl to delete on post, changes to logging by @reidsunderland in #1050
• adding new_pathWrite() as per Reid suggestion by @petersilva in #1049
• Issue1080 2 by @petersilva in #1083
• Add scheduled flow with http metadata by @reidsunderland in #1084
• Issue1035 retry too much, reduce cpu to some extent... by @petersilva in #1086
• fix typo rel_relPath -> new_relPath by @reidsunderland in #1092
• Scheduled flow with HTTP metadata: add variable expansion for the path by @reidsunderland in #1088
• honour statehost in one mkdir case by @petersilva in #1087
• fix #1091 download and/or sender plugin error crashes by @petersilva in #1093
• Doc feedback by @petersilva in #1094
• issue #982 detail in cleanup was not working when statehost set by @petersilva in #1096
• more robust validation of return value from destfnscript by @petersilva in #1085
• Add directory to send, etc. errors by @reidsunderland in #1099
• only run after_post after we actually post by @petersilva in #1101
• fix #1095 crash in sender when size missing from message by @petersilva in #1102
• Issue1089 fileAgeMin and fileAgeMax should be honoured by all components by @petersilva in #1100
• Finalize Azure Transfer tests by @gcglinton in #1051
• fix crash where 'dictionary changed size during iteration' by @reidsunderland in #1109
• Change chmod -> utime, i think it was a typo? by @reidsunderland in #1108
• Issue1104 msg rate cpu by @petersilva in #1105
• Random fixes by @petersilva in #1103
• Restoring blockmanifest transfers by @petersilva in #1110
• Issue1097 fractional rates by @petersilva in #1106

Full Changelog: v3.00.53pre2...v3.00.54rc1

Packages Available:

Ubuntu PPA: https://launchpad.net/~ssc-hpc-chp-spc/+archive/ubuntu/metpx-pre-release

PyPI: https://pypi.org/project/metpx-sr3/3.0.54rc1/
Note: uploaded pypi package from ubuntu 22.04... installation on older OS & python versions may
be compromised (error, will be corrected in next release.)

• (Coming Soon) RPM package for fedora core 39 (fc39)

• (Coming Soon) RPM package for Redhat Version 8 (rh8) attached here, install with rpm -ivh ... note:

  • Some dependencies are not readily available. May have to obtain packages
    from pip to get a fully functional installation. Verify with sr3 features after installation, to determine if everything needed is available.

• (Coming Soon) self-extracting windows installer also attached.