v3.7.0 release tracker #101

cognifloyd · 2022-05-02T17:16:47Z

Pre-Release PR Reviews

@cognifloyd reviewed all PRs across all st2 repos involved in the release and merged everything I could. I moved others from the 3.7 project+milestone to 3.8.

st2chatops release prep

We needed to get StackStorm/hubot-stackstorm#214 included in st2chatops which involved release hubot-stackstorm.

@cognifloyd released hubot-stackstorm v0.12.0:

created release+tag in github ui v0.12.0
ran npm login on cli and logged in as myself
ran npm publish in my clone of the hubot-stackstorm repo (checked out on master at tag v0.12.0)
verified that it was published: https://www.npmjs.com/package/hubot-stackstorm/v/0.12.0

Added that to st2chatops in StackStorm/st2chatops#175

@cognifloyd: I have never made a release on NPM. It would have been simpler to just log in as storminstanley on the command line, and then get the OTP from a senior maintainer. But I didn't know that so, I logged in first on the website. Then, no senior maintainers were available to get me another OTP to login on CLI which is required to publish a release. That's why I created my own account and added myself to hubot-stackstorm so I could make the release.

aside: the st2chatops mess will become much better with the transition from JS based hubot to Python based OpsDroid. I'll be working more on that later this year.

🔗 Testing Before Release

volunteers welcome - @cognifloyd will not volun-tell anyone to do this as I cannot know others' time constraints.

steps 4 and 5: create and e2e test the staging-stable packages

These tests highlighted a couple of issues:

We must turn on our EC2 VMs before running workflows: ubuntu-build-itest001, ubuntu-build002 and ubuntu-build003
The upgrade workflow does not automatically handle python upgrades.
- The RockyLinux 8 tests failed because we didn't run sudo st2ctl reload --register-recreate-virtualenvs.
- All the tests passed after running that on the test VM.
The upgrade workflow does not cleanup VMs on failure. So, I had to trigger st2cd.destroy_vm for a lot of VMs.
It did not cause issues, but I noticed that st2cd.setup_e2e_tests should be updated to use python3.8 by default on EL8.
The upgrade notes need to be updated to show --register-recreate-virtualenvs instead of --register-setup-recreate-virtualenvs.

🔗 step 6 staging-stable testing

BOX=ubuntu/bionic64 REPO_TYPE=staging RELEASE=stable vagrant up
BOX=ubuntu/focal64 REPO_TYPE=staging RELEASE=stable vagrant up
BOX=centos/7 REPO_TYPE=staging RELEASE=stable vagrant up
BOX=rockylinux/8 REPO_TYPE=staging RELEASE=stable vagrant up

volunteers welcome - @cognifloyd will not volun-tell anyone to do this as I cannot know others' time constraints.

🔗 step 8 stable testing

BOX=ubuntu/bionic64 RELEASE=stable vagrant up
BOX=ubuntu/focal64 RELEASE=stable vagrant up
BOX=centos/7 RELEASE=stable vagrant up
BOX=rockylinux/8 RELEASE=stable vagrant up

volunteers welcome - @cognifloyd will not volun-tell anyone to do this as I cannot know others' time constraints.

🔗 step 9 tag releases

st2client on pypi
st2 tag
packer-st2
- ❗ packet.net errors prevented the vagrant box build
- https://app.circleci.com/pipelines/github/StackStorm/packer-st2/1503/workflows/b28059d8-82c1-4248-b925-648c0a9eb005/jobs/4284
- https://app.circleci.com/pipelines/github/StackStorm/packer-st2/1502/workflows/e109e3e4-bdfd-4e0a-8260-a1b89830a808/jobs/4282
- https://app.vagrantup.com/stackstorm/boxes/st2
api-docs (new /clone endpoint for actions is present, so this was updated)
docker

Also merged these changelog PRs:

🔗 step 10 update st2cd and st2ci packs

created 3.7.0 Release Fixes st2cd#472 to capture all the changes I had to make throughout this release
bumped version in StackStorm/st2cd@7ee062a
I don't have sudo access, and the remote for st2cd was pointing to a checkout in /home/amanda so I reinstalled the packs with packs.install and these addresses:
- git@github.com:StackStorm/st2cd.git
- git@github.com:StackStorm/st2ci.git
validated that rules look okay.

step 12 create the release

https://github.com/StackStorm/st2/releases/tag/v3.7.0

step 13 make sure new dev builds succeed

step 14 Check PackageCloud for new packages in unstable channel

u18
u20
el8
el7

remaining steps

step 15 Prep development version on master branch of st2-dockerfiles
step 16 Ensure that the AWS instances EC2 instances are all stopped:
- ubuntu-build-itest001
- ubuntu-build002
- ubuntu-build003
step 17 Delete the datastore keys
step 18 Update the version numbers in steps 3, 12, and 13 for the next release manager

blog article

draft
review
publish

Close v3.7.0

close this issue
close https://github.com/orgs/StackStorm/projects/23

The text was updated successfully, but these errors were encountered:

cognifloyd · 2022-05-02T17:27:49Z

Notes from Testing Before Release:

bionic

bash installer in vagrant box:

❗ installing crudini also installs python2.7
configured repo

  st2bionic: 20220502T165935+0000 ##########################################################
  st2bionic: 20220502T165935+0000 #### Following versions of packages will be installed ####
  st2bionic: 20220502T165935+0000 st2=3.7dev-54
  st2bionic: 20220502T165935+0000 st2web=3.7dev-19
  st2bionic: 20220502T165935+0000 st2chatops=3.7dev-6
  st2bionic: 20220502T165935+0000 ##########################################################

while installing st2, had an error: edit: this is expected as st2chatops is not installed yet.

  st2bionic: 20220502T165953+0000 Failed to start st2chatops.service: Unit st2chatops.service not found.
  st2bionic: 20220502T165953+0000 ##### st2 components status #####
  ...
  st2bionic: 20220502T165954+0000 st2chatops is not running.

just after creating datastore key but before creating token, an st2 command seems to have failed: edit: nope. the verify_st2 function explicitly calls st2 -h.

  st2bionic: 20220502T170025+0000 Key written to /etc/st2/keys/datastore_key.json. Secure the permissions so only StackStorm API process and StackStorm admin access the file.
  st2bionic: 20220502T170040+0000 st2 3.7dev (abb694b85), on Python 3.6.9
  st2bionic: 20220502T170041+0000 usage: st2 [-h] [--version] [--url BASE_URL] [--auth-url AUTH_URL]
  ...
  st2bionic: 20220502T170042+0000 +----------+----------------------------------+
  st2bionic: 20220502T170042+0000 | Property | Value                            |
  st2bionic: 20220502T170042+0000 +----------+----------------------------------+
  st2bionic: 20220502T170042+0000 | user     | st2admin                         |

subsequent st2 calls seem to have succeeded, but the logs are not clear on what calls are being made (ie why is it running these things?) edit: the verify_st2 function should probably have set -x (future enhancement).
nodejs and st2chatops get installed later, so the earlier st2chatops error is ~~probably~~ not an error.
bash installer says it succeeded

st2-self-check:

reported OK
st2 execution list shows all executions succeeded

st2web manual testing:

history: drop down filters work as expected.
history: click magnifying glass, type something, hit enter -> page reloads; but only first time.
❗ typing anything in magnifying glass / search field does not do anything (no filtering of results).
for actions and rules env arg does not accept key1=val1,key2=val2 format anymore. It has to be a full JSON object, {"key1":"val2","key2":"val2"}.
JSON.parse: unexpected character at line 1 column 1 of the JSON data
clone/delete action modal dialogs do not account for window size correctly
clone dialog has an empty drop down list for destination pack, and I can't type a different pack name.
Why does the Flow UI still show "mistral-v2" as an option in the Runner Type drop down (under action meta)?
Why can't I type a new pack name in the packs drop down of Flow?
when creating a new action if I select something other than default, then I default is removed from the list (prob because nothing is in the default pack).

focal

bash installer in vagrant box:

looks like the installer is relying on netstat when that's not available

  st2focal: 20220502T194503+0000 ########################################################
  st2focal: 20220502T194503+0000           Installing StackStorm unstable 3.7dev
  st2focal: 20220502T194503+0000 ########################################################
  st2focal: 20220502T194503+0000 sudo: netstat: command not found
  st2focal: 20220502T194503+0000 sudo: netstat: command not found
  st2focal: 20220502T194503+0000 sudo: netstat: command not found
  st2focal: 20220502T194503+0000 sudo: netstat: command not found
  st2focal: 20220502T194503+0000 sudo: netstat: command not found
  st2focal: 20220502T194503+0000 sudo: netstat: command not found
  st2focal: 20220502T194503+0000 sudo: netstat: command not found
  st2focal: 20220502T194503+0000 sudo: netstat: command not found
  st2focal: 20220502T194503+0000 sudo: netstat: command not found
  st2focal: 20220502T194503+0000 sudo: netstat: command not found

  st2focal: 20220502T194712+0000 ##########################################################
  st2focal: 20220502T194712+0000 #### Following versions of packages will be installed ####
  st2focal: 20220502T194712+0000 st2=3.7dev-54
  st2focal: 20220502T194712+0000 st2web=3.7dev-19
  st2focal: 20220502T194712+0000 st2chatops=3.7dev-6
  st2focal: 20220502T194712+0000 ##########################################################

bash installer says it succeeded

st2-self-check:

reported OK
st2 execution list shows all executions succeeded

st2web manual testing:

history: click magnifying glass, type something, hit enter -> page reloads; but only first time. then I had to refresh before the drop downs would work.
can't use esc to close modal clone/delete dialogs
no diff in behavior vs bionic

rocky 8

  st2el8: 20220503T025435+0000 ##########################################################
  st2el8: 20220503T025435+0000 #### Following versions of packages will be installed ####
  st2el8: 20220503T025435+0000 st2-3.7dev-54
  st2el8: 20220503T025435+0000 st2web-3.7dev-19
  st2el8: 20220503T025435+0000 st2chatops-3.7dev-6
  st2el8: 20220503T025435+0000 ##########################################################

I had to increase the vagrant disk size from 5 to 6 GB using the vagrant-disksize plugin and these instructions

yum install nodejs st2chatops causes some ugly warnings:

  st2el8: 20220503T025655+0000 Invalid configuration value: failovermethod=priority in /etc/yum.repos.d/nodesource-el8.repo; Configuration: OptionBinding with id "failovermethod" does not exist
  st2el8: 20220503T025655+0000 Invalid configuration value: failovermethod=priority in /etc/yum.repos.d/nodesource-el8.repo; Configuration: OptionBinding with id "failovermethod" does not exist

bash installer says it succeeded

st2-self-check:

reported OK
st2 execution list shows all executions succeeded

st2web manual testing:

no diff in behavior vs bionic or focal

nzlosh · 2022-05-02T23:49:51Z

CentOS7

st2-self-check failed on the enquiry chain test. The response schema expects a string, but an integer is given.

Attempting Test tests.test_inquiry_chain...ERROR! (38s)
Test output: ..................
id: 62705b6b7483bb205ae6b875
action.ref: tests.test_inquiry_chain
parameters: 
  protocol: http
  token: 0521454b412b4c829e2ce14c5272c3ef
status: failed
result_task: valid_response
result: 
  failed: true
  return_code: 1
  stderr: ''
  stdout: 'ERROR: 500 Server Error: Internal Server Error
    MESSAGE: Unable to resume liveaction "6270558d7483bb205ae6b7a6" because it is in "pausing" state and not in "paused" state. for url: http://12
7.0.0.1:9101/inquiries/6270558e93aa732b28abc467
    '
  succeeded: false


  cmd: 'sleep 3 ; st2 inquiry respond -r ''{"secondfactor": 123}'' 6270558e93aa732b28abc467'
  env:
    ST2_API_URL: http://127.0.0.1:9101
    ST2_AUTH_TOKEN: 0521454b412b4c829e2ce14c5272c3ef
    ST2_AUTH_URL: http://127.0.0.1:9100
    ST2_BASE_URL: http://127.0.0.1
status: failed (4s elapsed)
start_timestamp: Mon, 02 May 2022 22:30:19 UTC
end_timestamp: Mon, 02 May 2022 22:30:23 UTC
log: 
  - status: requested
    timestamp: '2022-05-02T22:30:19.253000Z'
  - status: scheduled
    timestamp: '2022-05-02T22:30:19.517000Z'
  - status: running
    timestamp: '2022-05-02T22:30:19.735000Z'
  - status: failed
    timestamp: '2022-05-02T22:30:24.015000Z'
result: 
  failed: true
  return_code: 1
  stderr: ''
  stdout: "ERROR: 400 Client Error: Bad Request
MESSAGE: Response for inquiry "6270558e93aa732b28abc467" did not pass schema validation. 123 is not of type 'string'

Failed validating 'type' in schema['properties']['secondfactor']:
    {'description': 'Please enter second factor for authenticating to '
                    '"foo" service',
     'required': True,
     'type': 'string'}

Update: This issue appeared when using the vagrant repo install script https://raw.githubusercontent.com/StackStorm/st2vagrant/master/scripts/install_st2.sh. When using the officially documented install script https://stackstorm.com/packages/install.sh this issue is not encountered and the st2-self-check passes.

amanda11 · 2022-05-03T16:11:19Z

@cognifloyd Re web testing:

typing anything in magnifying glass / search field does not do anything (no filtering of results).
Amanda: Is it the problem reported here: filter function on the history page does not work st2web#925? (follow comments and I found the pull-down filters were ok, but not the edit search bit).
for actions and rules env arg does not accept key1=val1,key2=val2 format anymore. It has to be a full JSON object, {"key1":"val2","key2":"val2"}.
Amanda: Interesting. Perhaps we should add that to release notes, or to the upgrade notes in st2docs?
clone/delete action modal dialogs do not account for window size correctly
clone dialog has an empty drop down list for destination pack, and I can't type a different pack name.
Why does the Flow UI still show "mistral-v2" as an option in the Runner Type drop down (under action meta)?
Why can't I type a new pack name in the packs drop down of Flow?
Amanda: Do you mean when you create a new action in workflow composer? And is this so you can also create a new pack? Or just you can't type in to filter list.
when creating a new action if I select something other than default, then I default is removed from the list (prob because nothing is in the default pack).
Amanda: I can confirmed that if you did have actions in defaults, then you can still go back and choose it. Verified this is same behaviour as in ST2 3.6. Also you can select default, by the oddity of selecting the blank entry - if you choose the blank entry in the pull-down, it then populates the field with "default".
for actions and rules env arg does not accept key1=val1,key2=val2 format anymore. It has to be a full JSON object, {"key1":"val2","key2":"val2"}.
Amanda: verified this is same behaviour in ST2 3.6. So not new. Though we should raise an issue to at least change the help for this as the help says to add key=val1

cognifloyd · 2022-05-03T16:48:13Z

typing anything in magnifying glass / search field does not do anything (no filtering of results).
Amanda: Is it the problem reported here: StackStorm/st2web#925? (follow comments and I found the pull-down filters were ok, but not the edit search bit).

Yes, that's it. And I get the same behavior in installs of 3.4.1 and 3.6.0 so this is an old bug.

these two are interrelated:

Why can't I type a new pack name in the packs drop down of Flow?
Amanda: Do you mean when you create a new action in workflow composer? And is this so you can also create a new pack? Or just you can't type in to filter list.

and

when creating a new action if I select something other than default, then I default is removed from the list (prob because nothing is in the default pack).
Amanda: I can confirmed that if you did have actions in defaults, then you can still go back and choose it. Verified this is same behaviour as in ST2 3.6. Also you can select default, by the oddity of selecting the blank entry - if you choose the blank entry in the pull-down, it then populates the field with "default".

When I create or edit a workflow, changing the pack doesn't allow me to type in the pack field.
If there are no actions/rules etc in the default pack, then you can't select the default pack.

When I create rules in the UI (vs saving in a git repo) I like to use __webui__ as the pack name (or if it's an adhoc thing not created in the webui, I would use some other dunder name) -- that's my personal convention to remember that this resource is only in the database, not preserved in git. The workflow composer would create a file, but that file would still not be committed in git or pushed to the git server, so I would still like to use something like __adhoc__ when creating new actions in the UI.

Then again, I don't really use Flow much, and this is probably a new behavioral request, so it's not a blocker.

for actions and rules env arg does not accept key1=val1,key2=val2 format anymore. It has to be a full JSON object, {"key1":"val2","key2":"val2"}.
Amanda: Interesting. Perhaps we should add that to release notes, or to the upgrade notes in st2docs?

Amanda: verified this is same behaviour in ST2 3.6. So not new. Though we should raise an issue to at least change the help for this as the help says to add key=val1

The help text is still valid for CLI usage. We should probably add support for converting this key=val syntax to json in the webui.
Since it's not a regression in this release, it is definitely not a blocker.

cognifloyd · 2022-05-06T02:48:58Z

Dangit. We have a blocker for RockyLinux 8

$ /opt/stackstorm/st2/bin/python --version
Python 3.8.8
$ /opt/stackstorm/virtualenvs/st2/bin/python --version
Python 3.6.8

It's building virtualenvs with the wrong python. That means that the dependencies in /opt/stackstorm/st2 cannot be inherited by /opt/stackstorm/virtualenvs/* which cause a ModeuleNotFoundError: No module named 'orjson'

I'm going to try to duplicate this in the st2vagrant box.

edit: added results

Nope. st2vagrant had the correct version of python in all the virtualenvs.

This might have something to do with the upgrade. Once I rebuilt the virtualenv on st2-pkg-upgrade-staging-stable-el8, the action in question succeeded. So maybe the upgrade workflow does not include updating from python3.6 to python3.8...

cognifloyd · 2022-05-06T03:40:42Z

Basically, this workflow does not follow the migration guide for RockyLinux8:
https://github.com/StackStorm/st2ci/blob/master/actions/workflows/st2_pkg_upgrade_e2e_test.yaml
https://docs.stackstorm.com/3.7/install/upgrades.html#version-specific-changes-migration-scripts

It does not run:

sudo st2ctl reload --register-recreate-virtualenvs

Hmm. And the migration guide is wrong, it shows: --register-setup-recreate-virtualenvs instead of --register-recreate-virtualenvs.

So, I manually ran the rebuild command and re-triggered the e2e tests. Everything passed. So, it's not a blocker; we just need a minor docs update.

Also, we need to update https://github.com/StackStorm/st2cd/blob/master/actions/setup_e2e_tests.sh to use python3.8 on EL8

arm4b · 2022-05-06T08:34:16Z

That's a good catch before shipping it to prod! 👍

amanda11 · 2022-05-09T08:58:08Z

Basically, this workflow does not follow the migration guide for RockyLinux8: https://github.com/StackStorm/st2ci/blob/master/actions/workflows/st2_pkg_upgrade_e2e_test.yaml https://docs.stackstorm.com/3.7/install/upgrades.html#version-specific-changes-migration-scripts

It does not run:
sudo st2ctl reload --register-recreate-virtualenvs
Hmm. And the migration guide is wrong, it shows: --register-setup-recreate-virtualenvs instead of --register-recreate-virtualenvs.

So, I manually ran the rebuild command and re-triggered the e2e tests. Everything passed. So, it's not a blocker; we just need a minor docs update.

Also, we need to update https://github.com/StackStorm/st2cd/blob/master/actions/setup_e2e_tests.sh to use python3.8 on EL8

Possibly should have had the st2ctl command in:
"st2ci/actions/st2_pkg_upgrade_deps_el8.sh"

Those files are to do version specific things on upgrade. Though I guess we'll never run it again to do an upgrade...

cognifloyd · 2022-05-13T17:06:12Z

OK. Release announcement blog has been published: https://stackstorm.com/2022/05/10/stackstorm-3-7-0-released/

The only thing we didn't do was fix the packer-st2 build (see #101 (comment)). I'm closing this as complete.

cognifloyd added the release label May 2, 2022

cognifloyd added this to the 3.7.0 milestone May 2, 2022

cognifloyd self-assigned this May 2, 2022

arm4b mentioned this issue May 4, 2022

Stackstorm compatiblity with Mongo 4.4 StackStorm/st2#5639

Open

cognifloyd closed this as completed May 13, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

v3.7.0 release tracker #101

v3.7.0 release tracker #101

cognifloyd commented May 2, 2022 •

edited

Loading

cognifloyd commented May 2, 2022 •

edited

Loading

nzlosh commented May 2, 2022 •

edited

Loading

amanda11 commented May 3, 2022 •

edited

Loading

cognifloyd commented May 3, 2022 •

edited

Loading

cognifloyd commented May 6, 2022 •

edited

Loading

cognifloyd commented May 6, 2022 •

edited

Loading

arm4b commented May 6, 2022

amanda11 commented May 9, 2022

cognifloyd commented May 13, 2022

v3.7.0 release tracker #101

v3.7.0 release tracker #101

Comments

cognifloyd commented May 2, 2022 • edited Loading

Pre-Release PR Reviews

st2chatops release prep

🔗 Testing Before Release

steps 4 and 5: create and e2e test the staging-stable packages

🔗 step 6 staging-stable testing

🔗 step 8 stable testing

🔗 step 9 tag releases

🔗 step 10 update st2cd and st2ci packs

step 12 create the release

step 13 make sure new dev builds succeed

step 14 Check PackageCloud for new packages in unstable channel

remaining steps

blog article

Close v3.7.0

cognifloyd commented May 2, 2022 • edited Loading

bionic

focal

rocky 8

nzlosh commented May 2, 2022 • edited Loading

CentOS7

amanda11 commented May 3, 2022 • edited Loading

cognifloyd commented May 3, 2022 • edited Loading

cognifloyd commented May 6, 2022 • edited Loading

cognifloyd commented May 6, 2022 • edited Loading

arm4b commented May 6, 2022

amanda11 commented May 9, 2022

cognifloyd commented May 13, 2022

cognifloyd commented May 2, 2022 •

edited

Loading

cognifloyd commented May 2, 2022 •

edited

Loading

nzlosh commented May 2, 2022 •

edited

Loading

amanda11 commented May 3, 2022 •

edited

Loading

cognifloyd commented May 3, 2022 •

edited

Loading

cognifloyd commented May 6, 2022 •

edited

Loading

cognifloyd commented May 6, 2022 •

edited

Loading