Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add systemtest for livedata/postprocessing interaction #167

Merged
merged 6 commits into from
Jul 1, 2024

Conversation

rosswhitfield
Copy link
Member

@rosswhitfield rosswhitfield commented Jun 28, 2024

Ref: 5723: LiveDataServer: Add testing in WebMon test environment

This adds the livedata container to docker-compose, webmon assume it has ssl so a self-signed ssl cert is created and used for access to livedata. I set a static IP for the nginx container so that it can be accessed consistently from with the docker network and outside.

A systemtest was added that triggers the new data files request, this causes the autoreduction to run with was updated to publish a "plot" to livedata. The data is then verify that it can be accessed from webmon/outside.

You can manually test the interactions. First make clean and make all then start up docker-compose, docker-compose up --build. After which you can access webmon via http://localhost . You will need to visit livedata once by https://172.16.238.222/ and "accept" the self-signed cert (Proceed unsafe) for this to work.

After visit http://localhost/report/arcs/214583/ and you should see the "Example Plot Data" (which is just text).

2024-06-28-144445_1272x1060_scrot

You can then select "reduction" at the bottom of the page then wait, after a while the "plot data" will update and you should see a new timestamp. This verifies that autoreduction is correctly publishing to livedata.

2024-06-28-144519_1272x1060_scrot

This should work for any ARCS run

Short description of the changes:

Long description of the changes:

Check list for the pull request

  • I have read the [CONTRIBUTING]
  • I have read the [CODE_OF_CONDUCT]
  • I have added tests for my changes
  • I have updated the documentation accordingly

Check list for the reviewer

  • I have read the [CONTRIBUTING]
  • I have verified the proposed changes
  • best software practices
    • all internal functions have an underbar, as is python standard
    • clearly named variables (better to be verbose in variable names)
    • code comments explaining the intent of code blocks
  • All the tests are passing
  • The documentation is up to date
  • code comments added when explaining intent

Manual test for the reviewer

References

@rosswhitfield rosswhitfield changed the title Add systemtest for livedata/postprocessing interation Add systemtest for livedata/postprocessing interaction Jun 28, 2024
@rosswhitfield rosswhitfield marked this pull request as ready for review June 28, 2024 05:17
Copy link
Member

@jmborr jmborr left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Makefile Show resolved Hide resolved
@jmborr
Copy link
Member

jmborr commented Jun 28, 2024

@rosswhitfield Upon composing the images in the local environment, the build process complains of missing set variables.

❯ docker-compose up --build
WARNING: The DJANGO_SETTINGS_MODULE variable is not set. Defaulting to a blank string.
WARNING: The LDAP_SERVER_URI variable is not set. Defaulting to a blank string.
WARNING: The LDAP_DOMAIN_COMPONENT variable is not set. Defaulting to a blank string.
WARNING: The CATALOG_URL variable is not set. Defaulting to a blank string.
WARNING: The CATALOG_ID variable is not set. Defaulting to a blank string.
WARNING: The CATALOG_SECRET variable is not set. Defaulting to a blank string.

Perhap that's why the containers are unhealthy:

ERROR: for dasmon  Container "2b5bbfc386a4" is unhealthy.
ERROR: for nginx  Container "2b5bbfc386a4" is unhealthy.
ERROR: for amq_pv_gen  Container "2b5bbfc386a4" is unhealthy.
ERROR: for livedata  Container "2b5bbfc386a4" is unhealthy.
ERROR: for workflow  Container "2b5bbfc386a4" is unhealthy.

Are you setting these variables using the values from the secret CI variables stored in the webmon-deploy repo?
Also, which of the settings under reporting.reporting_app.settings did you pick for DJANGO_SETTINGS_MODULE

@rosswhitfield
Copy link
Member Author

@rosswhitfield Upon composing the images in the local environment, the build process complains of missing set variables.

❯ docker-compose up --build
WARNING: The DJANGO_SETTINGS_MODULE variable is not set. Defaulting to a blank string.
WARNING: The LDAP_SERVER_URI variable is not set. Defaulting to a blank string.
WARNING: The LDAP_DOMAIN_COMPONENT variable is not set. Defaulting to a blank string.
WARNING: The CATALOG_URL variable is not set. Defaulting to a blank string.
WARNING: The CATALOG_ID variable is not set. Defaulting to a blank string.
WARNING: The CATALOG_SECRET variable is not set. Defaulting to a blank string.

Perhap that's why the containers are unhealthy:

ERROR: for dasmon  Container "2b5bbfc386a4" is unhealthy.
ERROR: for nginx  Container "2b5bbfc386a4" is unhealthy.
ERROR: for amq_pv_gen  Container "2b5bbfc386a4" is unhealthy.
ERROR: for livedata  Container "2b5bbfc386a4" is unhealthy.
ERROR: for workflow  Container "2b5bbfc386a4" is unhealthy.

Are you setting these variables using the values from the secret CI variables stored in the webmon-deploy repo? Also, which of the settings under reporting.reporting_app.settings did you pick for DJANGO_SETTINGS_MODULE

Sorry, it looks like the dev docs need updating.

Do this first before running docker and the tests

export DJANGO_SETTINGS_MODULE=reporting.reporting_app.settings.envtest
export LDAP_SERVER_URI=.
export LDAP_DOMAIN_COMPONENT=.

You don't need to set the CATALOG ones.

@codecov-commenter
Copy link

codecov-commenter commented Jul 1, 2024

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 77.07%. Comparing base (b03c170) to head (df5069a).

Additional details and impacted files
@@           Coverage Diff           @@
##             next     #167   +/-   ##
=======================================
  Coverage   77.07%   77.07%           
=======================================
  Files          50       50           
  Lines        4616     4616           
=======================================
  Hits         3558     3558           
  Misses       1058     1058           

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

Copy link
Member

@jmborr jmborr left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Manual test works. Upon clicking in reduction, a "plot" is uploaded to livedata and the timestamp is updated
capture

@rosswhitfield rosswhitfield merged commit fc21b34 into next Jul 1, 2024
4 checks passed
@rosswhitfield rosswhitfield deleted the livedata_systemtest branch July 1, 2024 23:04
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants