Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Scoring: Use N/A for recommended/optional, 0 is only a failure for Mandatory #292

Merged
merged 1 commit into from
Oct 11, 2023

Conversation

manics
Copy link
Member

@manics manics commented Oct 10, 2023

✅ Checklist

  • This pull request has a meaningful title.
  • If your changes are not yet ready to merge, you have marked this pull request as a draft pull request.

☑️ Maintainers' checklist

  • This pull request has had the appropriate labels assigned
  • This pull request has been added to the SATRE backlog project board
  • This pull request has been assigned to one or more maintainers

⤴️ Summary

Clarifies the scoring system:

  • 0 is only a SATRE failure for Mandatory
  • N/A should be used if a statement is not relevant

🌂 Related issues

Closes #290

🙋 Acknowledging contributors

@manics manics added the proposed change A proposed change to the specification label Oct 10, 2023
:1 Sufficient: The TRE meets this requirement met but there is substantial scope for improvement
:2 Satisfied: The TRE meets this requirement met but there may still be scope for improvement
:N/A: Not applicable: The statement is not relevant to a TRE, may apply to **Recommended** or **Optional** statements
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Note if we return to #272 and use conditional Mandatory statements this will need to be updated, AFAIK currently all Mandatory statements are mandatory for all.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think this is right, there's probably statements in out eval where we gave ourselves a 2 and just said "we pass because it's not applicable" - I don't think this is a huge issue though

@manics manics requested a review from a team October 10, 2023 14:34
@manics manics merged commit 57e6f22 into sa-tre:main Oct 11, 2023
@manics manics deleted the eval-scoring-clarify branch October 11, 2023 15:05
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
proposed change A proposed change to the specification
Projects
Status: Done
Development

Successfully merging this pull request may close these issues.

[Discussion]: Evaluation scoring: unclear how to score non-mandatory items
2 participants