Scorecard Integration #1294

404-geek · 2024-06-26T11:05:43Z

ScoreCode Integration

This pull request integrates the ScoreCode Repo into SCIO, enabling the fetching of the latest OSSF Scorecard Data for discovered packages using their vcs_url. The current implementation supports github.com and gitlab.com VCS URLs.

Key Features:

Integration with ScoreCode Repo
Fetching of OSSF Scorecard Data using vcs_url
Support for github.com and gitlab.com VCS URLs

Related Issues:

This feature enhances SCIO's functionality by ensuring that users can retrieve the most up-to-date security scores for packages discovered in their projects, improving overall security assessment and management.

developed functions to check for availability nexB#598 Signed-off-by: 404-geek <[email protected]>

Signed-off-by: 404-geek <[email protected]>

…aving logic nexB#1283 Signed-off-by: 404-geek <[email protected]>

Signed-off-by: 404-geek <[email protected]>

…exB#1283 Signed-off-by: 404-geek <[email protected]>

Signed-off-by: 404-geek <[email protected]>

… nexB#598 Signed-off-by: 404-geek <[email protected]>

…ecard_integration

…up.cfg nexB#1283 Signed-off-by: 404-geek <[email protected]>

Signed-off-by: 404-geek <[email protected]>

…gration # Conflicts: # scanpipe/models.py # scanpipe/tests/test_models.py

Signed-off-by: 404-geek <[email protected]>

tdruez · 2024-08-21T05:10:24Z

scanpipe/migrations/0068_merge_20240820_1656.py

+# Generated by Django 5.0.7 on 2024-08-20 16:56
+
+from django.db import migrations
+
+
+class Migration(migrations.Migration):
+
+    dependencies = [
+        ('scanpipe', '0067_discoveredpackage_notes'),
+        ('scanpipe', '0067_packagescore_scorecardcheck'),
+    ]
+
+    operations = [
+    ]


The migrations file need to be consolidated.

Migrate your local db to 0066: ./manage.py migrate scanpipe 0066

Delete those migrations files (0067 and 0068) and regenerate a single clean migration file.

tdruez · 2024-08-21T05:10:43Z

scanpipe/models.py

+    selected_steps = models.JSONField(
        null=True, blank=True, validators=[validate_none_or_list]
    )
-    selected_steps = models.JSONField(
+    selected_groups = models.JSONField(


Unwanted change.

tdruez · 2024-08-21T05:12:15Z

scanpipe/models.py

+    )
+
+    @classmethod
+    @transaction.atomic()


@404-geek Could you provide an example that shows why atomic() is useful here?

tdruez · 2024-08-21T05:13:03Z

scanpipe/models.py

+
+    @classmethod
+    @transaction.atomic()
+    def create_from_data(cls, DiscoveredPackage, scorecard_data, scoring_tool=None):


Why provide the DiscoveredPackage class object as an arg? Isn't it already present in the context/scope?

tdruez · 2024-08-21T05:14:07Z

scanpipe/models.py

+    @classmethod
+    @transaction.atomic()
+    def create_from_data(cls, DiscoveredPackage, scorecard_data, scoring_tool=None):
+        """Create ScoreCard Object from ScoreCard Object"""


"Object" why using upper case here?

tdruez · 2024-08-21T05:18:27Z

scanpipe/pipelines/get_scorecard_info_packages.py

+    Attributes
+    ----------
+        download_inputs (bool): Indicates whether inputs should be downloaded.
+        is_addon (bool): Indicates whether this pipeline is an add-on.
+
+    Methods
+    -------
+        steps(cls):
+            Defines the steps for the pipeline.
+
+        check_scorecode_service_availability(self):
+            Checks if the ScoreCode service is configured and available.
+
+        lookup_save_packages_scorecode_info(self):
+            Fetches ScoreCode information for each discovered package in the project
+            and saves the information to the respective package instances.
+


This is not consistent with existing pipelines.
Also, the Pipeline docstring is used to render documentation in the UI.
The Attributes and Methods are not relevant in that context.

tdruez · 2024-08-21T05:20:43Z

scanpipe/pipelines/get_scorecard_info_packages.py

+        if not scorecard.is_available():
+            raise Exception("scorecode service is not available.")
+
+    def lookup_save_packages_scorecode_info(self):


fetch_packages_scorecode_info would be better.

tdruez · 2024-08-21T05:21:23Z

scanpipe/pipelines/get_scorecard_info_packages.py

+            )
+
+        else:
+            raise Exception("No Data Found for the packages")


Why mxing upper and lower case in a sentence?

tdruez · 2024-08-21T05:23:52Z

setup.cfg

@@ -98,6 +98,8 @@ install_requires =
    fontawesomefree==6.6.0
    # MatchCode-toolkit
    matchcode-toolkit==5.1.0
+    # ScoreCode
+    ScoreCode[full] @ git+https://github.com/nexB/ScoreCode.git@0ab078f18d83684c3a920095bcec8664d44cf028


What's the plan to issue a proper release of ScoreCode?

tdruez · 2024-08-21T05:25:05Z

setup.cfg

@@ -136,6 +139,7 @@ scancodeio_pipelines =
    collect_symbols_tree_sitter = scanpipe.pipelines.collect_symbols_tree_sitter:CollectSymbolsTreeSitter
    enrich_with_purldb = scanpipe.pipelines.enrich_with_purldb:EnrichWithPurlDB
    find_vulnerabilities = scanpipe.pipelines.find_vulnerabilities:FindVulnerabilities
+    get_scorecard_info_packages = scanpipe.pipelines.get_scorecard_info_packages:FetchScoreCodeInfo


fetch_scorecode_info would be much better and consistent with the pipeline name.

…#1283 Signed-off-by: 404-geek <[email protected]>

Signed-off-by: 404-geek <[email protected]>

pombredanne · 2024-09-29T22:21:01Z

@404-geek can you check the failing tests?

tdruez · 2024-10-29T06:30:51Z

Hey @404-geek , what's your latest status on this PR? Any chances we can complete and merge it before it diverges too much from the main branch?

404-geek · 2024-10-29T17:36:52Z

Hey @404-geek , what's your latest status on this PR? Any chances we can complete and merge it before it diverges too much from the main branch?

Hi @tdruez,

I’m planning to have this PR ready by next week. Most of the requested changes have been addressed; I just need to add a few test cases before pushing the final version.

Signed-off-by: 404-geek <[email protected]>

…ecard_integration

Signed-off-by: 404-geek <[email protected]>

404-geek added 30 commits June 26, 2024 13:20

Add ScoreCard config into settings.py

173db43

developed functions to check for availability nexB#598 Signed-off-by: 404-geek <[email protected]>

code style fix nexB#598

272b99c

Signed-off-by: 404-geek <[email protected]>

Merge branch 'refs/heads/ossf_api_template' into scorecard_integration

944cee2

settings.py code style fix nexB#598

bc445c1

Signed-off-by: 404-geek <[email protected]>

mixin import and models declaration nexB#1283

f241b3b

Signed-off-by: 404-geek <[email protected]>

transforming scorecard data into object for saving nexB#1283

605a5cf

Signed-off-by: 404-geek <[email protected]>

added test cases for saving scorecard data into models and modified s…

3833dca

…aving logic nexB#1283 Signed-off-by: 404-geek <[email protected]>

Merge branch 'main' into feat-models_integration

37380c8

added score checks mixin to models nexB#1283

5502b5f

Signed-off-by: 404-geek <[email protected]>

Merge branch 'nexB:main' into feat-models_integration

2aeb2a4

empty details in score response handled nexB#1283

103fca0

Signed-off-by: 404-geek <[email protected]>

changed class names to camel case for models and modified the tests n…

952e6a6

…exB#1283 Signed-off-by: 404-geek <[email protected]>

code formatted nexB#1283

3ba3db5

Signed-off-by: 404-geek <[email protected]>

code formatted nexB#1283

2f2b846

Signed-off-by: 404-geek <[email protected]>

code formatted nexB#1283

39a056d

Signed-off-by: 404-geek <[email protected]>

docstrings formatted nexB#1283

e5f3e7a

Signed-off-by: 404-geek <[email protected]>

created basic fetch and availability functions for scorecode pipeline…

c2f5c4d

… nexB#598 Signed-off-by: 404-geek <[email protected]>

Merge branch 'nexB:main' into scorecard_integration

6ee5b07

Merge remote-tracking branch 'origin/scorecard_integration' into scor…

760afc2

…ecard_integration

modified doc strings and models and imported ScoreCode package in set…

0dbc92f

…up.cfg nexB#1283 Signed-off-by: 404-geek <[email protected]>

setup.cfg nexB#1283

d652f42

Signed-off-by: 404-geek <[email protected]>

Merge branch 'nexB:main' into scorecard_integration

aa154c0

Merge branch 'refs/heads/feat-models_integration' into scorecard_inte…

37ad73a

…gration # Conflicts: # scanpipe/models.py # scanpipe/tests/test_models.py

reinstated deleted code during rebase nexB#1283

563991b

Signed-off-by: 404-geek <[email protected]>

code formatting nexB#1283

4632dfc

Signed-off-by: 404-geek <[email protected]>

database migrations for scorecard nexB#1283

923c834

Signed-off-by: 404-geek <[email protected]>

updated the scanpipe only fields nexB#1283

94bfcd0

Signed-off-by: 404-geek <[email protected]>

changed scorecode commit hash for latest pull nexB#1283

d129f73

Signed-off-by: 404-geek <[email protected]>

Merge branch 'nexB:main' into scorecard_integration

24c7be0

update pipeline code and changed scorecode hash commit nexB#1283

259f004

Signed-off-by: 404-geek <[email protected]>

modified lookup and save logic nexB#1283

29da290

Signed-off-by: 404-geek <[email protected]>

404-geek changed the title ~~Scorecard integration~~ Scorecard Integration Jul 26, 2024

404-geek mentioned this pull request Jul 26, 2024

Models Integration in SCIO #1304

Closed

404-geek added 8 commits July 27, 2024 15:46

merged migrations due to conflicts nexB#1283

9d72734

Signed-off-by: 404-geek <[email protected]>

updated migrations nexB#1283

301122e

Signed-off-by: 404-geek <[email protected]>

updated doc string for get_scorecard_info_packages.py nexB#1283

5812d97

Signed-off-by: 404-geek <[email protected]>

Added scorecard pipeline to SCIO with intergration test nexB#1283

df50416

Signed-off-by: 404-geek <[email protected]>

Merge branch 'main' into scorecard_integration

88a43dc

moved the data to be regenerated if reqiured nexB#1283

fc4945e

Signed-off-by: 404-geek <[email protected]>

updated urls for testing nexB#1283

c00b3b3

Signed-off-by: 404-geek <[email protected]>

added merged migration file nexB#1283

b97ff7a

Signed-off-by: 404-geek <[email protected]>

tdruez requested changes Aug 21, 2024

View reviewed changes

404-geek added 5 commits August 23, 2024 01:02

Changed docstring and renamed functions according to suggestions nexB…

496945b

…#1283 Signed-off-by: 404-geek <[email protected]>

class name changes in steps of pipeline nexB#1283

f86d5bb

Signed-off-by: 404-geek <[email protected]>

pipeline name updated nexB#1283

36e955a

Signed-off-by: 404-geek <[email protected]>

Merge branch 'aboutcode-org:main' into scorecard_integration

90c113a

update pipeline code and steps nexB#1283

43886a3

Signed-off-by: 404-geek <[email protected]>

404-geek added 10 commits October 29, 2024 21:20

Merge branch 'main' into scorecard_integration

7c134c7

update pipeline steps to work with scorecode 0.0.2 release nexB#1283

3d4d6ea

Signed-off-by: 404-geek <[email protected]>

Merge remote-tracking branch 'origin/scorecard_integration' into scor…

f4ed4b5

…ecard_integration

update migration nexB#1283

b9229a5

Signed-off-by: 404-geek <[email protected]>

Merge branch 'aboutcode-org:main' into scorecard_integration

f80f111

update migration nexB#1283

ee2ea14

Signed-off-by: 404-geek <[email protected]>

rename pipeline name with data parsing function nexB#1283

bd5b9b3

Signed-off-by: 404-geek <[email protected]>

code valid nexB#1283

2b4629e

Signed-off-by: 404-geek <[email protected]>

update setup.cfg nexB#1283

99ee48d

Signed-off-by: 404-geek <[email protected]>

optimize code while saving score checks nexB#1283

80153f8

Signed-off-by: 404-geek <[email protected]>

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Scorecard Integration #1294

Scorecard Integration #1294

404-geek commented Jun 26, 2024 •

edited

Loading

tdruez Aug 21, 2024

tdruez Aug 21, 2024

tdruez Aug 21, 2024

tdruez Aug 21, 2024

tdruez Aug 21, 2024

tdruez Aug 21, 2024

tdruez Aug 21, 2024

tdruez Aug 21, 2024

tdruez Aug 21, 2024

tdruez Aug 21, 2024

pombredanne commented Sep 29, 2024

tdruez commented Oct 29, 2024

404-geek commented Oct 29, 2024

Scorecard Integration #1294

Are you sure you want to change the base?

Scorecard Integration #1294

Conversation

404-geek commented Jun 26, 2024 • edited Loading

ScoreCode Integration

Key Features:

Related Issues:

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

pombredanne commented Sep 29, 2024

tdruez commented Oct 29, 2024

404-geek commented Oct 29, 2024

404-geek commented Jun 26, 2024 •

edited

Loading