Bulk clearing of detected licenses

[YanickNoblanc]

Hi,

Is there a way to do bulk clearing of licenses detected on packages ? On all files detected in a single package, or even better across all files of all analyzed packages ? I have a fossology server runing, analysing packages included in an ubuntu distribution (more than 2300 packages for each distrib) I cannot reasonably clear all detected licenses, package by package, and file by file inside each package. I have been clearing several detected licenses, trying to systematicaly use functionalities :

• Go through all files with licenses and no clearing result
• Apply decision to all future occurrences of this file
• Bulk recognition
• Ignore conflicts
• Scan files with findings
• Schedule Bulk Scan

But whatever I do, licenses cleared on a package are never concluded on any other scanned package. They are not even cleared 100% of the time, on all files where they were detected inside the same package.

Is there a way to automaticaly conclude ALL detected licenses on ALL packages using Web UI or REST API ?

[Akashsah2003]

Hey @YanickNoblanc,

When you upload and schedule scans with agents you can use Automatic Concluded License Decider options in the UI to automatically clear the files. For instance, if you are using Ojo License Analysis for detecting licenses you can use Scanners matches if Ojo or REUSE.Software findings are no contradiction with other findings option to automatically clear the files after scan.

[YanickNoblanc]

Thanks a lot, once again, what would be the corresponding parameters for a scanning using REST API ?

[Akashsah2003]

For scanning using REST API for the above example you can use something like this

In the scanOptions in analysis you can mention which agent you want to use for the upload and in decider you can mention how you want to clear the files in the upload. For more details, you can check out V1 API documentation in openapi.yaml.

[YanickNoblanc]

Thanks a lot for the info. Unfortunatly I am scanning via URL using following REST API command, for which adding this option as is does not work :

      UPLOADID=$(curl --connect-timeout "${CURL_TIMEOUT}" -k -s -S -L -X POST "${FOSSURL}/repo/api/v1/uploads" \
       -H "folderId: ${FOLDERID}" \
       -H 'uploadDescription: created by REST' \
       -H 'public: public' \
       -H 'Content-Type: application/json' \
       -H 'uploadType: url' \
       -H 'applyGlobal: true' \
       -H "Authorization: Bearer ${FOSSTOKEN}" \
       --data-raw "{
        \"location\": { \"url\": \"${PKGURL}\" },
        \"name\": \"${PKGFILENAME}\"
        }" \
        | jq -r '(.message)' || true)

[YanickNoblanc]

Any way to get this working when using REST API and scanning via url ?

[Akashsah2003]

curl -k -s -S -L -X POST "http://localhost/repo/api/v1/uploads" \
     -H "folderId: 1" \
     -H 'uploadDescription: created by REST' \
     -H 'public: public' \
     -H 'Content-Type: application/json' \
     -H 'uploadType: url' \
     -H 'applyGlobal: true' \
     -H "Authorization: Bearer eyJ0eXAiOiJKV1QiLCJhbGciOiJIUzI1NiJ9.eyJleHAiOjE3MTg5MDgxOTksIm5iZiI6MTcxNjQwMjYwMCwianRpIjoiTWk0eiIsInNjb3BlIjoid3JpdGUifQ.oycUbOs60YA9G3JRRsAmPPFrcgjVtgJkAha1HeUUnpY" \
     --data-raw '{
        "location": { "url": "https://cdn.kernel.org/pub/linux/kernel/v4.x/linux-4.19.314.tar.xz" },
        "scanOptions": {
        	"analysis": {
        		"ojo": true
        	},
        	"decider": {
        		"ojo_decider": true
        	}
        }
     }'

The above curl command works for me. It scans the package with ojo and uses ojo_decider to clear it.

Please check if its working for you.

[YanickNoblanc]

Thanks a lot, I probably had a syntax mistake. I successfully relaunched with same options. But licenses are still not automaticaly concluded.

[YanickNoblanc]

Is it normal that Job history does not display any line with 'ojo' ?

[Akashsah2003]

It seems like you maybe using another curl command like the one given below to schedule jobs after uploading. The curl command I mentioned above is an alternative to this. If you are using something like this you can modify -d option to schedule analysis. nomos, monk and ojo are agents for license analysis. nomos uses some part of license text or license name to search for licenses in the text, monk tries to match full license texts when scanning and the nomos_monk decider clears files if the nomos findings are within findings of monk. So this combination may not clear all the files. ojo searches for license declaration in the text like SPDX-License-Identifier: MIT to search for licenses and ojo_decider clears files if findings by other scanners are not in contradiction with ojo. So this combination may clear all the files but it cannot detect licenses if it is not declared by the author. So it depends on your use case which agent and decider you want to use. For more information on agents you can check out wiki.

curl -X 'POST' \
  'http://localhost/repo/api/v1/jobs' \
  -H 'accept: application/json' \
  -H 'folderId: 1' \
  -H 'uploadId: 2' \
  -H 'Content-Type: application/json' \
  -d '{
  "analysis": {
    "bucket": true,
    "copyright_email_author": true,
    "ecc": true,
    "keyword": true,
    "mime": true,
    "monk": true,
    "nomos": true,
    "ojo": false,
    "package": true,
  },
  "decider": {
    "nomos_monk": true,
  },
  "reuse": {
    "reuse_upload": 0,
    "reuse_group": "string",
    "reuse_main": true,
    "reuse_enhanced": true,
    "reuse_report": true,
    "reuse_copyright": true
  }
}'

[YanickNoblanc]

Not sure of what I am doing wrong, whatever I try, license are not automaticaly cleared until I manualy clear them.
Attaching my script. If you have a little time to check what is done wrong in ther, it would be fantastic. Thank you so much for all the help tou already provided.
foss_analyse_package_sh.txt

[Akashsah2003]

Hey @YanickNoblanc only using ojo agent will not give all the license results for your usecase. so I have written a script that clears all the files after scanning the upload using REST API. Attaching the script integrated to the script that you provided.

trace "  Clearing"
# Get the top item ID
response=$(curl -s -X 'GET' "http://localhost/repo/api/v1/uploads/${UPLOADID}/topitem" -H 'accept: application/json' -H "Authorization: Bearer ${FOSSTOKEN}")
itemid=$(echo $response | jq -r '.message')

# Initialize pageno
pageno=1

while true; do
    # Get the items in the tree view
    response=$(curl -s -X 'GET' "http://localhost/repo/api/v1/uploads/${UPLOADID}/item/${itemid}/tree/view?flatten=true" -H 'accept: application/json' -H "page: ${pageno}" -H 'limit: 50' -H "Authorization: Bearer ${FOSSTOKEN}")

    # Check if the response is an empty array
    if [ "$(echo $response | jq length)" -eq 0 ]; then
        break
    fi

    # Iterate through each item in the response
    echo $response | jq -c '.[]' | while read -r item; do
        itemtobeclearedid=$(echo $item | jq -r '.fileDetails.id')

        # Execute the clearing decision
        curl -s -X 'PUT' "http://localhost/repo/api/v1/uploads/${UPLOADID}/item/${itemtobeclearedid}/clearing-decision" \
            -H 'accept: application/json' \
            -H 'Content-Type: application/json' \
            -H "Authorization: Bearer ${FOSSTOKEN}" \
            -d '{
            "decisionType": 5,
            "globalDecision": true
        }'
    done

    # Increment the page number
    pageno=$((pageno + 1))
done

foss_analyse_package_modified_sh.txt

[YanickNoblanc]

After a few modifications, script works perfectly as expected.

Reporting a minor interface bug : when displaying a package's files, when displayed files are on several pages, if selecting a page>1, then page stays stuck on selected value and no more files are displayed when going to another place in same package or on different package with only one page to be showed.

Thank you so much for your help and time :-)