Skip to content

Commit

Permalink
feat: Add scanning of local files and folders (single scan) (#558)
Browse files Browse the repository at this point in the history 
* feat: Add scanning of local files and folders (single scan)

Signed-off-by: Oleg Kopysov <[email protected]>

* fix: Improve code coverage

Signed-off-by: Oleg Kopysov <[email protected]>

* fix: Apply fixes after code review

Signed-off-by: Oleg Kopysov <[email protected]>

* feat: Add scanning of local files and folders (single scan)

Signed-off-by: Oleg Kopysov <[email protected]>

* fix: Improve code coverage

Signed-off-by: Oleg Kopysov <[email protected]>

* fix: Apply fixes after code review

Signed-off-by: Oleg Kopysov <[email protected]>

* fix: Apply fixes after code review

Signed-off-by: Oleg Kopysov <[email protected]>

* fix: Update unit tests and fix the build process

Signed-off-by: Oleg Kopysov <[email protected]>

---------

Signed-off-by: Oleg Kopysov <[email protected]>
  • Loading branch information
@o-kopysov
o-kopysov authored Aug 7, 2024
1 parent 420cc39 commit 57c5e2b
Show file tree
Hide file tree
Showing 11 changed files with 685 additions and 118 deletions.
35 changes: 27 additions & 8 deletions doc/quick-start-guide-and-build.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -245,34 +245,53 @@ _LPVS_ is now built and running. You can create a new pull request or update an

#### 4.2 Single scan mode

Alternatively, you can perform a one-time scan on a specific pull request using the single scan mode. Follow these steps:
Alternatively, you can perform a one-time scan on a specific pull request or local files using the single scan mode. Follow these steps:

4.2.1 Begin by running the installation and navigating to the target directory, similar to the process in service mode (refer to steps 4.1.1 and 4.1.2):
4.2.1 Install and navigate to the target directory as described in step 4.1.1 and 4.1.2:

```bash
mvn clean install
cd target/
```

4.2.2 Execute the single scan with the following command:
4.2.2 Choose either to scan a specific pull request from GitHub or local files.

4.2.2.1 To scan a specific pull request from GitHub, execute the following command:

```bash
java -jar -Dgithub.token=<my-token> lpvs-*.jar --github.pull.request=<PR URL>
```

4.2.3 By default, the above command requires a pre-configured MySQL database. To avoid setting up the database, use the "singlescan" profile:
4.2.2.2 To scan local files or directories, execute the following command:

```bash
java -jar lpvs-*.jar --local.path=</path/to/file/or/folder>
```

4.2.3 By default, the above commands require a pre-configured MySQL database. Use the "singlescan" profile to skip setting up a pre-configured MySQL database:

```bash
java -jar -Dspring.profiles.active=singlescan -Dgithub.token=<my-token> lpvs-*.jar --github.pull.request=<PR URL>
java -jar -Dspring.profiles.active=singlescan lpvs-*.jar --local.path=</path/to/file/or/folder>
```

These steps streamline the process, allowing you to run a scan on a single pull request without the need for a preconfigured database.
4.2.4 Optionally, generate an HTML report and save it in a specified folder. Replace `path/to/your/folder` with the full path to the folder where you want to save the HTML report, and `your_report_filename.html` with the desired filename for the report.

4.2.4 Available option to generate an HTML report and save it in a specified folder. Replace `/path/to/your/folder` with the full path to the folder where you want to save the HTML report, and `your_report_filename.html` with the desired filename for the report.
```bash
java -jar -Dgithub.token=<my-token> lpvs-*.jar --github.pull.request=<PR URL> --build.html.report=</path/to/your/folder/your_report_filename.html>
java -jar -Dspring.profiles.active=singlescan -Dgithub.token=<my-token> lpvs-*.jar --github.pull.request=<PR URL> --build.html.report=</path/to/your/folder/your_report_filename.html>
java -jar -Dspring.profiles.active=singlescan lpvs-*.jar --local.path=</path/to/file/or/folder> --build.html.report=<your_report_filename.html>
```

These steps streamline the process, allowing you to run a scan on a single pull request without the need for a preconfigured database.
Note: Ensure that the specified folder exists before generating the HTML report.

4.2.5 Examples of commands:

```bash
java -jar -Dspring.profiles.active=singlescan lpvs-*.jar --github.pull.request=https://github.com/Samsung/LPVS/pull/2
java -jar -Dspring.profiles.active=singlescan lpvs-*.jar --github.pull.request=https://github.com/Samsung/LPVS/pull/2 --build.html.report=report.html
java -jar -Dspring.profiles.active=singlescan lpvs-*.jar --local.path=test.c
java -jar -Dspring.profiles.active=singlescan lpvs-*.jar --local.path=test --build.html.report=test/report.html
```

#### 4.3 Use of _LPVS_ JAR `lpvs-x.y.z.jar` in your project

Expand Down
16 changes: 6 additions & 10 deletions src/main/java/com/lpvs/service/LPVSLicenseService.java
View file
Original file line number Diff line number Diff line change
Expand Up @@ -15,6 +15,7 @@
import com.lpvs.util.LPVSExitHandler;

import com.lpvs.util.LPVSPayloadUtil;
import io.micrometer.common.util.StringUtils;
import lombok.NoArgsConstructor;
import lombok.extern.slf4j.Slf4j;
import org.springframework.beans.factory.annotation.Autowired;
Expand Down Expand Up @@ -230,7 +231,7 @@ protected LPVSLicense findLicenseByName(String name) {
if (license.getLicenseName().equalsIgnoreCase(name)) {
return license;
}
if (license.getAlternativeNames() != null && !license.getAlternativeNames().isBlank()) {
if (!StringUtils.isBlank(license.getAlternativeNames())) {
String[] names = license.getAlternativeNames().split(",");
for (String n : names) {
if (n.trim().equalsIgnoreCase(name)) {
Expand Down Expand Up @@ -258,15 +259,10 @@ public LPVSLicense getLicenseBySpdxIdAndName(
lic = findLicenseInOsoriDB(licenseSpdxId);
// If not found, create new license with default field values
if (lic == null) {
lic =
new LPVSLicense() {
{
setSpdxId(licenseSpdxId);
setLicenseName(licName);
setAlternativeNames(null);
setAccess("UNREVIEWED");
}
};
lic = new LPVSLicense();
lic.setSpdxId(licenseSpdxId);
lic.setLicenseName(licName);
lic.setAccess("UNREVIEWED");
}
// Save new license
lic = lpvsLicenseRepository.saveAndFlush(lic);
Expand Down
13 changes: 10 additions & 3 deletions src/main/java/com/lpvs/service/LPVSQueueProcessorService.java
View file
Original file line number Diff line number Diff line change
Expand Up @@ -7,6 +7,7 @@
package com.lpvs.service;

import com.lpvs.entity.LPVSQueue;
import io.micrometer.common.util.StringUtils;
import lombok.extern.slf4j.Slf4j;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.beans.factory.annotation.Value;
Expand All @@ -29,11 +30,17 @@ public class LPVSQueueProcessorService {
private LPVSQueueService queueService;

/**
* Trigger value obtained from application properties.
* Trigger value to start a single scan of a pull request (optional).
*/
@Value("${github.pull.request:}")
private String trigger;

/**
* Trigger value to start a single scan of local files or folder (optional).
*/
@Value("${local.path:}")
private String localPath;

/**
* Constructor for LPVSQueueProcessorService.
*
Expand All @@ -51,12 +58,12 @@ public class LPVSQueueProcessorService {
* @throws Exception If an exception occurs during queue processing.
*/
@EventListener(ApplicationReadyEvent.class)
private void queueProcessor() throws Exception {
protected void queueProcessor() throws Exception {
// Check for any pending elements in the LPVSQueue.
queueService.checkForQueue();

// Process LPVSQueue elements until the trigger is set.
while (trigger == null || trigger.isEmpty()) {
while (StringUtils.isBlank(trigger) && StringUtils.isBlank(localPath)) {
// Get the first element from the LPVSQueue.
LPVSQueue webhookConfig = queueService.getQueueFirstElement();
log.info("PROCESS Webhook id = " + webhookConfig.getId());
Expand Down
142 changes: 107 additions & 35 deletions src/main/java/com/lpvs/service/scan/LPVSDetectService.java
View file
Original file line number Diff line number Diff line change
Expand Up @@ -7,15 +7,16 @@
package com.lpvs.service.scan;

import java.io.File;
import java.nio.file.Files;
import java.nio.file.Path;
import java.nio.file.Paths;
import java.text.SimpleDateFormat;
import java.util.ArrayList;
import java.util.Date;
import java.util.List;

import com.lpvs.service.LPVSGitHubConnectionService;
import com.lpvs.service.LPVSGitHubService;
import com.lpvs.service.LPVSLicenseService;
import com.lpvs.util.LPVSFileUtil;
import io.micrometer.common.util.StringUtils;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.beans.factory.annotation.Value;
import org.springframework.boot.SpringApplication;
Expand Down Expand Up @@ -59,11 +60,17 @@ public class LPVSDetectService {
private LPVSScanService scanService;

/**
* GitHub pull request used to trigger a single license scan (optional).
* Trigger value to start a single scan of a pull request (optional).
*/
@Value("${github.pull.request:}")
private String trigger;

/**
* Trigger value to start a single scan of local files or folder (optional).
*/
@Value("${local.path:}")
private String localPath;

/**
* Optional parameter to save html report to specified location.
*/
Expand Down Expand Up @@ -105,52 +112,117 @@ public LPVSDetectService(
*/
@EventListener(ApplicationReadyEvent.class)
public void runSingleScan() {
if (trigger != null && !HtmlUtils.htmlEscape(trigger).isEmpty()) {
log.info("Triggered single scan operation");
// generateReport indicates that a report should be generated (HTML or command line output)
boolean generateReport = false;
LPVSQueue webhookConfig = null;
List<LPVSFile> scanResult = null;
List<LPVSLicenseService.Conflict<String, String>> detectedConflicts = null;

// Error case when both pull request scan and local files scan are set to true
if (!StringUtils.isBlank(trigger) && !StringUtils.isBlank(localPath)) {
log.error(
"Incorrect settings: both pull request scan and local files scan are set to true.");
SpringApplication.exit(ctx, () -> 0);

// Scan option - single pull request scan
} else if (!StringUtils.isBlank(trigger)) {
log.info("Triggered single scan of pull request.");
try {
licenseService.reloadFromTables();
LPVSQueue webhookConfig =
webhookConfig =
gitHubService.getInternalQueueByPullRequest(HtmlUtils.htmlEscape(trigger));

List<LPVSFile> scanResult =
scanResult =
this.runScan(
webhookConfig, gitHubService.getPullRequestFiles(webhookConfig));

List<LPVSLicenseService.Conflict<String, String>> detectedConflicts =
licenseService.findConflicts(webhookConfig, scanResult);

if (htmlReport != null && !HtmlUtils.htmlEscape(htmlReport).isEmpty()) {
Path buildReportPath = Paths.get(htmlReport);
Path parentDirectory = buildReportPath.getParent();

if (parentDirectory != null && Files.isDirectory(parentDirectory)) {
String report =
LPVSCommentUtil.buildHTMLComment(
webhookConfig, scanResult, detectedConflicts);
LPVSCommentUtil.saveHTMLToFile(report, buildReportPath.toString());
} else {
log.error(
"Error: The parent directory '"
+ parentDirectory
+ "' does not exist.");
}
detectedConflicts = licenseService.findConflicts(webhookConfig, scanResult);
generateReport = true;
log.info("Single scan of pull request completed.");
} catch (Exception ex) {
log.error("Single scan of pull request failed with error: " + ex.getMessage());
SpringApplication.exit(ctx, () -> 0);
}

// Scan option - single scan of local file or folder
} else if (!StringUtils.isBlank(localPath)) {
log.info("Triggered single scan of local file(s).");
try {
licenseService.reloadFromTables();
localPath = HtmlUtils.htmlEscape(localPath);
File localFile = new File(localPath);
if (localFile.exists()) {
// 1. Generate webhook config
webhookConfig = getInternalQueueByLocalPath();
// 2. Copy files
LPVSFileUtil.copyFiles(
localPath, LPVSFileUtil.getLocalDirectoryPath(webhookConfig));
// 3. Trigger scan
scanResult =
this.runScan(
webhookConfig,
LPVSFileUtil.getLocalDirectoryPath(webhookConfig));

detectedConflicts = licenseService.findConflicts(webhookConfig, scanResult);
generateReport = true;
log.info("Single scan of local file(s) completed.");
} else {
String report =
LPVSCommentUtil.reportCommentBuilder(
webhookConfig, scanResult, detectedConflicts);
if (report != null && !report.isEmpty()) {
log.info(report);
}
throw new Exception("File path does not exist: " + localPath);
}
log.info("Single scan completed.");

} catch (Exception ex) {
log.error("Single scan finished with errors.");
log.error("Can't trigger single scan: " + ex.getMessage());
log.error("Single scan of local file(s) failed with error: " + ex.getMessage());
SpringApplication.exit(ctx, () -> 0);
}
}

// Report generation
// 1. HTML format
if (generateReport && !StringUtils.isBlank(htmlReport)) {
File report = new File(HtmlUtils.htmlEscape(htmlReport));
String folderPath = report.getParent();
if (folderPath == null) {
folderPath = ".";
}
File folder = new File(folderPath);
if (folder.exists() && folder.isDirectory()) {
String reportFile =
LPVSCommentUtil.buildHTMLComment(
webhookConfig, scanResult, detectedConflicts);
LPVSCommentUtil.saveHTMLToFile(reportFile, report.getAbsolutePath());
} else {
log.error("Error: The parent directory '" + folder.getPath() + "' does not exist.");
}
SpringApplication.exit(ctx, () -> 0);
} else if (generateReport) {
// 2. Command line output
String report =
LPVSCommentUtil.reportCommentBuilder(
webhookConfig, scanResult, detectedConflicts);
if (!report.isEmpty()) {
log.info(report);
}
SpringApplication.exit(ctx, () -> 0);
}
}

/**
* Creates a new LPVSQueue object with default values for a local scan.
*
* @return the new LPVSQueue object
*/
private LPVSQueue getInternalQueueByLocalPath() {
LPVSQueue queue = new LPVSQueue();
queue.setDate(new Date());
queue.setUserId("Single scan of local files run");
queue.setReviewSystemType("local_scan");
SimpleDateFormat sdf = new SimpleDateFormat("yyyyMMdd-HHmmss");
String repoUrl = "local_scan_" + sdf.format(queue.getDate());
queue.setRepositoryUrl(repoUrl);
queue.setPullRequestUrl(repoUrl);
return queue;
}

/**
* Runs a license scan based on the selected scanner type.
*
Expand Down
Loading

0 comments on commit 57c5e2b

Please sign in to comment.