Update OCLint setup to use native tools instead of xcpretty (#40)

* Add sonar.apple.jsonCompilationDatabasePath parameter

* Fix AppleTestsSensorTest

* Add JSON Compilation Database merge logic

* Upgrade JProc version to 2.8.0

* Export some logic to a separate object OCLintJSONDatabaseBuilder

* Adding OCLintJSONDatabaseBuilderTest

* Create OCLintExtractor to run  oclint-json-compilation-database

* Adding parsing step to OCLintExtractor

* Update OCLintReportParser

* Merging all the piece into OCLintSensor

* Some fixes

* Update documentation

* Updating documentation

* Update OCLintReportParserTest

* Cleaning

* Add headers

* Improve OCLint regex for the include parameter

* Update README

* Fix error in unit tests

* Fix bug (from Sonar Cloud)

* Fix issues (from Sonar Cloud)

* Fix runtime failure, the file wasn't properly closed before it was read by another ressource

Author: gaelfoppolo

.gitignore

@@ -33,3 +33,6 @@ build/
 
 ### Mac OS ###
 .DS_Store
+
+### OCLint Tests ###
+objc-lang/src/test/resources/oclint/build

README.md

@@ -33,52 +33,60 @@ The plugin is designed to support Swift 5 syntax.
 
 ## Requirements
 
-### Xcode
+### Mandatory
 
-Download Xcode from [Apple Developer](https://developer.apple.com/download/).
+#### Xcode
 
-The plugin was tested with Xcode 13+, but should work with older versions.
+Xcode is required in order to build the project and run tests.
+You can download it from [Apple Developer](https://developer.apple.com/download/), but we strongly recommend to use a version manager such as [xcinfo](https://github.com/xcodereleases/xcinfo).
 
-### sonar-scanner (requires Java)
+> **Note**
+> The plugin was tested with Xcode 13+, but should work with older versions (down to Xcode 11).
 
-Install sonar-scanner as explained in the [official documentation]((https://docs.sonarqube.org/latest/analysis/scan/sonarscanner/)).
+#### sonar-scanner
 
-### xcpretty
+sonar-scanner is required to run this plugin, build and send the metrics to Sonar.  
+You can install it as explained in the [official documentation]((https://docs.sonarqube.org/latest/analysis/scan/sonarscanner/)), but we strongly recommend to install it with [Homebrew](https://github.com/Homebrew/brew).
 
-xcpretty is used to generate a JSON Compilation Database from ``xcodebuild`` log.
+> **Note**
+> sonar-scanner requires Java.
+> We recommend to use a Java environment manager such as `jenv` to install OCLint to control the version.
 
-See install instructions [here](https://github.com/xcpretty/xcpretty).
+### Optional
 
-### SwiftLint
+#### SwiftLint
 
 SwiftLint is used to analyse Swift source files.
+You can install it as explained in [here](https://github.com/realm/SwiftLint).
 
-See install instructions [here](https://github.com/realm/SwiftLint).
+> **Warning**
+> Without SwiftLint many issues will not be detected. This may decrease the quality of the analysis.  
 
-### OCLint
+#### OCLint
 
 OCLint is used to analyse Objective-C source files.
+You can install it as explained in [here](https://docs.oclint.org/en/stable/intro/homebrew.html).
 
-See install instructions [here](https://docs.oclint.org/en/stable/intro/homebrew.html).
+> **Note**
+> We recommend to use a `Gemfile` to install OCLint to control the version.
 
-Important: after initial installation, macOS will block usage of OCLint libraries. In order to get rid of the manual verification of each of them, use the following commands:
+> **Warning**
+> Important: after initial installation, macOS will block usage of OCLint libraries. In order to get rid of the manual verification of each of them, use the following commands:
+>
+> ```bash
+> $ sudo xattr -dr com.apple.quarantine /usr/local/lib/oclint/rules/lib*
+> $ sudo xattr -dr com.apple.quarantine /usr/local/lib/oclint/reporters/lib*
+> ```
 
-```bash
-$ sudo xattr -dr com.apple.quarantine /usr/local/lib/oclint/rules/lib*
-$ sudo xattr -dr com.apple.quarantine /usr/local/lib/oclint/reporters/lib*
-```
-
-### mobsfscan
+#### mobsfscan
 
 mobsfscan is used to analyse Swift & Objective-C source files to find insecure code patterns.
+You can install it as explained in [here](https://github.com/MobSF/mobsfscan).
 
-See install instructions [here](https://github.com/MobSF/mobsfscan).
-
-### Periphery
+#### Periphery
 
 Periphery is used to analyse Swift source files to find unused code.
-
-See install instructions [here](https://github.com/peripheryapp/periphery).
+You can install it as explained in [here](https://github.com/peripheryapp/periphery).
 
 ## Installation (on the server)
 
@@ -111,14 +119,15 @@ sonar.tests=iOSAppTests
 # Defaults to build/result.xcresult
 # sonar.apple.resultBundlePath=custom/path/to/file.xcresult
 
-# Path to xcodebuild.log file
-# Defaults to build
-# sonar.apple.xcodebuild.logPath=custom/path/to/file.log
-
 # Path to periphery.log file
 # Defaults to build
 # sonar.apple.periphery.logPath=custom/path/to/file.log
 
+# Path to the JSON Compilation Database folder
+# The path is relative to the project base directory.
+# Defaults to build/json_compilation_database
+# sonar.apple.jsonCompilationDatabasePath=custom/path/to/folder
+
 # Encoding of the source code. Default is default system encoding.
 sonar.sourceEncoding=UTF-8
 ```
@@ -130,18 +139,20 @@ For a complete list of available options, please refer to the [SonarQube documen
 Use the following commands from the root folder to start an analysis:
 
 ```bash
-
-# Run tests 
 # Don't forget to add -workspace to the build command if your project is part of a workspace
 # Don't forget to activate 'Gather coverage' option in the app scheme or add '-enableCodeCoverage YES' to the following command
-$ xcodebuild \
+
+# Run tests 
+$ xcrun xcodebuild \
   -project MyApp.xcodeproj \
   -scheme MyApp \
   -sdk iphonesimulator \
   -destination 'platform=iOS Simulator,name=iPhone 11 Pro' \
   -derivedDataPath ./derivedData \
   -resultBundlePath build/result.xcresult \
-   clean test
+  OTHER_CFLAGS="\$(inherited) -gen-cdb-fragment-path build/compilation_database" \
+  -quiet \
+  clean test
 
 # Saves Periphery log to build/periphery.log (this is necessary for Swift dead code analysis)
 # Don't forget to add --workspace to the build command if your project is part of a workspace
@@ -154,15 +165,6 @@ $ periphery scan \
   --format xcode \
   --quiet | tee build/periphery.log
 
-# This rebuild is required to perform Objective-C issue analysis
-# Skip it if your project doe not use Objective-C, or if you do want to report Objective-C issues
-$ xcodebuild \
-  -project MyApp.xcodeproj \
-  -scheme MyApp \
-  -sdk iphonesimulator \
-  -destination 'platform=iOS Simulator,name=iPhone 11 Pro' \
-   COMPILER_INDEX_STORE_ENABLE=NO clean test | tee build/xcodebuild.log
-  
 # Run the analysis and publish to the SonarQube server
 # Don't forget to specify `sonar.host.url` and `sonar.login` in `sonar-project.properties` or supply it to the following command.
 $ sonar-scanner

commons/src/main/java/fr/insideapp/sonarqube/apple/commons/ExtensionFileFilter.java

@@ -0,0 +1,35 @@
+/*
+ * SonarQube Apple Plugin - Enables analysis of Swift and Objective-C projects into SonarQube.
+ * Copyright © 2022 inside|app (contact@insideapp.fr)
+ *
+ * This program is free software: you can redistribute it and/or modify
+ * it under the terms of the GNU Lesser General Public License as published by
+ * the Free Software Foundation, either version 3 of the License, or
+ * (at your option) any later version.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+ * GNU Lesser General Public License for more details.
+ *
+ * You should have received a copy of the GNU Lesser General Public License
+ * along with this program.  If not, see <http://www.gnu.org/licenses/>.
+ */
+package fr.insideapp.sonarqube.apple.commons;
+
+import java.io.File;
+import java.io.FileFilter;
+
+public final class ExtensionFileFilter implements FileFilter {
+
+    private final String extension;
+
+    public ExtensionFileFilter(String extension) {
+        this.extension = extension;
+    }
+
+    @Override
+    public boolean accept(File pathname) {
+        return pathname.getPath().toLowerCase().endsWith("." + extension);
+    }
+}

commons/src/test/java/fr/insideapp/sonarqube/apple/commons/tests/AppleTestsSensorTest.java

@@ -105,7 +105,7 @@ public void executeSuccess() {
     private void assertContainer(Container container) {
         // update setting to get a custom location for Xcode result bundle path
         MapSettings settings = new MapSettings();
-        settings.setProperty(AppleResultSensor.RESULT_BUNDLE_PATH_KEY, "sensor.xcresult");
+        settings.setProperty(AppleResultSensor.RESULT_BUNDLE_PATH_KEY, container.resultBundlePath);
         context.setSettings(settings);
 
         String fullFileNamePath = container.fileNamePath + "." + EXTENSION;

objc-lang/src/main/java/fr/insideapp/sonarqube/objc/issues/oclint/OCLintExtractor.java

@@ -0,0 +1,88 @@
+/*
+ * SonarQube Apple Plugin - Enables analysis of Swift and Objective-C projects into SonarQube.
+ * Copyright © 2022 inside|app (contact@insideapp.fr)
+ *
+ * This program is free software: you can redistribute it and/or modify
+ * it under the terms of the GNU Lesser General Public License as published by
+ * the Free Software Foundation, either version 3 of the License, or
+ * (at your option) any later version.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+ * GNU Lesser General Public License for more details.
+ *
+ * You should have received a copy of the GNU Lesser General Public License
+ * along with this program.  If not, see <http://www.gnu.org/licenses/>.
+ */
+package fr.insideapp.sonarqube.objc.issues.oclint;
+
+import com.fasterxml.jackson.core.JsonProcessingException;
+import com.fasterxml.jackson.databind.ObjectMapper;
+import com.fasterxml.jackson.databind.SerializationFeature;
+import fr.insideapp.sonarqube.objc.ObjectiveC;
+import fr.insideapp.sonarqube.objc.issues.oclint.models.OCLintReport;
+import org.buildobjects.process.ProcBuilder;
+import org.sonar.api.batch.sensor.SensorContext;
+import org.sonar.api.utils.log.Logger;
+import org.sonar.api.utils.log.Loggers;
+
+import java.io.File;
+import java.util.Arrays;
+import java.util.stream.Collectors;
+
+import static com.fasterxml.jackson.databind.DeserializationFeature.FAIL_ON_UNKNOWN_PROPERTIES;
+
+public class OCLintExtractor {
+
+    private static final Logger LOGGER = Loggers.get(OCLintExtractor.class);
+    private static final int COMMAND_TIMEOUT = 30 * 60 * 1000;
+    private static final int COMMAND_EXIT_CODE = 0;
+    private final SensorContext context;
+
+    private ObjectMapper objectMapper;
+
+    public OCLintExtractor(SensorContext context) {
+        this.context = context;
+        objectMapper = new ObjectMapper()
+                .disable(FAIL_ON_UNKNOWN_PROPERTIES)
+                .enable(SerializationFeature.INDENT_OUTPUT);
+    }
+
+    public OCLintReport extract(File compileCommandsFolder) throws JsonProcessingException {
+        String jsonReport = new ProcBuilder("oclint-json-compilation-database")
+                .withArgs(buildSourceArguments())
+                .withArgs("-p", compileCommandsFolder.getAbsolutePath())
+                .withArgs("--")
+                .withArgs("-report-type", "json")
+                .withTimeoutMillis(COMMAND_TIMEOUT)
+                .withExpectedExitStatuses(COMMAND_EXIT_CODE)
+                .run()
+                .getOutputString();
+
+        OCLintReport oclintReport = objectMapper.readValue(jsonReport, OCLintReport.class);
+        LOGGER.info("OCLint found {} violation(s)", oclintReport.violations.length);
+        return oclintReport;
+    }
+
+    private String[] buildSourceArguments() {
+        // Retrieve all the sources specified
+        final String sourcesInput = context.config().get("sonar.sources").orElse(".");
+        // Retrieve all the file extensions for Objective-C
+        String fileExtensions = Arrays.stream(ObjectiveC.EXTENSIONS).collect(Collectors.joining("|"));
+        final String[] sources = sourcesInput.split(",");
+        final String[] sourceArgs = new String[sources.length * 2];
+        final File baseDirectory = context.fileSystem().baseDir();
+        // Build parameters for each source
+        for (int i = 0; i < sources.length; i++) {
+            sourceArgs[i * 2] = "--include";
+            String regexPath = String.format("%s/.*\\.(%s)", sources[i], fileExtensions);
+            File absoluteRegexPath = new File(baseDirectory, regexPath);
+            LOGGER.debug("For source '{}', following regex is used: {}", sources[i], absoluteRegexPath.getAbsolutePath());
+            // we use the absolute path since (same as JSON Compilation Database)
+            sourceArgs[i * 2 + 1] = absoluteRegexPath.getAbsolutePath();
+        }
+        return sourceArgs;
+    }
+
+}

objc-lang/src/main/java/fr/insideapp/sonarqube/objc/issues/oclint/OCLintJSONDatabaseBuilder.java

@@ -0,0 +1,78 @@
+/*
+ * SonarQube Apple Plugin - Enables analysis of Swift and Objective-C projects into SonarQube.
+ * Copyright © 2022 inside|app (contact@insideapp.fr)
+ *
+ * This program is free software: you can redistribute it and/or modify
+ * it under the terms of the GNU Lesser General Public License as published by
+ * the Free Software Foundation, either version 3 of the License, or
+ * (at your option) any later version.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+ * GNU Lesser General Public License for more details.
+ *
+ * You should have received a copy of the GNU Lesser General Public License
+ * along with this program.  If not, see <http://www.gnu.org/licenses/>.
+ */
+package fr.insideapp.sonarqube.objc.issues.oclint;
+
+import fr.insideapp.sonarqube.apple.commons.ExtensionFileFilter;
+import org.apache.commons.io.FileUtils;
+import org.apache.commons.lang3.RegExUtils;
+import org.sonar.api.utils.log.Logger;
+import org.sonar.api.utils.log.Loggers;
+
+import java.io.File;
+import java.io.FileFilter;
+import java.nio.charset.StandardCharsets;
+import java.util.Arrays;
+import java.util.Iterator;
+import java.util.regex.Pattern;
+
+public final class OCLintJSONDatabaseBuilder {
+
+    private static final Pattern CLEAN_PATTERN = Pattern.compile("(\"-index-store-path.*DataStore\", |\"-index-unit-output-path.*\\.o\", )");
+
+    private static final Logger LOGGER = Loggers.get(OCLintJSONDatabaseBuilder.class);
+
+    public String build(File jsonCompilationDatabaseFolder) {
+        // Retrieve the JSON Database fragments
+        FileFilter jsonFileFilter = new ExtensionFileFilter("json");
+        File[] jsonFiles = jsonCompilationDatabaseFolder.listFiles(jsonFileFilter);
+        Iterator<File> jsonFilesIterator = Arrays.asList(jsonFiles).iterator();
+
+        final StringBuilder compileCommandsBuilder = new StringBuilder();
+
+        // Beginning the JSON Database
+        compileCommandsBuilder.append("[");
+
+        // For each JSON Database fragment
+        while (jsonFilesIterator.hasNext()) {
+            File jsonFile = jsonFilesIterator.next();
+            try {
+                // Getting the content
+                String json = FileUtils.readFileToString(jsonFile, StandardCharsets.UTF_8);
+                // Cleaning the JSON Database
+                json = RegExUtils.removeAll(json, CLEAN_PATTERN);
+
+                /* For the last one, we remove the trailing comma
+                because fragments generated by clang have a comma before EOF
+                and we want a valid JSON in the end
+                 */
+                if (!jsonFilesIterator.hasNext()) {
+                    json = RegExUtils.removeFirst(json, ",$");
+                }
+                compileCommandsBuilder.append(json);
+            } catch (Exception e) {
+                LOGGER.error(e.getMessage(), e);
+            }
+        }
+
+        // Ending the JSON Database
+        compileCommandsBuilder.append("]");
+
+        return compileCommandsBuilder.toString();
+    }
+
+}