rename scenario_coverage to scenario_execution_coverage (#206)

.devcontainer/Dockerfile

@@ -41,11 +41,7 @@ RUN --mount=type=bind,target=/tmp_setup export DEBIAN_FRONTEND=noninteractive &&
     xargs -a /tmp_setup/deb_requirements.txt apt-get install -y --no-install-recommends && \
     xargs -a /tmp_setup/libs/scenario_execution_kubernetes/deb_requirements.txt apt-get install -y --no-install-recommends && \
     rosdep update --rosdistro="${ROS_DISTRO}" && \
-    for d in /tmp_setup/*; do \
-        [[ ! -d  "$d" ]] && continue; \
-        [[ "$(basename $d)" =~ ^(install|build|log)$ ]] && continue; \
-        rosdep install --rosdistro="${ROS_DISTRO}" --from-paths "$d" --ignore-src -r -y; \
-    done && \
+    rosdep install --rosdistro="${ROS_DISTRO}" --from-paths /tmp_setup --ignore-src -r -y && \
     rm -rf /var/lib/apt/lists/*
 
 ##############################################################################

.github/workflows/test_build.yml

@@ -58,7 +58,7 @@ jobs:
           colcon test --packages-select \
           scenario_execution \
           scenario_execution_os \
-          scenario_coverage \
+          scenario_execution_coverage \
           scenario_execution_test \
           --event-handlers console_direct+ \
           --return-code-on-test-failure \
@@ -125,7 +125,7 @@ jobs:
         run: |
           source /opt/ros/${{ github.event.pull_request.base.ref == 'main' && 'humble' || github.event.pull_request.base.ref }}/setup.bash
           source install/setup.bash
-          find . -name "*.osc" | grep -Ev "lib_osc/*|examples/example_scenario_variation|scenario_coverage|fail*|install|build" | while read -r file; do 
+          find . -name "*.osc" | grep -Ev "lib_osc/*|examples/example_scenario_variation|scenario_execution_coverage|fail*|install|build" | while read -r file; do 
             echo "$file"; 
             ros2 run scenario_execution scenario_execution "$file" -n; 
           done
@@ -543,7 +543,7 @@ jobs:
           comment_mode: always
           files: |
             downloaded-artifacts/test-scenario-execution/scenario_execution/TEST.xml
-            downloaded-artifacts/test-scenario-execution/scenario_coverage/TEST.xml
+            downloaded-artifacts/test-scenario-execution/scenario_execution_coverage/TEST.xml
             downloaded-artifacts/test-scenario-execution/libs/scenario_execution_os/TEST.xml
             downloaded-artifacts/test-scenario-execution/test/scenario_execution_test/TEST.xml
             downloaded-artifacts/test-scenario-execution-ros/scenario_execution_ros/TEST.xml

README.md

@@ -69,6 +69,6 @@ source install/setup.bash
 To launch a scenario with ROS2:
 
 ```bash
-ros2 launch scenario_execution_ros scenario_launch.py scenario:=examples/example_scenario/hello_world.osc live_tree:=True
+ros2 run scenario_execution_ros scenario_execution_ros examples/example_scenario/hello_world.osc -t
 ```
 

docs/architecture.rst

@@ -66,7 +66,7 @@ Modules
 - ``scenario_execution``: The base package for scenario execution. It provides the parsing of OpenSCENARIO 2 files and the conversion to py-trees. It's middleware agnostic and can therefore be used as a basis for more specific implementations (e.g. ROS). It also provides basic OpenSCENARIO 2 libraries and actions.
 - ``scenario_execution_ros``: This package uses ``scenario_execution`` as a basis and implements a ROS2 version of scenario execution. It provides a OpenSCENARIO 2 library with basic ROS2-related actions like publishing on a topic or calling a service.
 - ``scenario_execution_control``: Provides code to control scenario execution (in ROS2) from another application such as RViz.
-- ``scenario_coverage``: Provides tools to generate concrete scenarios from abstract OpenSCENARIO 2 scenario definition and execute them.
+- ``scenario_execution_coverage``: Provides tools to generate concrete scenarios from abstract OpenSCENARIO 2 scenario definition and execute them.
 - ``scenario_execution_gazebo``: Provides a `Gazebo <https://gazebosim.org/>`_-specific OpenSCENARIO 2 library with actions.
 - ``scenario_execution_interfaces``: Provides ROS2 `interfaces <https://docs.ros.org/en/rolling/Concepts/Basic/About-Interfaces.html>`__, more specifically, messages and services, which are used to interface ROS2 with the ``scenario_execution_control`` package.
 - ``scenario_execution_rviz``: Contains several `rviz <https://github.com/ros2/rviz>`__ plugins for visualizing and controlling scenarios when working with ROS2.

docs/how_to_run.rst

@@ -187,15 +187,15 @@ PyQtEngine works on your machine and render web pages correctly.
 
 Scenario Coverage
 -----------------
-The ``scenario_coverage`` package provides the ability to run variations of a scenario from a single scenario definition. It offers a fast and efficient method to test scenario with different attribute values, streamlining the development and testing process.
+The ``scenario_execution_coverage`` package provides the ability to run variations of a scenario from a single scenario definition. It offers a fast and efficient method to test scenario with different attribute values, streamlining the development and testing process.
 
-Below are the steps to run a scenario using ``scenario_coverage``..
+Below are the steps to run a scenario using ``scenario_execution_coverage``..
 
 First, build the packages:
 
 .. code-block:: bash
 
-   colcon build --packages-up-to scenario_coverage
+   colcon build --packages-up-to scenario_execution_coverage
    source install/setup.bash
 
 Then, generate the scenario files for each variation of scenario  using the ``scenario_variation`` executable, you can pass your own custom scenario as an input. For this exercise, we will use a scenario present in  :repo_link:`examples/example_scenario_variation/`.

docs/tutorials.rst

@@ -315,7 +315,7 @@ Create Scenarios with Variations
 --------------------------------
 In this example, we'll demonstrate how to generate and run multiple scenarios using only one scenario definition.
 
-For this we'll use the  :repo_link:`scenario_coverage/scenario_coverage/scenario_variation`. to save the intermediate scenario models in ``.sce`` extension file and then use :repo_link:`scenario_coverage/scenario_coverage/scenario_batch_execution` to execute each generated scenario.
+For this we'll use the  :repo_link:`scenario_execution_coverage/scenario_execution_coverage/scenario_variation`. to save the intermediate scenario models in ``.sce`` extension file and then use :repo_link:`scenario_execution_coverage/scenario_execution_coverage/scenario_batch_execution` to execute each generated scenario.
 
 The scenario file looks as follows:
 
@@ -332,14 +332,14 @@ The scenario file looks as follows:
 Here, a simple scenario variation example using log action plugin is created and two messages ``foo`` and
 ``bar`` using the array syntax are passed.
 
-As this is not a concrete scenario, ``scenario_execution`` won't be able to execute it. Instead we'll use ``scenario_variation`` from the ``scenario_coverage`` package to generate all variations and save them to intermediate scenario model files with ``.sce`` extension.
+As this is not a concrete scenario, ``scenario_execution`` won't be able to execute it. Instead we'll use ``scenario_variation`` from the ``scenario_execution_coverage`` package to generate all variations and save them to intermediate scenario model files with ``.sce`` extension.
 Afterwards we could either use ``scenario_execution`` to run each created scenario manually or make use of ``scenario_batch_execution`` which reads all scenarios within a directory and executes them one after the other.
 
-Now, lets try to run this scenario. To do this, first build Packages ``scenario_execution`` and ``scenario_coverage``:
+Now, lets try to run this scenario. To do this, first build Packages ``scenario_execution`` and ``scenario_execution_coverage``:
 
 .. code-block::
 
-    colcon build --packages-up-to scenario_execution_ros && colcon build --packages-up-to scenario_coverage
+    colcon build --packages-up-to scenario_execution_ros && colcon build --packages-up-to scenario_execution_coverage
 
 
 * Now, create intermediate scenarios with ``.sce`` extension using the command:

examples/example_scenario_variation/README.md

@@ -1,9 +1,9 @@
 # Example Scenario Variation
 
-To run the Example Scenario Variation with scenario, first build Packages `scenario_execution` and `scenario_coverage`:
+To run the Example Scenario Variation with scenario, first build Packages `scenario_execution` and `scenario_execution_coverage`:
 
 ```bash
-colcon build --packages-up-to scenario_execution && colcon build --packages-up-to scenario_coverage
+colcon build --packages-up-to scenario_execution && colcon build --packages-up-to scenario_execution_coverage
 ```
 
 Source the workspace:

scenario_execution_coverage/CHANGELOG.rst

@@ -0,0 +1,8 @@
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+Changelog for package scenario_execution_coverage
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+1.2.0 (2024-10-02)
+------------------
+* Initial creation of coverage package for scenario execution
+

scenario_coverage/README.md → scenario_execution_coverage/README.md

@@ -1,6 +1,6 @@
-# Scenario Coverage
+# Scenario Execution Coverage
 
-The `scenario_coverage` packages provides two tools:
+The `scenario_execution_coverage` packages provides two tools:
 
 - `scenario_variation`: Create concrete scenarios out of scenario with variation definition
 - `scenario_batch_execution`: Execute multiple scenarios, one after the other.

scenario_coverage/package.xml → scenario_execution_coverage/package.xml

@@ -1,7 +1,7 @@
 <?xml version="1.0"?>
 <?xml-model href="http://download.ros.org/schema/package_format3.xsd" schematypens="http://www.w3.org/2001/XMLSchema"?>
 <package format="3">
-  <name>scenario_coverage</name>
+  <name>scenario_execution_coverage</name>
   <version>1.2.0</version>
   <description>Robotics Scenario Execution Coverage Tools</description>
   <author email="[email protected]">Intel Labs</author>

scenario_execution_coverage/scenario_execution_coverage/__init__.py

@@ -0,0 +1,15 @@
+# Copyright (C) 2024 Intel Corporation
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions
+# and limitations under the License.
+#
+# SPDX-License-Identifier: Apache-2.0

scenario_execution_coverage/scenario_execution_coverage/scenario_batch_execution.py

@@ -0,0 +1,223 @@
+#! /usr/bin/env python3
+
+# Copyright (C) 2024 Intel Corporation
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions
+# and limitations under the License.
+#
+# SPDX-License-Identifier: Apache-2.0
+
+import os
+import sys
+import argparse
+import subprocess  # nosec B404
+from threading import Thread
+from copy import deepcopy
+import signal
+from defusedxml import ElementTree as ETparse
+import xml.etree.ElementTree as ET  # nosec B405
+import logging
+
+
+class ScenarioBatchExecution(object):
+
+    def __init__(self, args) -> None:
+        self.ignore_process_return_value = args.ignore_process_return_value
+        if not os.path.isdir(args.output_dir):
+            try:
+                os.mkdir(args.output_dir)
+            except OSError as e:
+                raise ValueError(f"Could not create output directory: {e}") from e
+        if not os.access(args.output_dir, os.W_OK):
+            raise ValueError(f"Output directory '{args.output_dir}' not writable.")
+        if os.path.exists(os.path.join(args.output_dir, 'test.xml')):
+            os.remove(os.path.join(args.output_dir, 'test.xml'))
+        self.output_dir = args.output_dir
+
+        dir_content = os.listdir(args.scenario_dir)
+        self.scenarios = []
+        for entry in dir_content:
+            if entry.endswith(".sce") or entry.endswith(".osc"):
+                self.scenarios.append(os.path.join(args.scenario_dir, entry))
+        if not self.scenarios:
+            raise ValueError(f"Directory {args.scenario_dir} does not contain any scenarios.")
+        self.scenarios.sort()
+        print(f"Detected {len(self.scenarios)} scenarios.")
+        self.launch_command = args.launch_command
+        if self.get_launch_command("", "") is None:
+            raise ValueError("Launch command does not contain {SCENARIO} and {OUTPUT_DIR}: " + " ".join(args.launch_command))
+        print(f"Launch command: {self.launch_command}")
+
+    def get_launch_command(self, scenario_name, output_dir):
+        launch_command = deepcopy(self.launch_command)
+        scenario_replaced = False
+        output_dir_replaced = False
+        for i in range(0, len(launch_command)):  # pylint: disable=consider-using-enumerate
+            if "{SCENARIO}" in launch_command[i]:
+                launch_command[i] = launch_command[i].replace('{SCENARIO}', scenario_name)
+                scenario_replaced = True
+            if "{OUTPUT_DIR}" in launch_command[i]:
+                launch_command[i] = launch_command[i].replace('{OUTPUT_DIR}', output_dir)
+                output_dir_replaced = True
+        if scenario_replaced and output_dir_replaced:
+            return launch_command
+        else:
+            return None
+
+    def run(self) -> bool:
+        def log_output(out, logger):
+            try:
+                for line in iter(out.readline, b''):
+                    msg = line.decode().strip()
+                    print(msg)
+                    logger.info(msg)
+                out.close()
+            except ValueError:
+                pass
+
+        def configure_logger(log_file_path):
+            logger = logging.getLogger(log_file_path)
+            if logger.hasHandlers():
+                logger.handlers.clear()
+            file_handler = logging.FileHandler(filename=log_file_path, mode='a')
+            file_handler.setFormatter(logging.Formatter('%(message)s'))
+            file_handler.setLevel(logging.INFO)
+            logger.addHandler(file_handler)
+            logger.setLevel(logging.INFO)
+            return logger
+
+        ret = True
+        for scenario in self.scenarios:
+            scenario_name = os.path.splitext(os.path.basename(scenario))[0]
+            output_file_path = os.path.join(self.output_dir, scenario_name)
+            if not os.path.isdir(output_file_path):
+                os.mkdir(output_file_path)
+            launch_command = self.get_launch_command(scenario, output_file_path)
+            log_cmd = " ".join(launch_command)
+            print(f"### For scenario {scenario}, executing process: '{log_cmd}'")
+            process = subprocess.Popen(launch_command, stdout=subprocess.PIPE, stderr=subprocess.PIPE, stdin=subprocess.PIPE)
+            file_handler = logging.FileHandler(filename=os.path.join(output_file_path, scenario_name + '.log'), mode='w')
+            logger = configure_logger(os.path.join(output_file_path, scenario_name + '.log'))
+            log_stdout_thread = Thread(target=log_output, args=(process.stdout, logger, ))
+            log_stdout_thread.daemon = True  # die with the program
+            log_stdout_thread.start()
+            log_stderr_thread = Thread(target=log_output, args=(process.stderr, logger, ))
+            log_stderr_thread.daemon = True  # die with the program
+            log_stderr_thread.start()
+
+            print(f"### Waiting for process to finish...")
+            try:
+                process.wait()
+                if process.returncode:
+                    print("### Process failed.")
+                    ret = False
+                else:
+                    print("### Process finished successfully.")
+            except KeyboardInterrupt:
+                print("### Interrupted by user. Sending SIGINT...")
+                process.send_signal(signal.SIGINT)
+                try:
+                    process.wait(timeout=20)
+                    return False
+                except subprocess.TimeoutExpired:
+                    print("### Process not stopped after 20s. Sending SIGKILL...")
+                    process.send_signal(signal.SIGKILL)
+                try:
+                    process.wait(timeout=10)
+                    return False
+                except subprocess.TimeoutExpired:
+                    print("### Process not stopped after 10s.")
+                return False
+            file_handler.flush()
+            file_handler.close()
+        xml_ret = self.combine_test_xml()
+        if self.ignore_process_return_value:
+            return xml_ret
+        else:
+            return xml_ret and ret
+
+    def combine_test_xml(self):
+        print(f"### Writing combined tests to '{self.output_dir}/test.xml'.....")
+        tree = ET.Element('testsuite')
+        total_time = 0
+        total_errors = 0
+        total_failures = 0
+        total_tests = 0
+        for scenario in self.scenarios:
+            scenario_name = os.path.splitext(os.path.basename(scenario))[0]
+            test_file = os.path.join(self.output_dir, scenario_name, 'test.xml')
+            parsed_successfully = False
+            if os.path.exists(test_file):
+                root = None
+                try:
+                    test_tree = ETparse.parse(test_file)
+                    root = test_tree.getroot()
+                except ETparse.ParseError:
+                    print(f"### Error XML file {test_file} could not be parsed")
+                if root is not None:
+                    parsed_successfully = True
+                    total_errors += int(root.attrib.get('errors', 0))
+                    total_failures += int(root.attrib.get('failures', 0))
+                    total_time += float(root.attrib.get('time', 0))
+                    total_tests += int(root.attrib.get('tests', 0))
+                    for testcase in root.findall('testcase'):
+                        testcase.set('name', str(scenario_name))
+                        tree.append(testcase)
+                else:
+                    print(f"### XML file has no 'testsuite' element. {test_file}")
+
+            if not parsed_successfully:
+                total_errors += 1
+                missing_test_elem = ET.Element('testcase')
+                missing_test_elem.set("classname", "tests.scenario")
+                missing_test_elem.set("name", "no_test_result")
+                missing_test_elem.set("time", "0.0")
+                failure_elem = ET.Element('failure')
+                failure_elem.set("message", f"expected file {test_file} not found")
+                missing_test_elem.append(failure_elem)
+                tree.append(missing_test_elem)
+        tree.set('errors', str(total_errors))
+        tree.set('failures', str(total_failures))
+        tree.set('time', str(total_time))
+        tree.set('tests', str(total_tests))
+        combined_tests = ET.ElementTree(tree)
+        ET.indent(combined_tests, space="\t", level=0)
+        combined_tests.write(os.path.join(self.output_dir, "test.xml"), encoding='utf-8', xml_declaration=True)
+        return total_errors == 0 and total_failures == 0
+
+
+def main():
+    """
+    main function
+    """
+    parser = argparse.ArgumentParser()
+    parser.add_argument('-i', '--scenario-dir', type=str, help='Directory containing the scenarios')
+    parser.add_argument('-o', '--output-dir', type=str, help='Directory containing the output', default='out')
+    parser.add_argument('-r', '--ignore-process-return-value', action='store_true',
+                        help='Should a non-zero return value of the executed process result in a failure?')
+    parser.add_argument('launch_command', nargs='+')
+    args = parser.parse_args(sys.argv[1:])
+
+    try:
+        scenario_batch_execution = ScenarioBatchExecution(args)
+    except Exception as e:  # pylint: disable=broad-except
+        print(f"Error while initializing batch execution: {e}")
+        sys.exit(1)
+    if scenario_batch_execution.run():
+        sys.exit(0)
+    else:
+        print("Error during batch executing!")
+        sys.exit(1)
+
+
+if __name__ == '__main__':
+    main()