Add: Extraction of VIDA scripts

These VIDA scripts directly implement some functionality of the program interacting with the vehicles. From this, one can learn about the usage of the individual CAN messages.

Author: spnda

.gitignore

@@ -2,3 +2,4 @@ sql/__pycache__/
 ecu/
 csv/
 images/
+extracted_scripts/

README.md

@@ -9,7 +9,7 @@ The Python scripts use duckdb and pandas to work with the data.
 ### Extracting data from the database
 
 Depending on wether PowerShell 7 and the SQLServer module are installed, you can either use `generate_csv.ps1` or `generate_csv.bat`.
-The PowerShell script takes significantly longer than the Batch script does to generate, but is capable of outputting completely correct files.
+The PowerShell scripts can take longer than the Batch script does to generate, but is capable of outputting completely correct files.
 The PowerShell script uses the `UseQuotes` option on `Export-Csv` which was introduced in PowerShell 7. On Windows 7 Professional the last working version of PowerShell 7 is 7.2.9.
 The SQL credentials are already written into the scripts and won't need to be changed.
 
@@ -33,3 +33,11 @@ The CSV file will contain all known CAN parameters, their localized name, their
 
 The conversion methods found in the outputted CSV file can be applied with the `scripts/evaluate_conversion.py` script.
 It parses the expression into a simple executable AST that supports the feature set required for the expressions from the VIDA database.
+
+### Extracting scripts
+
+The VIDA database contains XML-based scripts that can read and write data from and to the car.
+These use the CAN protocol and are used to perform certain actions, but can also be used to figure out how the protocol works.
+Reading these scripts can allow us to figure out how the CAN-based protocol works and how we can make use of it. 
+
+The `sql/extract_scripts.ps1` can be used to extract all script metadata into CSV files and the actual scripts into decompressed XML files.

scripts/filter_scripts.py

@@ -0,0 +1,57 @@
+import sys
+import duckdb
+import pandas as pd
+from read_csv import get_csv, DatabaseFile
+from vin_decoder import decode_vin, Vehicle
+
+def get_filtered_scripts_for_profile(profile: str) -> list[str]:
+    """
+    Returns all script IDs available for given profile
+    """
+    get_csv(DatabaseFile.script_profile_map)
+    return duckdb.sql(f"""
+        SELECT fkScript FROM script_profile_map
+        WHERE TRIM(fkProfile) = '{profile}'
+    """).df()['fkScript'].tolist()
+
+
+def get_filtered_scripts(profiles: list[str]) -> pd.DataFrame:
+    """
+    Returns all script IDs available for all given profiles.
+    This is intended where one vehicle is represented by multiple profiles,
+    not for querying scripts for multiple vehicles.
+    """
+    get_csv(DatabaseFile.script_profile_map)
+    profiles_df = pd.DataFrame(profiles, columns=['fkProfile'])
+    return duckdb.sql("""
+        SELECT DISTINCT fkScript FROM script_profile_map
+        INNER JOIN profiles_df ON profiles_df.fkProfile=TRIM(script_profile_map.fkProfile)
+    """).df()
+
+
+def get_script_descriptions(scripts: pd.DataFrame) -> pd.DataFrame:
+    get_csv(DatabaseFile.script_content)
+    return duckdb.sql("""
+        SELECT DISTINCT scripts.fkScript,script_content.DisplayText FROM scripts
+        INNER JOIN script_content ON script_content.fkScript=scripts.fkScript
+    """).df()
+
+
+def get_script_description(script_id: str) -> str:
+    get_csv(DatabaseFile.script_content)
+    return duckdb.sql(f"SELECT fkScript,DisplayText FROM script_content WHERE fkScript='{script_id}'").df()['DisplayText'].iloc[0]
+
+
+if __name__ == '__main__':
+    profiles = []
+
+    if len(profiles) == 0 and len(sys.argv) == 1:
+        print("Usage: filter_scripts.py <VIN>")
+    else:
+        if len(profiles) == 0:
+            vehicle = decode_vin(sys.argv[1], 1002)
+            profiles = vehicle.get_vehicle_profiles()
+        
+        scripts = get_filtered_scripts(profiles)
+        scripts = get_script_descriptions(scripts)
+        duckdb.write_csv(scripts, "ecu/scripts.csv")

scripts/read_csv.py

@@ -26,6 +26,12 @@ class DatabaseFile(str, enum.Enum):
     t162 = 'carcom/T162_ProfileValue'
     t163 = 'carcom/T163_ProfileValueType'
     t191 = 'carcom/T191_TextData'
+    script = 'DiagSwdlRepository/Script'
+    script_car_function = 'DiagSwdlRepository/ScriptCarFunction'
+    script_content = 'DiagSwdlRepository/ScriptContent'
+    script_profile_map = 'DiagSwdlRepository/ScriptProfileMap'
+    script_type = 'DiagSwdlRepository/ScriptType'
+    script_variant = 'DiagSwdlRepository/ScriptVariant'
 
 def get_csv(csv_file: DatabaseFile) -> duckdb.DuckDBPyRelation:
     """
@@ -34,6 +40,6 @@ def get_csv(csv_file: DatabaseFile) -> duckdb.DuckDBPyRelation:
     if csv_file in loaded_csv_dictionary:
         return loaded_csv_dictionary[csv_file]
     else:
-        loaded_csv_dictionary[csv_file] = duckdb.read_csv(f'csv/{csv_file}.csv', encoding='utf-8')
+        loaded_csv_dictionary[csv_file] = duckdb.read_csv(f'csv/{csv_file}.csv', header=True, encoding='utf-8')
         duckdb.register(csv_file.name, loaded_csv_dictionary[csv_file])
         return loaded_csv_dictionary[csv_file]

sql/extract_scripts.ps1

@@ -0,0 +1,40 @@
+. $PSScriptRoot\sql_server.ps1
+
+New-Item -ItemType Directory -Path 'csv/DiagSwdlRepository/' -ErrorAction SilentlyContinue
+
+# First, extract metadata into CSV files
+$db_tables = 'Script','ScriptCarFunction','ScriptProfileMap','ScriptType','ScriptVariant'
+foreach ($table in $db_tables) {
+    Write-Output "Writing CSV '$($table)'..."
+    $file_path = 'csv/DiagSwdlRepository/' + $table + '.csv'
+    $rows = Invoke-Sqlcmd @parameters -Query @"
+        set nocount on;
+        SELECT * FROM DiagSwdlRepository.dbo.$($table)
+"@
+    $rows | Export-Csv -Encoding UTF8 -UseQuotes AsNeeded -NoTypeInformation -path $file_path
+}
+
+# Now, extract the ScriptContent table but write the binary data into separate files
+$target_language = 15 # This is en-US. For values, check DiagSwdlRepository.dbo.Language
+$rows = Invoke-Sqlcmd @parameters -MaxBinaryLength 10000000 -Query @"
+    SELECT * FROM DiagSwdlRepository.dbo.ScriptContent
+    WHERE fkLanguage=$($target_language)
+"@
+
+$rows | Export-Csv -Encoding UTF8 -UseQuotes AsNeeded -NoTypeInformation -path 'csv/DiagSwdlRepository/ScriptContent.csv'
+
+# The methodology of extracting the scripts I found here: http://www.stevediraddo.com/2019/01/13/volvo-canbus-tinkering/
+New-Item -ItemType Directory -Path 'extracted_scripts/' -ErrorAction SilentlyContinue
+foreach ($row in $rows) {
+    Write-Output "Writing script '$($row.fkScript)'..."
+
+    # Write the zip file data to a file
+    # TODO: Using MemoryStream and ZipArchive, it should be possible to do this in memory with .NET
+    $zip_file = "$PWD\extracted_scripts\$($row.fkScript).zip"
+    [System.IO.File]::WriteAllBytes($zip_file, $row.XmlDataCompressed)
+    
+    # Extract the script XML from the archive, and give it an xml ending
+    Expand-Archive -Path $zip_file -DestinationPath "$PWD\extracted_scripts"
+    Rename-Item -Path "$PWD\extracted_scripts\$($row.fkScript)" -NewName "$PWD\extracted_scripts\$($row.fkScript).xml"
+    Remove-Item -Path $zip_file
+}

sql/generate_csv.ps1

@@ -16,10 +16,7 @@ foreach ($dbname in $dbnames) {
 
     # Make sure the target directory exists 
     $folder = 'csv/' + $dbname + '/'
-    if (!(Test-Path -PathType container $folder))
-    {
-        New-Item -ItemType Directory -Path $folder | Out-Null
-    }
+    New-Item -ItemType Directory -Path $folder -ErrorAction SilentlyContinue
 
     # Write the contents of each table as CSVs
     # TODO: This script takes VERY long to execute. Use threading?