Plaso storage issue

[vxsh4d0w]

Hey there,
yesterday after creating a plaso file, I tried to upload it on Timesketch and I got this message:

Original filename: dc.plaso
File on disk: /usr/share/timesketch/upload/23b8bc9904be40a09e3537d7eecea7b1
File size: 3.56GB
Uploaded by: jdoe
Provider: WebUpload
Context: dc.plaso
Status: fail
Total File Events: 0
Error message: Traceback (most recent call last): File "/usr/bin/pinfo.py", line 85, in if not Main(): File "/usr/bin/pinfo.py", line 62, in Main tool.PrintStorageInformation() File "/usr/lib/python3/dist-packages/plaso/cli/pinfo_tool.py", line 1673, in PrintStorageInformation storage_reader = self._GetStorageReader(self._storage_file_path) File "/usr/lib/python3/dist-packages/plaso/cli/pinfo_tool.py", line 587, in _GetStorageReader storage_factory.StorageFactory.CreateStorageReaderForFile(path)) File "/usr/lib/python3/dist-packages/plaso/storage/factory.py", line 49, in CreateStorageReaderForFile return sqlite_reader.SQLiteStorageFileReader(path) File "/usr/lib/python3/dist-packages/plaso/storage/sqlite/reader.py", line 20, in init self._store.Open(path=path) File "/usr/lib/python3/dist-packages/acstore/sqlite_store.py", line 999, in Open self._ReadAndCheckStorageMetadata(check_readable_only=True) File "/usr/lib/python3/dist-packages/plaso/storage/sqlite/sqlite_file.py", line 231, in _ReadAndCheckStorageMetadata self._CheckStorageMetadata( File "/usr/lib/python3/dist-packages/plaso/storage/sqlite/sqlite_file.py", line 53, in _CheckStorageMetadata super(SQLiteStorageFile, self)._CheckStorageMetadata( File "/usr/lib/python3/dist-packages/acstore/sqlite_store.py", line 199, in _CheckStorageMetadata raise IOError(( OSError: Format version: 20230327 is too new and not yet supported, minimum supported version: 20230107.

The file has been generated with plaso - log2timeline version 20230717, while Timeskecth version is 20230721 (runnning on Docker).
How can I fix this issue?

[jaegeral]

Hey, something is not really making sense, the error message states the file is created in version 20230327 can you please run https://plaso.readthedocs.io/en/latest/sources/user/Using-pinfo.html on the file and provide the version?

And can you go into the docker container and do:
https://timesketch.org/guides/admin/troubleshooting/#issues-importing-plaso-file

And run log2timeline.py --troubles within the docker container as well?

Thx

[vxsh4d0w]

Here all info:

************************** Plaso Storage Information ***************************
Filename : dc.plaso
Format version : 20230327
Serialization format : json

*********************************** Sessions ***********************************
bec1414e-e1ee-4a9e-942e-1f8e20a46169 : 2023-08-01T19:28:28.164274+00:00

******************************** Event sources *********************************
Total : 246947

************************* Events generated per parser **************************
Parser (plugin) name : Number of events

                         amcache : 885
                  appcompatcache : 1019
                          bagmru : 59
           explorer_mountpoints2 : 4
          explorer_programscache : 2
                        filestat : 987538
                             lnk : 890
                  mrulist_string : 2
       mrulistex_shell_item_list : 2
                mrulistex_string : 6
 mrulistex_string_and_shell_item : 5

mrulistex_string_and_shell_item_list : 1
msie_webcache : 48
msie_zone : 60
network_drives : 2
networks : 8
olecf_automatic_destinations : 35
olecf_default : 161
olecf_document_summary : 12
olecf_summary : 80
pe : 38502
setupapi : 2184
shell_items : 405
user_access_logging : 25202
userassist : 41
usnjrnl : 4147158
windows_boot_execute : 8
windows_run : 10
windows_sam_users : 4
windows_services : 1652
windows_shutdown : 8
windows_task_cache : 340
windows_timezone : 4
windows_typed_urls : 2
windows_version : 6
winevtx : 229996
winlogon : 12
winrar_mru : 1
winreg_default : 689247
Total : 6125601

No events labels stored.

******************* Extraction warnings generated per parser *******************
Parser (plugin) name : Number of warnings

           esedb : 1
          winreg : 1

---

************** Path specifications with most extraction warnings ***************
Number of warnings : Pathspec

             1 : type: OS, location: /mnt/d/SharedVM/XXXXXXXX_C.raw
               : type: RAW
               : type: TSK_PARTITION, location: /p2, part index: 3, start
                 offset: 0x15f00000
               : type: NTFS, location: \Windows\NTDS\temp.edb, MFT
                 attribute: 1, MFT entry: 35433
             1 : type: OS, location: /mnt/d/SharedVM/XXXXXXXX_C.raw
               : type: RAW
               : type: TSK_PARTITION, location: /p2, part index: 3, start
                 offset: 0x15f00000
               : type: NTFS, location: \Windows\System32\config\SOFTWARE,
                 MFT attribute: 1, MFT entry: 98432

---

******************** Recovery warnings generated per parser ********************
Parser (plugin) name : Number of warnings

         winevtx : 1163

---

*************** Path specifications with most recovery warnings ****************
Number of warnings : Pathspec

           147 : type: OS, location: /mnt/d/SharedVM/XXXXXXXX_C.raw
               : type: RAW
               : type: TSK_PARTITION, location: /p2, part index: 3, start
                 offset: 0x15f00000
               : type: NTFS, location:
                 \Windows\System32\winevt\Logs\Microsoft-Windows-PowerShell%4Operational.evtx,
                 MFT attribute: 2, MFT entry: 625
           129 : type: OS, location: /mnt/d/SharedVM/XXXXXXXX_C.raw
               : type: RAW
               : type: TSK_PARTITION, location: /p2, part index: 3, start
                 offset: 0x15f00000
               : type: NTFS, location:
                 \Windows\System32\winevt\Logs\Microsoft-Windows-AppReadiness%4Operational.evtx,
                 MFT attribute: 2, MFT entry: 95602
           116 : type: OS, location: /mnt/d/SharedVM/XXXXXXXX_C.raw
               : type: RAW
               : type: TSK_PARTITION, location: /p2, part index: 3, start
                 offset: 0x15f00000
               : type: NTFS, location:
                 \Windows\System32\winevt\Logs\Microsoft-Windows-WinRM%4Operational.evtx,
                 MFT attribute: 2, MFT entry: 108249
            93 : type: OS, location: /mnt/d/SharedVM/XXXXXXXX_C.raw
               : type: RAW
               : type: TSK_PARTITION, location: /p2, part index: 3, start
                 offset: 0x15f00000
               : type: NTFS, location:
                 \Windows\System32\winevt\Logs\Microsoft-Windows-CodeIntegrity%4Operational.evtx,
                 MFT attribute: 2, MFT entry: 36124
            86 : type: OS, location: /mnt/d/SharedVM/XXXXXXXX_C.raw
               : type: RAW
               : type: TSK_PARTITION, location: /p2, part index: 3, start
                 offset: 0x15f00000
               : type: NTFS, location:
                 \Windows\System32\winevt\Logs\Microsoft-Windows-International%4Operational.evtx,
                 MFT attribute: 2, MFT entry: 94680
            76 : type: OS, location: /mnt/d/SharedVM/XXXXXXXX_C.raw
               : type: RAW
               : type: TSK_PARTITION, location: /p2, part index: 3, start
                 offset: 0x15f00000
               : type: NTFS, location:
                 \Windows\System32\winevt\Logs\Microsoft-Windows-TerminalServices-LocalSessionManager%4Operational.evtx,
                 MFT attribute: 2, MFT entry: 108255
            74 : type: OS, location: /mnt/d/SharedVM/XXXXXXXX_C.raw
               : type: RAW
               : type: TSK_PARTITION, location: /p2, part index: 3, start
                 offset: 0x15f00000
               : type: NTFS, location:
                 \Windows\System32\winevt\Logs\Microsoft-Windows-TaskScheduler%4Maintenance.evtx,
                 MFT attribute: 2, MFT entry: 108340
            60 : type: OS, location: /mnt/d/SharedVM/XXXXXXXX_C.raw
               : type: RAW
               : type: TSK_PARTITION, location: /p2, part index: 3, start
                 offset: 0x15f00000
               : type: NTFS, location:
                 \Windows\System32\winevt\Logs\Microsoft-Windows-WindowsUpdateClient%4Operational.evtx,
                 MFT attribute: 2, MFT entry: 479
            50 : type: OS, location: /mnt/d/SharedVM/XXXXXXXX_C.raw
               : type: RAW
               : type: TSK_PARTITION, location: /p2, part index: 3, start
                 offset: 0x15f00000
               : type: NTFS, location:
                 \Windows\System32\winevt\Logs\Microsoft-Windows-Bits-Client%4Operational.evtx,
                 MFT attribute: 2, MFT entry: 495
            47 : type: OS, location: /mnt/d/SharedVM/XXXXXXXX_C.raw
               : type: RAW
               : type: TSK_PARTITION, location: /p2, part index: 3, start
                 offset: 0x15f00000
               : type: NTFS, location:
                 \Windows\System32\winevt\Logs\Microsoft-Windows-GroupPolicy%4Operational.evtx,
                 MFT attribute: 2, MFT entry: 108253

---

No analysis reports stored.

$ sudo docker exec timesketch-web log2timeline.py --troubles
2023-08-02 10:01:29,604 [INFO] (MainProcess) PID:85 <data_location> Determined data location: /usr/share/plaso
Using Python version 3.10.6 (main, May 29 2023, 11:10:38) [GCC 11.3.0]

Path: /usr/bin/log2timeline.py

plaso - log2timeline version 20230311

Checking availability and versions of dependencies.
[OK] acstore version: 20230226
[OK] artifacts version: 20221219
[OK] bencode
[OK] certifi version: 2023.05.07
[OK] cryptography version: 41.0.2
[OK] dateutil version: 2.8.1
[OK] defusedxml version: 0.7.1
[OK] dfdatetime version: 20230225
[OK] dfvfs version: 20221224
[OK] dfwinreg version: 20221218
[OK] dtfabric version: 20221218
[OPTIONAL] unable to determine version information for: flor
[OK] future version: 0.18.2
[OK] lz4 version: 4.3.2
[OK] opensearchpy
[OK] pefile version: 2023.2.7
[OK] psutil version: 5.9.4
[OK] pybde version: 20221031
[OK] pycreg version: 20221022
[OK] pyesedb version: 20220806
[OK] pyevt version: 20221022
[OK] pyevtx version: 20221101
[OK] pyewf version: 20140814
[OK] pyfsapfs version: 20221102
[OK] pyfsext version: 20220829
[OK] pyfsfat version: 20220925
[OK] pyfshfs version: 20220831
[OK] pyfsntfs version: 20221023
[OK] pyfsxfs version: 20220829
[OK] pyfvde version: 20220915
[OK] pyfwnt version: 20220922
[OK] pyfwsi version: 20230114
[OK] pylnk version: 20230205
[OK] pyluksde version: 20221103
[OK] pymodi version: 20221023
[OK] pymsiecf version: 20221024
[OK] pyolecf version: 20221024
[OK] pyparsing version: 2.4.7
[OK] pyphdi version: 20221025
[OK] pyqcow version: 20221124
[OK] pyregf version: 20221026
[OK] pyscca version: 20221027
[OK] pysigscan version: 20230109
[OK] pysmdev version: 20221028
[OK] pysmraw version: 20221028
[OK] pytsk3 version: 20221228
[OK] pytz
[OK] pyvhdi version: 20221124
[OK] pyvmdk version: 20221124
[OK] pyvsgpt version: 20221029
[OK] pyvshadow version: 20221030
[OK] pyvslvm version: 20221025
[OK] redis version: 4.4.4
[OK] requests version: 2.31.0
[OK] six version: 1.12.0
[OK] urllib3 version: 1.26.16
[OK] xattr version: 0.10.1
[OK] xlsxwriter version: 3.0.8
[OK] yaml version: 6.0.1
[OK] yara version: 4.2.3
[OK] zmq version: 25.0.0

Also see: https://plaso.readthedocs.io/en/latest/sources/user/Troubleshooting.html

sudo docker exec timesketch-web pinfo.py /usr/share/timesketch/upload/23b8bc9904be40a09e3537d7eecea7b1
Traceback (most recent call last):
File "/usr/bin/pinfo.py", line 85, in
if not Main():
File "/usr/bin/pinfo.py", line 62, in Main
tool.PrintStorageInformation()
File "/usr/lib/python3/dist-packages/plaso/cli/pinfo_tool.py", line 1673, in PrintStorageInformation
storage_reader = self._GetStorageReader(self._storage_file_path)
File "/usr/lib/python3/dist-packages/plaso/cli/pinfo_tool.py", line 587, in _GetStorageReader
storage_factory.StorageFactory.CreateStorageReaderForFile(path))
File "/usr/lib/python3/dist-packages/plaso/storage/factory.py", line 49, in CreateStorageReaderForFile
return sqlite_reader.SQLiteStorageFileReader(path)
File "/usr/lib/python3/dist-packages/plaso/storage/sqlite/reader.py", line 20, in init
self._store.Open(path=path)
File "/usr/lib/python3/dist-packages/acstore/sqlite_store.py", line 999, in Open
self._ReadAndCheckStorageMetadata(check_readable_only=True)
File "/usr/lib/python3/dist-packages/plaso/storage/sqlite/sqlite_file.py", line 231, in _ReadAndCheckStorageMetadata
self._CheckStorageMetadata(
File "/usr/lib/python3/dist-packages/plaso/storage/sqlite/sqlite_file.py", line 53, in _CheckStorageMetadata
super(SQLiteStorageFile, self)._CheckStorageMetadata(
File "/usr/lib/python3/dist-packages/acstore/sqlite_store.py", line 199, in _CheckStorageMetadata
raise IOError((
OSError: Format version: 20230327 is too new and not yet supported, minimum supported version: 20230107.

[jaegeral]

Yes so you have 20230311 in your Timesketch installation, yet the file was created with 20230327 which is newer then what you have in Timesketch, thus is can not read it.
So to get it ingested you need to update Timesketch / Plaso in your Timesketch container.

[vxsh4d0w]

Well, I should check the entire configuration. This docker comes from this repo https://github.com/blueteam0ps/AllthingsTimesketch. (https://github.com/blueteam0ps/AllthingsTimesketch/blob/master/tsplaso_docker_install.sh).

[vxsh4d0w]

So, I tried to update manually Plaso but the old version is automatically provided with Timesketck image downloaded from the original repo (us-docker.pkg.dev/osdfir-registry/timesketch/timesketch). So I suppose that the image should be updated including last version of Plaso. At the moment I solved converting everything in csv format using psort. I tried to update Plaso within the container but I got the same issue, even if the command log2timeline.py -V returned

plaso - log2timeline version 20230717

[msktyshha]

The Log2Timeline.py utility is already included in the Timesketch Docker image. To ensure compatibility, you should use this preinstalled version to process your disk images or other files into .plaso format. Perform this conversion from within the Timesketch container itself, as this ensures version compatibility with Timesketch. I hope this clarifies things.