Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Timesketch auto-renames timeline uploads with the same timeline name #3052

Open
puffyCid opened this issue Mar 22, 2024 · 7 comments
Open

Timesketch auto-renames timeline uploads with the same timeline name #3052

puffyCid opened this issue Mar 22, 2024 · 7 comments
Labels

Comments

@puffyCid
Copy link

puffyCid commented Mar 22, 2024

Describe the bug
When uploading mutliple timelines with the same name to a sketch, Timesketch renames the timeline name by appending 4 random characters.
When uploading timelines with the same name, i was assuming the additional uploads with just get added to the timeline data source instead of a new name

To Reproduce
This seems to happen mainly when using async uploads.
However, i was able to trigger using sync uploads by running the upload multiple times
I was able to trigger it using the python code below (replacing with Timesketch instance, and with home directory
I also had to spawn it several times to "mimic" async a bit

from timesketch_api_client import client as timesketch_client
from timesketch_import_client import importer
import os


def list_files(path, client, sketch):
    files = os.scandir(path)

    for file in files:
        entry = {"message": file.name, "datetime":"1970-01-01T00:00:00.000Z", "timestamp_desc": "test"}
        with importer.ImportStreamer() as streamer:
            streamer.set_sketch(sketch)
            streamer.set_timeline_name('uploadA')
            streamer.add_dict(entry)

        if file.is_dir():
            print(path+"/"+file.name)
            list_files(path+"/"+file.name, client, sketch)

        


def main():
    client = timesketch_client.TimesketchApi(host_uri='<IP>', username='sketchy', password='password')
    sketch = client.get_sketch(6)
    start = "<home path>"
    entries = []
    list_files(start, client, sketch)


if __name__ == "__main__":
    main()

its alot easier to trigger with async code is seems

Expected behavior
Timesketch continues to upload data to the same Timeline name and increment the data source

Screenshots
Image below is from the python code. I had to spawn multiple instances to trigger the rename
python

Image below is from async code i was using (mixture of TypeScript and Rust)
async

Desktop (please complete the following information):
Running Timesketch on Ubuntu 22.04 VM

Additional context
I brought this up in Timesketch Slack channel, and it was mentioned to open a Github issue.

Could this possibly happen because mutliple uploads are being submitted at once (or too quickly) and there is some kind of brief lock on the timeline name and when the second upload occurs the lock triggers timesketch/opensearch to rename it?

Let me know if additional info is required
Thanks

@puffyCid puffyCid added the Bug label Mar 22, 2024
@puffyCid puffyCid changed the title Timesketch auto-renames timeline uploads Timesketch auto-renames timeline uploads with the same timeline name Mar 22, 2024
@justzh

This comment was marked as spam.

@justzh

This comment was marked as spam.

@justzh

This comment was marked as spam.

@justzh

This comment was marked as spam.

@justzh

This comment was marked as spam.

@justzh

This comment was marked as spam.

@mbartle-sf
Copy link

I'm having this issue, too. It occurs whenever the client makes a request to upload data to a sketch while the search index for the sketch is in use. This makes it common when uploading asynchronously, but possible even when uploading synchronously (as long as the upload rate is faster than the OpenSearch indexing rate).

If the index for a given timeline is in use, Timesketch will create a new index and timeline for the data. The new timeline is given the "original name plus 5 random characters" name. If the user uploads more data and the original search index is still in use, but the secondary index is not, Timesketch is able to find and use the secondary index, but still always creates a new timeline.

On my team, we create our own JSONL timelines and send them to Timesketch in batches, of which there can be thousands. This bug leads us to have several hundred timelines, even when there are only 2-3 indexes on the sketch.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

No branches or pull requests

3 participants