Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Mail indexing optimization: loading mails & data across multiple days #8599

Open
7 tasks
charlag opened this issue Feb 24, 2025 · 0 comments · May be fixed by #8622
Open
7 tasks

Mail indexing optimization: loading mails & data across multiple days #8599

charlag opened this issue Feb 24, 2025 · 0 comments · May be fixed by #8622
Assignees
@paw-hub @BijinDev
Labels
performance performance related issues

Comments

@charlag
Copy link
Contributor

charlag commented Feb 24, 2025 •
edited by paw-hub
Loading

Mail indexing is loading data day-by-day. In #8565 we changed to load MailSetEntrys and load them always in full chunks.

Unfortunately loading the actual data (Mails, MailDetails, Files) is still done day-by-day. It would be more optimal to collect a certain number of MailSetEntrys and then execute the download in bigger chunks.

Note: there's a similar optimization where we don't write emails to IndexedDB until we've collected a certain number of them.

Test notes

  • Delete all credentials
  • Log in as [email protected] and wait for it to do the initial synchronization.
  • Open up developer tools.
  • Search for "[email protected]", setting the search range to the year 2000, but don't start indexing just yet.
  • Clear the network tab.
  • Now start indexing.
  • Verify it only takes a couple of minutes to do this, that there are only ~233 requests, and that the oldest mail is from 2018
@charlag charlag added bug broken functionality, usability problems, unexpected errors performance performance related issues and removed bug broken functionality, usability problems, unexpected errors labels Feb 24, 2025
@charlag charlag mentioned this issue Feb 24, 2025
Open
@paw-hub paw-hub self-assigned this Feb 25, 2025
paw-hub added a commit that referenced this issue Feb 27, 2025
@paw-hub
Process indexed mails once we've reached some amount
7c13346 
Instead of processing day-by-day, we can process after we've reached a
certain amount. We should still iterate by day, but we instead index
multiple days at once.

This reduces the number of requests by over 80% on some mailboxes.

Closes #8599
paw-hub added a commit that referenced this issue Feb 27, 2025
@paw-hub
Process indexed mails once we've reached some amount
3aa9130 
Instead of indexing day-by-day, we can process after we've reached some
threshold. We should still iterate by day, but we should index multiple
days at once.

This reduces the number of requests by over 80% on some mailboxes.

Closes #8599
paw-hub added a commit that referenced this issue Feb 27, 2025
@paw-hub
Process indexed mails once we've reached some amount
8eb68bf 
Instead of indexing day-by-day, we can process after we've reached some
threshold. We should still iterate by day, but we should index multiple
days at once.

This reduces the number of requests by over 80% on some mailboxes, and
it may cut loading time in time in such cases.

Closes #8599
paw-hub added a commit that referenced this issue Feb 27, 2025
@paw-hub
Process indexed mails once we've reached some amount
80377fc 
Instead of indexing day-by-day, we can process after we've reached some
threshold. We should still iterate by day, but we should index multiple
days at once.

This reduces the number of requests by over 80% on some mailboxes, and
it may cut loading time in half in such cases.

Closes #8599
@paw-hub paw-hub linked a pull request Feb 27, 2025 that will close this issue
Open
paw-hub added a commit that referenced this issue Feb 27, 2025
@paw-hub
Process indexed mails once we've reached some amount
2c0dc11 
Instead of indexing day-by-day, we can process after we've reached some
threshold. We should still iterate by day, but we should index multiple
days at once.

This reduces the number of requests by over 80% on some mailboxes, and
it may cut loading time in half in such cases.

Closes #8599
paw-hub added a commit that referenced this issue Feb 27, 2025
@paw-hub
Process indexed mails once we've reached some amount
a965a1d 
Instead of indexing day-by-day, we can process after we've reached some
threshold. We should still iterate by day, but we should index multiple
days at once.

This reduces the number of requests by over 80% on some mailboxes, and
it may cut loading time in half in such cases.

Closes #8599
paw-hub added a commit that referenced this issue Feb 27, 2025
@paw-hub @BijinDev
Process indexed mails once we've reached some amount
8ddee04 
Instead of indexing day-by-day, we can process after we've reached some
threshold. We should still iterate by day, but we should index multiple
days at once.

This reduces the number of requests by over 80% on some mailboxes, and
it may cut loading time in half in such cases.

Closes #8599

Co-authored-by: bir <[email protected]>
paw-hub added a commit that referenced this issue Feb 27, 2025
@paw-hub @BijinDev
Process indexed mails once we've reached some amount
e79855e 
Instead of indexing day-by-day, we can process after we've reached some
threshold. We should still iterate by day, but we should index multiple
days at once.

This reduces the number of requests by over 80% on some mailboxes, and
it may cut loading time in half in such cases.

Closes #8599

Co-authored-by: bir <[email protected]>
@paw-hub paw-hub linked a pull request Feb 27, 2025 that will close this issue
Optimize mail loading #8622
Open
paw-hub added a commit that referenced this issue Feb 27, 2025
@paw-hub @BijinDev
Process indexed mails once we've reached some amount
670e3b6 
Instead of indexing day-by-day, we can process after we've reached some
threshold. We should still iterate by day, but we should index multiple
days at once.

This reduces the number of requests by over 80% on some mailboxes, and
it may cut loading time in half in such cases.

Closes #8599

Co-authored-by: bir <[email protected]>
paw-hub added a commit that referenced this issue Feb 27, 2025
@paw-hub @BijinDev
Process indexed mails once we've reached some amount
d13bcb1 
Instead of indexing day-by-day, we can process after we've reached some
threshold. We should still iterate by day, but we should index multiple
days at once.

This reduces the number of requests by over 80% on some mailboxes, and
it may cut loading time in half in such cases.

Closes #8599

Co-authored-by: bir <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@paw-hub paw-hub

@BijinDev BijinDev

Labels
performance performance related issues
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Optimize mail loading tutao/tutanota
3 participants
@charlag @paw-hub @BijinDev