Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

High memory consumption at restore #3460

Open
arouene opened this issue Nov 21, 2023 · 1 comment · May be fixed by #3465
Open

High memory consumption at restore #3460

arouene opened this issue Nov 21, 2023 · 1 comment · May be fixed by #3465

Comments

@arouene
Copy link
Contributor

arouene commented Nov 21, 2023

Hello,

When using kopia restore we had up to about 20 GB of used memory, which prevented us to restore.

Here is a capture of the heap allocation while restoring (from a subsequent restoration which is taking a bit less memory).

Screenshot from 2023-11-21 11-15-48

About the process of restoration: We are migrating the last snapshot on the disk, and then we restore that snapshot.

Why so much memory is used while restoring ? What can we do about it ?
@arouene
Copy link
Contributor Author

arouene commented Nov 23, 2023

In method copyDirectoryContent. (

kopia/snapshot/restore/restore.go

Line 265 in f3a621d

func (c *copier) copyDirectoryContent(ctx context.Context, d fs.Directory, targetPath string, currentdepth, maxdepth int32, onCompletion parallelwork.CallbackFunc) error {
)

Would you see any inconvenience in performing a breadth-first processing rather than stacking all the entries?

		if e.IsDir() {
			c.stats.EnqueuedDirCount.Add(1)
			// enqueue directories first, so that we quickly determine the total number and size of items.
			c.q.EnqueueBack(ctx, func() error {
				return c.copyEntry(ctx, e, path.Join(targetPath, e.Name()), currentdepth, maxdepth, onItemCompletion)
			})
		} else {
			if isSymlink(e) {
				c.stats.EnqueuedSymlinkCount.Add(1)
			} else {
				c.stats.EnqueuedFileCount.Add(1)
			}

			c.stats.EnqueuedTotalFileSize.Add(e.Size())

			c.q.EnqueueFront(ctx, func() error {
				return c.copyEntry(ctx, e, path.Join(targetPath, e.Name()), currentdepth, maxdepth, onItemCompletion)
			})

By inverting EnqueueBack and EnqueueFront ?

In my tests, doing that, I use "only" 1.5GB- 2GB of memory instead of 10GB.

@arouene arouene linked a pull request Nov 23, 2023 that will close this issue
Open
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

fix(snapshots): optimise memory consumption for restores arouene/kopia
1 participant
@arouene