Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Restic backup on local machine hangs while scanning files #3156

Closed
mqp opened this issue Dec 9, 2020 · 4 comments
Closed

Restic backup on local machine hangs while scanning files #3156

mqp opened this issue Dec 9, 2020 · 4 comments
Labels
state: need feedback waiting for feedback, e.g. from the submitter

Comments

@mqp
Copy link

mqp commented Dec 9, 2020

Output of restic version

restic 0.11.0 compiled with go1.15.3 on linux/amd64

How did you run restic exactly?

restic -r /mnt/backups --verbose --verbose backup /mnt/ext

The first time I did this, restic ran for about 20 minutes, scanned a few dozen gigabytes, and then hung indefinitely, displaying the ETA increasing and the same two MP3s it was scanning as the last output, while making no progress. I attempted to kill it, but it didn't respond to even a kill -9. The only way I could get it to die is by umount -f /mnt/ext on the disk it was backing up (the first time failed with "target is busy"; the second time worked.) I then tried it again and it scanned for about 40 minutes before hanging with the same symptoms. htop showed that lots of memory was available and restic was no longer using any CPU. Plenty of disk is available.

The second time, I sent a SIGABRT and retrieved a stack trace, which you can see at the bottom of this issue.

What backend/server/service did you use to store the repository?

My machine is running Arch Linux with the latest version of restic from the AUR. The path it was backing up, /mnt/ext, is a exFAT filesystem mounted from a local USB disk. The repository target, /mnt/backups, is an ext4 filesystem mounted from another local USB disk. Its cache is on a third local disk.

Expected behavior

Restic creates a new snapshot in the /mnt/backups repository.

Actual behavior

Restic hangs apparently indefinitely.

Steps to reproduce the behavior

So far this didn't happen when I was creating the repository (last week) but it's happened every time I tried to add a snapshot.

Do you have any idea what may have caused this?

Nope. Smells like some kind of deadlock.

Do you have an idea how to solve the issue?

Nope.

Did restic help you today? Did it make you happy in any way?

If I can get restic working I will be very happy to have an incremental backup tool I can rely on!

Output after SIGABRT:

goroutine 0 [idle]:
runtime.futex(0x555d7c201a68, 0x80, 0x0, 0x0, 0x7ffd00000000, 0x555d7ab6dabe, 0x41a3c, 0x20cbc5f3, 0x7ffd5564c458, 0x555d7ab0bec5, ...)
        runtime/sys_linux_amd64.s:587 +0x21
runtime.futexsleep(0x555d7c201a68, 0x555d00000000, 0xffffffffffffffff)
        runtime/os_linux.go:45 +0x46
runtime.notesleep(0x555d7c201a68)
        runtime/lock_futex.go:159 +0xa5
runtime.stopm()
        runtime/proc.go:1910 +0xc8
runtime.findrunnable(0xc000042800, 0x0)
        runtime/proc.go:2471 +0xa7f
runtime.schedule()
        runtime/proc.go:2669 +0x2d7
runtime.park_m(0xc000001c80)
        runtime/proc.go:2837 +0xab
runtime.mcall(0x0)
        runtime/asm_amd64.s:318 +0x53

goroutine 1 [syscall, 68 minutes]:
syscall.Syscall6(0x101, 0xffffffffffffff9c, 0xc0010636d0, 0xa0000, 0x0, 0x0, 0x0, 0x0, 0xed394920a, 0x555d7c200880)
        syscall/asm_linux_amd64.s:41 +0x5
syscall.openat(0xffffffffffffff9c, 0xc0010635e0, 0x46, 0xa0000, 0x0, 0x0, 0x0, 0x1000)
        syscall/zsyscall_linux_amd64.go:68 +0xc8
syscall.Open(...)
        syscall/syscall_linux.go:152
os.openFileNolog(0xc0010635e0, 0x46, 0x20000, 0x555d00000000, 0x0, 0xec58f81ec, 0x555d7c200880)
        os/file_unix.go:200 +0x91
os.OpenFile(0xc0010635e0, 0x46, 0x20000, 0x0, 0x0, 0x1000, 0x2400)
        os/file.go:327 +0x65
github.com/restic/restic/internal/fs.Local.OpenFile(0xc0010635e0, 0x46, 0x20000, 0xe00000000, 0x555d7c200880, 0x0, 0xec58f81ec, 0x555d7c200880)
        github.com/restic/restic/internal/fs/fs_local.go:36 +0x4d
github.com/restic/restic/internal/archiver.(*Archiver).Save(0xc0000dc750, 0x555d7bc7e8e0, 0xc000084400, 0xc001063630, 0x46, 0xc0010635e0, 0x46, 0xc00172db80, 0xc001063630, 0x46, ...)
        github.com/restic/restic/internal/archiver/archiver.go:402 +0x419
github.com/restic/restic/internal/archiver.(*Archiver).SaveDir(0xc0000dc750, 0x555d7bc7e8e0, 0xc000084400, 0xc0015d37a0, 0x30, 0x555d7bc845a0, 0xc0010e2410, 0xc0015d3770, 0x30, 0xc0010c1c60, ...)
        github.com/restic/restic/internal/archiver/archiver.go:247 +0x3ba
github.com/restic/restic/internal/archiver.(*Archiver).Save(0xc0000dc750, 0x555d7bc7e8e0, 0xc000084400, 0xc0015d37a0, 0x30, 0xc0015d3770, 0x30, 0xc0014efe00, 0xc0015d37a0, 0x30, ...)
        github.com/restic/restic/internal/archiver/archiver.go:453 +0x110b
github.com/restic/restic/internal/archiver.(*Archiver).SaveDir(0xc0000dc750, 0x555d7bc7e8e0, 0xc000084400, 0xc0013998a0, 0x1b, 0x555d7bc845a0, 0xc000fe7790, 0xc001399840, 0x1b, 0xc0010832a0, ...)
        github.com/restic/restic/internal/archiver/archiver.go:247 +0x3ba
github.com/restic/restic/internal/archiver.(*Archiver).Save(0xc0000dc750, 0x555d7bc7e8e0, 0xc000084400, 0xc0013998a0, 0x1b, 0xc001399840, 0x1b, 0xc00171ab40, 0xc0013998a0, 0x1b, ...)
        github.com/restic/restic/internal/archiver/archiver.go:453 +0x110b
github.com/restic/restic/internal/archiver.(*Archiver).SaveDir(0xc0000dc750, 0x555d7bc7e8e0, 0xc000084400, 0xc0016e9a70, 0xe, 0x555d7bc845a0, 0xc00170cf70, 0xc0016e9a60, 0xe, 0xc001715940, ...)
        github.com/restic/restic/internal/archiver/archiver.go:247 +0x3ba
github.com/restic/restic/internal/archiver.(*Archiver).Save(0xc0000dc750, 0x555d7bc7e8e0, 0xc000084400, 0xc0016e9a70, 0xe, 0xc0016e9a60, 0xe, 0xc0000c4f00, 0xc0016e9a70, 0xe, ...)
        github.com/restic/restic/internal/archiver/archiver.go:453 +0x110b
github.com/restic/restic/internal/archiver.(*Archiver).SaveDir(0xc0000dc750, 0x555d7bc7e8e0, 0xc000084400, 0xc0002fb088, 0x8, 0x555d7bc845a0, 0xc0002f6000, 0x7ffd5564e536, 0x8, 0xc0000b6900, ...)
        github.com/restic/restic/internal/archiver/archiver.go:247 +0x3ba
github.com/restic/restic/internal/archiver.(*Archiver).Save(0xc0000dc750, 0x555d7bc7e8e0, 0xc000084400, 0xc0002fb088, 0x8, 0x7ffd5564e536, 0x8, 0xc0000c4640, 0xc0002fb088, 0x8, ...)
        github.com/restic/restic/internal/archiver/archiver.go:453 +0x110b
github.com/restic/restic/internal/archiver.(*Archiver).SaveTree(0xc0000dc750, 0x555d7bc7e8e0, 0xc000084400, 0xc0002fb080, 0x4, 0xc0001265c0, 0xc0000b6740, 0x555d7bc26060, 0xc0001ca820, 0x555d7ae51173)
        github.com/restic/restic/internal/archiver/archiver.go:563 +0x4d4
github.com/restic/restic/internal/archiver.(*Archiver).SaveTree(0xc0000dc750, 0x555d7bc7e8e0, 0xc000084400, 0x555d7b44600e, 0x1, 0xc0000843c0, 0xc0000b6260, 0xc0000b6260, 0x555d7ab690bb, 0xee9cb0ee4942)
        github.com/restic/restic/internal/archiver/archiver.go:594 +0xc28
github.com/restic/restic/internal/archiver.(*Archiver).Snapshot.func1(0xc0000dc750, 0x555d7bc7e8e0, 0xc000084400, 0xc0000843c0, 0x0, 0x0, 0x0, 0xc0000365fa, 0x6, 0x0, ...)
        github.com/restic/restic/internal/archiver/archiver.go:790 +0xe5
github.com/restic/restic/internal/archiver.(*Archiver).Snapshot(0xc0000dc750, 0x555d7bc7e8e0, 0xc000126580, 0xc00003f220, 0x1, 0x1, 0x0, 0x0, 0x0, 0xc0000365fa, ...)
        github.com/restic/restic/internal/archiver/archiver.go:800 +0x238
main.runBackup(0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
        github.com/restic/restic/cmd/restic/cmd_backup.go:631 +0x1085
main.glob..func2(0x555d7c1f0de0, 0xc000118000, 0x1, 0x5, 0x0, 0x0)
        github.com/restic/restic/cmd/restic/cmd_backup.go:71 +0x1ba
github.com/spf13/cobra.(*Command).execute(0x555d7c1f0de0, 0xc00009a980, 0x5, 0x8, 0x555d7c1f0de0, 0xc00009a980)
        github.com/spf13/[email protected]/command.go:826 +0x47c
github.com/spf13/cobra.(*Command).ExecuteC(0x555d7c1f3d60, 0xc00003f190, 0x4, 0x10)
        github.com/spf13/[email protected]/command.go:914 +0x30b
github.com/spf13/cobra.(*Command).Execute(...)
        github.com/spf13/[email protected]/command.go:864
main.main()
        github.com/restic/restic/cmd/restic/main.go:98 +0x45

goroutine 4 [chan receive, 109 minutes]:
github.com/restic/restic/internal/restic.init.0.func1.1()
        github.com/restic/restic/internal/restic/lock.go:258 +0xb5
created by github.com/restic/restic/internal/restic.init.0.func1
        github.com/restic/restic/internal/restic/lock.go:255 +0x37

goroutine 20 [syscall, 109 minutes]:
os/signal.signal_recv(0x0)
        runtime/sigqueue.go:147 +0x9e
os/signal.loop()
        os/signal/signal_unix.go:23 +0x25
created by os/signal.Notify.func1.1
        os/signal/signal.go:150 +0x46

goroutine 21 [chan receive, 109 minutes]:
github.com/restic/restic/internal/restic.init.2.func1(0xc0001022a0)
        github.com/restic/restic/internal/restic/progress_unix.go:17 +0x65
created by github.com/restic/restic/internal/restic.init.2
        github.com/restic/restic/internal/restic/progress_unix.go:16 +0xad

goroutine 23 [select]:
go.opencensus.io/stats/view.(*worker).start(0xc00009a700)
        [email protected]/stats/view/worker.go:276 +0x105
created by go.opencensus.io/stats/view.init.0
        [email protected]/stats/view/worker.go:34 +0x6a

goroutine 26 [chan receive, 109 minutes]:
main.CleanupHandler(0xc000086900)
        github.com/restic/restic/cmd/restic/cleanup.go:59 +0x4e
created by main.init.0
        github.com/restic/restic/cmd/restic/cleanup.go:21 +0x69

goroutine 6 [select]:
github.com/restic/restic/internal/ui/termstatus.(*Terminal).run(0xc000102cc0, 0x555d7bc7e8e0, 0xc000084200)
        github.com/restic/restic/internal/ui/termstatus/status.go:97 +0x167
github.com/restic/restic/internal/ui/termstatus.(*Terminal).Run(0xc000102cc0, 0x555d7bc7e8e0, 0xc000084200)
        github.com/restic/restic/internal/ui/termstatus/status.go:86 +0x72
main.glob..func2.1(0x0, 0x0)
        github.com/restic/restic/cmd/restic/cmd_backup.go:69 +0x6f
gopkg.in/tomb%2ev2.(*Tomb).run(0xc0001180a0, 0xc00000e6c0)
        gopkg.in/[email protected]/tomb.go:163 +0x3a
created by gopkg.in/tomb%2ev2.(*Tomb).Go
        gopkg.in/[email protected]/tomb.go:159 +0xbc

goroutine 7 [select]:
github.com/restic/restic/internal/ui.(*Backup).Run(0xc00038a1c0, 0x555d7bc7e8e0, 0xc000126000, 0x0, 0x0)
        github.com/restic/restic/internal/ui/backup.go:98 +0x378
main.runBackup.func2(0x0, 0x0)
        github.com/restic/restic/cmd/restic/cmd_backup.go:499 +0x86
gopkg.in/tomb%2ev2.(*Tomb).run(0xc0001180f0, 0xc000233140)
        gopkg.in/[email protected]/tomb.go:163 +0x3a
created by gopkg.in/tomb%2ev2.(*Tomb).Go
        gopkg.in/[email protected]/tomb.go:159 +0xbc

goroutine 8 [select, 5 minutes]:
main.refreshLocks(0x555d7c200300, 0xc00008aea0)
        github.com/restic/restic/cmd/restic/lock.go:70 +0x1c5
created by main.lockRepository
        github.com/restic/restic/cmd/restic/lock.go:47 +0x275

goroutine 34 [select, 68 minutes]:
github.com/restic/restic/internal/archiver.(*BlobSaver).worker(0xc000371020, 0x555d7bc7e8e0, 0xc000084440, 0xc000104900, 0xc000084440, 0x0)
        github.com/restic/restic/internal/archiver/blob_saver.go:117 +0xfc
github.com/restic/restic/internal/archiver.NewBlobSaver.func1(0x0, 0x0)
        github.com/restic/restic/internal/archiver/blob_saver.go:34 +0x85
gopkg.in/tomb%2ev2.(*Tomb).run(0xc000118280, 0xc000518690)
        gopkg.in/[email protected]/tomb.go:163 +0x3a
created by gopkg.in/tomb%2ev2.(*Tomb).Go
        gopkg.in/[email protected]/tomb.go:159 +0xbc

goroutine 35 [select, 68 minutes]:
github.com/restic/restic/internal/archiver.(*BlobSaver).worker(0xc000371020, 0x555d7bc7e8e0, 0xc000084440, 0xc000104900, 0xc000084440, 0x0)
        github.com/restic/restic/internal/archiver/blob_saver.go:117 +0xfc
github.com/restic/restic/internal/archiver.NewBlobSaver.func1(0x0, 0x0)
        github.com/restic/restic/internal/archiver/blob_saver.go:34 +0x85
gopkg.in/tomb%2ev2.(*Tomb).run(0xc000118280, 0xc0005186c0)
        gopkg.in/[email protected]/tomb.go:163 +0x3a
created by gopkg.in/tomb%2ev2.(*Tomb).Go
        gopkg.in/[email protected]/tomb.go:159 +0xbc

goroutine 36 [chan receive, 109 minutes]:
github.com/restic/restic/internal/archiver.NewBufferPool.func1(0x555d7bc7e8e0, 0xc000084400, 0xc0005186f0)
        github.com/restic/restic/internal/archiver/buffer.go:40 +0x4a
created by github.com/restic/restic/internal/archiver.NewBufferPool
        github.com/restic/restic/internal/archiver/buffer.go:39 +0xac

goroutine 37 [syscall, 68 minutes]:
syscall.Syscall(0x0, 0x6, 0xc000ec4000, 0x80000, 0x19857a763ca85340, 0x19a776d174f9cc7b, 0x19c16338ac0b6d36)
        syscall/asm_linux_amd64.s:18 +0x5
syscall.read(0x6, 0xc000ec4000, 0x80000, 0x80000, 0x1af1cc766b9e675e, 0x1b0f8e38d0a64327, 0x1b2d829f98f7dc1c)
        syscall/zsyscall_linux_amd64.go:686 +0x5c
syscall.Read(...)
        syscall/syscall_unix.go:187
internal/poll.(*FD).Read.func1(0x72, 0x0, 0x0)
        internal/poll/fd_unix.go:155 +0x4e
internal/poll.ignoringEINTR(0xc0000ed898, 0x72, 0xc0000ed801, 0x0)
        internal/poll/fd_unix.go:567 +0x29
internal/poll.(*FD).Read(0xc0017b9680, 0xc000ec4000, 0x80000, 0x80000, 0x0, 0x0, 0x0)
        internal/poll/fd_unix.go:155 +0x13e
os.(*File).read(...)
        os/file_posix.go:31
os.(*File).Read(0xc0000106d0, 0xc000ec4000, 0x80000, 0x80000, 0x2, 0x2, 0x81)
        os/file.go:116 +0x73
io.ReadAtLeast(0x555d7bc688e0, 0xc0000106d0, 0xc000ec4000, 0x80000, 0x80000, 0x80000, 0x555d7ab14d5b, 0xc0000edb10, 0x555d7ab14ffb)
        io/io.go:314 +0x89
io.ReadFull(...)
        io/io.go:333
github.com/restic/chunker.(*Chunker).Next(0xc000351300, 0xc002478000, 0xd00d, 0x800000, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
        github.com/restic/[email protected]/chunker.go:238 +0x954
github.com/restic/restic/internal/archiver.(*FileSaver).saveFile(0xc000518720, 0x555d7bc7e8e0, 0xc000084440, 0xc000351300, 0xc0010c6a40, 0x3d, 0x555d7bc879c0, 0xc0000106d0, 0x555d7bc845a0, 0xc0010e2a90, ...)
        github.com/restic/restic/internal/archiver/file_saver.go:167 +0x39e
github.com/restic/restic/internal/archiver.(*FileSaver).worker(0xc000518720, 0x555d7bc7e8e0, 0xc000084440, 0xc000104960)
        github.com/restic/restic/internal/archiver/file_saver.go:235 +0x305
github.com/restic/restic/internal/archiver.NewFileSaver.func2(0x0, 0x0)
        github.com/restic/restic/internal/archiver/file_saver.go:88 +0x7c
gopkg.in/tomb%2ev2.(*Tomb).run(0xc000118280, 0xc000518750)
        gopkg.in/[email protected]/tomb.go:163 +0x3a
created by gopkg.in/tomb%2ev2.(*Tomb).Go
        gopkg.in/[email protected]/tomb.go:159 +0xbc

goroutine 38 [syscall, 68 minutes]:
syscall.Syscall(0x0, 0x8, 0xc000f44000, 0x80000, 0x0, 0x0, 0x0)
        syscall/asm_linux_amd64.s:18 +0x5
syscall.read(0x8, 0xc000f44000, 0x80000, 0x80000, 0x0, 0x0, 0x0)
        syscall/zsyscall_linux_amd64.go:686 +0x5c
syscall.Read(...)
        syscall/syscall_unix.go:187
internal/poll.(*FD).Read.func1(0x72, 0x0, 0x0)
        internal/poll/fd_unix.go:155 +0x4e
internal/poll.ignoringEINTR(0xc00188f898, 0x72, 0x1, 0x0)
        internal/poll/fd_unix.go:567 +0x29
internal/poll.(*FD).Read(0xc0017b9740, 0xc000f44000, 0x80000, 0x80000, 0x0, 0x0, 0x0)
        internal/poll/fd_unix.go:155 +0x13e
os.(*File).read(...)
        os/file_posix.go:31
os.(*File).Read(0xc0000106e0, 0xc000f44000, 0x80000, 0x80000, 0x0, 0x0, 0x0)
        os/file.go:116 +0x73
io.ReadAtLeast(0x555d7bc688e0, 0xc0000106e0, 0xc000f44000, 0x80000, 0x80000, 0x80000, 0x555d7ab14d5b, 0xc00188fb10, 0x555d7ab14ffb)
        io/io.go:314 +0x89
io.ReadFull(...)
        io/io.go:333
github.com/restic/chunker.(*Chunker).Next(0xc000352600, 0xc001c78000, 0x0, 0x800000, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
        github.com/restic/[email protected]/chunker.go:238 +0x954
github.com/restic/restic/internal/archiver.(*FileSaver).saveFile(0xc000518720, 0x555d7bc7e8e0, 0xc000084440, 0xc000352600, 0xc0010ee0a0, 0x45, 0x555d7bc879c0, 0xc0000106e0, 0x555d7bc845a0, 0xc0010e2c30, ...)
        github.com/restic/restic/internal/archiver/file_saver.go:167 +0x39e
github.com/restic/restic/internal/archiver.(*FileSaver).worker(0xc000518720, 0x555d7bc7e8e0, 0xc000084440, 0xc000104960)
        github.com/restic/restic/internal/archiver/file_saver.go:235 +0x305
github.com/restic/restic/internal/archiver.NewFileSaver.func2(0x0, 0x100000000000000)
        github.com/restic/restic/internal/archiver/file_saver.go:88 +0x7c
gopkg.in/tomb%2ev2.(*Tomb).run(0xc000118280, 0xc000518780)
        gopkg.in/[email protected]/tomb.go:163 +0x3a
created by gopkg.in/tomb%2ev2.(*Tomb).Go
        gopkg.in/[email protected]/tomb.go:159 +0xbc

goroutine 39 [select, 68 minutes]:
github.com/restic/restic/internal/archiver.(*TreeSaver).worker(0xc000371060, 0x555d7bc7e8e0, 0xc000084440, 0xc0001049c0, 0xc000084440, 0x0)
        github.com/restic/restic/internal/archiver/tree_saver.go:148 +0x1a5
github.com/restic/restic/internal/archiver.NewTreeSaver.func1(0x0, 0x100000000000000)
        github.com/restic/restic/internal/archiver/tree_saver.go:61 +0x85
gopkg.in/tomb%2ev2.(*Tomb).run(0xc000118280, 0xc0005187b0)
        gopkg.in/[email protected]/tomb.go:163 +0x3a
created by gopkg.in/tomb%2ev2.(*Tomb).Go
        gopkg.in/[email protected]/tomb.go:159 +0xbc

goroutine 40 [select, 68 minutes]:
github.com/restic/restic/internal/archiver.(*TreeSaver).worker(0xc000371060, 0x555d7bc7e8e0, 0xc000084440, 0xc0001049c0, 0xc000084440, 0x0)
        github.com/restic/restic/internal/archiver/tree_saver.go:148 +0x1a5
github.com/restic/restic/internal/archiver.NewTreeSaver.func1(0x0, 0x100000000000000)
        github.com/restic/restic/internal/archiver/tree_saver.go:61 +0x85
gopkg.in/tomb%2ev2.(*Tomb).run(0xc000118280, 0xc0005187e0)
        gopkg.in/[email protected]/tomb.go:163 +0x3a
created by gopkg.in/tomb%2ev2.(*Tomb).Go
        gopkg.in/[email protected]/tomb.go:159 +0xbc

goroutine 41 [select, 68 minutes]:
github.com/restic/restic/internal/archiver.(*TreeSaver).worker(0xc000371060, 0x555d7bc7e8e0, 0xc000084440, 0xc0001049c0, 0xc000084440, 0xc000038200)
        github.com/restic/restic/internal/archiver/tree_saver.go:148 +0x1a5
github.com/restic/restic/internal/archiver.NewTreeSaver.func1(0x0, 0x100000000000000)
        github.com/restic/restic/internal/archiver/tree_saver.go:61 +0x85
gopkg.in/tomb%2ev2.(*Tomb).run(0xc000118280, 0xc000518810)
        gopkg.in/[email protected]/tomb.go:163 +0x3a
created by gopkg.in/tomb%2ev2.(*Tomb).Go
        gopkg.in/[email protected]/tomb.go:159 +0xbc

goroutine 42 [select, 68 minutes]:
github.com/restic/restic/internal/archiver.(*TreeSaver).worker(0xc000371060, 0x555d7bc7e8e0, 0xc000084440, 0xc0001049c0, 0xc000084440, 0x0)
        github.com/restic/restic/internal/archiver/tree_saver.go:148 +0x1a5
github.com/restic/restic/internal/archiver.NewTreeSaver.func1(0x0, 0x0)
        github.com/restic/restic/internal/archiver/tree_saver.go:61 +0x85
gopkg.in/tomb%2ev2.(*Tomb).run(0xc000118280, 0xc000518840)
        gopkg.in/[email protected]/tomb.go:163 +0x3a
created by gopkg.in/tomb%2ev2.(*Tomb).Go
        gopkg.in/[email protected]/tomb.go:159 +0xbc

goroutine 43 [select, 68 minutes]:
github.com/restic/restic/internal/archiver.(*TreeSaver).worker(0xc000371060, 0x555d7bc7e8e0, 0xc000084440, 0xc0001049c0, 0xc000084440, 0x0)
        github.com/restic/restic/internal/archiver/tree_saver.go:148 +0x1a5
github.com/restic/restic/internal/archiver.NewTreeSaver.func1(0x0, 0x0)
        github.com/restic/restic/internal/archiver/tree_saver.go:61 +0x85
gopkg.in/tomb%2ev2.(*Tomb).run(0xc000118280, 0xc000518870)
        gopkg.in/[email protected]/tomb.go:163 +0x3a
created by gopkg.in/tomb%2ev2.(*Tomb).Go
        gopkg.in/[email protected]/tomb.go:159 +0xbc

goroutine 44 [select, 68 minutes]:
github.com/restic/restic/internal/archiver.(*TreeSaver).worker(0xc000371060, 0x555d7bc7e8e0, 0xc000084440, 0xc0001049c0, 0xc000084440, 0x0)
        github.com/restic/restic/internal/archiver/tree_saver.go:148 +0x1a5
github.com/restic/restic/internal/archiver.NewTreeSaver.func1(0x0, 0x0)
        github.com/restic/restic/internal/archiver/tree_saver.go:61 +0x85
gopkg.in/tomb%2ev2.(*Tomb).run(0xc000118280, 0xc0005188a0)
        gopkg.in/[email protected]/tomb.go:163 +0x3a
created by gopkg.in/tomb%2ev2.(*Tomb).Go
        gopkg.in/[email protected]/tomb.go:159 +0xbc

goroutine 45 [select, 68 minutes]:
github.com/restic/restic/internal/archiver.(*TreeSaver).worker(0xc000371060, 0x555d7bc7e8e0, 0xc000084440, 0xc0001049c0, 0xc000084440, 0x555d7bc7e8e0)
        github.com/restic/restic/internal/archiver/tree_saver.go:148 +0x1a5
github.com/restic/restic/internal/archiver.NewTreeSaver.func1(0x0, 0x0)
        github.com/restic/restic/internal/archiver/tree_saver.go:61 +0x85
gopkg.in/tomb%2ev2.(*Tomb).run(0xc000118280, 0xc0005188d0)
        gopkg.in/[email protected]/tomb.go:163 +0x3a
created by gopkg.in/tomb%2ev2.(*Tomb).Go
        gopkg.in/[email protected]/tomb.go:159 +0xbc

goroutine 46 [select, 68 minutes]:
github.com/restic/restic/internal/archiver.(*TreeSaver).worker(0xc000371060, 0x555d7bc7e8e0, 0xc000084440, 0xc0001049c0, 0xc000084440, 0x0)
        github.com/restic/restic/internal/archiver/tree_saver.go:148 +0x1a5
github.com/restic/restic/internal/archiver.NewTreeSaver.func1(0x0, 0x0)
        github.com/restic/restic/internal/archiver/tree_saver.go:61 +0x85
gopkg.in/tomb%2ev2.(*Tomb).run(0xc000118280, 0xc000518900)
        gopkg.in/[email protected]/tomb.go:163 +0x3a
created by gopkg.in/tomb%2ev2.(*Tomb).Go
        gopkg.in/[email protected]/tomb.go:159 +0xbc

goroutine 47 [select, 68 minutes]:
github.com/restic/restic/internal/archiver.(*TreeSaver).worker(0xc000371060, 0x555d7bc7e8e0, 0xc000084440, 0xc0001049c0, 0xc000084440, 0x0)
        github.com/restic/restic/internal/archiver/tree_saver.go:148 +0x1a5
github.com/restic/restic/internal/archiver.NewTreeSaver.func1(0x0, 0x0)
        github.com/restic/restic/internal/archiver/tree_saver.go:61 +0x85
gopkg.in/tomb%2ev2.(*Tomb).run(0xc000118280, 0xc000518930)
        gopkg.in/[email protected]/tomb.go:163 +0x3a
created by gopkg.in/tomb%2ev2.(*Tomb).Go
        gopkg.in/[email protected]/tomb.go:159 +0xbc

goroutine 48 [select, 68 minutes]:
github.com/restic/restic/internal/archiver.(*TreeSaver).worker(0xc000371060, 0x555d7bc7e8e0, 0xc000084440, 0xc0001049c0, 0xc000084440, 0x0)
        github.com/restic/restic/internal/archiver/tree_saver.go:148 +0x1a5
github.com/restic/restic/internal/archiver.NewTreeSaver.func1(0x0, 0x0)
        github.com/restic/restic/internal/archiver/tree_saver.go:61 +0x85
gopkg.in/tomb%2ev2.(*Tomb).run(0xc000118280, 0xc000518960)
        gopkg.in/[email protected]/tomb.go:163 +0x3a
created by gopkg.in/tomb%2ev2.(*Tomb).Go
        gopkg.in/[email protected]/tomb.go:159 +0xbc

goroutine 49 [select, 68 minutes]:
github.com/restic/restic/internal/archiver.(*TreeSaver).worker(0xc000371060, 0x555d7bc7e8e0, 0xc000084440, 0xc0001049c0, 0xc000084440, 0x0)
        github.com/restic/restic/internal/archiver/tree_saver.go:148 +0x1a5
github.com/restic/restic/internal/archiver.NewTreeSaver.func1(0x0, 0x0)
        github.com/restic/restic/internal/archiver/tree_saver.go:61 +0x85
gopkg.in/tomb%2ev2.(*Tomb).run(0xc000118280, 0xc000518990)
        gopkg.in/[email protected]/tomb.go:163 +0x3a
created by gopkg.in/tomb%2ev2.(*Tomb).Go
        gopkg.in/[email protected]/tomb.go:159 +0xbc

goroutine 50 [select, 68 minutes]:
github.com/restic/restic/internal/archiver.(*TreeSaver).worker(0xc000371060, 0x555d7bc7e8e0, 0xc000084440, 0xc0001049c0, 0xc000084440, 0x0)
        github.com/restic/restic/internal/archiver/tree_saver.go:148 +0x1a5
github.com/restic/restic/internal/archiver.NewTreeSaver.func1(0x0, 0x0)
        github.com/restic/restic/internal/archiver/tree_saver.go:61 +0x85
gopkg.in/tomb%2ev2.(*Tomb).run(0xc000118280, 0xc0005189c0)
        gopkg.in/[email protected]/tomb.go:163 +0x3a
created by gopkg.in/tomb%2ev2.(*Tomb).Go
        gopkg.in/[email protected]/tomb.go:159 +0xbc

goroutine 51 [select, 68 minutes]:
github.com/restic/restic/internal/archiver.(*TreeSaver).worker(0xc000371060, 0x555d7bc7e8e0, 0xc000084440, 0xc0001049c0, 0xc000084440, 0x22a619f664bb4ffe)
        github.com/restic/restic/internal/archiver/tree_saver.go:148 +0x1a5
github.com/restic/restic/internal/archiver.NewTreeSaver.func1(0x640740c859974f14, 0x905bfc7a563f4096)
        github.com/restic/restic/internal/archiver/tree_saver.go:61 +0x85
gopkg.in/tomb%2ev2.(*Tomb).run(0xc000118280, 0xc0005189f0)
        gopkg.in/[email protected]/tomb.go:163 +0x3a
created by gopkg.in/tomb%2ev2.(*Tomb).Go
        gopkg.in/[email protected]/tomb.go:159 +0xbc

goroutine 52 [select, 68 minutes]:
github.com/restic/restic/internal/archiver.(*TreeSaver).worker(0xc000371060, 0x555d7bc7e8e0, 0xc000084440, 0xc0001049c0, 0xc000084440, 0xe3656639d1f7c0a8)
        github.com/restic/restic/internal/archiver/tree_saver.go:148 +0x1a5
github.com/restic/restic/internal/archiver.NewTreeSaver.func1(0x838ee2b84d3cd165, 0xa5f6c0be4e578d29)
        github.com/restic/restic/internal/archiver/tree_saver.go:61 +0x85
gopkg.in/tomb%2ev2.(*Tomb).run(0xc000118280, 0xc000518a20)
        gopkg.in/[email protected]/tomb.go:163 +0x3a
created by gopkg.in/tomb%2ev2.(*Tomb).Go
        gopkg.in/[email protected]/tomb.go:159 +0xbc

goroutine 53 [select, 68 minutes]:
github.com/restic/restic/internal/archiver.(*TreeSaver).worker(0xc000371060, 0x555d7bc7e8e0, 0xc000084440, 0xc0001049c0, 0xc000084440, 0xf30a193e336966ff)
        github.com/restic/restic/internal/archiver/tree_saver.go:148 +0x1a5
github.com/restic/restic/internal/archiver.NewTreeSaver.func1(0xcafde83c5a73926b, 0x7b57a6eddb970b4a)
        github.com/restic/restic/internal/archiver/tree_saver.go:61 +0x85
gopkg.in/tomb%2ev2.(*Tomb).run(0xc000118280, 0xc000518a50)
        gopkg.in/[email protected]/tomb.go:163 +0x3a
created by gopkg.in/tomb%2ev2.(*Tomb).Go
        gopkg.in/[email protected]/tomb.go:159 +0xbc

goroutine 54 [select, 68 minutes]:
github.com/restic/restic/internal/archiver.(*TreeSaver).worker(0xc000371060, 0x555d7bc7e8e0, 0xc000084440, 0xc0001049c0, 0xc000084440, 0xa771f5973c0f826e)
        github.com/restic/restic/internal/archiver/tree_saver.go:148 +0x1a5
github.com/restic/restic/internal/archiver.NewTreeSaver.func1(0xc6f7757f787f7623, 0xdb3824d50ac4d356)
        github.com/restic/restic/internal/archiver/tree_saver.go:61 +0x85
gopkg.in/tomb%2ev2.(*Tomb).run(0xc000118280, 0xc000518a80)
        gopkg.in/[email protected]/tomb.go:163 +0x3a
created by gopkg.in/tomb%2ev2.(*Tomb).Go
        gopkg.in/[email protected]/tomb.go:159 +0xbc

goroutine 55 [select, 68 minutes]:
github.com/restic/restic/internal/archiver.(*TreeSaver).worker(0xc000371060, 0x555d7bc7e8e0, 0xc000084440, 0xc0001049c0, 0xc000084440, 0x8be6d65b39863983)
        github.com/restic/restic/internal/archiver/tree_saver.go:148 +0x1a5
github.com/restic/restic/internal/archiver.NewTreeSaver.func1(0xe960bc0f0be5dc53, 0x8613427d4aaad374)
        github.com/restic/restic/internal/archiver/tree_saver.go:61 +0x85
gopkg.in/tomb%2ev2.(*Tomb).run(0xc000118280, 0xc000518ab0)
        gopkg.in/[email protected]/tomb.go:163 +0x3a
created by gopkg.in/tomb%2ev2.(*Tomb).Go
        gopkg.in/[email protected]/tomb.go:159 +0xbc

goroutine 56 [select, 68 minutes]:
github.com/restic/restic/internal/archiver.(*TreeSaver).worker(0xc000371060, 0x555d7bc7e8e0, 0xc000084440, 0xc0001049c0, 0xc000084440, 0x0)
        github.com/restic/restic/internal/archiver/tree_saver.go:148 +0x1a5
github.com/restic/restic/internal/archiver.NewTreeSaver.func1(0x0, 0x0)
        github.com/restic/restic/internal/archiver/tree_saver.go:61 +0x85
gopkg.in/tomb%2ev2.(*Tomb).run(0xc000118280, 0xc000518ae0)
        gopkg.in/[email protected]/tomb.go:163 +0x3a
created by gopkg.in/tomb%2ev2.(*Tomb).Go
        gopkg.in/[email protected]/tomb.go:159 +0xbc

goroutine 57 [select, 68 minutes]:
github.com/restic/restic/internal/archiver.(*TreeSaver).worker(0xc000371060, 0x555d7bc7e8e0, 0xc000084440, 0xc0001049c0, 0xc000084440, 0x0)
        github.com/restic/restic/internal/archiver/tree_saver.go:148 +0x1a5
github.com/restic/restic/internal/archiver.NewTreeSaver.func1(0x0, 0x0)
        github.com/restic/restic/internal/archiver/tree_saver.go:61 +0x85
gopkg.in/tomb%2ev2.(*Tomb).run(0xc000118280, 0xc000518b10)
        gopkg.in/[email protected]/tomb.go:163 +0x3a
created by gopkg.in/tomb%2ev2.(*Tomb).Go
        gopkg.in/[email protected]/tomb.go:159 +0xbc

goroutine 58 [select, 68 minutes]:
github.com/restic/restic/internal/archiver.(*TreeSaver).worker(0xc000371060, 0x555d7bc7e8e0, 0xc000084440, 0xc0001049c0, 0xc000084440, 0xd4e6aebeac367f43)
        github.com/restic/restic/internal/archiver/tree_saver.go:148 +0x1a5
github.com/restic/restic/internal/archiver.NewTreeSaver.func1(0x33b4f4f8fa0d240, 0xceea6af0e3c1c29e)
        github.com/restic/restic/internal/archiver/tree_saver.go:61 +0x85
gopkg.in/tomb%2ev2.(*Tomb).run(0xc000118280, 0xc000518b40)
        gopkg.in/[email protected]/tomb.go:163 +0x3a
created by gopkg.in/tomb%2ev2.(*Tomb).Go
        gopkg.in/[email protected]/tomb.go:159 +0xbc

goroutine 59 [select, 68 minutes]:
github.com/restic/restic/internal/archiver.(*TreeSaver).worker(0xc000371060, 0x555d7bc7e8e0, 0xc000084440, 0xc0001049c0, 0xc000084440, 0x0)
        github.com/restic/restic/internal/archiver/tree_saver.go:148 +0x1a5
github.com/restic/restic/internal/archiver.NewTreeSaver.func1(0x404020303020302, 0xa4593bf10cf9d56b)
        github.com/restic/restic/internal/archiver/tree_saver.go:61 +0x85
gopkg.in/tomb%2ev2.(*Tomb).run(0xc000118280, 0xc000518b70)
        gopkg.in/[email protected]/tomb.go:163 +0x3a
created by gopkg.in/tomb%2ev2.(*Tomb).Go
        gopkg.in/[email protected]/tomb.go:159 +0xbc

goroutine 60 [select, 68 minutes]:
github.com/restic/restic/internal/archiver.(*TreeSaver).worker(0xc000371060, 0x555d7bc7e8e0, 0xc000084440, 0xc0001049c0, 0xc000084440, 0xb10b3b8de021e79)
        github.com/restic/restic/internal/archiver/tree_saver.go:148 +0x1a5
github.com/restic/restic/internal/archiver.NewTreeSaver.func1(0x26fb040ea79e264a, 0x54be44ff81f4a44b)
        github.com/restic/restic/internal/archiver/tree_saver.go:61 +0x85
gopkg.in/tomb%2ev2.(*Tomb).run(0xc000118280, 0xc000518ba0)
        gopkg.in/[email protected]/tomb.go:163 +0x3a
created by gopkg.in/tomb%2ev2.(*Tomb).Go
        gopkg.in/[email protected]/tomb.go:159 +0xbc

goroutine 61 [select, 68 minutes]:
github.com/restic/restic/internal/archiver.(*TreeSaver).worker(0xc000371060, 0x555d7bc7e8e0, 0xc000084440, 0xc0001049c0, 0xc000084440, 0x0)
        github.com/restic/restic/internal/archiver/tree_saver.go:148 +0x1a5
github.com/restic/restic/internal/archiver.NewTreeSaver.func1(0x46435402edd12622, 0xfacaaca51a651305)
        github.com/restic/restic/internal/archiver/tree_saver.go:61 +0x85
gopkg.in/tomb%2ev2.(*Tomb).run(0xc000118280, 0xc000518bd0)
        gopkg.in/[email protected]/tomb.go:163 +0x3a
created by gopkg.in/tomb%2ev2.(*Tomb).Go
        gopkg.in/[email protected]/tomb.go:159 +0xbc

goroutine 62 [select, 68 minutes]:
github.com/restic/restic/internal/archiver.(*TreeSaver).worker(0xc000371060, 0x555d7bc7e8e0, 0xc000084440, 0xc0001049c0, 0xc000084440, 0xe33c919fd8fec0fc)
        github.com/restic/restic/internal/archiver/tree_saver.go:148 +0x1a5
github.com/restic/restic/internal/archiver.NewTreeSaver.func1(0xb0951ade8ae3aa2a, 0x24b49cb185aefc2c)
        github.com/restic/restic/internal/archiver/tree_saver.go:61 +0x85
gopkg.in/tomb%2ev2.(*Tomb).run(0xc000118280, 0xc000518c00)
        gopkg.in/[email protected]/tomb.go:163 +0x3a
created by gopkg.in/tomb%2ev2.(*Tomb).Go
        gopkg.in/[email protected]/tomb.go:159 +0xbc

goroutine 63 [select, 68 minutes]:
github.com/restic/restic/internal/archiver.(*TreeSaver).worker(0xc000371060, 0x555d7bc7e8e0, 0xc000084440, 0xc0001049c0, 0xc000084440, 0x0)
        github.com/restic/restic/internal/archiver/tree_saver.go:148 +0x1a5
github.com/restic/restic/internal/archiver.NewTreeSaver.func1(0xa6e8a12a4974b58d, 0x65a557f6a76f3aed)
        github.com/restic/restic/internal/archiver/tree_saver.go:61 +0x85
gopkg.in/tomb%2ev2.(*Tomb).run(0xc000118280, 0xc000518c30)
        gopkg.in/[email protected]/tomb.go:163 +0x3a
created by gopkg.in/tomb%2ev2.(*Tomb).Go
        gopkg.in/[email protected]/tomb.go:159 +0xbc

goroutine 64 [select, 68 minutes]:
github.com/restic/restic/internal/archiver.(*TreeSaver).worker(0xc000371060, 0x555d7bc7e8e0, 0xc000084440, 0xc0001049c0, 0xc000084440, 0x3dd5a736b37f8000)
        github.com/restic/restic/internal/archiver/tree_saver.go:148 +0x1a5
github.com/restic/restic/internal/archiver.NewTreeSaver.func1(0xbb8ef6dfbaa2ed8a, 0xb612564a62cd57ba)
        github.com/restic/restic/internal/archiver/tree_saver.go:61 +0x85
gopkg.in/tomb%2ev2.(*Tomb).run(0xc000118280, 0xc000518c60)
        gopkg.in/[email protected]/tomb.go:163 +0x3a
created by gopkg.in/tomb%2ev2.(*Tomb).Go
        gopkg.in/[email protected]/tomb.go:159 +0xbc

goroutine 65 [select, 68 minutes]:
github.com/restic/restic/internal/archiver.(*TreeSaver).worker(0xc000371060, 0x555d7bc7e8e0, 0xc000084440, 0xc0001049c0, 0xc000084440, 0x1c68dbf999da67d5)
        github.com/restic/restic/internal/archiver/tree_saver.go:148 +0x1a5
github.com/restic/restic/internal/archiver.NewTreeSaver.func1(0x793abe84adce1aeb, 0x626970987ff54b83)
        github.com/restic/restic/internal/archiver/tree_saver.go:61 +0x85
gopkg.in/tomb%2ev2.(*Tomb).run(0xc000118280, 0xc000518c90)
        gopkg.in/[email protected]/tomb.go:163 +0x3a
created by gopkg.in/tomb%2ev2.(*Tomb).Go
        gopkg.in/[email protected]/tomb.go:159 +0xbc

goroutine 66 [select, 68 minutes]:
github.com/restic/restic/internal/archiver.(*TreeSaver).worker(0xc000371060, 0x555d7bc7e8e0, 0xc000084440, 0xc0001049c0, 0xc000084440, 0x0)
        github.com/restic/restic/internal/archiver/tree_saver.go:148 +0x1a5
github.com/restic/restic/internal/archiver.NewTreeSaver.func1(0xf9a0c6b56f8d5d7c, 0x1e02f4e09cc58e27)
        github.com/restic/restic/internal/archiver/tree_saver.go:61 +0x85
gopkg.in/tomb%2ev2.(*Tomb).run(0xc000118280, 0xc000518cc0)
        gopkg.in/[email protected]/tomb.go:163 +0x3a
created by gopkg.in/tomb%2ev2.(*Tomb).Go
        gopkg.in/[email protected]/tomb.go:159 +0xbc

goroutine 67 [select, 68 minutes]:
github.com/restic/restic/internal/archiver.(*TreeSaver).worker(0xc000371060, 0x555d7bc7e8e0, 0xc000084440, 0xc0001049c0, 0xc000084440, 0xb1dc50921db37c47)
        github.com/restic/restic/internal/archiver/tree_saver.go:148 +0x1a5
github.com/restic/restic/internal/archiver.NewTreeSaver.func1(0x1a4fb671de74fc79, 0x3e45b8da77944622)
        github.com/restic/restic/internal/archiver/tree_saver.go:61 +0x85
gopkg.in/tomb%2ev2.(*Tomb).run(0xc000118280, 0xc000518cf0)
        gopkg.in/[email protected]/tomb.go:163 +0x3a
created by gopkg.in/tomb%2ev2.(*Tomb).Go
        gopkg.in/[email protected]/tomb.go:159 +0xbc

goroutine 68 [select, 68 minutes]:
github.com/restic/restic/internal/archiver.(*TreeSaver).worker(0xc000371060, 0x555d7bc7e8e0, 0xc000084440, 0xc0001049c0, 0xc000084440, 0x7bbee8c86ca41c55)
        github.com/restic/restic/internal/archiver/tree_saver.go:148 +0x1a5
github.com/restic/restic/internal/archiver.NewTreeSaver.func1(0x59719c1d0bd508ad, 0x172908b15436ab16)
        github.com/restic/restic/internal/archiver/tree_saver.go:61 +0x85
gopkg.in/tomb%2ev2.(*Tomb).run(0xc000118280, 0xc000518d20)
        gopkg.in/[email protected]/tomb.go:163 +0x3a
created by gopkg.in/tomb%2ev2.(*Tomb).Go
        gopkg.in/[email protected]/tomb.go:159 +0xbc

goroutine 69 [select, 68 minutes]:
github.com/restic/restic/internal/archiver.(*TreeSaver).worker(0xc000371060, 0x555d7bc7e8e0, 0xc000084440, 0xc0001049c0, 0xc000084440, 0x2ec65f8301ab4566)
        github.com/restic/restic/internal/archiver/tree_saver.go:148 +0x1a5
github.com/restic/restic/internal/archiver.NewTreeSaver.func1(0xa44c52d13ad99c58, 0x550651477cad963b)
        github.com/restic/restic/internal/archiver/tree_saver.go:61 +0x85
gopkg.in/tomb%2ev2.(*Tomb).run(0xc000118280, 0xc000518d50)
        gopkg.in/[email protected]/tomb.go:163 +0x3a
created by gopkg.in/tomb%2ev2.(*Tomb).Go
        gopkg.in/[email protected]/tomb.go:159 +0xbc

goroutine 70 [select, 68 minutes]:
github.com/restic/restic/internal/archiver.(*TreeSaver).worker(0xc000371060, 0x555d7bc7e8e0, 0xc000084440, 0xc0001049c0, 0xc000084440, 0x9fad07c5c2a9fb08)
        github.com/restic/restic/internal/archiver/tree_saver.go:148 +0x1a5
github.com/restic/restic/internal/archiver.NewTreeSaver.func1(0xdd866e09b515807b, 0xb9f40831132ab31)
        github.com/restic/restic/internal/archiver/tree_saver.go:61 +0x85
gopkg.in/tomb%2ev2.(*Tomb).run(0xc000118280, 0xc000518d80)
        gopkg.in/[email protected]/tomb.go:163 +0x3a
created by gopkg.in/tomb%2ev2.(*Tomb).Go
        gopkg.in/[email protected]/tomb.go:159 +0xbc

goroutine 71 [select, 68 minutes]:
github.com/restic/restic/internal/archiver.(*TreeSaver).worker(0xc000371060, 0x555d7bc7e8e0, 0xc000084440, 0xc0001049c0, 0xc000084440, 0x0)
        github.com/restic/restic/internal/archiver/tree_saver.go:148 +0x1a5
github.com/restic/restic/internal/archiver.NewTreeSaver.func1(0x0, 0x0)
        github.com/restic/restic/internal/archiver/tree_saver.go:61 +0x85
gopkg.in/tomb%2ev2.(*Tomb).run(0xc000118280, 0xc000518db0)
        gopkg.in/[email protected]/tomb.go:163 +0x3a
created by gopkg.in/tomb%2ev2.(*Tomb).Go
        gopkg.in/[email protected]/tomb.go:159 +0xbc

goroutine 72 [select, 68 minutes]:
github.com/restic/restic/internal/archiver.(*TreeSaver).worker(0xc000371060, 0x555d7bc7e8e0, 0xc000084440, 0xc0001049c0, 0xc000084440, 0x2b7ba3d2951de265)
        github.com/restic/restic/internal/archiver/tree_saver.go:148 +0x1a5
github.com/restic/restic/internal/archiver.NewTreeSaver.func1(0xca5fa9a4773dfdca, 0x8a84b8608621b2b1)
        github.com/restic/restic/internal/archiver/tree_saver.go:61 +0x85
gopkg.in/tomb%2ev2.(*Tomb).run(0xc000118280, 0xc000518de0)
        gopkg.in/[email protected]/tomb.go:163 +0x3a
created by gopkg.in/tomb%2ev2.(*Tomb).Go
        gopkg.in/[email protected]/tomb.go:159 +0xbc

goroutine 73 [select, 68 minutes]:
github.com/restic/restic/internal/archiver.(*TreeSaver).worker(0xc000371060, 0x555d7bc7e8e0, 0xc000084440, 0xc0001049c0, 0xc000084440, 0x0)
        github.com/restic/restic/internal/archiver/tree_saver.go:148 +0x1a5
github.com/restic/restic/internal/archiver.NewTreeSaver.func1(0x0, 0x0)
        github.com/restic/restic/internal/archiver/tree_saver.go:61 +0x85
gopkg.in/tomb%2ev2.(*Tomb).run(0xc000118280, 0xc000518e10)
        gopkg.in/[email protected]/tomb.go:163 +0x3a
created by gopkg.in/tomb%2ev2.(*Tomb).Go
        gopkg.in/[email protected]/tomb.go:159 +0xbc

goroutine 74 [select, 68 minutes]:
github.com/restic/restic/internal/archiver.(*TreeSaver).worker(0xc000371060, 0x555d7bc7e8e0, 0xc000084440, 0xc0001049c0, 0xc000084440, 0x0)
        github.com/restic/restic/internal/archiver/tree_saver.go:148 +0x1a5
github.com/restic/restic/internal/archiver.NewTreeSaver.func1(0x5ab7d41328920ebf, 0x3265e1fe520ee731)
        github.com/restic/restic/internal/archiver/tree_saver.go:61 +0x85
gopkg.in/tomb%2ev2.(*Tomb).run(0xc000118280, 0xc000518e40)
        gopkg.in/[email protected]/tomb.go:163 +0x3a
created by gopkg.in/tomb%2ev2.(*Tomb).Go
        gopkg.in/[email protected]/tomb.go:159 +0xbc

goroutine 75 [select, 68 minutes]:
github.com/restic/restic/internal/archiver.(*TreeSaver).worker(0xc000371060, 0x555d7bc7e8e0, 0xc000084440, 0xc0001049c0, 0xc000084440, 0xe27e2e521ce06cb1)
        github.com/restic/restic/internal/archiver/tree_saver.go:148 +0x1a5
github.com/restic/restic/internal/archiver.NewTreeSaver.func1(0x0, 0x0)
        github.com/restic/restic/internal/archiver/tree_saver.go:61 +0x85
gopkg.in/tomb%2ev2.(*Tomb).run(0xc000118280, 0xc000518e70)
        gopkg.in/[email protected]/tomb.go:163 +0x3a
created by gopkg.in/tomb%2ev2.(*Tomb).Go
        gopkg.in/[email protected]/tomb.go:159 +0xbc

goroutine 76 [select, 68 minutes]:
github.com/restic/restic/internal/archiver.(*TreeSaver).worker(0xc000371060, 0x555d7bc7e8e0, 0xc000084440, 0xc0001049c0, 0xc000084440, 0x0)
        github.com/restic/restic/internal/archiver/tree_saver.go:148 +0x1a5
github.com/restic/restic/internal/archiver.NewTreeSaver.func1(0x6f75adcf6d833eef, 0xa113ae20eb8d8d80)
        github.com/restic/restic/internal/archiver/tree_saver.go:61 +0x85
gopkg.in/tomb%2ev2.(*Tomb).run(0xc000118280, 0xc000518ea0)
        gopkg.in/[email protected]/tomb.go:163 +0x3a
created by gopkg.in/tomb%2ev2.(*Tomb).Go
        gopkg.in/[email protected]/tomb.go:159 +0xbc

goroutine 77 [select, 68 minutes]:
github.com/restic/restic/internal/archiver.(*TreeSaver).worker(0xc000371060, 0x555d7bc7e8e0, 0xc000084440, 0xc0001049c0, 0xc000084440, 0x0)
        github.com/restic/restic/internal/archiver/tree_saver.go:148 +0x1a5
github.com/restic/restic/internal/archiver.NewTreeSaver.func1(0x0, 0x0)
        github.com/restic/restic/internal/archiver/tree_saver.go:61 +0x85
gopkg.in/tomb%2ev2.(*Tomb).run(0xc000118280, 0xc000518ed0)
        gopkg.in/[email protected]/tomb.go:163 +0x3a
created by gopkg.in/tomb%2ev2.(*Tomb).Go
        gopkg.in/[email protected]/tomb.go:159 +0xbc

goroutine 78 [select, 68 minutes]:
github.com/restic/restic/internal/archiver.(*TreeSaver).worker(0xc000371060, 0x555d7bc7e8e0, 0xc000084440, 0xc0001049c0, 0xc000084440, 0x4935c987acd97d1f)
        github.com/restic/restic/internal/archiver/tree_saver.go:148 +0x1a5
github.com/restic/restic/internal/archiver.NewTreeSaver.func1(0x0, 0x0)
        github.com/restic/restic/internal/archiver/tree_saver.go:61 +0x85
gopkg.in/tomb%2ev2.(*Tomb).run(0xc000118280, 0xc000518f00)
        gopkg.in/[email protected]/tomb.go:163 +0x3a
created by gopkg.in/tomb%2ev2.(*Tomb).Go
        gopkg.in/[email protected]/tomb.go:159 +0xbc

rax    0xca
rbx    0x555d7c201920
rcx    0x555d7ab6dee3
rdx    0x0
rdi    0x555d7c201a68
rsi    0x80
rbp    0x7ffd5564c420
rsp    0x7ffd5564c3d8
r8     0x0
r9     0x0
r10    0x0
r11    0x286
r12    0xc00005fed8
r13    0x1
r14    0x0
r15    0xc00008a780
rip    0x555d7ab6dee1
rflags 0x286
cs     0x33
fs     0x0
gs     0x0
@MichaelEischer
Copy link
Member

The stacktrace looks like several syscalls got stuck, at least two read and one open syscall are active for 68 minutes. Combined with the fact that kill -9 didn't work this looks like a deadlock in the kernel. kill -9 is supposed to kill an application without giving the application any way to prevent that. So I don't think this is a bug in restic, but that you've rather run into a bug in the exFAT filesystem.

@MichaelEischer MichaelEischer added the state: need feedback waiting for feedback, e.g. from the submitter label Dec 11, 2020
@mqp
Copy link
Author

mqp commented Dec 11, 2020

That's useful to know -- I could plausibly switch that disk to use a different filesystem without too much trouble. If I do, I'll update with whether the problem recurs or not.

@mqp
Copy link
Author

mqp commented Dec 25, 2020

I switched to ext4 on the source disk and have made several backups without issue.

@MichaelEischer
Copy link
Member

Thanks for the feedback. So this looks like an issue of the underlying filesystem. For such problems it might be worth a try to set the environment variable GODEBUG to asyncpreemptoff=1 as in #2659. The async preemption feature of Go for goroutine seems to trigger quite a few bugs/problems. I'll close the issue for now. Feel free to reopen it if the issue resurfaces.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
state: need feedback waiting for feedback, e.g. from the submitter
Projects
None yet
Development

No branches or pull requests

2 participants