Lost data on reboot from degraded mirror

[piecler]

Hello,

I'm trying to understand what went wrong on my server.
Already tried to find it out with claude code on the server, but no luck yet.

Years ago I installed Ubuntu 22.04 LTS with zfs mirrored on 2 boot drives.
Just added luks crypt to the data partition. The boot partition was also zfs but not encrypted.
Server booted fine from either drive.
On december 10th 15:03 one drive failed.
In that time the server kept running fine in a degraded state.
A few terabytes were written in that period on the working drive.
Eventually on december 27th I realized that a drive had failed.
Got a new drive, and shutdown the server to replace the drive.
Shutdown completed and server did not crash, but did not see if there were error messages during shutdown.
Replaced the faulty drive und booted again.
System booted from the still working mirror drive.
But it rolled back all changes that were made after the drive failed on december 10th 15:03.
All logs on the server show nothing after 15:03, just a gap till the reboot.
Also no errors can be found anywhere. The errors were in the logs after 15:03, since I've seen them prior to reboot myself.
The server kept running fine in that time, since we worked daily on it without problems.
It did not slow down or anything. Behaved just fine.

Any ideas why that rollback could have happened?

`zdb -l /dev/mapper/root2_crypt

LABEL 0

version: 5000
name: 'rpool'
state: 0
txg: 54710867
pool_guid: 10393663121414298557
errata: 0
hostid: 2120435735
hostname: 'lager'
top_guid: 15377459437090521160
guid: 5395025176851143102
vdev_children: 1
vdev_tree:
    type: 'mirror'
    id: 0
    guid: 15377459437090521160
    metaslab_array: 132
    metaslab_shift: 33
    ashift: 12
    asize: 995350282240
    is_log: 0
    create_txg: 4
    children[0]:
        type: 'disk'
        id: 0
        guid: 1299657707244334166
        path: '/dev/mapper/root3_crypt'
        whole_disk: 0
        DTL: 2139
        create_txg: 4
    children[1]:
        type: 'disk'
        id: 1
        guid: 5395025176851143102
        path: '/dev/mapper/root2_crypt'
        whole_disk: 0
        DTL: 184
        create_txg: 4
features_for_read:
    com.delphix:hole_birth
    com.delphix:embedded_data
labels = 0 1 2 3

`

[IvanVolosyuk]

Sounds like you removed a wrong drive, which failed intermittently, e.g. loose cable. The other drive which you removed should have all the data you lost :)

[piecler]

It may sound like that, but I am sure the correct drive has been removed.
I've verified the Serialnumbers quite a few times and wrote them down.

However I'll check again as soon as i have the drive in my hands again.
Let's assume the not failing ssd got replaced.
How would I import the rpool now without risking damage to the live rpool to copy the files?
Should I do that on another computer?