Instructions for using upgrade tool in docker environment

[KaiRo-at]

When running OpenEthereum as a docker container, a DB upgrade should be possible and supported. As the data is of course in a volume outside the container, it's accessible to a different container, so if instructions would be published on how to build and use a container with the upgrade tool, that would be very helpful.

[KaiRo-at]

I have tried the following, FWIW (excuse having a number of "parity" paths in there, that's leftover from "ye ole times"):

---

docker-compose setup

  oe-upgrade:
    image: rust
    volumes:
      - /data/parity/data:/data/io.parity.ethereum/
      - /data/parity/upgrade:/tool/

Build upgrade tool

Run the below within the container, i.e. after
docker-compose -f /data/docker-compose.yml run oe-upgrade bash:

apt update && apt install -y clang cmake
git clone https://github.com/openethereum/3.1-db-upgrade-tool.git
cd 3.1-db-upgrade-tool
cargo install --path . --root ../tool/

Run the actual update

docker-compose -f /data/docker-compose.yml run oe-upgrade tool/bin/oe-upgrade-db-3-1 /data/io.parity.ethereum/chains/ropsten/db/ae90623718e47d66/overlayrecent/

---

This isn't the nicest or cleanest way, but all that's needed is a one-off after all, and it runs the migration successfully on a DB previously run with OE v3.0.0 - but then running openethereum/openethereum:v3.1.1 on the upgraded DB gives the following error:
thread 'main' panicked at 'Low-level database error.: Custom { kind: Other, error: "Corruption: Unknown Footer version. Maybe this file was created with newer version of RocksDB?" }', util/journaldb/src/overlayrecentdb.rs:195:28

And when I restart OE to repair the DB, it ends up not being able to recover with this:

2021-01-27 15:04:20 UTC DB has been previously marked as corrupted, attempting repair
2021-01-27 15:28:33 UTC DB corrupted: Invalid argument: You have to open all column families. Column families not opened: col1, col3, col0, col2, attempting repair
Failed to open database Custom { kind: Other, error: "Received null column family handle from DB." }
Error: 1

Is this a problem of how I'm running this or an issue with the DB upgrade tool itself (in the latter case, should probably move to a different issue)?

[KaiRo-at]

Is this a problem of how I'm running this or an issue with the DB upgrade tool itself (in the latter case, should probably move to a different issue)?

Thanks to @rakita for answering some questions of me on chat recently, confirming this should be a problem of my setup here, that he tested upgrades successfully (which I assumed anyhow but wanted to make sure) and that RocksDB versions should not be an issue.

I had an idea this morning of what may be the cause here, and it may come down to permissions: The parity-ethereum/openethereum docker containers are running their binaries inside the container under the user ID 1000, which means that user ID is what is and needs to be on all files in the docker volume (and on-disk files structure) used for the DB. Now, with the setup I gave above, the upgrade tool runs as root user inside its container, and writing with the root user ID on the DB. When the new openethereum container tries to run with that DB, there may be some files it cannot write to (maybe even read) because they are owned by root, and that may cause the issue. Given that it can write most of the DB, it doesn't flag the issue right away but fails later. If this suspicion is correct, a chmod -R 1000:1000 /path/to/db should fix it, I will test that tomorrow.

[KaiRo-at]

OK, that idea did not work out. Despite corrected permissions, the same errors do come again, at least on this Ropsten DB I used for testing. And as there's no time left until Berlin to figure more things out, I guess I'll have to re-sync all our nodes from scratch and we'll need to pay a lump of money and invest a lot of syncing time to run more archive nodes as we otherwise can't have event logs for the last 2 years available any more (which our DApp needs, at least for our contracts).
This DB "upgrade" story has been a huge disappointment for me, and I'm not sure I can recommend I OpenEthereum in good consciousness any more as I have no idea if such a disappointment will happen again in the future. Sorry.

[rakita]

@KaiRo-at please try this pull request for updated lib: openethereum/openethereum#329
It seems if you started syncing with v3.0.1 you are maybe incompatible with older version of db

[KaiRo-at]

@KaiRo-at please try this pull request for updated lib: openethereum/openethereum#329

Thanks, I will try once a release with those changes is out as a docker container. Unfortunately, due to Berlin, I'm out of testnet machines/DBs to try with, and had to start re-synching from scratch on mainnet archive nodes, so the only machines I still could try a DB upgrade on are the two that our production DApp is depending on right now...