rollback migration cause data loss

[OuYangJinTing]

As the title, it destroy data when happen migration rollback.
Can the data be automatically exported before rollback, and then imported back when migrating again？

PS: English is not my native language; please excuse typing errors.

[ka8725]

@OuYangJinTing thanks for adding your issue. Can you please specify your real scenario? That would allow me to think of a concrete fix rather than implementing an abstract solution. Thank you!

[OuYangJinTing]

@ka8725 The actual scenario is as follows:

git checkout -b feat main
rails generate model Feat name:string
rails db:migrate   # create feats table
rails runner 'Feat.create(name: :test)'
git add -A && git commit -m 'stash'
git checkout -b other main
rails generate model Other name:string
rails db:migrate   # create others table and drop feats table
git add -A && git commit -m 'stash'
git checkout feat
rails db:migrate   # create feats table and drop others table
rails runner 'puts Feat.count'   # ==> 0

[ka8725]

@OuYangJinTing if the original migration had the seeding data populated automatically would it work for you? There is a solution for seeding data within migrations - https://github.com/ka8725/migration_data

This is what you could do after installing this gem:

1. add "#data" into the "create feats table" migration that does this: Feat.create(name: :test)
2. optionally, you can enable it only in development mode, like specified here

It might sound too annoying. But, I don't think it's possible to implement something that also would have the data not touched. Even if implement automatic dumping data on rolling migration back and later reinstating this data, there is no guarantee this data will be valid. There are many reasons why it could be invalid:

• If it was a deleted table, it can be updated so that a new required column is added, but there is no data in the dump.
• If it was an enum column with one set of values that were changed later, we have a problem again.

So, my recommendation is seeding data inside migrations for dev purposes. Or handle these cases manually. If you have a better idea, I'm open for discussion. Thank you!

[OuYangJinTing]

@ka8725 Thank for you reply. the migration_data is good.
But it maybe not handle all scenario, because the some data is generated during manual operation.
I also think import that export data maybe is invalid too, but sometimes it is valid. eg: I gave earlier example.
I think that if there is no good solution, it should require confirmed by the user before performing the rollback automatically.

[ka8725]

@OuYangJinTing WDYT about this approach: when it rolls back the first migration automatically, it makes the DB dump behind the scene. So that on each run of rollback_branches there is a DB dump that remembers this most recent migration ID. When db:migrate runs up and tries to reinstall the associated DB dump. This approach would require implementing tasks that make a dump and restore it. Also, these tasks would need adapters for different DBs. Note, even this solution is not ideal though, as it drops recent data as well. Consider the scenario:

git checkout
rails generate model User email:string
rails db:migrate   # create users table
rails runner 'User.create(email: "[email protected]")'
git add -A && git commit -m 'stash'

git checkout -b feat main
rails generate model Feat name:string
rails db:migrate   # create feats table
rails runner 'Feat.create(name: :test)'
rails runner 'User.create(email: "[email protected]")'
git add -A && git commit -m 'stash'

git checkout -b other main
rails generate model Other name:string
rails db:migrate   # create others table and drop feats table, dumped data
git add -A && git commit -m 'stash'

git checkout feat
rails db:migrate   # create feats table, dump restored
rails runner 'puts Feat.count'   # ==> 1
rails runner 'puts User.count'   # ==> 1 but it's expected to be 2 (!)

I thought about DB dump diff, that would probably work in case of dropping/restoring the table and its data. But when it comes to dropping/restoring columns - again there is a problem. It can't generate column data for new rows automatically. This way, IMO the issue is reduced to proper implementation of DB dump diff. I'm not sure if there is any, I'm researching the topic.

[OuYangJinTing]

@ka8725 Sorry. I just saw this comment today. I'm a little confused that users table has always existed, why the data is lost?
Maybe I didn't understand the above example, can you help me explain again? Thank you.

[ka8725]

@OuYangJinTing because the solution implied to have full DB dump of rollback. If there is a solution that can generate a DB data diff, it would solve the problem.

[OuYangJinTing]

@ka8725 I think the solution may be too complex, maybe it's better to simply give a warm prompt and ask whether to continue rollback. What do you think?

[ka8725]

@OuYangJinTing is a good suggestion, I like it. Also, it actually could be configurable so that someone can set it up to rollback always without confirmation. Also, it could check if a deleting table/column really has data in there.