Migrations are missing for 5.0

[VdeJong]

I got this message after upgrading to 5.0.

 Your models in app(s): 'robots' have changes that are not yet reflected in a migration, and so won't be applied.
  Run 'manage.py makemigrations' to make new migrations, and then re-run 'manage.py migrate' to apply them.

The migration is forgotten to include in the commit.

[michaeljcollinsuk]

I also have this issue, seems that a 0003 migration should be checked in. Without this makemigrations --check --dry-run fails

[tony]

Thanks for the report. This needs more basic information. I am guessing this is #112 related. I didn't sit on that code review nor merge it but I'll take responsibility as the one who cut the 5.0 release.

@VdeJong @michaeljcollinsuk, anyone else experiencing this:

• Do you have DEFAULT_AUTO_FIELD set? If so, what to?
• DB + DB Version?
• Django version?
• Python version?
• If it tries to create a migration file, what's the content of the file?
• Any further context?

I have a preliminary PR started at #129 but I want to be deliberate about understanding the what and why before we publish a fix.

Anything else on your environments and isolating this would be helpful. It's appreciated!

[fabien-michel]

Same issue here after upgrading to django-robots==5.0

• DEFAULT_AUTO_FIELD = 'django.db.models.AutoField'
• sqlite
• django 3.2
• python 3.8

[fabien-michel]

When trying to run makemigrations it try to create a migration file for robots:

manage.py makemigrations --dry-run
Migrations for 'robots':
  /usr/local/lib/python3.8/site-packages/robots/migrations/0003_auto_20220516_1206.py
    - Alter field id on rule
    - Alter field id on url

[amourousias]

When trying to run makemigrations it try to create a migration file for robots:

manage.py makemigrations --dry-run
Migrations for 'robots':
  /usr/local/lib/python3.8/site-packages/robots/migrations/0003_auto_20220516_1206.py
    - Alter field id on rule
    - Alter field id on url

Having same issue, trying to make migration with those same 2 changes

• DEFAULT_AUTO_FIELD not set
• postgresql 12.7
• django 3.2
• python 3.7

[tony]

@fabien-michel @amourousias Thank you for the additional feedback

[tony]

@fabien-michel @amourousias Can you provide the content of the file?

e.g. what's inside of /usr/local/lib/python3.8/site-packages/robots/migrations/0003_auto_20220516_1206.py?

[tony]

Hi @umarmughal824 and @smithdc1

In regards to #112, by the looks of it there's a missing migration.

Any suggestions for how we approach this? Any reason why a migration would be missing for rule and id despite src/robots/migrations/0002_alter_id_fields.py being added in #112?

[amourousias]

@fabien-michel @amourousias Can you provide the content of the file?

e.g. what's inside of /usr/local/lib/python3.8/site-packages/robots/migrations/0003_auto_20220516_1206.py?

from django.db import migrations, models


class Migration(migrations.Migration):

    dependencies = [
        ('robots', '0002_alter_id_fields'),
    ]

    operations = [
        migrations.AlterField(
            model_name='rule',
            name='id',
            field=models.AutoField(auto_created=True, primary_key=True, serialize=False, verbose_name='ID'),
        ),
        migrations.AlterField(
            model_name='url',
            name='id',
            field=models.AutoField(auto_created=True, primary_key=True, serialize=False, verbose_name='ID'),
        ),
    ]```

[jan-szejko-steelseries]

@tony

I think this is related: https://docs.djangoproject.com/en/4.1/releases/3.2/#customizing-type-of-auto-created-primary-keys

The 0002 migration changed the auto fields to BigAutoField, but it's not set as the default for the app. So it works only for projects with DEFAULT_AUTO_FIELD set to BigAutoField.

You can set the default in src/robots/apps.py:

from django.apps import AppConfig

class RobotsConfig(AppConfig):
    default_auto_field = 'django.db.models.BigAutoField'
    name = 'robots'

so it works for everyone.

[tony]

@jan-szejko-steelseries Are you open to making a pull request?

In regards to any PRs that'd work for existing and new installations - across django versions: PRs and tests are welcome.

[jan-szejko-steelseries]

@tony Here you go.

[jan-szejko-steelseries]

@tony What's the progress? I can't run makemigrations because of this issue.

[tony]

@jan-szejko-steelseries Sorry for the delay - I'm not available during the weekdays (I will give this my undivided attention on Saturday)

[tony]

Looking for testers for 6.0b0 (PyPI, GitHub release, changes)

I've released 6.0b0, which has @jan-szejko-steelseries's patch from #134

I am looking for feedback on this before release☝️

5.0 users: Getting migrations back on track

If you already applied a custom migration, you may need to resolve this (once)

If you ran makemigrations and applied the custom migration - there may be another migration conflict yet to solve to get back on track with django-robots. If you see a migration row in django_migrations that isn't in src/robots/migrations, that may be the case.

If you do run into migration issues (conflicts with custom migrations) I'm also interested in steps you took to resolve the conflicts.

cc: @amourousias @fabien-michel @VdeJong @jan-szejko-steelseries @michaeljcollinsuk @mattaustin @jonprindiville, @svandeneertwegh @justin-ven from #125

[fdemmer]

Hey @tony, thank you for the beta release! Just wanted to confirm, that it solves the "missing" migrations issue and I see no adverse effects.

[tony]

@fdemmer Thank you for the feedback!

I am deliberating whether to release 6.0 now as its been a few months, we may not be able to count on more reviewers.

Any more reviewers available to try 6.0b? (PyPI, GitHub release, changes)

[kraig-droid]

If you do run into migration issues (conflicts with custom migrations) I'm also interested in steps you took to resolve the conflicts.

Here's what I did. With the beta release installed and with only two files in /env/lib/python3.9/site-packages/robots/migrations/, showmigrations doesn't show any problems:

$ ./manage.py showmigrations robots
robots
 [X] 0001_initial
 [X] 0002_alter_id_fields

But looking at the DB, there are problems:

> from django.db.migrations.recorder import MigrationRecorder
> pp(list(MigrationRecorder.Migration.objects.filter(app='robots').values()))
[{'id': 618,
  'app': 'robots',
  'name': '0001_initial',
  'applied': datetime.datetime(2021, 8, 18, 13, 56, 24, 74487, tzinfo=datetime.timezone.utc)},
 {'id': 859,
  'app': 'robots',
  'name': '0002_alter_id_fields',
  'applied': datetime.datetime(2022, 1, 25, 19, 41, 47, 400304, tzinfo=datetime.timezone.utc)},
 {'id': 963,
  'app': 'robots',
  'name': '0003_auto_20220419_1535',
  'applied': datetime.datetime(2022, 6, 7, 21, 24, 27, 326008, tzinfo=datetime.timezone.utc)},
 {'id': 1016,
  'app': 'robots',
  'name': '0003_alter_rule_id_alter_url_id',
  'applied': datetime.datetime(2022, 10, 11, 19, 22, 58, 916935, tzinfo=datetime.timezone.utc)}]

Oof. It shows four migrations, two of which we don't have source files for anymore. (Make sure you specifically look and see what files you have. I don't think installing the beta clobbers the 'extra' migrations if you have them. I went overboard and rm -rf'd the robots directory and lost them.)

Hmm. Let's see the column types in the DB (./manage.py dbshell)

=> \d+ robots_url
                                                        Table "public.robots_url"
 Column  |          Type          | Collation | Nullable |                Default                 | Storage  | Stats target | Description 
---------+------------------------+-----------+----------+----------------------------------------+----------+--------------+-------------
 id      | integer                |           | not null | nextval('robots_url_id_seq'::regclass) | plain    |              | 
 pattern | character varying(255) |           | not null |                                        | extended |              | 

Same for robots_rule. That type should be bigint instead of integer. So how to get there? Here's one way:

Our goal is to manage.py migrate --fake robots 0001, since at 0001 the id field is integer. Then manage.py migrate robots 0002. But there are complications. If you have all the files corresponding to the migration .values() above, you might be all set. I didn't. All I had was 0001 and 0002. So here's what I did:

$ cd env/lib/python3.9/site-packages/robots/migrations
$ cp 0002_alter_id_fields.py 0003_alter_rule_id_alter_url_id.py # filename shown in the .values() above
$ cp 0002_alter_id_fields.py 0003_auto_20220419_1535.py # ditto
$ ./manage.py makemigrations --merge robots # required because there are multiple leaf nodes in the migration graph
Merging robots
  Branch 0002_alter_id_fields
    - Alter field id on rule
    - Alter field id on url
  Branch 0003_alter_rule_id_alter_url_id
    - Alter field id on rule
    - Alter field id on url
  Branch 0003_auto_20220419_1535
    - Alter field id on rule
    - Alter field id on url

Merging will only work if the operations printed above do not conflict
with each other (working on different fields or models)
Should these migration branches be merged? [y/N] y

Created new merge migration env/lib/python3.9/site-packages/robots/migrations/0004_merge_20230728_1649.py
$ ./manage.py migrate --fake robots 0001
Operations to perform:
  Target specific migration: 0001_initial, from robots
Running migrations:
  Rendering model states... DONE
  Unapplying robots.0002_alter_id_fields... FAKED
  Unapplying robots.0003_alter_rule_id_alter_url_id... FAKED
  Unapplying robots.0003_auto_20220419_1535... FAKED

Now, back in django shell:

>>> pp(list(MigrationRecorder.Migration.objects.filter(app='robots').values()))
[{'id': 618,
  'app': 'robots',
  'name': '0001_initial',
  'applied': datetime.datetime(2021, 8, 18, 13, 56, 24, 74487, tzinfo=datetime.timezone.utc)}]

Looking good, only one migration showing in the DB. Let's check the column type

=> \d+ robots_url
                                                        Table "public.robots_url"
 Column  |          Type          | Collation | Nullable |                Default                 | Storage  | Stats target | Description 
---------+------------------------+-----------+----------+----------------------------------------+----------+--------------+-------------
 id      | integer                |           | not null | nextval('robots_url_id_seq'::regclass) | plain    |              | 
 pattern | character varying(255) |           | not null |                                        | extended |              | 

So now we can ./manage.py migrate robots 0002 then remove migrations 0003's and 0004 from the filesystem, and we're all done.

[svandeneertwegh]

I am too missing migrations running the new django-robots 5.0 version

[jonprindiville]

Any more reviewers available to try 6.0b?

Hey hey 👋 I have got a production Django 3.2 app which still sets its own DEFAULT_AUTO_FIELD to django.db.models.AutoField.

A while back when trying to move from django-robots==4.0 to django-robots==5.0 I did encounter the "missing migrations" problem described in this thread (so I just remained on 4.0.)

A few days ago I successfully transitioned the app from using django-robots==4.0 to instead use that django-robots==6.0b0 release -- I have not encountered any problems with that move.

cc @tony @jan-szejko-steelseries thanks!

[tony]

@jonprindiville Thank you for getting back!

I will publish django-robots==6.0 this weekend, then

[tony]

django-robots 6.0 is live: PyPI, releases, changelog

Based on that, I will officially close out this PR as resolved, absent any further issues. If there are, please make a new issue referencing this one. Thank you!