Defer index creation

[svenvarkel]

Creating indexes should be deferred / made in the background (with asyncio.ensure_future, for example). In case of large collections creation of indexes takes a very long time (tens of minutes or even hours). The application bootstrap would be blocked for that time and this is really bad.

Thus I request a feature:

• init_beanie() could have a boolean argument "create_indexes_in_background". Default "True"
• init_beanie() could have a boolean argument "skip_create_indexes" - that would skip creation of indexes totally. Default "False". Perhaps this could also be model / collection specific because I definitely don't want Beanie to initiate creation of indexes on a multi-gigabyte collection with hundreds of millions of documents at application bootstrap.

[svenvarkel]

One more piece of information here.

Right now I'm getting stuck with a large legacy collection that has indexes created before. I cannot start Beanie with that collection because in Beanie's "init.py" there's a method init_indexes() and in there it tries to load existing indexes. HOWEVER - for whatever reason existing indexes have float direction in their BSON definition, not int. Now the pymongo helper tries to parse it and it fails.

"skip_create_indexes" per collection / model would be of great help here.

It fails there:

File "/Users/sven/Library/Caches/pypoetry/virtualenvs/api-Dn4TtnJM-py3.11/lib/python3.11/site-packages/pymongo/helpers.py", line 128, in _index_document
    raise TypeError(
TypeError: second item in each key pair must be 1, -1, '2d', or another valid MongoDB index specifier.

[roman-right]

Hi @svenvarkel,

That's a great point. I will add this feature after the Pydantic v2 support.

[svenvarkel]

I implemented it here for my own needs:
https://github.com/svenvarkel/beanie/blob/db17bd067b6feae438be141eff6acdff9542fcc4/beanie/odm/utils/init.py#L398

This is "per model index toggle" or something. I didn't create that global flag to skip indexes as whole.

If you want, I can easily create that myself and provide you a pull request if that makes your life easier.

[jorma16]

Hey! Something new about this issue or the related PR? I need this... In development this wasn't cause me ant problem but when I went to production database the application doesn't have permissions to create indexes 😢