N+1

[elyas-hedayat]

Hi.

I have a question about N+1 that happens when using DjangoObjecttypes.

Based on documentation problems https://github.com/graphql-python/graphene-django/blob/main/docs/authorization.rst#global-filtering DjangoObjectType has some problems when overriding get_queryset in Objecttypes

Does this repo solve all the problems that are pointed out here?

[MrThearMan]

Hi, thanks for the question.

I think you must be referring to this line:

... Note that because of this, benefits from using select_related in objects that define a relation to this DjangoObjectType will be canceled out. In the case of prefetch_related, the benefits of the optimization will be lost only if the custom get_queryset modifies the queryset. ...

As you can see here, graphene-django-query-optimizers DjangoObjectType will also override the get_queryset method. When using GraphQL Connection fields, this is where the optimization happens, so if you override it, you must make sure to call the parent method.

I'm a bit confused what the part about select_related means. Simply defining the method again and returning the same queryset should not result in the query data being lost. I verified this by overriding get_queryset and get_node in my testing setup and test still show green.

Now, since the optimized database queries are run on the first ObjectType in the query, cached, and fetched from the cache on the child ObjectTypes, adding additional filtering will of course affect this, since we need to run an additional query to fetch the filtered amount of data. The QuerySet._result_cache is purged when new flters are added (django.db.models.query.QuerySet._clone creates a new instance and doesn't pass _result_cache, so it's set to None), resulting in additional queries.

I don't think there is any way around this currently. The optimizer only knows what to optimize based on the GraphQL schema, and uses standard queryset caching (i.e. QuerySet._result_cache) to forward prefetched queryset data to the child resolvers. Adding additional filering on from get_queryset will result in additional queries, but I think there is still some room for optimization. In essence, I would need to hint to the optimizer that additional filtering has been done, which means the optimizer should go through the cache for each entity in the filtered queryset and update the _result_cache manually based on what is in there. This is how "x-to-one" relationships already work with get_node.

If you'd like, you can make an issue about this, I'll get to implementing it in the near future.

[MrThearMan]

Ok, I had a crack at this, and the initial idea I had didn't work, but I came up with a much cleaner one which doesn't need any additional queries.

Essentially, you can now define a classmethod filter_queryset on the ObjectType, which the optimizer will find and call when optimizing, whether the object type is the top level model, or a child prefetched model.

class ExampleType(DjangoObjectType):
    @classmethod
    def filter_queryset(cls, queryset: QuerySet, info: GQLInfo) -> QuerySet:
        return queryset.filter(...)

Released this in 0.0.9

[elyas-hedayat]

i faced with an other problem
assume that we have two models :

class Menu(models.Model):
    description = models.CharField(max_length=125, null=True, blank=True)
    menu_name = models.CharField(max_length=125)


class SubMenu(models.Model):
    description = models.CharField(max_length=125, null=True, blank=True)
    menu = models.ForeignKey(
        Menu,
        on_delete=models.PROTECT,
        related_name="sub_menus",
        limit_choices_to={"is_active": True, "is_deleted": False},
        null=True,
    )

when we want to write query like this to get submenu of each menu:

query{
  menuList{
    edges{
      node{
        id
        subMenus{
          edges{
             node{
              name
            }
          }
        }
      }
    }
  }
}

when we try to get submenus it's generate extra query for each item (it you check : https://docs.graphene-python.org/projects/django/en/latest/debug/ it will be more clear i think for you )

do you have any solution ?

[MrThearMan]

I'll need to see your ObjectTypes and how those are configured.

[elyas-hedayat]

here you're:

class MenuType(DjangoObjectType):
    class Meta:
        model = Menu
        interfaces = (CustomNode,)
        filterset_class = MenuFilter


class SubMenuMainType(DjangoObjectType):
    class Meta:
        model = SubMenu
        interfaces = (CustomNode,)
        filterset_class = SubMenuFilter

[MrThearMan]

Looks OK to me. Might be an issue with nested connection fields. The issue is that the connection field resolver adds the pagination queryset limits, and therefore we cannot cache these parts of the queryset when the optimization is done. For the top level field, this is fine, since the connection resolver is only run once, but for nested connection fields they are run for each nested item.

Currently, I'm only using connection fields for the top level queries, and using regular ListFields for nested items (see schema in the tests directory). With this, the optimization works fine, but I'd like to have nested connection fields work as well. I'll have a look when I have time.

[MrThearMan]

Ok, I managed to reproduce the issue, and found a fix which seems to work. Released in 0.1.1.