feature: Significant performance boost for filer directory listings (d…

…jango-cms#1353)

* remove mptt

* Remove deprecated function

* Update test

* Update flake8 errors

* Fix:	update move folder admin action

* Remove n+1 issue from move files/folders admin

* Remove unneeded ordering by mptt fields

* Remove spaces

* Optimize qs, delay thumbnail creation

* Add back clipboard

* Add url cache for file selector box
Add upscaled images

* Update package.json

* Add performance test

* Ensure annotation of thumbnail urls works in tests

* Update pre-commit.yaml for current isort version

* fix isort issue

* Update docs wrt paginator default

* Improved qs heuristics

* Update pre-commit hooks, add Django 4.2 tests

* Update changelog

* Update tox.ini

* feat: Allow uploads to be configured properly

* Fix isort issue

.github/workflows/test.yml

@@ -8,20 +8,21 @@ jobs:
     strategy:
       fail-fast: false
       matrix:
-        python-version: [ '3.7', '3.8', '3.9', '3.10', '3.11' ]
+        python-version: ['3.8', '3.9', '3.10', '3.11' ]
         requirements-file: [
           django-2.2.txt,
-          django-3.0.txt,
-          django-3.1.txt,
           django-3.2.txt,
           django-4.0.txt,
-          django-4.1.txt
+          django-4.1.txt,
+          django-4.2.txt,
         ]
         exclude:
           - python-version: 3.7
             requirements-file: django-4.0.txt
           - python-version: 3.7
             requirements-file: django-4.1.txt
+          - python-version: 3.7
+            requirements-file: django-4.2.txt
           - python-version: 3.9
             requirements-file: django-2.2.txt
           - python-version: 3.10

.pre-commit-config.yaml

@@ -26,7 +26,7 @@ repos:
       - id: flake8
 
   - repo: https://github.com/asottile/yesqa
-    rev: v1.4.0
+    rev: v1.5.0
     hooks:
       - id: yesqa
 

CHANGELOG.rst

@@ -2,9 +2,13 @@
 CHANGELOG
 =========
 
-unreleased
-==========
+Unpublished
+===========
 
+* Refactored directory list view for significant performance increses
+* Remove thumbnail generation from the directory list view request response cylce
+* Add Django 4.2 support
+* Add thumbnail view for faster visual management of image libraries
 * Fix File.objects.only() query required for deleting user who own files.
 
 2.2.5 (2023-06-11)

addon.json

@@ -3,7 +3,6 @@
     "installed-apps": [
         "filer",
         "easy_thumbnails",
-        "mptt",
         "polymorphic"
     ]
 }

docs/installation.rst

@@ -15,7 +15,6 @@ Dependencies
 ------------
 
 * `Django`_ >= 2.2
-* `django-mptt`_ >=0.6
 * `easy_thumbnails`_ >= 2.0
 * `django-polymorphic`_ >= 0.7
 * `Pillow`_ >=2.3.0 (with JPEG and ZLIB support, `PIL`_ may work but is not supported)
@@ -41,7 +40,6 @@ Add ``"filer"`` and related apps to your project's ``INSTALLED_APPS`` setting an
         ...
         'easy_thumbnails',
         'filer',
-        'mptt',
         ...
     ]
 
@@ -147,7 +145,6 @@ generation errors,  two options are provided to help when working with ``django-
 .. _django-polymorphic: https://github.com/bconstantin/django_polymorphic
 .. _easy_thumbnails: https://github.com/SmileyChris/easy-thumbnails
 .. _sorl.thumbnail: http://thumbnail.sorl.net/
-.. _django-mptt: https://github.com/django-mptt/django-mptt/
 .. _Pillow: http://pypi.python.org/pypi/Pillow/
 .. _Pillow doc: https://pillow.readthedocs.io/en/latest/installation.html
 .. _PIL: http://www.pythonware.com/products/pil/

docs/settings.rst

@@ -6,7 +6,7 @@ Settings
 ``FILER_ENABLE_PERMISSIONS``
 ----------------------------
 
-Activate the or not the Permission check on the files and folders before 
+Activate the or not the Permission check on the files and folders before
 displaying them in the admin. When set to ``False`` it gives all the authorization
 to staff members based on standard Django model permissions.
 
@@ -73,7 +73,7 @@ Public storage uses ``DEFAULT_FILE_STORAGE`` as default storage backend.
 
 ``UPLOAD_TO`` is the function to generate the path relative to the storage root. The
 default generates a random path like ``1d/a5/1da50fee-5003-46a1-a191-b547125053a8/filename.jpg``. This
-will be applied whenever a file is uploaded or moved between public (without permission checks) and 
+will be applied whenever a file is uploaded or moved between public (without permission checks) and
 private (with permission checks) storages. Defaults to ``'filer.utils.generate_filename.randomized'``.
 
 Overriding single keys is possible, for example just set your custom ``UPLOAD_TO``::
@@ -117,12 +117,12 @@ Defaults to using the DefaultServer (doh)! This will serve the files with the dj
 The number of items (Folders, Files) that should be displayed per page in
 admin.
 
-Defaults to ``20``
+Defaults to ``100``
 
 ``FILER_SUBJECT_LOCATION_IMAGE_DEBUG``
 --------------------------------------
 
-Draws a red circle around to point in the image that was used to do the 
+Draws a red circle around to point in the image that was used to do the
 subject location aware image cropping.
 
 Defaults to ``False``
@@ -153,6 +153,15 @@ Defines the path element common to all canonical file URLs.
 Defaults to ``'canonical/'``
 
 
+``FILER_UPLOADER_MAX_FILES``
+----------------------------
+
+Limit of files to upload by one drag and drop event. This is to avoid
+extensive accidental uploads, e.g. by dragging to root direcory onto an
+upload field.
+
+Defaults to ``100``.
+
 ``FILER_UPLOADER_CONNECTIONS``
 ------------------------------
 
@@ -161,3 +170,10 @@ Number of simultaneous AJAX uploads. Defaults to 3.
 If your database backend is SQLite it would be set to 1 by default. This allows
 to avoid ``database is locked`` errors on SQLite during multiple simultaneous
 file uploads.
+
+``FILER_UPLOADER_MAX_FILE_SIZE``
+--------------------------------
+
+Limits the maximal file size if set. Takes an integer (file size in MB).
+
+Defaults to ``None``.

filer/admin/fileadmin.py

@@ -1,12 +1,17 @@
 from django import forms
 from django.contrib.admin.utils import unquote
-from django.http import HttpResponseRedirect
-from django.urls import reverse
+from django.http import Http404, HttpResponse, HttpResponseRedirect
+from django.shortcuts import get_object_or_404
+from django.urls import path, reverse
 from django.utils.safestring import mark_safe
 from django.utils.translation import gettext as _
 
+from easy_thumbnails.files import get_thumbnailer
+from easy_thumbnails.options import ThumbnailOptions
+
 from .. import settings
-from ..models import File
+from ..models import BaseImage, File
+from ..settings import DEFERRED_THUMBNAIL_SIZES
 from .permissions import PrimitivePermissionAwareModelAdmin
 from .tools import AdminContext, admin_url_params_encoded, popup_status
 
@@ -147,5 +152,25 @@ def display_canonical(self, instance):
     display_canonical.allow_tags = True
     display_canonical.short_description = _('canonical URL')
 
+    def get_urls(self):
+        return super().get_urls() + [
+            path("icon/<int:file_id>/<int:size>",
+                 self.admin_site.admin_view(self.icon_view),
+                 name="filer_file_fileicon")
+        ]
+
+    def icon_view(self, request, file_id: int, size: int) -> HttpResponse:
+        if size not in DEFERRED_THUMBNAIL_SIZES:
+            # Only allow icon sizes relevant for the admin
+            raise Http404
+        file = get_object_or_404(File, pk=file_id)
+        if not isinstance(file, BaseImage):
+            raise Http404()
+
+        thumbnailer = get_thumbnailer(file)
+        thumbnail_options = ThumbnailOptions({'size': (size, size), "crop": True})
+        thumbnail = thumbnailer.get_thumbnail(thumbnail_options, generate=True)
+        return HttpResponseRedirect(thumbnail.url)
+
 
 FileAdmin.fieldsets = FileAdmin.build_fieldsets()

filer/admin/folderadmin.py

@@ -13,6 +13,7 @@
 from django.core.exceptions import PermissionDenied, ValidationError
 from django.core.paginator import EmptyPage, PageNotAnInteger, Paginator
 from django.db import models, router
+from django.db.models import OuterRef, Subquery
 from django.http import HttpResponse, HttpResponseRedirect
 from django.shortcuts import get_object_or_404, render
 from django.urls import re_path, reverse
@@ -22,9 +23,11 @@
 from django.utils.translation import gettext_lazy as _
 from django.utils.translation import ngettext_lazy
 
+from easy_thumbnails.models import Thumbnail
+
 from .. import settings
 from ..models import File, Folder, FolderPermission, FolderRoot, ImagesWithMissingData, UnsortedImages, tools
-from ..settings import FILER_IMAGE_MODEL, FILER_PAGINATE_BY
+from ..settings import FILER_IMAGE_MODEL, FILER_PAGINATE_BY, TABLE_LIST_TYPE
 from ..thumbnail_processors import normalize_subject_location
 from ..utils.compatibility import get_delete_permission
 from ..utils.filer_easy_thumbnails import FilerActionThumbnailer
@@ -54,7 +57,7 @@ class Meta:
 class FolderAdmin(PrimitivePermissionAwareModelAdmin):
     list_display = ('name',)
     exclude = ('parent',)
-    list_per_page = 20
+    list_per_page = 100
     list_filter = ('owner',)
     search_fields = ['name']
     autocomplete_fields = ['owner']
@@ -258,6 +261,14 @@ def directory_listing(self, request, folder_id=None, viewtype=None):
             folder = get_object_or_404(self.get_queryset(request), id=folder_id)
         request.session['filer_last_folder_id'] = folder_id
 
+        list_type = get_directory_listing_type(request) or settings.FILER_FOLDER_ADMIN_DEFAULT_LIST_TYPE
+        if list_type == TABLE_LIST_TYPE:
+            size = "40x40"  # Prefetch thumbnails for listing
+            size_x2 = "80x80"
+        else:
+            size = "160x160"  # Prefetch thumbnails for thumbnail view
+            size_x2 = "320x320"
+
         # Check actions to see if any are available on this changelist
         actions = self.get_actions(request)
 
@@ -302,17 +313,16 @@ def directory_listing(self, request, folder_id=None, viewtype=None):
             file_qs = folder.files.all()
             show_result_count = False
 
-        folder_qs = folder_qs.order_by('name')
+        folder_qs = folder_qs.order_by('name').select_related("owner")
         order_by = request.GET.get('order_by', None)
-        if order_by is not None:
-            order_by = order_by.split(',')
-            order_by = [field for field in order_by
-                        if re.sub(r'^-', '', field) in self.order_by_file_fields]
-            if len(order_by) > 0:
-                file_qs = file_qs.order_by(*order_by)
-
-        folder_children = []
-        folder_files = []
+        if order_by is None:
+            order_by = "file"
+        order_by = order_by.split(',')
+        order_by = [field for field in order_by
+                    if re.sub(r'^-', '', field) in self.order_by_file_fields]
+        if len(order_by) > 0:
+            file_qs = file_qs.order_by(*order_by)
+
         if folder.is_root and not search_mode:
             virtual_items = folder.virtual_folders
         else:
@@ -332,8 +342,20 @@ def directory_listing(self, request, folder_id=None, viewtype=None):
         if folder.is_root:
             folder_qs = folder_qs.exclude(**root_exclude_kwargs)
 
-        folder_children += folder_qs
-        folder_files += file_qs
+        # Annotate thumbnail status
+        thumbnail_qs = (
+            Thumbnail.objects
+            .filter(
+                source__name=OuterRef("file"),
+                modified__gte=OuterRef("modified_at"),
+            )
+            .exclude(name__contains="upscale")  # TODO: Check WHY not used by directory listing
+            .order_by("-modified")
+        )
+        file_qs = file_qs.annotate(
+            thumbnail_name=Subquery(thumbnail_qs.filter(name__contains=f"__{size}_").values_list("name")[:1]),
+            thumbnailx2_name=Subquery(thumbnail_qs.filter(name__contains=f"__{size_x2}_").values_list("name")[:1])
+        ).select_related("owner")
 
         try:
             permissions = {
@@ -345,16 +367,13 @@ def directory_listing(self, request, folder_id=None, viewtype=None):
         except:  # noqa
             permissions = {}
 
-        if order_by is None or len(order_by) == 0:
-            folder_files.sort()
-
-        items = folder_children + folder_files
+        items = list(itertools.chain(folder_qs, file_qs))
         paginator = Paginator(items, FILER_PAGINATE_BY)
 
         # Are we moving to clipboard?
         if request.method == 'POST' and '_save' not in request.POST:
             # TODO: Refactor/remove clipboard parts
-            for f in folder_files:
+            for f in folder_qs:
                 if "move-to-clipboard-%d" % (f.id,) in request.POST:
                     clipboard = tools.get_user_clipboard(request.user)
                     if f.has_edit_permission(request):
@@ -409,7 +428,6 @@ def directory_listing(self, request, folder_id=None, viewtype=None):
         except EmptyPage:
             paginated_items = paginator.page(paginator.num_pages)
 
-        list_type = get_directory_listing_type(request) or settings.FILER_FOLDER_ADMIN_DEFAULT_LIST_TYPE
         context = self.admin_site.each_context(request)
         context.update({
             'folder': folder,
@@ -420,15 +438,17 @@ def directory_listing(self, request, folder_id=None, viewtype=None):
             'paginated_items': paginated_items,
             'virtual_items': virtual_items,
             'uploader_connections': settings.FILER_UPLOADER_CONNECTIONS,
+            'max_files': settings.FILER_UPLOADER_MAX_FILES,
+            'max_filesize': settings.FILER_UPLOADER_MAX_FILE_SIZE,
             'permissions': permissions,
             'permstest': userperms_for_request(folder, request),
             'current_url': request.path,
             'title': _('Directory listing for %(folder_name)s') % {'folder_name': folder.name},
             'search_string': ' '.join(search_terms),
             'q': urlquote(q),
             'show_result_count': show_result_count,
-            'folder_children': folder_children,
-            'folder_files': folder_files,
+            'folder_children': folder_qs,
+            'folder_files': file_qs,
             'limit_search_to_folder': limit_search_to_folder,
             'is_popup': popup_status(request),
             'filer_admin_context': AdminContext(request),
@@ -744,8 +764,7 @@ def delete_files_or_folders(self, request, files_queryset, folders_queryset):
                 folder_ids = set()
                 for folder in folders_queryset:
                     folder_ids.add(folder.id)
-                    folder_ids.update(
-                        folder.get_descendants().values_list('id', flat=True))
+                    folder_ids.update(folder.get_descendants_ids())
                 for f in File.objects.filter(folder__in=folder_ids):
                     self.log_deletion(request, f, force_str(f))
                     f.delete()
@@ -869,12 +888,8 @@ def _list_all_destination_folders(self, request, folders_queryset, current_folde
         return list(self._list_all_destination_folders_recursive(request, folders_queryset, current_folder, root_folders, allow_self, 0))
 
     def _move_files_and_folders_impl(self, files_queryset, folders_queryset, destination):
-        for f in files_queryset:
-            f.folder = destination
-            f.save()
-        for f in folders_queryset:
-            f.move_to(destination, 'last-child')
-            f.save()
+        files_queryset.update(folder=destination)
+        folders_queryset.update(parent=destination)
 
     def move_files_and_folders(self, request, files_queryset, folders_queryset):
         opts = self.model._meta

filer/migrations/0016_alter_folder_index_together_remove_folder_level_and_more.py

@@ -0,0 +1,33 @@
+# Generated by Django 5.0.dev20230606055133 on 2023-06-07 14:14
+
+from django.db import migrations
+
+
+class Migration(migrations.Migration):
+
+    dependencies = [
+        ('filer', '0015_alter_file_owner_alter_file_polymorphic_ctype_and_more'),
+    ]
+
+    operations = [
+        migrations.AlterIndexTogether(
+            name='folder',
+            index_together=set(),
+        ),
+        migrations.RemoveField(
+            model_name='folder',
+            name='level',
+        ),
+        migrations.RemoveField(
+            model_name='folder',
+            name='lft',
+        ),
+        migrations.RemoveField(
+            model_name='folder',
+            name='rght',
+        ),
+        migrations.RemoveField(
+            model_name='folder',
+            name='tree_id',
+        ),
+    ]