Improve native event handling in the toolbar area.

[Andre-lbc]

The issue:

Interacting (double-clicking or dragging) with areas in the Flutter app where the native macOS toolbar would typically be triggers native actions like maximizing or moving the window. While this is expected behavior on "empty" areas, it's undesirable when interacting with interactive widgets, such as buttons and draggable interfaces.

The solution:

I've developed a solution that involves creating and managing invisible native macOS UI elements (NSViews) in Swift. These elements intercept the native toolbar events, preventing them from being processed natively and instead passing them to the Flutter engine for handling within the app.

I've implemented this fix in my app project, and I'd like to share it upstream for further discussion (e.g.: better naming, code improvements, etc) and potential integration into the package.

Preview:

With debug layers

Screen.Recording.2024-10-31.at.09.42.51.mp4

Overlay legend:

• Yellow: total usable toolbar area.
• Green: "Passthrough" UI (absorbs native events and send them to Flutter)

Without debug overlays

Screen.Recording.2024-10-31.at.09.48.03.mp4

Key code points:

• MacosToolbarPassthrough: A widget that watches its child and creates an invisible "passthrough" equivalent on the native side. Mouse events on this "passthrough" item do not trigger native events like expanding or dragging the window. Most layout changes in the child automatically trigger an update on the native equivalent, but you can also manually trigger an update using requestUpdate from the widget's state (MacosToolbarPassthroughState).
  Most simple UI (e.g.: static size and fixed positions) may only need to use one or multiple MacosToolbarPassthrough. More complex use cases may also require the addition of a MacosToolbarPassthroughScope.
• MacosToolbarPassthroughScope (optional): Flutter is optimized to avoid unnecessarily drawing UI elements, so detecting some changes in UI can be expensive or cumbersome. Even worse, sometimes the implementation may not be able to detect these changes at all (such as in some scrolling widgets or widgets that either move or change size in uncommon ways), resulting in flutter widgets being out of sync relative to their native counterpart.
  Wrapping multiple MacosToolbarPassthroughunder a MacosToolbarPassthroughScope allows you to both:
  1- Trigger a reevaluation on all scoped items when any of them internally requests a reevaluation.
  2- Manually trigger these reevaluation events if needed (using notifyChangesOf).
  When a scoped item detects a change that would normally trigger a reevaluation of their position and size, they instead notify their parent scope, which then notifies all their scoped items to reevaluate and update their positions and sizes.
• I am using NSTitlebarAccessoryViewController instead of NSToolbar because the later has a padding that I could not remove and also was much more difficult to work with (e.g.: position items).

Code:

MainFlutterWindow.swift

import Cocoa
import FlutterMacOS
import window_manager 
 
class MainFlutterWindow: NSWindow {
    private var hasToolbarDebugLayers: Bool = false
    private var toolbarPassthroughContainer: NSView?
    private var toolbarPassthroughViews: [String: PassthroughView] = [:]

  override func awakeFromNib() {
    let flutterViewController = FlutterViewController.init()
    let windowFrame = self.frame
    self.contentViewController = flutterViewController
    self.setFrame(windowFrame, display: true)

    RegisterGeneratedPlugins(registry: flutterViewController)
      
    setupToolbarPassthroughViewPlugin(flutterViewController: flutterViewController)

    super.awakeFromNib()
  }
    
    func setupToolbarPassthroughViewPlugin(flutterViewController: FlutterViewController) {
        
    let pluginChannel = FlutterMethodChannel(
          name: "[PLUGIN_NAMESPACE]/toolbar_passthrough",
          binaryMessenger: flutterViewController.engine.binaryMessenger)
        pluginChannel.setMethodCallHandler { (call, result) in
            switch call.method {
                case "updateView":
                  if let args = call.arguments as? [String: Any],
                     let id = args["id"] as? String,
                     let x = args["x"] as? CGFloat,
                     let y = args["y"] as? CGFloat,
                     let width = args["width"] as? CGFloat,
                     let height = args["height"] as? CGFloat {
                      self.updateToolbarPassthroughView(id: id, x: x, y: y, width: width, height: height, flutterViewController: flutterViewController)
                    result(nil)
                  } else {
                    result(FlutterError(code: "INVALID_ARGUMENTS", message: "Invalid arguments for updateToolbarPassthroughView", details: nil))
                  }
                case "removeView":
                  if let args = call.arguments as? [String: Any],
                     let id = args["id"] as? String {
                    self.removeToolbarPassthroughView(id: id)
                    result(nil)
                  } else {
                    result(FlutterError(code: "INVALID_ARGUMENTS", message: "Invalid arguments for removeToolbarPassthroughView", details: nil))
                  }
                case "initialize":
                    // Clean up necessary for flutter hot restart
                   self.toolbarPassthroughViews.removeAll()
                   self.toolbarPassthroughContainer = nil
                    
                    #if DEBUG
                      let args = call.arguments as? [String: Any]
                      let showDebugLayers = args?["showDebugLayers"] as? Bool
                      self.hasToolbarDebugLayers = showDebugLayers ?? false
                    #endif
                   
                    // Get the count of accessory view controllers
                    let accessoryCount = self.titlebarAccessoryViewControllers.count
                
                    // Iterate through the indices in reverse order to avoid index shifting
                    for index in stride(from: accessoryCount - 1, through: 0, by: -1) {
                        self.removeTitlebarAccessoryViewController(at: index)
                    }
                  result(nil)
                default:
                  result(FlutterMethodNotImplemented)
                }
        }
        
    }
    
    func updateToolbarPassthroughView(id: String, x: CGFloat, y: CGFloat, width: CGFloat, height: CGFloat, flutterViewController: FlutterViewController) {
      DispatchQueue.main.async {
          let window = self
          
          if self.toolbarPassthroughContainer == nil {
              // Initialize the view if it is nil
              let accessoryViewController = NSTitlebarAccessoryViewController()
              accessoryViewController.layoutAttribute = .top
              
              self.toolbarPassthroughContainer = NSView()
              if (self.hasToolbarDebugLayers){
                  self.toolbarPassthroughContainer!.wantsLayer = true
                  self.toolbarPassthroughContainer!.layer?.backgroundColor = NSColor.yellow.withAlphaComponent(0.2).cgColor
              }
              self.toolbarPassthroughContainer!.translatesAutoresizingMaskIntoConstraints = false
              
              // Assign the custom view to the accessory view controller
              accessoryViewController.view = self.toolbarPassthroughContainer!

              // Add the accessory view controller to the window
              window.addTitlebarAccessoryViewController(accessoryViewController)
          }
          
          if let containerView = self.toolbarPassthroughContainer {
             let windowHeight = window.frame.height
             
             // Convert Flutter coordinates to macOS coordinates
             let macY = windowHeight - y - height
             
             let flutterToggleInvertedPosition = CGRect(x: x, y: macY, width: width, height: height)
             let frame = containerView.convert(flutterToggleInvertedPosition, from: nil)

           var view: PassthroughView
           if let existingView = self.toolbarPassthroughViews[id] {
             view = existingView
             view.frame = frame
           } else {
               view = PassthroughView(frame: frame, flutterViewController:flutterViewController)
             if (self.hasToolbarDebugLayers){
                 view.wantsLayer = true
                 view.layer?.backgroundColor = NSColor.green.withAlphaComponent(0.2).cgColor
                 
             }

             // Add the view to the containerView
             containerView.addSubview(view)
             self.toolbarPassthroughViews[id] = view
           }
         }
      }
    }

    func removeToolbarPassthroughView(id: String) {
      DispatchQueue.main.async {
        if let view = self.toolbarPassthroughViews[id] {
          view.removeFromSuperview()
          self.toolbarPassthroughViews.removeValue(forKey: id)
        }
      }
  }

  override public func order(_ place: NSWindow.OrderingMode, relativeTo otherWin: Int) {
      super.order(place, relativeTo: otherWin)
      hiddenWindowAtLaunch()
  }
}

class PassthroughView: NSView {
  var flutterViewController: FlutterViewController?
    
    
    required init(frame: CGRect, flutterViewController: FlutterViewController) {
        super.init(frame: frame)
        self.flutterViewController = flutterViewController
    }
    
    required init?(coder decoder: NSCoder) {
       fatalError("init(coder:) has not been implemented")
     }
    

  override func mouseDown(with event: NSEvent) {
    flutterViewController!.mouseDown(with: event)
  }

  override func mouseUp(with event: NSEvent) {
    flutterViewController!.mouseUp(with: event)
  }
    
    override var mouseDownCanMoveWindow: Bool {
        return false
    }
}

macos_toolbar_passthrough.dart

/// A collection of widgets that intercept and handle native macOS toolbar
/// events (e.g.: double-click to maximize, dragging to move window) so they can
/// be processed within a Flutter app.
///
/// The issue: interacting (double-clicking or dragging) with the area within
/// a flutter app where the native macOS toolbar would typically be triggers
/// native actions like maximizing or moving the window. This is the expected
/// behavior on "empty" areas, but undesirable when interacting with an input,
/// such as a button.
///
/// This solution involves creating and managing invisible native macOS UI
/// elements (Mainly NSViews) in Swift that intercept these events, preventing
/// them from being processed natively and instead passing them to the Flutter
/// engine for further handling.
library;

import 'dart:async';
import 'dart:collection';

import 'package:flutter/material.dart';
import 'package:flutter/rendering.dart';
import 'package:flutter/services.dart';

const String kMacosToolbarPassthroughMethodChannel =
    '[PLUGIN_NAMESPACE]/toolbar_passthrough';

const _debounceDuration = Duration(milliseconds: 10);

/// An optional "scope" for toolbar items.
///
/// When a scoped item detects a change that would normally trigger a
/// reevaluation of their position and size, they instead notify their parent
/// scope, which then notifies all their scoped items to reevaluate and update
/// their positions and sizes.
///
/// Since Flutter is optimized to avoid unnecessarily drawing UI elements,
/// detecting some changes in UI can be expensive or cumbersome. Even worse,
/// sometimes the implementation may not be able to detect these changes at all
/// (such as in some scrolling widgets or widgets that either move or change
/// size in uncommon ways), resulting in flutter widgets being out of sync
/// relative to their native counterpart.
///
/// Wrapping multiple [MacosToolbarPassthrough] under a
/// [MacosToolbarPassthroughScope] allows you to both:
/// 1- Trigger a reevaluation on all scoped items when any of them internally
/// requests a reevaluation.
/// 2- Manually trigger these reevaluation events if needed (see:
/// [notifyChangesOf]).
class MacosToolbarPassthroughScope extends StatefulWidget {
  const MacosToolbarPassthroughScope({super.key, required this.child});

  final Widget child;

  static void Function() notifyChangesOf(BuildContext context) {
    final result = maybeNotifyChangesOf(context);
    assert(result != null, 'No MacosToolbarPassthroughScope found in context');
    return result!;
  }

  static void Function()? maybeNotifyChangesOf(BuildContext context) {
    return _DescendantRegistry.maybeOf(context)?.notifyChanges;
  }

  @override
  State<MacosToolbarPassthroughScope> createState() =>
      _MacosToolbarPassthroughScopeState();
}

class _MacosToolbarPassthroughScopeState
    extends State<MacosToolbarPassthroughScope> {
  late final _DescendantRegistry _registry;

  @override
  void initState() {
    super.initState();
    _registry = _DescendantRegistry(_onReevaluationRequested);
  }

  @override
  void dispose() {
    _registry.dispose();
    super.dispose();
  }

  void _onReevaluationRequested() {
    for (final descendant in _registry.all.values) {
      descendant.requestUpdate();
    }
  }

  @override
  Widget build(BuildContext context) {
    return _InheritedDescendantRegistry(
      registry: _registry,
      child: widget.child,
    );
  }
}

class _DescendantRegistry {
  _DescendantRegistry(void Function() onReevaluationRequested)
      : _onReevaluationRequested = onReevaluationRequested;

  final void Function() _onReevaluationRequested;
  Timer? _debounceTimer;

  final Map<String, MacosToolbarPassthroughState> _descendantsById = {};

  void register(final MacosToolbarPassthroughState descendant) {
    _descendantsById.putIfAbsent(descendant._key.toString(), () => descendant);
  }

  void unregister(final String descendantKey) {
    _descendantsById.remove(descendantKey);
  }

  UnmodifiableMapView<String, MacosToolbarPassthroughState> get all =>
      UnmodifiableMapView(_descendantsById);

  void notifyChanges() {
    // Debounce change notifications
    _debounceTimer?.cancel();
    _debounceTimer = Timer(
      _debounceDuration,
      _onReevaluationRequested,
    );
  }

  void dispose() {
    _debounceTimer?.cancel();
  }

  static _DescendantRegistry? maybeOf(BuildContext context) {
    return context
        .dependOnInheritedWidgetOfExactType<_InheritedDescendantRegistry>()
        ?.registry;
  }
}

class _InheritedDescendantRegistry extends InheritedWidget {
  const _InheritedDescendantRegistry({
    required super.child,
    required this.registry,
  });

  final _DescendantRegistry registry;

  @override
  bool updateShouldNotify(covariant InheritedWidget oldWidget) => false;
}

/// A widget that intercepts and handles native macOS toolbar events (e.g.:
/// double-click to maximize, dragging to move window), forwarding them so they
/// can be processed within Flutter only.
///
/// Most simple UI (e.g.: static size and fixed positions) may only need to use
/// one or multiple [MacosToolbarPassthrough]. More complex use cases may also
/// require the addition of a [MacosToolbarPassthroughScope].
///
/// You can manually trigger the reevaluation of a [MacosToolbarPassthrough] by
/// calling [MacosToolbarPassthroughState.requestUpdate] method from its state,
/// which you may access by using a [GlobalKey] as its `key`.
class MacosToolbarPassthrough extends StatefulWidget {
  const MacosToolbarPassthrough({
    super.key,
    required this.child,
  });

  final Widget child;

  @override
  State<MacosToolbarPassthrough> createState() =>
      MacosToolbarPassthroughState();
}

class MacosToolbarPassthroughState extends State<MacosToolbarPassthrough>
    with WidgetsBindingObserver {
  /// A unique key identifying the content to be measured.
  final GlobalKey _key = GlobalKey();

  static const platform = MethodChannel(kMacosToolbarPassthroughMethodChannel);

  bool _isDisposed = false;
  _DescendantRegistry? _registry;
  Timer? _debounceTimer;
  ScrollableState? _scrollable;
  Offset? _lastPosition;
  Size? _lastSize;

  @override
  void initState() {
    super.initState();
    WidgetsBinding.instance.addObserver(this);
  }

  @override
  void didChangeDependencies() {
    super.didChangeDependencies();

    // Register this instance if scoped
    _registry = _DescendantRegistry.maybeOf(context);
    _registry?.register(this);
    _scrollable = Scrollable.maybeOf(context);
  }

  @override
  void dispose() {
    _sendRemoveMessage();
    _isDisposed = true;
    WidgetsBinding.instance.removeObserver(this);
    _registry?.unregister(_key.toString());
    _debounceTimer?.cancel();
    super.dispose();
  }

  @override
  void didChangeMetrics() => _onChange();

  void _onChange() {
    // If under a parent scope
    if (_registry case _DescendantRegistry registry) {
      // Notify scope of changes
      registry.notifyChanges();
    }
    // If standalone item
    else {
      // Debounce updates to native UI counterpart
      _debounceTimer?.cancel();
      _debounceTimer = Timer(
        _debounceDuration,
        () {
          requestUpdate();
        },
      );
    }
  }

  void requestUpdate() {
    // Immediately update position and size
    _sendUpdateMessage();
    WidgetsBinding.instance.addPostFrameCallback((_) {
      // Wait for next frame to update position and size
      _sendUpdateMessage();
    });
  }

  void _sendUpdateMessage() async {
    if (_isDisposed ||
        !mounted ||
        !context.mounted ||
        _key.currentContext?.mounted != true) {
      return;
    }

    final RenderBox? renderBox =
        _key.currentContext?.findRenderObject() as RenderBox?;

    if (renderBox == null || !renderBox.attached) {
      return;
    }

    Offset? position = renderBox.localToGlobal(Offset.zero);
    Size? size = renderBox.size;

    // If item is child of a scrollable
    if (_scrollable?.position case ScrollPosition scrollPosition) {
      // With the viewport related to the scroll area
      if (RenderAbstractViewport.maybeOf(renderBox) case final viewport?) {
        final double viewportExtent;
        final double sizeExtent;
        switch (scrollPosition.axis) {
          case Axis.horizontal:
            viewportExtent = viewport.paintBounds.width;
            sizeExtent = size.width;
            break;

          case Axis.vertical:
            viewportExtent = viewport.paintBounds.height;
            sizeExtent = size.height;
            break;
        }

        final RevealedOffset viewportOffset = viewport
            .getOffsetToReveal(renderBox, 0.0, axis: scrollPosition.axis);

        // Get viewport deltas
        final double deltaStart = viewportOffset.offset - scrollPosition.pixels;
        final double deltaEnd = deltaStart + sizeExtent;

        // Check if item is within viewport
        final bool isWithinViewport =
            (deltaStart >= 0.0 && deltaStart < viewportExtent) ||
                (deltaEnd > 0.0 && deltaEnd < viewportExtent);

        // If this item is within the scrollable viewport
        if (isWithinViewport) {
          final double startClipped = deltaStart < 0.0 ? -deltaStart : 0.0;
          final double endClipped =
              deltaEnd > viewportExtent ? deltaEnd - viewportExtent : 0.0;

          // Clip overextending content
          switch (scrollPosition.axis) {
            case Axis.horizontal:
              // Clip content is overextending horizontally
              position = position.translate(startClipped, 0.0);
              size = Size(size.width - startClipped - endClipped, size.height);
              break;

            case Axis.vertical:
              // Clip content is overextending vertically
              position = position.translate(0.0, startClipped);
              size = Size(size.width, size.height - startClipped - endClipped);
              break;
          }
        }
        // If this item is not within the scrollable viewport
        else {
          position = null;
          size = null;
        }
      }
    }

    // Update native view if was removed or changed position or size
    if (_lastPosition != position || _lastSize != size) {
      // If item is not within the scrollable viewport
      if (position == null || size == null) {
        _sendRemoveMessage();
      }
      // Update item position and size
      else {
        await platform.invokeMethod('updateView', {
          'id': _key.toString(),
          'x': position.dx,
          'y': position.dy,
          'width': size.width,
          'height': size.height,
        });
      }
      _lastPosition = position;
      _lastSize = size;
    }
  }

  void _sendRemoveMessage() async {
    await platform.invokeMethod('removeView', {
      'id': _key.toString(),
    });
  }

  @override
  Widget build(BuildContext context) {
    return LayoutBuilder(
      builder: (context, constraints) {
        // Triggers evaluation on layout changed event
        _onChange();
        return SizedBox(
          key: _key,
          child: widget.child,
        );
      },
    );
  }
}

main.dart

// ...

  const config = macos.MacosWindowUtilsConfig(
    toolbarStyle: macos.NSWindowToolbarStyle.unified,
  );
  await config.apply();

  if (kIsMacOS) {
    const platform = MethodChannel(kMacosToolbarPassthroughMethodChannel);
    await platform.invokeMethod(
      'initialize',
      kDebugMode
          ? {
              'showDebugLayers': false,
            }
          : null,
    );
  }

// ...

This builds upon improvements made in #43

[Adrian-Samoticha]

This looks great. Do you think it’d be possible to implement this into macos_window_utils in such a way that users wouldn’t need to update their MainFlutterWindow.swift file?

Right now, macos_window_utils is usable without touching any Swift code (unless you’re targeting older macOS versions, see #44). It’d be great if your proposal could be implemented in such a way that this continues to be the case.

[Andre-lbc]

I think so. I am not so familiar with macos_window_utils plugin structure (I usually prototype this way and migrate to a plugin using pigeon), but I believe we can modify MacOSWindowUtilsPlugin.swift and add handles for these methods there (merge both initialize methods and add the equivalent of updateView and removeView, but with better naming).

So, maybe we can edit MacOSWindowUtilsPlugin.swift, adding these changes into its handle method:

• Add updatePassthroughView: Or a better name, with what I've called "updateView" in the code above.
• Add removePassthroughView: Or a better name, with what I've called "removeView" in the code above.
• Modify existing initialize: Maybe add a parameter to enable "passthrough views" (perhaps in place of BlockingToolbar) and adding the code from what I've called "initialize" below the existing initialization code.

Then it is just a matter of organizing the new methods/classes/variables according to the project structure.

[Adrian-Samoticha]

If I understand correctly, your PassthroughViews can be added to any toolbar, so maybe it would indeed be best to add methods for updating and removing them, alongside a parameter to enable their debug colors.

[Andre-lbc]

Yes, they can be added to any toolbar (as far as I tested).

I would also like to share the solutions to some problems I had after upgrading to Sequoia.

The problem is that right clicking on the toolbar area is no longer passed to flutter because it now opens a native menu (i.e.: it consumes the event and does not open my menu implementation on flutter side).

To fix this issue, add the following changes to the code above:

MainFlutterWindow.swift

On method channel setup:

//... Same as before

                case "initialize":
                    // Clean up necessary for flutter hot restart
                   self.toolbarPassthroughViews.removeAll()
                   self.toolbarPassthroughContainer = nil

+                    // Disable toolbar customizations and remove native context menu (right click)
+                    if let toolbar = self.toolbar {
+                        toolbar.allowsUserCustomization = false
+                        toolbar.allowsExtensionItems = false
+                        if #available(macOS 15.0, *) {
+                            toolbar.allowsDisplayModeCustomization = false
+                        }
+                    }
                    
                    #if DEBUG
                      let args = call.arguments as? [String: Any]
                      let showDebugLayers = args?["showDebugLayers"] as? Bool
                      self.hasToolbarDebugLayers = showDebugLayers ?? false
                    #endif
                   
                    // Get the count of accessory view controllers
                    let accessoryCount = self.titlebarAccessoryViewControllers.count
                
                    // Iterate through the indices in reverse order to avoid index shifting
                    for index in stride(from: accessoryCount - 1, through: 0, by: -1) {
                        self.removeTitlebarAccessoryViewController(at: index)
                    }
                  result(nil)

    //... Same as before 

On PassthroughView:

class PassthroughView: NSView {
  var flutterViewController: FlutterViewController?
    
    required init(frame: CGRect, flutterViewController: FlutterViewController) {
        super.init(frame: frame)
        self.flutterViewController = flutterViewController
    }
    
    required init?(coder decoder: NSCoder) {
       fatalError("init(coder:) has not been implemented")
     }
    

    override func mouseDown(with event: NSEvent) {
      flutterViewController!.mouseDown(with: event)
    }

    override func mouseUp(with event: NSEvent) {
      flutterViewController!.mouseUp(with: event)
    }
    
    override var mouseDownCanMoveWindow: Bool {
        return false
    }
    
+    override func rightMouseUp(with event: NSEvent) {
+        flutterViewController!.rightMouseUp(with: event)
+    }
+    
+    override func rightMouseDown(with event: NSEvent) {
+        flutterViewController!.rightMouseDown(with: event)
+    }
    
}

[Adrian-Samoticha]

@Andre-lbc Excuse the delay, I was a little busy lately. You’re right, right clicking on a toolbar in macOS Sequoia seems to open a context menu that allows the user to choose whether toolbar items appear as icons or text. I’d argue that is an issue in and of itself and needs to be fixed as well, so thanks for the heads up.