feat: GPU shared texture offscreen rendering

docs/api/structures/offscreen-shared-texture.md

@@ -0,0 +1,14 @@
+# OffscreenSharedTexture Object
+
+* `textureInfo` Object - The shared texture info.
+  * `widgetType` string - The widget type of the texture, could be `popup` or `frame`.
+  * `pixelFormat` string - The pixel format of the texture, could be `rgba` or `bgra`.
+  * `sharedTextureHandle` string - _Windows_ _macOS_ The handle to the shared texture.
+  * `planes` Object[] - _Linux_ Each plane's info of the shared texture.
+    * `stride` number - The strides and offsets in bytes to be used when accessing the buffers via a memory mapping. One per plane per entry.
+    * `offset` number - The strides and offsets in bytes to be used when accessing the buffers via a memory mapping. One per plane per entry.
+    * `size` number - Size in bytes of the plane. This is necessary to map the buffers.
+    * `fd` number - File descriptor for the underlying memory object (usually dmabuf).
+  * `modifier` string - _Linux_ The modifier is retrieved from GBM library and passed to EGL driver.
+* `release` Function - Release the resources. The `texture` cannot be directly passed to another process, users need to maintain texture lifecycles in
+main process, but it is safe to pass the `textureInfo` to another process.

docs/api/structures/web-preferences.md

@@ -83,6 +83,10 @@
   window. Defaults to `false`. See the
   [offscreen rendering tutorial](../../tutorial/offscreen-rendering.md) for
   more details.
+* `offscreenUseSharedTexture` boolean (optional) - Whether to use GPU shared texture for accelerated
+   paint event. Defaults to `false`. See the
+  [offscreen rendering tutorial](../../tutorial/offscreen-rendering.md) for
+  more details.
 * `contextIsolation` boolean (optional) - Whether to run Electron APIs and
   the specified `preload` script in a separate JavaScript context. Defaults
   to `true`. The context that the `preload` script runs in will only have

docs/api/web-contents.md

@@ -869,19 +869,33 @@ app.whenReady().then(() => {
 
 Returns:
 
-* `event` Event
+* `details` Event\<\>
+  * `texture` [OffscreenSharedTexture](structures/offscreen-shared-texture.md) - The GPU shared texture of the frame, when `webPreferences.offscreenUseSharedTexture` is `true`.
 * `dirtyRect` [Rectangle](structures/rectangle.md)
 * `image` [NativeImage](native-image.md) - The image data of the whole frame.
 
-Emitted when a new frame is generated. Only the dirty area is passed in the
-buffer.
+Emitted when a new frame is generated. Only the dirty area is passed in the buffer.
+
+When using shared texture, it is possible to pass texture to other processes through IPC, or handle the event in async handler.
+It is important to call `texture.release()` when the texture is no longer needed as soon as possible, before the underlying
+frame pool is drained. By managing the lifecycle by yourself, you can safely pass the `texture.textureInfo` to other processes.
 
 ```js
 const { BrowserWindow } = require('electron')
 
-const win = new BrowserWindow({ webPreferences: { offscreen: true } })
-win.webContents.on('paint', (event, dirty, image) => {
-  // updateBitmap(dirty, image.getBitmap())
+const win = new BrowserWindow({ webPreferences: { offscreen: true, offscreenUseSharedTexture: true } })
+win.webContents.on('paint', async (e, dirty, image) => {
+  if (e.texture) {
+    // You can handle the event in async handler
+    await new Promise(resolve => setTimeout(resolve, 50))
+    // importTextureHandle(dirty, e.texture.textureInfo)
+    // You can also pass the `textureInfo` to other processes (not `texture`, the `release` function is not passable)
+    // You have to release the texture at this process when you are done with it
+    e.texture.release()
+  } else {
+    // details.texture will be null when offscreenUseSharedTexture is false
+    // importBitmap(dirty, image.getBitmap())
+  }
 })
 win.loadURL('https://github.com')
 ```

docs/fiddles/features/offscreen-rendering/main.js

@@ -9,13 +9,30 @@ function createWindow () {
     width: 800,
     height: 600,
     webPreferences: {
-      offscreen: true
+      offscreen: true,
+      offscreenUseSharedTexture: true // or false
     }
   })
 
   win.loadURL('https://github.com')
-  win.webContents.on('paint', (event, dirty, image) => {
-    fs.writeFileSync('ex.png', image.toPNG())
+  win.webContents.on('paint', async (event, dirty, image) => {
+    if (event.texture) {
+      // Import the shared texture handle to your own rendering world.
+      // importSharedHandle(event.texture.textureInfo)
+
+      // Example plugin to import shared texture by native addon:
+      // https://github.com/electron/electron/tree/main/spec/fixtures/native-addon/osr-gpu
+
+      // You can handle the event in async handler
+      await new Promise(resolve => setTimeout(resolve, 50))
+
+      // You can also pass the `textureInfo` to other processes (not `texture`, the `release` function is not passable)
+      // You have to release the texture at this process when you are done with it
+      event.texture.release()
+    } else {
+      // texture will be null when `offscreenUseSharedTexture` is false.
+      fs.writeFileSync('ex.png', image.toPNG())
+    }
   })
   win.webContents.setFrameRate(60)
   console.log(`The screenshot has been successfully saved to ${path.join(process.cwd(), 'ex.png')}`)

docs/tutorial/offscreen-rendering.md

@@ -3,7 +3,8 @@
 ## Overview
 
 Offscreen rendering lets you obtain the content of a `BrowserWindow` in a
-bitmap, so it can be rendered anywhere, for example, on texture in a 3D scene.
+bitmap or a shared GPU texture, so it can be rendered anywhere, for example,
+on texture in a 3D scene.
 The offscreen rendering in Electron uses a similar approach to that of the
 [Chromium Embedded Framework](https://bitbucket.org/chromiumembedded/cef)
 project.
@@ -23,16 +24,28 @@ losses with no benefits.
 
 #### GPU accelerated
 
-GPU accelerated rendering means that the GPU is used for composition. Because of
-that, the frame has to be copied from the GPU which requires more resources,
-thus this mode is slower than the Software output device. The benefit of this
-mode is that WebGL and 3D CSS animations are supported.
+GPU accelerated rendering means that the GPU is used for composition. The benefit
+of this mode is that WebGL and 3D CSS animations are supported. There're two
+different approaches depends on whether `webPreferences.offscreenUseSharedTexture`
+is set to true.
+
+1. Use GPU shared texture
+
+    The frame are directly copied in GPU textures, thus this mode is very fast because
+    there's no CPU-GPU memory copies overhead and you can directly import the shared
+    texture to your own rendering program. The texture is passed in `texture` param of
+    `paint` event.
+
+2. Use CPU shared memory bitmap
+
+    The frame has to be copied from the GPU to the CPU bitmap which requires more
+    resources, thus this mode is slower than the Software output device. But is supports
+    GPU related functionalities.
 
 #### Software output device
 
 This mode uses a software output device for rendering in the CPU, so the frame
-generation is much faster. As a result, this mode is preferred over the GPU
-accelerated one.
+generation is faster than shared memory GPU accelerated mode.
 
 To enable this mode, GPU acceleration has to be disabled by calling the
 [`app.disableHardwareAcceleration()`][disablehardwareacceleration] API.
@@ -51,13 +64,30 @@ function createWindow () {
     width: 800,
     height: 600,
     webPreferences: {
-      offscreen: true
+      offscreen: true,
+      offscreenUseSharedTexture: true // or false
     }
   })
 
   win.loadURL('https://github.com')
-  win.webContents.on('paint', (event, dirty, image) => {
-    fs.writeFileSync('ex.png', image.toPNG())
+  win.webContents.on('paint', async (event, dirty, image) => {
+    if (event.texture) {
+      // Import the shared texture handle to your own rendering world.
+      // importSharedHandle(event.texture.textureInfo)
+
+      // Example plugin to import shared texture by native addon:
+      // https://github.com/electron/electron/tree/main/spec/fixtures/native-addon/osr-gpu
+
+      // You can handle the event in async handler
+      await new Promise(resolve => setTimeout(resolve, 50))
+
+      // You can also pass the `textureInfo` to other processes (not `texture`, the `release` function is not passable)
+      // You have to release the texture at this process when you are done with it
+      event.texture.release()
+    } else {
+      // texture will be null when `offscreenUseSharedTexture` is false.
+      fs.writeFileSync('ex.png', image.toPNG())
+    }
   })
   win.webContents.setFrameRate(60)
   console.log(`The screenshot has been successfully saved to ${path.join(process.cwd(), 'ex.png')}`)

filenames.auto.gni

@@ -107,6 +107,7 @@ auto_filenames = {
     "docs/api/structures/mouse-wheel-input-event.md",
     "docs/api/structures/notification-action.md",
     "docs/api/structures/notification-response.md",
+    "docs/api/structures/offscreen-shared-texture.md",
     "docs/api/structures/open-external-permission-request.md",
     "docs/api/structures/payment-discount.md",
     "docs/api/structures/permission-request.md",

filenames.gni

@@ -455,6 +455,8 @@ filenames = {
     "shell/browser/notifications/platform_notification_service.h",
     "shell/browser/osr/osr_host_display_client.cc",
     "shell/browser/osr/osr_host_display_client.h",
+    "shell/browser/osr/osr_paint_event.cc",
+    "shell/browser/osr/osr_paint_event.h",
     "shell/browser/osr/osr_render_widget_host_view.cc",
     "shell/browser/osr/osr_render_widget_host_view.h",
     "shell/browser/osr/osr_video_consumer.cc",
@@ -601,6 +603,8 @@ filenames = {
     "shell/common/gin_converters/net_converter.cc",
     "shell/common/gin_converters/net_converter.h",
     "shell/common/gin_converters/optional_converter.h",
+    "shell/common/gin_converters/osr_converter.cc",
+    "shell/common/gin_converters/osr_converter.h",
     "shell/common/gin_converters/serial_port_info_converter.h",
     "shell/common/gin_converters/std_converter.h",
     "shell/common/gin_converters/time_converter.cc",

patches/chromium/.patches

@@ -130,3 +130,4 @@ fix_font_face_resolution_when_renderer_is_blocked.patch
 feat_enable_passing_exit_code_on_service_process_crash.patch
 chore_remove_reference_to_chrome_browser_themes.patch
 feat_enable_customizing_symbol_color_in_framecaptionbutton.patch
+osr_shared_texture_remove_keyed_mutex_on_win_dxgi.patch

patches/chromium/osr_shared_texture_remove_keyed_mutex_on_win_dxgi.patch

@@ -0,0 +1,85 @@
+From 0000000000000000000000000000000000000000 Mon Sep 17 00:00:00 2001
+From: Reito <[email protected]>
+Date: Fri, 19 Apr 2024 11:46:47 +0800
+Subject: Remove DXGI GMB keyed-mutex
+
+This CL is only for reference for a vendor specific change. It is kept for rebasing.
+
+On CEF/electron, FrameSinkVideoCapturer is used in GPU accelerated OSR, and it can be faster
+if we remove the mutex on the shared resource for Windows.
+
+Maintain this patch in https://crrev.com/c/5465148
+
+Change-Id: Id90acd587f17acd228ae7cb5ef93005eb8388b80
+
+diff --git a/gpu/ipc/service/gpu_memory_buffer_factory_dxgi.cc b/gpu/ipc/service/gpu_memory_buffer_factory_dxgi.cc
+index 58df3e9ad67b2b7835fe64835f9e589a563ee073..d08d4611361fc1387e94f553ba88c20a8cd87741 100644
+--- a/gpu/ipc/service/gpu_memory_buffer_factory_dxgi.cc
++++ b/gpu/ipc/service/gpu_memory_buffer_factory_dxgi.cc
+@@ -180,7 +180,8 @@ gfx::GpuMemoryBufferHandle GpuMemoryBufferFactoryDXGI::CreateGpuMemoryBuffer(
+   // so make sure that the usage is one that we support.
+   DCHECK(usage == gfx::BufferUsage::GPU_READ ||
+          usage == gfx::BufferUsage::SCANOUT ||
+-         usage == gfx::BufferUsage::SCANOUT_CPU_READ_WRITE)
++         usage == gfx::BufferUsage::SCANOUT_CPU_READ_WRITE ||
++         usage == gfx::BufferUsage::SCANOUT_VEA_CPU_READ)
+       << "Incorrect usage, usage=" << gfx::BufferUsageToString(usage);
+
+   D3D11_TEXTURE2D_DESC desc = {
+@@ -194,7 +195,9 @@ gfx::GpuMemoryBufferHandle GpuMemoryBufferFactoryDXGI::CreateGpuMemoryBuffer(
+       D3D11_BIND_SHADER_RESOURCE | D3D11_BIND_RENDER_TARGET,
+       0,
+       D3D11_RESOURCE_MISC_SHARED_NTHANDLE |
+-          D3D11_RESOURCE_MISC_SHARED_KEYEDMUTEX};
++          static_cast<UINT>(usage == gfx::BufferUsage::SCANOUT_VEA_CPU_READ
++                                ? D3D11_RESOURCE_MISC_SHARED
++                                : D3D11_RESOURCE_MISC_SHARED_KEYEDMUTEX)};
+
+   Microsoft::WRL::ComPtr<ID3D11Texture2D> d3d11_texture;
+
+diff --git a/media/video/renderable_gpu_memory_buffer_video_frame_pool.cc b/media/video/renderable_gpu_memory_buffer_video_frame_pool.cc
+index c007238f1c7392042ae91a82f3d70751ba258422..0a65f20f65ed1d7cc140d0a54042fc4a41138774 100644
+--- a/media/video/renderable_gpu_memory_buffer_video_frame_pool.cc
++++ b/media/video/renderable_gpu_memory_buffer_video_frame_pool.cc
+@@ -198,7 +198,7 @@ gfx::Size GetBufferSizeInPixelsForVideoPixelFormat(
+ bool FrameResources::Initialize() {
+   auto* context = pool_->GetContext();
+
+-  constexpr gfx::BufferUsage kBufferUsage =
++  gfx::BufferUsage buffer_usage =
+ #if BUILDFLAG(IS_MAC) || BUILDFLAG(IS_CHROMEOS)
+       gfx::BufferUsage::SCANOUT_VEA_CPU_READ
+ #else
+@@ -212,13 +212,30 @@ bool FrameResources::Initialize() {
+   const gfx::Size buffer_size_in_pixels =
+       GetBufferSizeInPixelsForVideoPixelFormat(format_, coded_size_);
+
++#if BUILDFLAG(IS_WIN)
++  // For CEF OSR feature, currently there's no other place in chromium use RGBA.
++  // If the format is RGBA, currently CEF do not write to the texture anymore
++  // once the GMB is returned from CopyRequest. So there will be no race
++  // condition on that texture. We can request a GMB without a keyed mutex to
++  // accelerate and probably prevent some driver deadlock.
++  if (format_ == PIXEL_FORMAT_ARGB || format_ == PIXEL_FORMAT_ABGR) {
++    // This value is 'borrowed', SCANOUT_VEA_CPU_READ is probably invalid
++    // cause there's no real SCANOUT on Windows. We simply use this enum as a
++    // flag to disable mutex in the GMBFactoryDXGI because this enum is also
++    // used above in macOS and CrOS for similar usage (claim no other one will
++    // concurrently use the resource).
++    // https://chromium-review.googlesource.com/c/chromium/src/+/5302103
++    buffer_usage = gfx::BufferUsage::SCANOUT_VEA_CPU_READ;
++  }
++#endif
++
+   // Create the GpuMemoryBuffer.
+   gpu_memory_buffer_ = context->CreateGpuMemoryBuffer(
+-      buffer_size_in_pixels, buffer_format, kBufferUsage);
++      buffer_size_in_pixels, buffer_format, buffer_usage);
+   if (!gpu_memory_buffer_) {
+     DLOG(ERROR) << "Failed to allocate GpuMemoryBuffer for frame: coded_size="
+                 << coded_size_.ToString()
+-                << ", usage=" << static_cast<int>(kBufferUsage);
++                << ", usage=" << static_cast<int>(buffer_usage);
+     return false;
+   }
+

shell/browser/api/electron_api_web_contents.cc

@@ -120,6 +120,7 @@
 #include "shell/common/gin_converters/image_converter.h"
 #include "shell/common/gin_converters/net_converter.h"
 #include "shell/common/gin_converters/optional_converter.h"
+#include "shell/common/gin_converters/osr_converter.h"
 #include "shell/common/gin_converters/value_converter.h"
 #include "shell/common/gin_helper/dictionary.h"
 #include "shell/common/gin_helper/object_template_builder.h"
@@ -767,6 +768,9 @@ WebContents::WebContents(v8::Isolate* isolate,
   if (options.Get(options::kOffscreen, &b) && b)
     type_ = Type::kOffScreen;
 
+  if (options.Get(options::kOffscreenUseSharedTexture, &b))
+    offscreen_use_shared_texture_ = b;
+
   // Init embedder earlier
   options.Get("embedder", &embedder_);
 
@@ -801,7 +805,7 @@ WebContents::WebContents(v8::Isolate* isolate,
 
     if (embedder_ && embedder_->IsOffScreen()) {
       auto* view = new OffScreenWebContentsView(
-          false,
+          false, offscreen_use_shared_texture_,
           base::BindRepeating(&WebContents::OnPaint, base::Unretained(this)));
       params.view = view;
       params.delegate_view = view;
@@ -821,7 +825,7 @@ WebContents::WebContents(v8::Isolate* isolate,
 
     content::WebContents::CreateParams params(session->browser_context());
     auto* view = new OffScreenWebContentsView(
-        transparent,
+        transparent, offscreen_use_shared_texture_,
         base::BindRepeating(&WebContents::OnPaint, base::Unretained(this)));
     params.view = view;
     params.delegate_view = view;
@@ -3525,8 +3529,21 @@ bool WebContents::IsOffScreen() const {
   return type_ == Type::kOffScreen;
 }
 
-void WebContents::OnPaint(const gfx::Rect& dirty_rect, const SkBitmap& bitmap) {
-  Emit("paint", dirty_rect, gfx::Image::CreateFrom1xBitmap(bitmap));
+void WebContents::OnPaint(const gfx::Rect& dirty_rect,
+                          const SkBitmap& bitmap,
+                          const OffscreenSharedTexture& tex) {
+  v8::Isolate* isolate = JavascriptEnvironment::GetIsolate();
+  v8::HandleScope handle_scope(isolate);
+
+  gin::Handle<gin_helper::internal::Event> event =
+      gin_helper::internal::Event::New(isolate);
+  v8::Local<v8::Object> event_object = event.ToV8().As<v8::Object>();
+
+  gin_helper::Dictionary dict(isolate, event_object);
+  dict.Set("texture", tex);
+
+  EmitWithoutEvent("paint", event, dirty_rect,
+                   gfx::Image::CreateFrom1xBitmap(bitmap));
 }
 
 void WebContents::StartPainting() {

shell/browser/api/electron_api_web_contents.h

@@ -43,6 +43,7 @@
 #include "shell/browser/background_throttling_source.h"
 #include "shell/browser/event_emitter_mixin.h"
 #include "shell/browser/extended_web_contents_observer.h"
+#include "shell/browser/osr/osr_paint_event.h"
 #include "shell/browser/ui/inspectable_web_contents.h"
 #include "shell/browser/ui/inspectable_web_contents_delegate.h"
 #include "shell/browser/ui/inspectable_web_contents_view_delegate.h"
@@ -302,7 +303,9 @@ class WebContents : public ExclusiveAccessContext,
 
   // Methods for offscreen rendering
   bool IsOffScreen() const;
-  void OnPaint(const gfx::Rect& dirty_rect, const SkBitmap& bitmap);
+  void OnPaint(const gfx::Rect& dirty_rect,
+               const SkBitmap& bitmap,
+               const OffscreenSharedTexture& info);
   void StartPainting();
   void StopPainting();
   bool IsPainting() const;
@@ -832,6 +835,9 @@ class WebContents : public ExclusiveAccessContext,
 
   bool offscreen_ = false;
 
+  // Whether offscreen rendering use gpu shared texture
+  bool offscreen_use_shared_texture_ = false;
+
   // Whether window is fullscreened by HTML5 api.
   bool html_fullscreen_ = false;
 

shell/browser/osr/osr_host_display_client.cc

@@ -70,7 +70,7 @@ void LayeredWindowUpdater::Draw(const gfx::Rect& damage_rect,
 
   if (active_ && canvas_->peekPixels(&pixmap)) {
     bitmap.installPixels(pixmap);
-    callback_.Run(damage_rect, bitmap);
+    callback_.Run(damage_rect, bitmap, {});
   }
 
   std::move(draw_callback).Run();