feat: GPU shared texture offscreen rendering

Author: reitowo

docs/api/structures/offscreen-shared-texture.md

@@ -0,0 +1,10 @@
+# OffscreenSharedTexture Object
+
+* `pixelFormat` string - The pixel format of the texture, could be RGBA or BGRA.
+* `sharedTextureHandle` string - _macOS_ _Windows_ The handle to the shared texture.
+* `planes` Object[] - _Linux_ Each plane's info of the shared texture.
+  * `stride` number - The strides and offsets in bytes to be used when accessing the buffers via a memory mapping. One per plane per entry.
+  * `offset` number - The strides and offsets in bytes to be used when accessing the buffers via a memory mapping. One per plane per entry.
+  * `size` number - Size in bytes of the plane. This is necessary to map the buffers.
+  * `fd` number - File descriptor for the underlying memory object (usually dmabuf).
+* `modifier` string - _Linux_ The modifier is retrieved from GBM library and passed to EGL driver.

docs/api/structures/web-preferences.md

@@ -83,6 +83,10 @@
   window. Defaults to `false`. See the
   [offscreen rendering tutorial](../../tutorial/offscreen-rendering.md) for
   more details.
+* `offscreenUseSharedTexture` boolean (optional) - Whether to use GPU shared texture for accelerated
+   paint event. Defaults to `false`. See the
+  [offscreen rendering tutorial](../../tutorial/offscreen-rendering.md) for
+  more details.
 * `contextIsolation` boolean (optional) - Whether to run Electron APIs and
   the specified `preload` script in a separate JavaScript context. Defaults
   to `true`. The context that the `preload` script runs in will only have

docs/api/web-contents.md

@@ -874,6 +874,7 @@ Returns:
 * `event` Event
 * `dirtyRect` [Rectangle](structures/rectangle.md)
 * `image` [NativeImage](native-image.md) - The image data of the whole frame.
+* `texture` [OffscreenSharedTexture](structures/offscreen-shared-texture.md) - The GPU shared texture of the frame. `null` if `webPreferences.offscreenUseSharedTexture` is `false`.
 
 Emitted when a new frame is generated. Only the dirty area is passed in the
 buffer.
@@ -882,7 +883,7 @@ buffer.
 const { BrowserWindow } = require('electron')
 
 const win = new BrowserWindow({ webPreferences: { offscreen: true } })
-win.webContents.on('paint', (event, dirty, image) => {
+win.webContents.on('paint', (event, dirty, image, texture) => {
   // updateBitmap(dirty, image.getBitmap())
 })
 win.loadURL('https://github.com')

docs/fiddles/features/offscreen-rendering/main.js

@@ -9,13 +9,19 @@ function createWindow () {
     width: 800,
     height: 600,
     webPreferences: {
-      offscreen: true
+      offscreen: true,
+      offscreenUseSharedTexture: true // or false
     }
   })
 
   win.loadURL('https://github.com')
-  win.webContents.on('paint', (event, dirty, image) => {
-    fs.writeFileSync('ex.png', image.toPNG())
+  win.webContents.on('paint', (event, dirty, image, texture) => {
+    if (texture) {
+      // This will be not null when `offscreenUseSharedTexture` is true.
+      // Import the shared texture handle to your own rendering world.
+    } else {
+      fs.writeFileSync('ex.png', image.toPNG())
+    }
   })
   win.webContents.setFrameRate(60)
   console.log(`The screenshot has been successfully saved to ${path.join(process.cwd(), 'ex.png')}`)

docs/tutorial/offscreen-rendering.md

@@ -3,7 +3,8 @@
 ## Overview
 
 Offscreen rendering lets you obtain the content of a `BrowserWindow` in a
-bitmap, so it can be rendered anywhere, for example, on texture in a 3D scene.
+bitmap or a shared GPU texture, so it can be rendered anywhere, for example,
+on texture in a 3D scene.
 The offscreen rendering in Electron uses a similar approach to that of the
 [Chromium Embedded Framework](https://bitbucket.org/chromiumembedded/cef)
 project.
@@ -23,10 +24,22 @@ losses with no benefits.
 
 #### GPU accelerated
 
-GPU accelerated rendering means that the GPU is used for composition. Because of
-that, the frame has to be copied from the GPU which requires more resources,
-thus this mode is slower than the Software output device. The benefit of this
-mode is that WebGL and 3D CSS animations are supported.
+GPU accelerated rendering means that the GPU is used for composition. The benefit
+of this mode is that WebGL and 3D CSS animations are supported. There're two
+different approaches depends on whether `webPreferences.offscreenUseSharedTexture`
+is set to true.
+
+1. Use shared texture
+
+    The frame are directly copied in GPU textures, thus this mode is very fast because
+    there's no CPU-GPU memory copies overhead and you can directly import the shared
+    texture to your own rendering program. The texture is passed in `texture` param of
+    `paint` event.
+
+2. Use shared memory bitmap
+
+    The frame has to be copied from the GPU to the CPU bitmap which requires more
+    resources, thus this mode is slower than the Software output device.
 
 #### Software output device
 
@@ -51,13 +64,19 @@ function createWindow () {
     width: 800,
     height: 600,
     webPreferences: {
-      offscreen: true
+      offscreen: true,
+      offscreenUseSharedTexture: true // or false
     }
   })
 
   win.loadURL('https://github.com')
-  win.webContents.on('paint', (event, dirty, image) => {
-    fs.writeFileSync('ex.png', image.toPNG())
+  win.webContents.on('paint', (event, dirty, image, texture) => {
+    if (texture) {
+      // This will be not null when `offscreenUseSharedTexture` is true.
+      // Import the shared texture handle to your own rendering world.
+    } else {
+      fs.writeFileSync('ex.png', image.toPNG())
+    }
   })
   win.webContents.setFrameRate(60)
   console.log(`The screenshot has been successfully saved to ${path.join(process.cwd(), 'ex.png')}`)

filenames.auto.gni

@@ -108,6 +108,7 @@ auto_filenames = {
     "docs/api/structures/mouse-wheel-input-event.md",
     "docs/api/structures/notification-action.md",
     "docs/api/structures/notification-response.md",
+    "docs/api/structures/offscreen-shared-texture.md",
     "docs/api/structures/open-external-permission-request.md",
     "docs/api/structures/payment-discount.md",
     "docs/api/structures/permission-request.md",

filenames.gni

@@ -451,6 +451,8 @@ filenames = {
     "shell/browser/notifications/platform_notification_service.h",
     "shell/browser/osr/osr_host_display_client.cc",
     "shell/browser/osr/osr_host_display_client.h",
+    "shell/browser/osr/osr_paint_event.cc",
+    "shell/browser/osr/osr_paint_event.h",
     "shell/browser/osr/osr_render_widget_host_view.cc",
     "shell/browser/osr/osr_render_widget_host_view.h",
     "shell/browser/osr/osr_video_consumer.cc",
@@ -597,6 +599,8 @@ filenames = {
     "shell/common/gin_converters/net_converter.cc",
     "shell/common/gin_converters/net_converter.h",
     "shell/common/gin_converters/optional_converter.h",
+    "shell/common/gin_converters/osr_converter.cc",
+    "shell/common/gin_converters/osr_converter.h",
     "shell/common/gin_converters/serial_port_info_converter.h",
     "shell/common/gin_converters/std_converter.h",
     "shell/common/gin_converters/time_converter.cc",

shell/browser/api/electron_api_web_contents.cc

@@ -119,6 +119,7 @@
 #include "shell/common/gin_converters/image_converter.h"
 #include "shell/common/gin_converters/net_converter.h"
 #include "shell/common/gin_converters/optional_converter.h"
+#include "shell/common/gin_converters/osr_converter.h"
 #include "shell/common/gin_converters/value_converter.h"
 #include "shell/common/gin_helper/dictionary.h"
 #include "shell/common/gin_helper/object_template_builder.h"
@@ -753,6 +754,9 @@ WebContents::WebContents(v8::Isolate* isolate,
   if (options.Get(options::kOffscreen, &b) && b)
     type_ = Type::kOffScreen;
 
+  if (options.Get(options::kOffscreenUseSharedTexture, &b))
+    offscreen_use_shared_texture_ = b;
+
   // Init embedder earlier
   options.Get("embedder", &embedder_);
 
@@ -787,7 +791,7 @@ WebContents::WebContents(v8::Isolate* isolate,
 
     if (embedder_ && embedder_->IsOffScreen()) {
       auto* view = new OffScreenWebContentsView(
-          false,
+          false, offscreen_use_shared_texture_,
           base::BindRepeating(&WebContents::OnPaint, base::Unretained(this)));
       params.view = view;
       params.delegate_view = view;
@@ -807,7 +811,7 @@ WebContents::WebContents(v8::Isolate* isolate,
 
     content::WebContents::CreateParams params(session->browser_context());
     auto* view = new OffScreenWebContentsView(
-        transparent,
+        transparent, offscreen_use_shared_texture_,
         base::BindRepeating(&WebContents::OnPaint, base::Unretained(this)));
     params.view = view;
     params.delegate_view = view;
@@ -3509,8 +3513,10 @@ bool WebContents::IsOffScreen() const {
   return type_ == Type::kOffScreen;
 }
 
-void WebContents::OnPaint(const gfx::Rect& dirty_rect, const SkBitmap& bitmap) {
-  Emit("paint", dirty_rect, gfx::Image::CreateFrom1xBitmap(bitmap));
+void WebContents::OnPaint(const gfx::Rect& dirty_rect,
+                          const SkBitmap& bitmap,
+                          const OffscreenSharedTexture& tex) {
+  Emit("paint", dirty_rect, gfx::Image::CreateFrom1xBitmap(bitmap), tex);
 }
 
 void WebContents::StartPainting() {

shell/browser/api/electron_api_web_contents.h

@@ -43,6 +43,7 @@
 #include "shell/browser/background_throttling_source.h"
 #include "shell/browser/event_emitter_mixin.h"
 #include "shell/browser/extended_web_contents_observer.h"
+#include "shell/browser/osr/osr_paint_event.h"
 #include "shell/browser/ui/inspectable_web_contents.h"
 #include "shell/browser/ui/inspectable_web_contents_delegate.h"
 #include "shell/browser/ui/inspectable_web_contents_view_delegate.h"
@@ -300,7 +301,9 @@ class WebContents : public ExclusiveAccessContext,
 
   // Methods for offscreen rendering
   bool IsOffScreen() const;
-  void OnPaint(const gfx::Rect& dirty_rect, const SkBitmap& bitmap);
+  void OnPaint(const gfx::Rect& dirty_rect,
+               const SkBitmap& bitmap,
+               const OffscreenSharedTexture& info);
   void StartPainting();
   void StopPainting();
   bool IsPainting() const;
@@ -833,6 +836,9 @@ class WebContents : public ExclusiveAccessContext,
 
   bool offscreen_ = false;
 
+  // Whether offscreen rendering use gpu shared texture
+  bool offscreen_use_shared_texture_ = false;
+
   // Whether window is fullscreened by HTML5 api.
   bool html_fullscreen_ = false;
 

shell/browser/osr/osr_host_display_client.cc

@@ -70,7 +70,7 @@ void LayeredWindowUpdater::Draw(const gfx::Rect& damage_rect,
 
   if (active_ && canvas_->peekPixels(&pixmap)) {
     bitmap.installPixels(pixmap);
-    callback_.Run(damage_rect, bitmap);
+    callback_.Run(damage_rect, bitmap, {});
   }
 
   std::move(draw_callback).Run();