feat: GPU shared texture offscreen rendering

docs/api/structures/offscreen-shared-texture.md

@@ -0,0 +1,11 @@
+# OffscreenSharedTexture Object
+
+* `widgetType` string - The widget type of the texture, could be `popup` or `frame`.
+* `pixelFormat` string - The pixel format of the texture, could be `rgba` or `bgra`.
+* `sharedTextureHandle` string - _macOS_ _Windows_ The handle to the shared texture.
+* `planes` Object[] - _Linux_ Each plane's info of the shared texture.
+ * `stride` number - The strides and offsets in bytes to be used when accessing the buffers via a memory mapping. One per plane per entry.
+ * `offset` number - The strides and offsets in bytes to be used when accessing the buffers via a memory mapping. One per plane per entry.
+ * `size` number - Size in bytes of the plane. This is necessary to map the buffers.
+ * `fd` number - File descriptor for the underlying memory object (usually dmabuf).
+* `modifier` string - _Linux_ The modifier is retrieved from GBM library and passed to EGL driver.

docs/api/structures/web-preferences.md

@@ -83,6 +83,10 @@
  window. Defaults to `false`. See the
  [offscreen rendering tutorial](../../tutorial/offscreen-rendering.md) for
  more details.
+* `offscreenUseSharedTexture` boolean (optional) - Whether to use GPU shared texture for accelerated
+ paint event. Defaults to `false`. See the
+ [offscreen rendering tutorial](../../tutorial/offscreen-rendering.md) for
+ more details.
 * `contextIsolation` boolean (optional) - Whether to run Electron APIs and
  the specified `preload` script in a separate JavaScript context. Defaults
  to `true`. The context that the `preload` script runs in will only have

docs/api/web-contents.md

@@ -874,16 +874,22 @@ Returns:
 * `event` Event
 * `dirtyRect` [Rectangle](structures/rectangle.md)
 * `image` [NativeImage](native-image.md) - The image data of the whole frame.
+* `texture` [OffscreenSharedTexture](structures/offscreen-shared-texture.md) - The GPU shared texture of the frame. `null` if `webPreferences.offscreenUseSharedTexture` is `false`.
 
 Emitted when a new frame is generated. Only the dirty area is passed in the
 buffer.
 
 ```js
 const { BrowserWindow } = require('electron')
 
-const win = new BrowserWindow({ webPreferences: { offscreen: true } })
-win.webContents.on('paint', (event, dirty, image) => {
- // updateBitmap(dirty, image.getBitmap())
+const win = new BrowserWindow({ webPreferences: { offscreen: true, offscreenUseSharedTexture: true } })
+win.webContents.on('paint', (event, dirty, image, texture) => {
+ if (texture) {
+ // importTextureHandle(dirty, texture)
+ } else {
+ // texture will be null when offscreenUseSharedTexture is false
+ // updateBitmap(dirty, image.getBitmap())
+ }
 })
 win.loadURL('https://github.com')
 ```

docs/fiddles/features/offscreen-rendering/main.js

@@ -9,13 +9,19 @@ function createWindow () {
  width: 800,
  height: 600,
  webPreferences: {
- offscreen: true
+ offscreen: true,
+ offscreenUseSharedTexture: true // or false
  }
  })
 
  win.loadURL('https://github.com')
- win.webContents.on('paint', (event, dirty, image) => {
- fs.writeFileSync('ex.png', image.toPNG())
+ win.webContents.on('paint', (event, dirty, image, texture) => {
+ if (texture) {
+ // Import the shared texture handle to your own rendering world.
+ } else {
+ // texture will be null when `offscreenUseSharedTexture` is false.
+ fs.writeFileSync('ex.png', image.toPNG())
+ }
  })
  win.webContents.setFrameRate(60)
  console.log(`The screenshot has been successfully saved to ${path.join(process.cwd(), 'ex.png')}`)

docs/tutorial/offscreen-rendering.md

@@ -3,7 +3,8 @@
 ## Overview
 
 Offscreen rendering lets you obtain the content of a `BrowserWindow` in a
-bitmap, so it can be rendered anywhere, for example, on texture in a 3D scene.
+bitmap or a shared GPU texture, so it can be rendered anywhere, for example,
+on texture in a 3D scene.
 The offscreen rendering in Electron uses a similar approach to that of the
 [Chromium Embedded Framework](https://bitbucket.org/chromiumembedded/cef)
 project.
@@ -23,10 +24,22 @@ losses with no benefits.
 
 #### GPU accelerated
 
-GPU accelerated rendering means that the GPU is used for composition. Because of
-that, the frame has to be copied from the GPU which requires more resources,
-thus this mode is slower than the Software output device. The benefit of this
-mode is that WebGL and 3D CSS animations are supported.
+GPU accelerated rendering means that the GPU is used for composition. The benefit
+of this mode is that WebGL and 3D CSS animations are supported. There're two
+different approaches depends on whether `webPreferences.offscreenUseSharedTexture`
+is set to true.
+
+1. Use shared texture
+
+ The frame are directly copied in GPU textures, thus this mode is very fast because
+ there's no CPU-GPU memory copies overhead and you can directly import the shared
+ texture to your own rendering program. The texture is passed in `texture` param of
+ `paint` event.
+
+2. Use shared memory bitmap
+
+ The frame has to be copied from the GPU to the CPU bitmap which requires more
+ resources, thus this mode is slower than the Software output device.
 
 #### Software output device
 
@@ -51,13 +64,19 @@ function createWindow () {
  width: 800,
  height: 600,
  webPreferences: {
- offscreen: true
+ offscreen: true,
+ offscreenUseSharedTexture: true // or false
  }
  })
 
  win.loadURL('https://github.com')
- win.webContents.on('paint', (event, dirty, image) => {
- fs.writeFileSync('ex.png', image.toPNG())
+ win.webContents.on('paint', (event, dirty, image, texture) => {
+ if (texture) {
+ // Import the shared texture handle to your own rendering world.
+ } else {
+ // texture will be null when `offscreenUseSharedTexture` is false.
+ fs.writeFileSync('ex.png', image.toPNG())
+ }
  })
  win.webContents.setFrameRate(60)
  console.log(`The screenshot has been successfully saved to ${path.join(process.cwd(), 'ex.png')}`)

filenames.auto.gni

@@ -93,6 +93,7 @@ auto_filenames = {
  "docs/api/structures/gpu-feature-status.md",
  "docs/api/structures/hid-device.md",
  "docs/api/structures/input-event.md",
+ "docs/api/structures/io-counters.md",
  "docs/api/structures/ipc-main-event.md",
  "docs/api/structures/ipc-main-invoke-event.md",
  "docs/api/structures/ipc-renderer-event.md",
@@ -108,6 +109,7 @@ auto_filenames = {
  "docs/api/structures/mouse-wheel-input-event.md",
  "docs/api/structures/notification-action.md",
  "docs/api/structures/notification-response.md",
+ "docs/api/structures/offscreen-shared-texture.md",
  "docs/api/structures/open-external-permission-request.md",
  "docs/api/structures/payment-discount.md",
  "docs/api/structures/permission-request.md",

filenames.gni

@@ -451,6 +451,8 @@ filenames = {
  "shell/browser/notifications/platform_notification_service.h",
  "shell/browser/osr/osr_host_display_client.cc",
  "shell/browser/osr/osr_host_display_client.h",
+ "shell/browser/osr/osr_paint_event.cc",
+ "shell/browser/osr/osr_paint_event.h",
  "shell/browser/osr/osr_render_widget_host_view.cc",
  "shell/browser/osr/osr_render_widget_host_view.h",
  "shell/browser/osr/osr_video_consumer.cc",
@@ -597,6 +599,8 @@ filenames = {
  "shell/common/gin_converters/net_converter.cc",
  "shell/common/gin_converters/net_converter.h",
  "shell/common/gin_converters/optional_converter.h",
+ "shell/common/gin_converters/osr_converter.cc",
+ "shell/common/gin_converters/osr_converter.h",
  "shell/common/gin_converters/serial_port_info_converter.h",
  "shell/common/gin_converters/std_converter.h",
  "shell/common/gin_converters/time_converter.cc",

shell/browser/api/electron_api_web_contents.cc

@@ -119,6 +119,7 @@
 #include "shell/common/gin_converters/image_converter.h"
 #include "shell/common/gin_converters/net_converter.h"
 #include "shell/common/gin_converters/optional_converter.h"
+#include "shell/common/gin_converters/osr_converter.h"
 #include "shell/common/gin_converters/value_converter.h"
 #include "shell/common/gin_helper/dictionary.h"
 #include "shell/common/gin_helper/object_template_builder.h"
@@ -753,6 +754,9 @@ WebContents::WebContents(v8::Isolate* isolate,
  if (options.Get(options::kOffscreen, &b) && b)
  type_ = Type::kOffScreen;
 
+ if (options.Get(options::kOffscreenUseSharedTexture, &b))
+ offscreen_use_shared_texture_ = b;
+
  // Init embedder earlier
  options.Get("embedder", &embedder_);
 
@@ -787,7 +791,7 @@ WebContents::WebContents(v8::Isolate* isolate,
 
  if (embedder_ && embedder_->IsOffScreen()) {
  auto* view = new OffScreenWebContentsView(
- false,
+ false, offscreen_use_shared_texture_,
  base::BindRepeating(&WebContents::OnPaint, base::Unretained(this)));
  params.view = view;
  params.delegate_view = view;
@@ -807,7 +811,7 @@ WebContents::WebContents(v8::Isolate* isolate,
 
  content::WebContents::CreateParams params(session->browser_context());
  auto* view = new OffScreenWebContentsView(
- transparent,
+ transparent, offscreen_use_shared_texture_,
  base::BindRepeating(&WebContents::OnPaint, base::Unretained(this)));
  params.view = view;
  params.delegate_view = view;
@@ -3509,8 +3513,10 @@ bool WebContents::IsOffScreen() const {
  return type_ == Type::kOffScreen;
 }
 
-void WebContents::OnPaint(const gfx::Rect& dirty_rect, const SkBitmap& bitmap) {
- Emit("paint", dirty_rect, gfx::Image::CreateFrom1xBitmap(bitmap));
+void WebContents::OnPaint(const gfx::Rect& dirty_rect,
+ const SkBitmap& bitmap,
+ const OffscreenSharedTexture& tex) {
+ Emit("paint", dirty_rect, gfx::Image::CreateFrom1xBitmap(bitmap), tex);
 }
 
 void WebContents::StartPainting() {

shell/browser/api/electron_api_web_contents.h

@@ -43,6 +43,7 @@
 #include "shell/browser/background_throttling_source.h"
 #include "shell/browser/event_emitter_mixin.h"
 #include "shell/browser/extended_web_contents_observer.h"
+#include "shell/browser/osr/osr_paint_event.h"
 #include "shell/browser/ui/inspectable_web_contents.h"
 #include "shell/browser/ui/inspectable_web_contents_delegate.h"
 #include "shell/browser/ui/inspectable_web_contents_view_delegate.h"
@@ -300,7 +301,9 @@ class WebContents : public ExclusiveAccessContext,
 
  // Methods for offscreen rendering
  bool IsOffScreen() const;
- void OnPaint(const gfx::Rect& dirty_rect, const SkBitmap& bitmap);
+ void OnPaint(const gfx::Rect& dirty_rect,
+ const SkBitmap& bitmap,
+ const OffscreenSharedTexture& info);
  void StartPainting();
  void StopPainting();
  bool IsPainting() const;
@@ -833,6 +836,9 @@ class WebContents : public ExclusiveAccessContext,
 
  bool offscreen_ = false;
 
+ // Whether offscreen rendering use gpu shared texture
+ bool offscreen_use_shared_texture_ = false;
+
  // Whether window is fullscreened by HTML5 api.
  bool html_fullscreen_ = false;
 

shell/browser/osr/osr_host_display_client.cc

@@ -70,7 +70,7 @@ void LayeredWindowUpdater::Draw(const gfx::Rect& damage_rect,
 
  if (active_ && canvas_->peekPixels(&pixmap)) {
  bitmap.installPixels(pixmap);
- callback_.Run(damage_rect, bitmap);
+ callback_.Run(damage_rect, bitmap, {});
  }
 
  std::move(draw_callback).Run();

shell/browser/osr/osr_host_display_client.h

@@ -11,15 +11,13 @@
 #include "base/memory/shared_memory_mapping.h"
 #include "components/viz/host/host_display_client.h"
 #include "services/viz/privileged/mojom/compositing/layered_window_updater.mojom.h"
+#include "shell/browser/osr/osr_paint_event.h"
 #include "third_party/skia/include/core/SkBitmap.h"
 #include "third_party/skia/include/core/SkCanvas.h"
 #include "ui/gfx/native_widget_types.h"
 
 namespace electron {
 
-typedef base::RepeatingCallback<void(const gfx::Rect&, const SkBitmap&)>
- OnPaintCallback;
-
 class LayeredWindowUpdater : public viz::mojom::LayeredWindowUpdater {
  public:
  explicit LayeredWindowUpdater(

shell/browser/osr/osr_host_display_client_mac.mm

@@ -30,7 +30,7 @@
  kPremul_SkAlphaType),
  pixels, stride);
  bitmap.setImmutable();
- callback_.Run(ca_layer_params.damage, bitmap);
+ callback_.Run(ca_layer_params.damage, bitmap, {});
  }
 }
 

shell/browser/osr/osr_paint_event.cc

@@ -0,0 +1,25 @@
+
+// Copyright (c) 2024 GitHub, Inc.
+// Use of this source code is governed by the MIT license that can be found in
+// the LICENSE file.
+
+#include "shell/browser/osr/osr_paint_event.h"
+
+namespace electron {
+
+NativePixmapPlaneInfo::NativePixmapPlaneInfo() = default;
+NativePixmapPlaneInfo::~NativePixmapPlaneInfo() = default;
+NativePixmapPlaneInfo::NativePixmapPlaneInfo(
+ const NativePixmapPlaneInfo& other) = default;
+NativePixmapPlaneInfo::NativePixmapPlaneInfo(uint32_t stride,
+ uint64_t offset,
+ uint64_t size,
+ int fd)
+ : stride(stride), offset(offset), size(size), fd(fd) {}
+
+OffscreenSharedTextureValue::OffscreenSharedTextureValue() = default;
+OffscreenSharedTextureValue::~OffscreenSharedTextureValue() = default;
+OffscreenSharedTextureValue::OffscreenSharedTextureValue(
+ const OffscreenSharedTextureValue& other) = default;
+
+} // namespace electron

shell/browser/osr/osr_paint_event.h

@@ -0,0 +1,70 @@
+
+// Copyright (c) 2024 GitHub, Inc.
+// Use of this source code is governed by the MIT license that can be found in
+// the LICENSE file.
+
+#ifndef ELECTRON_SHELL_BROWSER_OSR_GPU_SHARED_TEXTURE_H
+#define ELECTRON_SHELL_BROWSER_OSR_GPU_SHARED_TEXTURE_H
+
+#include "base/functional/callback_helpers.h"
+#include "media/base/video_types.h"
+#include "third_party/skia/include/core/SkCanvas.h"
+#include "ui/gfx/canvas.h"
+#include "ui/gfx/native_widget_types.h"
+
+#include <cstdint>
+
+#include "content/public/common/widget_type.h"
+
+namespace electron {
+
+struct NativePixmapPlaneInfo {
+ // The strides and offsets in bytes to be used when accessing the buffers
+ // via a memory mapping. One per plane per entry. Size in bytes of the
+ // plane is necessary to map the buffers.
+ uint32_t stride;
+ uint64_t offset;
+ uint64_t size;
+
+ // File descriptor for the underlying memory object (usually dmabuf).
+ int fd;
+
+ NativePixmapPlaneInfo();
+ ~NativePixmapPlaneInfo();
+ NativePixmapPlaneInfo(const NativePixmapPlaneInfo& other);
+ NativePixmapPlaneInfo(uint32_t stride,
+ uint64_t offset,
+ uint64_t size,
+ int fd);
+};
+
+struct OffscreenSharedTextureValue {
+ OffscreenSharedTextureValue();
+ ~OffscreenSharedTextureValue();
+ OffscreenSharedTextureValue(const OffscreenSharedTextureValue& other);
+
+ // It is user's responsibility to compose popup widget textures.
+ content::WidgetType widget_type;
+
+ // The pixel format of the shared texture, RGBA or BGRA depends on platform.
+ media::VideoPixelFormat pixel_format;
+
+#if BUILDFLAG(IS_WIN) || BUILDFLAG(IS_MAC)
+ // On Windows it is a HANDLE to the shared D3D11 texture.
+ // On macOS it is a IOSurface* to the shared IOSurface.
+ uintptr_t shared_texture_handle;
+#elif BUILDFLAG(IS_LINUX)
+ std::vector<NativePixmapPlaneInfo> planes;
+ uint64_t modifier;
+#endif
+};
+
+typedef std::optional<OffscreenSharedTextureValue> OffscreenSharedTexture;
+
+typedef base::RepeatingCallback<
+ void(const gfx::Rect&, const SkBitmap&, const OffscreenSharedTexture&)>
+ OnPaintCallback;
+
+} // namespace electron
+
+#endif // ELECTRON_SHELL_BROWSER_OSR_GPU_SHARED_TEXTURE_H