Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat(session-replay): Add experimental flags to use a more efficient view renderer for Session Replay #4940

Open
wants to merge 5 commits into
base: main
Choose a base branch
from

Conversation

philprime
Copy link
Contributor

@philprime philprime commented Mar 4, 2025

📜 Description

TL;DR: Reduces the time required to render a session replay frame from ~160ms to ~30-36ms (benchmarked on an iPhone 8)

  • Implements a custom graphics image renderer.
  • Adds a new experimental feature enableExperimentalViewRenderer to choose the view renderer (enables our graphics renderer)
  • Adds a new experimental feature enableFastViewRenderer to choose the view draw method (switches from UIKit to CoreGraphics rendering, might be incomplete)
  • Adds an experimental graphics image renderer
  • Adds a view renderer using the experimental graphics image renderer
  • Adds a mask renderer using the experimental graphics image renderer

Note

The mask renderer is using a points-to-pixel scale of 1, therefore any higher resolution view render will be downscaled.

💡 Motivation and Context

See #4000 for more information for the main issue of frame performance.

To understand the changes, we need to look at the implementation of the SentryDefaultViewRenderer used by the SentryViewPhotographer:

@objcMembers
class SentryDefaultViewRenderer: NSObject, SentryViewRenderer {
    func render(view: UIView) -> UIImage {
        // START BLOCK - Setup
        let image = UIGraphicsImageRenderer(size: view.bounds.size).image { context in
            // START BLOCK - Draw
            view.drawHierarchy(in: view.bounds, afterScreenUpdates: false)
            // END BLOCK - Draw
        }
        // END BLOCK - Setup
        return image
    }
}

The rendering consists of two nested blocks:

  • Setup: to create a rendering context which is then used to convert rendered bitmap data into an in image
  • Draw: which is drawing the view hierarchy into the bitmap context

Both need to be analysed for their performance and potential improvement candidates.

In addition we need to reconsider how the resolutions and coordinate system works:

  • View has a size of a width and height in points (i.e. 375pt x 667pt for iPhone 8)
  • Windows have a scale at which the UI should be rendered (i.e. 2.0 for iPhone 8)
  • Screens have a resolution of size * scale pixels (i.e. 750px x 1334px for iPhone 8)

Therefore I also analyzed the impact of creating a scaled vs unscaled image, because the difference can be seen visually not only due to changes in resolution, but also due to blurriness:

Example - iOS-Swift at scale 1.0:
scale-1

Example - iOS-Swift at scale 2.0:
scale-2

The drawHierarchy performance has also been discussed in the Apple Developer Forums, leading to ReplayKit as an alternative (TBD)

💚 How did you test it?

I tried to create unit tests to test the view photographer using a complex UI built in a Storyboard file. I was not able to get it working, because unit tests have no render server, therefore trying to render a view to an image fails with the following error:

Rendering a view (0x105223a80, UILayoutContainerView) that has not been committed to render server is not supported.

Instead I performed manual testing by adapting the SentryViewPhotographer to use this method, then looking at the values over time when running the sample iOS-Swift on an iPhone 8:

func image(view: UIView, onComplete: @escaping ScreenshotCallback) {
    let viewSize = view.bounds.size

    let startTime = DispatchTime.now().uptimeNanoseconds
    let startRedactTime = DispatchTime.now().uptimeNanoseconds
    let redact = redactBuilder.redactRegionsFor(view: view)
    let endRedactTime = DispatchTime.now().uptimeNanoseconds

    // The render method is synchronous and must be called on the main thread.
    // This is because the render method accesses the view hierarchy which is managed from the main thread.
    let startRenderTime = DispatchTime.now().uptimeNanoseconds
    let renderedScreenshot = renderer.render(view: view)
    let endRenderTime = DispatchTime.now().uptimeNanoseconds
    let endTime = DispatchTime.now().uptimeNanoseconds

    printSummary(
        redactDiff: endRedactTime - startRedactTime,
        renderDiff: endRenderTime - startRenderTime,
        totalDiff: endTime - startTime
    )

    dispatchQueue.dispatchAsync { [maskRenderer] in
        // The mask renderer does not need to be on the main thread.
        // Moving it to a background thread to avoid blocking the main thread, therefore reducing the performance
        // impact/lag of the user interface.
        let maskedScreenshot = maskRenderer.maskScreenshot(screenshot: renderedScreenshot, size: viewSize, masking: redact)
        onComplete(maskedScreenshot)
    }
}

private let sampleLimit = 120
private var redactDiffHistory = [UInt64]()
private var renderDiffHistory = [UInt64]()
private var totalDiffHistory = [UInt64]()

func printSummary(redactDiff: UInt64, renderDiff: UInt64, totalDiff: UInt64) {
    redactDiffHistory = (redactDiffHistory + [redactDiff]).suffix(sampleLimit)
    renderDiffHistory = (renderDiffHistory + [renderDiff]).suffix(sampleLimit)
    totalDiffHistory = (totalDiffHistory + [totalDiff]).suffix(sampleLimit)

    let redactDiffHistoryMin = redactDiffHistory.min() ?? 0
    let redactDiffHistoryMax = redactDiffHistory.max() ?? 0
    let redactDiffHistoryAverage = Double(redactDiffHistory.reduce(0, +)) / Double(max(redactDiffHistory.count, 1))
    let sortedRedactDiffHistory = redactDiffHistory.sorted()
    let redactDiffHistoryP50 = sortedRedactDiffHistory[redactDiffHistory.count / 2]
    let redactDiffHistoryP75 = sortedRedactDiffHistory[Int(Double(redactDiffHistory.count) * 0.75)]
    let redactDiffHistoryP95 = sortedRedactDiffHistory[Int(Double(redactDiffHistory.count) * 0.95)]
    let redactDiffHistoryLast = redactDiffHistory.last ?? 0

    let renderDiffHistoryMin = renderDiffHistory.min() ?? 0
    let renderDiffHistoryMax = renderDiffHistory.max() ?? 0
    let renderDiffHistoryAverage = Double(renderDiffHistory.reduce(0, +)) / Double(max(renderDiffHistory.count, 1))
    let sortedRenderDiffHistory = renderDiffHistory.sorted()
    let renderDiffHistoryP50 = sortedRenderDiffHistory[renderDiffHistory.count / 2]
    let renderDiffHistoryP75 = sortedRenderDiffHistory[Int(Double(renderDiffHistory.count) * 0.75)]
    let renderDiffHistoryP95 = sortedRenderDiffHistory[Int(Double(renderDiffHistory.count) * 0.95)]
    let renderDiffHistoryLast = renderDiffHistory.last ?? 0

    let totalDiffHistoryMin = totalDiffHistory.min() ?? 0
    let totalDiffHistoryMax = totalDiffHistory.max() ?? 0
    let totalDiffHistoryAverage = Double(totalDiffHistory.reduce(0, +)) / Double(max(totalDiffHistory.count, 1))
    let sortedTotalDiffHistory = totalDiffHistory.sorted()
    let totalDiffHistoryP50 = sortedTotalDiffHistory[totalDiffHistory.count / 2]
    let totalDiffHistoryP75 = sortedTotalDiffHistory[Int(Double(totalDiffHistory.count) * 0.75)]
    let totalDiffHistoryP95 = sortedTotalDiffHistory[Int(Double(totalDiffHistory.count) * 0.95)]
    let totalDiffHistoryLast = totalDiffHistory.last ?? 0
    
    func f(_ value: UInt64) -> String {
        String(format: "%8.4f ms", Double(value) / 1_000_000.0)
    }

    func f(_ value: Double) -> String {
        String(format: "%8.4f ms", Double(value) / 1_000_000.0)
    }

    let samples = String(format: "%4.i Samples", redactDiffHistory.count)
    print("| \(samples) | Redact      | Render      | Total       |")
    print("|--------------|-------------|-------------|-------------|")
    print("| Min          | \(f(redactDiffHistoryMin)) | \(f(renderDiffHistoryMin)) | \(f(totalDiffHistoryMin)) |")
    print("| Max          | \(f(redactDiffHistoryMax)) | \(f(renderDiffHistoryMax)) | \(f(totalDiffHistoryMax)) |")
    print("| Avg          | \(f(redactDiffHistoryAverage)) | \(f(renderDiffHistoryAverage)) | \(f(totalDiffHistoryAverage)) |")
    print("| p50          | \(f(redactDiffHistoryP50)) | \(f(renderDiffHistoryP50)) | \(f(totalDiffHistoryP50)) |")
    print("| p75          | \(f(redactDiffHistoryP75)) | \(f(renderDiffHistoryP75)) | \(f(totalDiffHistoryP75)) |")
    print("| p95          | \(f(redactDiffHistoryP95)) | \(f(renderDiffHistoryP95)) | \(f(totalDiffHistoryP95)) |")
    print("| Last         | \(f(redactDiffHistoryLast)) | \(f(renderDiffHistoryLast)) | \(f(totalDiffHistoryLast)) |")
}

Baseline Performance

These are the results of using the currently released SentryViewPhotographer.image(view:onComplete:). We are not measuring the maskRenderer.maskScreenshot(...) because it is run on the background thread and therefore not blocking the main thread.

If a screen uses 60 frames per seconds to render graphics, the maximum time used to render one frame should be less than 1 / 60 = 16.667 ms. If a frame render takes longer, the next frame will be not be rendered (= a dropped frame) to catch up the lost time. This results in a stuttering user interface and visible to the user.

Results at screen scale (i.e. 2.0 for iPhone 8):

let image = UIGraphicsImageRenderer(size: view.bounds.size).image { _ in
    view.drawHierarchy(in: view.bounds, afterScreenUpdates: false)
}

The data shows that we have a significant frame delay with up to 161.5842ms = ~9 frames dropped every second.

120 Samples Redact Render Total
Min 3.0583 ms 145.3815 ms 151.4525 ms
Max 6.5138 ms 155.8338 ms 161.8351 ms
Avg 5.8453 ms 149.8243 ms 155.6732 ms
p50 6.0484 ms 149.2103 ms 154.1397 ms
p75 6.1136 ms 151.9487 ms 158.0255 ms
p95 6.2567 ms 155.3496 ms 161.3549 ms
Last 6.1190 ms 150.4965 ms 156.6198 ms

Results at scale 1.0:

let image = UIGraphicsImageRenderer(size: view.bounds.size, format: .init(for: .init(displayScale: 1))).image { _ in
    view.drawHierarchy(in: view.bounds, afterScreenUpdates: false)
}

No significant changes compared to the native screen scale.

120 Samples Redact Render Total
Min 5.9785 ms 146.0440 ms 152.2056 ms
Max 6.4990 ms 156.3045 ms 162.4112 ms
Avg 6.1184 ms 149.6858 ms 155.8079 ms
p50 6.1055 ms 148.6769 ms 155.1323 ms
p75 6.1640 ms 151.9197 ms 158.0497 ms
p95 6.3104 ms 155.7726 ms 161.8109 ms
Last 5.9875 ms 147.1455 ms 153.1360 ms

Alternative Attempts

Using UIView.snapshotView(afterScreenUpdates:)

The first attempt to optimize the performance was using the method UIView.snapshotView(afterScreenUpdates:) to create a dummy view hierarchy as stated in the documentation, to see if the dummy view would render faster than the normal one:

This method very efficiently captures the current rendered appearance of a view and uses it to build a new snapshot view. You can use the returned view as a visual stand-in for the current view in your app. [..] Because the content is captured from the already rendered content, this method reflects the current visual appearance of the view and is not updated to reflect animations that are scheduled or in progress. However, calling this method is faster than trying to render the contents of the current view into a bitmap image yourself.

During testing the snapshot view never displayed any visual data when rendered into an UIImage.

An Apple Engineer mentioned in the Apple Developer Forum that a snapshot can not be rendered to an image due to security reasons.

Results:

Discarded this approach because not possible.

Reuse the UIGraphicsImageRenderer

The documentation mentions that this renderer uses a built-in cache, therefore recommends to reuse renderer instances:

After initializing an image renderer, you can use it to draw multiple images with the same configuration. An image renderer keeps a cache of Core Graphics contexts, so reusing the same renderer can be more efficient than creating new renderers.

But the UIGraphicsImageRenderer is set to a fixed size and in our case the size of the view could eventually change.

To reduce the memory footprint I decided to have a maximum of one cached renderer, discarding and re-creating it whenever the size changes. In the worst case it has to re-create the renderer every time, yielding the same performance as without the cache. In the best case the size never changes and the renderer is reused forever.

Results:

There is no significant change compared to the baseline.

120 Samples Redact Render Total
Min 5.9499 ms 146.9310 ms 153.0000 ms
Max 6.7538 ms 156.0019 ms 161.9819 ms
Avg 6.0744 ms 149.5189 ms 155.5966 ms
p50 6.0541 ms 148.0545 ms 154.0874 ms
p75 6.1128 ms 151.6945 ms 157.7179 ms
p95 6.2532 ms 155.3220 ms 161.3656 ms
Last 6.0766 ms 147.0542 ms 153.1340 ms

Experimental View Renderer

UIKit is built on top of CoreGraphics and CoreAnimation also known as the QuartzCore.

The UIGraphicsImageRenderer has been introduced with iOS 10.0 to wrap around the CGContext provided by CoreGraphics, as the setup can be tedious and complicated.

The SentryGraphicsImageRenderer is a custom implementation creating a CGContext pixel buffer and also converting it into an UIImage afterwards.

Results at native scale (i.e. 2.0):

let scale = (view as? UIWindow ?? view.window)?.screen.scale ?? 1
let image = SentryGraphicsImageRenderer(size: view.bounds.size, scale: scale).image { context in
    view.drawHierarchy(in: view.bounds, afterScreenUpdates: false)
}
  • Significant decrease in render time compared to base line performance.
120 Samples Redact Render Total
Min 2.1284 ms 14.7570 ms 16.8885 ms
Max 6.7335 ms 32.5775 ms 38.6342 ms
Avg 6.0055 ms 25.4156 ms 31.4264 ms
p50 6.0432 ms 24.5575 ms 30.6436 ms
p75 6.0950 ms 27.3424 ms 33.3627 ms
p95 6.2285 ms 30.3218 ms 36.5009 ms
Last 6.0572 ms 24.3487 ms 30.4101 ms

Results at 1.0 scale:

let scale = 1.0
let image = SentryGraphicsImageRenderer(size: view.bounds.size, scale: scale).image { context in
    view.drawHierarchy(in: view.bounds, afterScreenUpdates: false)
}
  • Significant decrease in render time compared to base line performance.
  • Slower than native scale, probably due to downscaling of internal graphics data
120 Samples Redact Render Total
Min 2.1135 ms 27.0460 ms 29.1636 ms
Max 7.2708 ms 48.6616 ms 54.7815 ms
Avg 6.0159 ms 38.8028 ms 44.8226 ms
p50 6.1011 ms 38.4735 ms 44.7876 ms
p75 6.2221 ms 40.3663 ms 46.3873 ms
p95 6.4524 ms 44.4222 ms 50.5087 ms
Last 6.1820 ms 42.6977 ms 48.8843 ms

Replacing view.drawHierarchy(in:afterScreenUpdates:) with view.layer.render(in:)

Instead of drawing the view using the drawHierarchy provided on the UIKit-level, we can directly call the rendering of the layer used by the UIView provided by CoreGraphics.

Warning

During testing we noticed that the render is incomplete, in particular the tab bar icons have not been rendered. The exact impact is not known and can vary.

image_5

Results at native scale (i.e. 2.0):

let image = UIGraphicsImageRenderer(size: view.bounds.size).image { context in
    view.layer.render(in: context.cgContext)
}
  • Significant decrease in render time compared to the base line performance.
  • Faster than the SentryGraphicsImageRenderer + drawHierarchy
120 Samples Redact Render Total
Min 2.0926 ms 13.1221 ms 15.2177 ms
Max 6.3896 ms 24.6149 ms 30.6909 ms
Avg 6.0791 ms 20.5175 ms 26.6001 ms
p50 6.0992 ms 19.8446 ms 25.9533 ms
p75 6.1418 ms 21.5083 ms 27.6816 ms
p95 6.3071 ms 23.0865 ms 29.1678 ms
Last 6.0721 ms 24.6149 ms 30.6909 ms

Results at 1.0 scale:

let image = UIGraphicsImageRenderer(size: view.bounds.size, format: .init(for: .init(displayScale: 1))).image { context in
    view.layer.render(in: context.cgContext)
}
  • Slightly slower than native scale, probably due to downscaling of internal graphics data
120 Samples Redact Render Total
Min 2.6087 ms 12.5871 ms 15.1984 ms
Max 6.7715 ms 26.2617 ms 32.3283 ms
Avg 6.0303 ms 22.2413 ms 28.2754 ms
p50 6.0662 ms 21.3212 ms 27.3122 ms
p75 6.1138 ms 24.0657 ms 30.1942 ms
p95 6.3048 ms 26.2106 ms 32.2714 ms
Last 6.7715 ms 20.5220 ms 27.2970 ms

Experimental Renderer + view.layer.render(in:)

Combining the previous two approaches.

Warning

During testing we noticed that the render is incomplete, in particular the tab bar icons have not been rendered. The exact impact is not known and can vary.

Results at native scale (i.e. 2.0):

let scale = (view as? UIWindow ?? view.window)?.screen.scale ?? 1
let image = SentryGraphicsImageRenderer(size: view.bounds.size, scale: scale).image { context in
    view.layer.render(in: context.cgContext)
}
  • Significant decrease in render time compared to the base line performance.
  • Large improvements compared to experimental renderer with view.drawHierarchy(...)
  • No improvements compared to UIGraphicsImageRenderer + view.layer.render(in:)
120 Samples Redact Render Total
Min 5.9651 ms 18.5304 ms 24.5483 ms
Max 6.5470 ms 24.9161 ms 30.9267 ms
Avg 6.0764 ms 20.7397 ms 26.8196 ms
p50 6.0500 ms 19.8425 ms 26.0733 ms
p75 6.1109 ms 22.4187 ms 28.5533 ms
p95 6.3331 ms 24.6614 ms 30.6779 ms
Last 6.0334 ms 18.7154 ms 24.7522 ms

Results at 1.0 scale:

let scale = 1.0
let image = SentryGraphicsImageRenderer(size: view.bounds.size, scale: scale).image { context in
    view.layer.render(in: context.cgContext)
}
  • Slower than native scale, probably due to downscaling of internal graphics data
120 Samples Redact Render Total
Min 5.9296 ms 19.2106 ms 25.2905 ms
Max 6.2822 ms 27.0288 ms 33.0322 ms
Avg 6.0384 ms 21.6527 ms 27.6942 ms
p50 6.0178 ms 20.7255 ms 26.7658 ms
p75 6.0690 ms 22.8727 ms 28.9003 ms
p95 6.2106 ms 26.0749 ms 32.0846 ms
Last 6.1801 ms 23.6582 ms 29.8419 ms

Detailed Analysis of Implementation

Looking at one of the samples using drawViewHierarchyInRect in detail we can analyze the duration of the different calls in detail. In particular we notice that the most expensive calls are inside UIKit and CoreGraphics.

26.00 ms  100,0 % Sentry            |  SentryGraphicsImageRenderer.image(actions:)
25.00 ms   96,2 % Sentry            |   closure #1 in SentryExperimentalViewRenderer.render(view:)
25.00 ms   96,2 % UIKitCore         |    -[UIView drawViewHierarchyInRect:afterScreenUpdates:]
21.00 ms   80,8 % UIKitCore         |     -[UIImage drawInRect:blendMode:alpha:]
21.00 ms   80,8 % CoreGraphics      |      CGContextDrawImageWithOptions
20.00 ms   76,9 % CoreGraphics      |       ripc_DrawImage
15.00 ms   57,7 % CoreGraphics      |        ripc_AcquireRIPImageData
 5.00 ms   19,2 % CoreGraphics      |        RIPLayerBltImage
 1.00 ms    3,8 % libdispatch.dylib |       _dispatch_once_callout
 4.00 ms   15,4 % UIKitCore         |     _UIRenderViewImageAfterCommit
 1.00 ms    3,8 % Sentry            |   SentryGraphicsImageRenderer.Context.currentImage.getter
 1.00 ms    3,8 % CoreGraphics      |    CGBitmapContextCreateImage

It seems like UIView is backed by an UIImage from a previous render, then draws it again into the bitmap context. As this is is a private API, we can not access it directly. No further optimization possible.

@philprime philprime self-assigned this Mar 4, 2025
Copy link

github-actions bot commented Mar 4, 2025

Messages
📖 Do not forget to update Sentry-docs with your feature once the pull request gets approved.

Generated by 🚫 dangerJS against 5a2f253

Copy link

github-actions bot commented Mar 5, 2025

Performance metrics 🚀

  Plain With Sentry Diff
Startup time 1231.92 ms 1255.00 ms 23.08 ms
Size 22.30 KiB 830.26 KiB 807.96 KiB

Baseline results on branch: main

Startup times

Revision Plain With Sentry Diff
ab0012c 1209.06 ms 1228.78 ms 19.72 ms
825f0cb 1220.53 ms 1236.18 ms 15.65 ms
ae9c51b 1244.85 ms 1264.33 ms 19.47 ms
8aec30e 1249.29 ms 1252.63 ms 3.35 ms
b9fc537 1219.69 ms 1231.80 ms 12.12 ms
dbb4b19 1246.02 ms 1259.87 ms 13.85 ms
9202018 1225.77 ms 1250.06 ms 24.29 ms
e887ddc 1234.71 ms 1244.22 ms 9.50 ms
866c529 1223.76 ms 1244.27 ms 20.52 ms
fff4a70 1236.59 ms 1249.11 ms 12.52 ms

App size

Revision Plain With Sentry Diff
ab0012c 22.85 KiB 415.09 KiB 392.24 KiB
825f0cb 22.31 KiB 771.42 KiB 749.10 KiB
ae9c51b 22.85 KiB 411.13 KiB 388.28 KiB
8aec30e 21.58 KiB 616.76 KiB 595.18 KiB
b9fc537 21.58 KiB 676.19 KiB 654.61 KiB
dbb4b19 22.30 KiB 750.03 KiB 727.73 KiB
9202018 22.30 KiB 749.70 KiB 727.40 KiB
e887ddc 21.58 KiB 616.76 KiB 595.18 KiB
866c529 22.31 KiB 780.84 KiB 758.53 KiB
fff4a70 21.58 KiB 707.28 KiB 685.70 KiB

Previous results on branch: philprime/session-replay-custom-graphics-renderer

Startup times

Revision Plain With Sentry Diff
12fe555 1221.86 ms 1238.86 ms 17.00 ms
5e61c3b 1235.49 ms 1253.18 ms 17.69 ms
46b6763 1224.19 ms 1242.85 ms 18.66 ms
4b2daf3 1211.94 ms 1235.12 ms 23.19 ms

App size

Revision Plain With Sentry Diff
12fe555 22.30 KiB 832.44 KiB 810.13 KiB
5e61c3b 22.30 KiB 826.76 KiB 804.46 KiB
46b6763 22.30 KiB 826.23 KiB 803.92 KiB
4b2daf3 22.30 KiB 832.44 KiB 810.13 KiB

@@ -27,7 +27,8 @@ class SentryMaskingPreviewView: UIView {
init(redactOptions: SentryRedactOptions) {
self.photographer = SentryViewPhotographer(
renderer: PreviewRenderer(),
redactOptions: redactOptions
redactOptions: redactOptions,
enableExperimentalMasking: false
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Not sure how we could pass in the experimental masking/renderer flag here

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We have access to the redactOptions which stem from the SentryOptions. So in the SentrySessionReplayIntegration we should have access to the SentryOptions and then this flag.


class SentryExperimentalMaskRenderer: NSObject, SentryMaskRenderer {
func maskScreenshot(screenshot image: UIImage, size: CGSize, masking: [RedactRegion]) -> UIImage {
let image = SentryGraphicsImageRenderer(size: size, scale: 1).image { context in
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The SentryDefaultMaskRenderer is also using an display scale of 1

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please add this info as a comment.

Copy link

codecov bot commented Mar 5, 2025

Codecov Report

Attention: Patch coverage is 1.32450% with 149 lines in your changes missing coverage. Please review.

Project coverage is 8.959%. Comparing base (c964075) to head (5a2f253).

Files with missing lines Patch % Lines
...s/ViewCapture/SentryExperimentalMaskRenderer.swift 0.000% 53 Missing ⚠️
...ools/ViewCapture/SentryGraphicsImageRenderer.swift 0.000% 51 Missing ⚠️
Sources/Sentry/SentrySessionReplayIntegration.m 0.000% 18 Missing ⚠️
...s/ViewCapture/SentryExperimentalViewRenderer.swift 0.000% 10 Missing ⚠️
...es/Swift/Helper/SentryEnabledFeaturesBuilder.swift 0.000% 6 Missing ⚠️
...ift/Tools/ViewCapture/SentryViewPhotographer.swift 0.000% 6 Missing ⚠️
Sources/Sentry/SentryScreenshot.m 0.000% 3 Missing ⚠️
...ssionReplay/Preview/SentryMaskingPreviewView.swift 0.000% 2 Missing ⚠️

❗ There is a different number of reports uploaded between BASE (c964075) and HEAD (5a2f253). Click for more details.

HEAD has 2 uploads less than BASE
Flag BASE (c964075) HEAD (5a2f253)
3 1
Additional details and impacted files

Impacted file tree graph

@@              Coverage Diff              @@
##              main    #4940        +/-   ##
=============================================
- Coverage   92.470%   8.959%   -83.512%     
=============================================
  Files          666      357       -309     
  Lines        78926    25293     -53633     
  Branches     28572       94     -28478     
=============================================
- Hits         72983     2266     -70717     
- Misses        5845    23027     +17182     
+ Partials        98        0        -98     
Files with missing lines Coverage Δ
Sources/Swift/SentryExperimentalOptions.swift 80.000% <100.000%> (+30.000%) ⬆️
...ssionReplay/Preview/SentryMaskingPreviewView.swift 0.000% <0.000%> (-86.000%) ⬇️
Sources/Sentry/SentryScreenshot.m 0.000% <0.000%> (-81.968%) ⬇️
...es/Swift/Helper/SentryEnabledFeaturesBuilder.swift 0.000% <0.000%> (-100.000%) ⬇️
...ift/Tools/ViewCapture/SentryViewPhotographer.swift 0.000% <0.000%> (-70.000%) ⬇️
...s/ViewCapture/SentryExperimentalViewRenderer.swift 0.000% <0.000%> (ø)
Sources/Sentry/SentrySessionReplayIntegration.m 0.000% <0.000%> (-88.706%) ⬇️
...ools/ViewCapture/SentryGraphicsImageRenderer.swift 0.000% <0.000%> (ø)
...s/ViewCapture/SentryExperimentalMaskRenderer.swift 0.000% <0.000%> (ø)

... and 652 files with indirect coverage changes


Continue to review full report in Codecov by Sentry.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update c964075...5a2f253. Read the comment docs.

🚀 New features to boost your workflow:
  • Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@philprime philprime force-pushed the philprime/session-replay-custom-graphics-renderer branch from eef381e to 0c33278 Compare March 5, 2025 16:12
@philprime philprime changed the title feat(session-replay): Add experimental graphics renderer feat(session-replay): Add experimental flags to use a more efficient view renderer for Session Replay Mar 5, 2025
@philprime philprime marked this pull request as ready for review March 5, 2025 17:11
Copy link
Member

@philipphofmann philipphofmann left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This looks great. I think it's OK to skip automated test for now. Once we know the solution is working correctly, we can come up with proper tests to avoid regressions when changing the code.

@@ -27,7 +27,8 @@ class SentryMaskingPreviewView: UIView {
init(redactOptions: SentryRedactOptions) {
self.photographer = SentryViewPhotographer(
renderer: PreviewRenderer(),
redactOptions: redactOptions
redactOptions: redactOptions,
enableExperimentalMasking: false
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We have access to the redactOptions which stem from the SentryOptions. So in the SentrySessionReplayIntegration we should have access to the SentryOptions and then this flag.

* - Experiment: This is an experimental feature and is therefore disabled by default. In case you are noticing issues with the experimental
* view renderer, please report the issue on [GitHub](https://github.com/getsentry/sentry-cocoa).
*/
public var enableExperimentalViewRenderer = false
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

m: Shouldn't these rather be moved to SentryReplayOptions cause they are related to SR?

* - Experiment: This is an experimental feature and is therefore disabled by default. In case you are noticing issues with the experimental
* view renderer, please report the issue on [GitHub](https://github.com/getsentry/sentry-cocoa).
*/
public var enableFastViewRenderer = false
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

m: Why do we need two options? Why not enable the fastViewRenderer also with enableExperimentalViewRenderer or vice versa? What's the use case for only enabling one of the two introduced options?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I considered introducing only a single feature flag to reduce the amount of configuration options a customer needs to consider. There is a major difference between the two flags:

  • enableExperimentalViewRenderer uses our custom graphics renderer (I will rename the flag to enableCustomViewRenderer) instead of the one provided by UIKit, which should act the same and produce the same output, but be faster. The flag is intended so that customers can opt-in and provide real-world feedback on its stability. Eventually we will want to switch from the default view renderer to the custom one.
  • enableFastViewRenderer has known issues as it does not fully render the UIKit view hierarchy, but instead renders the underlying CoreGraphics/CoreAnimation layers. This has performance advantages, but my research has shown that the rendered UI is incomplete. This feature will most likely stay off, even after experimental state, due to this issue. But we still want to provide customers an option who prefer faster performance rather than complete UI renders, as it can still provide value to see partial screens. I decided against changing this flag based on some heuristics to leave the decision when to use it to the customer.

I also updated the PR description.


class SentryExperimentalMaskRenderer: NSObject, SentryMaskRenderer {
func maskScreenshot(screenshot image: UIImage, size: CGSize, masking: [RedactRegion]) -> UIImage {
let image = SentryGraphicsImageRenderer(size: size, scale: 1).image { context in
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please add this info as a comment.


import UIKit

class SentryExperimentalMaskRenderer: NSObject, SentryMaskRenderer {
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

m: Isn't this class almost a duplicate of SentryDefaultMaskRenderer? The only difference at a first glance is that this class uses the SentryGraphicsImageRenderer. If yes, can't we simply pass in the SentryGraphicsImageRenderer instead of the UIGraphicsImageRenderer and ditch the duplication?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

At this point in time, they are mostly equal while using a different renderer. I did consider merging it and only switch the renderer, so we can do that.

But I do not want to change the default implementation which is GA now and make the mask renderer a replaceable/injectable component.

I will re-evaluate to DRY

Comment on lines +9 to +10
* We introduced this class, because the ``UIGraphicsImageRenderer`` caused performance issues due to internal caching mechanisms.
* During testing we noticed a significant performance improvement by creating the bitmap context directly.
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

m: Please elaborate on what exactly the internal caching mechanisms are. Are they iOS specific or do they live in our SDK?

let maskedScreenshot = maskRenderer.maskScreenshot(screenshot: renderedScreenshot, size: viewSize, masking: redact)
onComplete(maskedScreenshot)
}
}

func image(view: UIView) -> UIImage {
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

m: Can't we also move masking the screenshot here to a bg thread? Currenlty this is only called from capturing a screenshot, so maybe adding a comment explaining why we don't move it to a bg thread here would be helpful.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
Status: In Progress
Development

Successfully merging this pull request may close these issues.

3 participants