-
-
Notifications
You must be signed in to change notification settings - Fork 343
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
feat(session-replay): Add experimental flags to use a more efficient view renderer for Session Replay #4940
base: main
Are you sure you want to change the base?
Conversation
|
Performance metrics 🚀
|
Revision | Plain | With Sentry | Diff |
---|---|---|---|
ab0012c | 1209.06 ms | 1228.78 ms | 19.72 ms |
825f0cb | 1220.53 ms | 1236.18 ms | 15.65 ms |
ae9c51b | 1244.85 ms | 1264.33 ms | 19.47 ms |
8aec30e | 1249.29 ms | 1252.63 ms | 3.35 ms |
b9fc537 | 1219.69 ms | 1231.80 ms | 12.12 ms |
dbb4b19 | 1246.02 ms | 1259.87 ms | 13.85 ms |
9202018 | 1225.77 ms | 1250.06 ms | 24.29 ms |
e887ddc | 1234.71 ms | 1244.22 ms | 9.50 ms |
866c529 | 1223.76 ms | 1244.27 ms | 20.52 ms |
fff4a70 | 1236.59 ms | 1249.11 ms | 12.52 ms |
App size
Revision | Plain | With Sentry | Diff |
---|---|---|---|
ab0012c | 22.85 KiB | 415.09 KiB | 392.24 KiB |
825f0cb | 22.31 KiB | 771.42 KiB | 749.10 KiB |
ae9c51b | 22.85 KiB | 411.13 KiB | 388.28 KiB |
8aec30e | 21.58 KiB | 616.76 KiB | 595.18 KiB |
b9fc537 | 21.58 KiB | 676.19 KiB | 654.61 KiB |
dbb4b19 | 22.30 KiB | 750.03 KiB | 727.73 KiB |
9202018 | 22.30 KiB | 749.70 KiB | 727.40 KiB |
e887ddc | 21.58 KiB | 616.76 KiB | 595.18 KiB |
866c529 | 22.31 KiB | 780.84 KiB | 758.53 KiB |
fff4a70 | 21.58 KiB | 707.28 KiB | 685.70 KiB |
Previous results on branch: philprime/session-replay-custom-graphics-renderer
Startup times
Revision | Plain | With Sentry | Diff |
---|---|---|---|
12fe555 | 1221.86 ms | 1238.86 ms | 17.00 ms |
5e61c3b | 1235.49 ms | 1253.18 ms | 17.69 ms |
46b6763 | 1224.19 ms | 1242.85 ms | 18.66 ms |
4b2daf3 | 1211.94 ms | 1235.12 ms | 23.19 ms |
App size
Revision | Plain | With Sentry | Diff |
---|---|---|---|
12fe555 | 22.30 KiB | 832.44 KiB | 810.13 KiB |
5e61c3b | 22.30 KiB | 826.76 KiB | 804.46 KiB |
46b6763 | 22.30 KiB | 826.23 KiB | 803.92 KiB |
4b2daf3 | 22.30 KiB | 832.44 KiB | 810.13 KiB |
@@ -27,7 +27,8 @@ class SentryMaskingPreviewView: UIView { | |||
init(redactOptions: SentryRedactOptions) { | |||
self.photographer = SentryViewPhotographer( | |||
renderer: PreviewRenderer(), | |||
redactOptions: redactOptions | |||
redactOptions: redactOptions, | |||
enableExperimentalMasking: false |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Not sure how we could pass in the experimental masking/renderer flag here
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
We have access to the redactOptions which stem from the SentryOptions. So in the SentrySessionReplayIntegration
we should have access to the SentryOptions and then this flag.
|
||
class SentryExperimentalMaskRenderer: NSObject, SentryMaskRenderer { | ||
func maskScreenshot(screenshot image: UIImage, size: CGSize, masking: [RedactRegion]) -> UIImage { | ||
let image = SentryGraphicsImageRenderer(size: size, scale: 1).image { context in |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
The SentryDefaultMaskRenderer
is also using an display scale of 1
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Please add this info as a comment.
Codecov ReportAttention: Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## main #4940 +/- ##
=============================================
- Coverage 92.470% 8.959% -83.512%
=============================================
Files 666 357 -309
Lines 78926 25293 -53633
Branches 28572 94 -28478
=============================================
- Hits 72983 2266 -70717
- Misses 5845 23027 +17182
+ Partials 98 0 -98
... and 652 files with indirect coverage changes Continue to review full report in Codecov by Sentry.
🚀 New features to boost your workflow:
|
eef381e
to
0c33278
Compare
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This looks great. I think it's OK to skip automated test for now. Once we know the solution is working correctly, we can come up with proper tests to avoid regressions when changing the code.
@@ -27,7 +27,8 @@ class SentryMaskingPreviewView: UIView { | |||
init(redactOptions: SentryRedactOptions) { | |||
self.photographer = SentryViewPhotographer( | |||
renderer: PreviewRenderer(), | |||
redactOptions: redactOptions | |||
redactOptions: redactOptions, | |||
enableExperimentalMasking: false |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
We have access to the redactOptions which stem from the SentryOptions. So in the SentrySessionReplayIntegration
we should have access to the SentryOptions and then this flag.
* - Experiment: This is an experimental feature and is therefore disabled by default. In case you are noticing issues with the experimental | ||
* view renderer, please report the issue on [GitHub](https://github.com/getsentry/sentry-cocoa). | ||
*/ | ||
public var enableExperimentalViewRenderer = false |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
m
: Shouldn't these rather be moved to SentryReplayOptions
cause they are related to SR?
* - Experiment: This is an experimental feature and is therefore disabled by default. In case you are noticing issues with the experimental | ||
* view renderer, please report the issue on [GitHub](https://github.com/getsentry/sentry-cocoa). | ||
*/ | ||
public var enableFastViewRenderer = false |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
m
: Why do we need two options? Why not enable the fastViewRenderer also with enableExperimentalViewRenderer
or vice versa? What's the use case for only enabling one of the two introduced options?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I considered introducing only a single feature flag to reduce the amount of configuration options a customer needs to consider. There is a major difference between the two flags:
enableExperimentalViewRenderer
uses our custom graphics renderer (I will rename the flag toenableCustomViewRenderer
) instead of the one provided by UIKit, which should act the same and produce the same output, but be faster. The flag is intended so that customers can opt-in and provide real-world feedback on its stability. Eventually we will want to switch from the default view renderer to the custom one.enableFastViewRenderer
has known issues as it does not fully render the UIKit view hierarchy, but instead renders the underlying CoreGraphics/CoreAnimation layers. This has performance advantages, but my research has shown that the rendered UI is incomplete. This feature will most likely stay off, even after experimental state, due to this issue. But we still want to provide customers an option who prefer faster performance rather than complete UI renders, as it can still provide value to see partial screens. I decided against changing this flag based on some heuristics to leave the decision when to use it to the customer.
I also updated the PR description.
|
||
class SentryExperimentalMaskRenderer: NSObject, SentryMaskRenderer { | ||
func maskScreenshot(screenshot image: UIImage, size: CGSize, masking: [RedactRegion]) -> UIImage { | ||
let image = SentryGraphicsImageRenderer(size: size, scale: 1).image { context in |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Please add this info as a comment.
|
||
import UIKit | ||
|
||
class SentryExperimentalMaskRenderer: NSObject, SentryMaskRenderer { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
m
: Isn't this class almost a duplicate of SentryDefaultMaskRenderer
? The only difference at a first glance is that this class uses the SentryGraphicsImageRenderer
. If yes, can't we simply pass in the SentryGraphicsImageRenderer
instead of the UIGraphicsImageRenderer
and ditch the duplication?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
At this point in time, they are mostly equal while using a different renderer. I did consider merging it and only switch the renderer, so we can do that.
But I do not want to change the default implementation which is GA now and make the mask renderer a replaceable/injectable component.
I will re-evaluate to DRY
* We introduced this class, because the ``UIGraphicsImageRenderer`` caused performance issues due to internal caching mechanisms. | ||
* During testing we noticed a significant performance improvement by creating the bitmap context directly. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
m
: Please elaborate on what exactly the internal caching mechanisms are. Are they iOS specific or do they live in our SDK?
let maskedScreenshot = maskRenderer.maskScreenshot(screenshot: renderedScreenshot, size: viewSize, masking: redact) | ||
onComplete(maskedScreenshot) | ||
} | ||
} | ||
|
||
func image(view: UIView) -> UIImage { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
m
: Can't we also move masking the screenshot here to a bg thread? Currenlty this is only called from capturing a screenshot, so maybe adding a comment explaining why we don't move it to a bg thread here would be helpful.
📜 Description
TL;DR: Reduces the time required to render a session replay frame from ~160ms to ~30-36ms (benchmarked on an iPhone 8)
enableExperimentalViewRenderer
to choose the view renderer (enables our graphics renderer)enableFastViewRenderer
to choose the view draw method (switches from UIKit to CoreGraphics rendering, might be incomplete)Note
The mask renderer is using a points-to-pixel scale of 1, therefore any higher resolution view render will be downscaled.
💡 Motivation and Context
See #4000 for more information for the main issue of frame performance.
To understand the changes, we need to look at the implementation of the
SentryDefaultViewRenderer
used by theSentryViewPhotographer
:The rendering consists of two nested blocks:
Setup
: to create a rendering context which is then used to convert rendered bitmap data into an in imageDraw
: which is drawing the view hierarchy into the bitmap contextBoth need to be analysed for their performance and potential improvement candidates.
In addition we need to reconsider how the resolutions and coordinate system works:
size
of awidth
andheight
in points (i.e.375pt x 667pt
for iPhone 8)scale
at which the UI should be rendered (i.e.2.0
for iPhone 8)size * scale
pixels (i.e.750px x 1334px
for iPhone 8)Therefore I also analyzed the impact of creating a scaled vs unscaled image, because the difference can be seen visually not only due to changes in resolution, but also due to blurriness:
Example - iOS-Swift at scale

1.0
:Example - iOS-Swift at scale

2.0
:The
drawHierarchy
performance has also been discussed in the Apple Developer Forums, leading to ReplayKit as an alternative (TBD)💚 How did you test it?
I tried to create unit tests to test the view photographer using a complex UI built in a Storyboard file. I was not able to get it working, because unit tests have no render server, therefore trying to render a view to an image fails with the following error:
Instead I performed manual testing by adapting the
SentryViewPhotographer
to use this method, then looking at the values over time when running the sampleiOS-Swift
on an iPhone 8:Baseline Performance
These are the results of using the currently released
SentryViewPhotographer.image(view:onComplete:)
. We are not measuring themaskRenderer.maskScreenshot(...)
because it is run on the background thread and therefore not blocking the main thread.If a screen uses 60 frames per seconds to render graphics, the maximum time used to render one frame should be less than
1 / 60 = 16.667 ms
. If a frame render takes longer, the next frame will be not be rendered (= a dropped frame) to catch up the lost time. This results in a stuttering user interface and visible to the user.Results at screen scale (i.e.
2.0
for iPhone 8):The data shows that we have a significant frame delay with up to 161.5842ms = ~9 frames dropped every second.
Results at scale
1.0
:No significant changes compared to the native screen scale.
Alternative Attempts
Using
UIView.snapshotView(afterScreenUpdates:)
The first attempt to optimize the performance was using the method UIView.snapshotView(afterScreenUpdates:) to create a dummy view hierarchy as stated in the documentation, to see if the dummy view would render faster than the normal one:
During testing the snapshot view never displayed any visual data when rendered into an
UIImage
.An Apple Engineer mentioned in the Apple Developer Forum that a snapshot can not be rendered to an image due to security reasons.
Results:
Discarded this approach because not possible.
Reuse the UIGraphicsImageRenderer
The documentation mentions that this renderer uses a built-in cache, therefore recommends to reuse renderer instances:
But the
UIGraphicsImageRenderer
is set to a fixed size and in our case the size of the view could eventually change.To reduce the memory footprint I decided to have a maximum of one cached renderer, discarding and re-creating it whenever the size changes. In the worst case it has to re-create the renderer every time, yielding the same performance as without the cache. In the best case the size never changes and the renderer is reused forever.
Results:
There is no significant change compared to the baseline.
Experimental View Renderer
UIKit is built on top of CoreGraphics and CoreAnimation also known as the
QuartzCore
.The
UIGraphicsImageRenderer
has been introduced withiOS 10.0
to wrap around theCGContext
provided by CoreGraphics, as the setup can be tedious and complicated.The
SentryGraphicsImageRenderer
is a custom implementation creating aCGContext
pixel buffer and also converting it into anUIImage
afterwards.Results at native scale (i.e.
2.0
):Results at
1.0
scale:Replacing
view.drawHierarchy(in:afterScreenUpdates:)
withview.layer.render(in:)
Instead of drawing the view using the
drawHierarchy
provided on the UIKit-level, we can directly call the rendering of thelayer
used by the UIView provided by CoreGraphics.Warning
During testing we noticed that the render is incomplete, in particular the tab bar icons have not been rendered. The exact impact is not known and can vary.
Results at native scale (i.e.
2.0
):SentryGraphicsImageRenderer
+drawHierarchy
Results at
1.0
scale:Experimental Renderer +
view.layer.render(in:)
Combining the previous two approaches.
Warning
During testing we noticed that the render is incomplete, in particular the tab bar icons have not been rendered. The exact impact is not known and can vary.
Results at native scale (i.e.
2.0
):view.drawHierarchy(...)
UIGraphicsImageRenderer
+view.layer.render(in:)
Results at
1.0
scale:Detailed Analysis of Implementation
Looking at one of the samples using
drawViewHierarchyInRect
in detail we can analyze the duration of the different calls in detail. In particular we notice that the most expensive calls are inside UIKit and CoreGraphics.It seems like
UIView
is backed by anUIImage
from a previous render, then draws it again into the bitmap context. As this is is a private API, we can not access it directly. No further optimization possible.