Off CPU profiling #196

florianl · 2024-10-20T17:21:01Z

This is the code that backs
#144. It can be reused to add features like requested in #33 and therefore can be an alternative to
#192.

The idea that enables off CPU profiling is, that perf event and kprobe eBPF programs are quite similar and can be converted. This allows, with the dynamic rewrite of tail call maps, the reuse of existing eBPF programs and concepts.

This proposal adds the new flag '-off-cpu-threshold' that enables off CPU profiling and attaches the two additional hooks, as discussed in Option B in #144.

florianl · 2024-10-20T17:24:05Z

support/ebpf/native_stack_trace.ebpf.c

@@ -4,14 +4,6 @@
 #include "tracemgmt.h"
 #include "stackdeltatypes.h"

-#ifndef __USER32_CS


This got moved to tracemgmt.h to make it available to other entry points.

florianl · 2024-10-20T17:24:16Z

support/ebpf/native_stack_trace.ebpf.c

@@ -602,151 +594,6 @@ static ErrorCode unwind_one_frame(u64 pid, u32 frame_idx, struct UnwindState *st
  #error unsupported architecture
 #endif

-// Initialize state from pt_regs


This got moved to tracemgmt.h to make it available to other entry points.

florianl · 2024-10-20T17:27:29Z

support/ebpf/tracemgmt.h

+#endif // TESTING_COREDUMP
+
+static inline
+int collect_trace(struct pt_regs *ctx, TraceOrigin origin, u32 pid, u32 tid, u64 off_cpu_time) {


With moving collect_trace() to tracemgmt.h its arguments changed. origin was added to further keep track what triggered the stack unwinding. pid and tid also got added as argument, as the entry point for off CPU profiling does some filtering on these parameters, so it didn't make sense to call the helper for this information multiple times. And finally off_cpu_time got added as argument, to forward the information for how long a trace was off CPU and store the information in the Trace struct.

florianl · 2024-10-20T17:29:30Z

tracer/tracer.go

-func initializeMapsAndPrograms(includeTracers types.IncludedTracers,
-	kernelSymbols *libpf.SymbolMap, filterErrorFrames bool, mapScaleFactor int,
-	kernelVersionCheck bool, debugTracer bool, bpfVerifierLogLevel uint32) (
+func initializeMapsAndPrograms(kernelSymbols *libpf.SymbolMap, cfg *Config) (


With adding an additional argument from *Config to initializeMapsAndPrograms() it made sense to just pass *Config rather than every single argument by itself.

tracer/tracer.go

umanwizard

I'll give my thought despite not being a maintainer.

umanwizard · 2024-10-23T18:36:29Z

support/ebpf/off_cpu.ebpf.c

+  if (bpf_get_prandom_u32()%OFF_CPU_THRESHOLD_MAX > syscfg->off_cpu_threshold) {
+    return 0;
+  }


I'm not sure the numbers will be statistically valid given this check. Shouldn't the probability of sampling be based on the amount of time spent? (Similar to how e.g. the probability of heap allocation samplers recording a stack is determined by the amount of bytes allocated).

Imagine if the off-CPU time of some process is dominated by one call to read that hangs forever (e.g. due to networking misconfiguration). Unless you get lucky enough that you hit this threshold on that call, you will not see it reflected in the profile.

Imagine if the off-CPU time of some process is dominated by one call to read that hangs forever (e.g. due to networking misconfiguration). Unless you get lucky enough that you hit this threshold on that call, you will not see it reflected in the profile.

I think you describe a good example of the general risk of a sampling approach. In #144 a sampling approach is proposed, as handling every scheduling event in both hooks is a too high risk to overload the system. If the sampling should be done on the off-CPU time, then there is also a management overhead to correctly size the eBPF map that communicates the start of the off-CPU time to the end of the off-CPU time. So yes, using a sampling approach to drop some events in the first hook will miss out on some events.
But similar to regular sampling based on-CPU profiling, if something is misconfigured, then there will be not a single event but multiple ones and so the issue will be visible to the profiler. The same applies to off-CPU profiling, I think. Overall, sampling based profiling provides valuable insights into the hot paths of the system for (expected or unexpected) things that happen often. Sampling is not an approach that catches every event, otherwise it would qualify as debugging or security utility - but sampling does not satisfy this requirements.

I completely agree that the feature must use sampling. However the strategy used for sampling makes a difference.

E.g., see here where a similar analysis was done for jemalloc: https://github.com/jemalloc/jemalloc/blob/2a693b83d2d1631b6a856d178125e1c47c12add9/doc_internal/PROFILING_INTERNALS.md#L4

The authors concluded that they needed to sample per-byte, rather than per-allocation, because sampling per-allocation increases the variance significantly in a scenario where the allocations can have significantly different sizes.

The corresponding approach here would be to sample per-nanosecond, rather than per-event.

The downside is then you would have to call bpf_ktime_get_ns on the begin and end unconditionally (to know how long the task was parked for and then do your sampling logic), whereas now we skip calling bpf_ktime_get_ns when the sampling doesn't hit. I'm not sure what the overhead of that would be.

I will try to put together a test based on your code to see how much this affects profiling variance in the real world.

Thanks for the link to the research that got into the sampling strategy for jemalloc.

From my perspective, there are major differences in the purpose of sampling off CPU profiling vs jemalloc that lead to the differences in the sampling strategies.
jemalloc considers small allocations as “cheaper” whereas larger allocations result in more effort. For off CPU profiling the effort to take computing resources from a task and later reassign computing resources are always the same and the time a task is off CPU does not make a difference. So the sampling strategy of jemalloc takes the effort into account while for off CPU profiling the effort is constant.

The general event based sampling approach also comes with the advantage that it lets users predict the direct storage impact that off CPU profiling will have. The storage requirements are linearly correlated to configuration of the CLI flag -off-cpu-threshold. Changing the sampling strategy will make the storage predictions more complex and harder.

We can also not make assumptions about the environment, where off CPU profiling will be used. Will the environment be dominated by short interrupts, with small off CPU times for tasks, or will there be more larger off CPU times, e.g. if spinning disks are used instead of other faster memory solutions. The general event based sampling approach does not make a difference between these two kinds.

Switching to sampling on the off CPU time will also introduce a management overhead. First eBPF maps need to be scaled larger, resulting in a larger memory requirement for the eBPF Profiling agent and secondly there will be more computing overhead. To make a fair sampling decision on the off CPU time we need to make sure, that every tasked that is handled by the scheduler fits into the eBPF maps. The additional computing overhead will be on calling bpf_ktime_get_ns twice to get the off CPU time for each task before making a sampling decision on it, and more eBPF map updates and lookups to forward the off CPU time information from the first to the second hook.

With #144 a general sampling approach was accepted, which is implemented in this current state of code proposal.
Maybe @open-telemetry/profiling-maintainers can provide their views on the different kinds of sampling strategies and provide guidance moving on on this topic.

Thanks for the detailed response. Your points about the differences in constraints between jemalloc and opentelemetry-ebpf-profiler are compelling.

I still think in the future it is worth testing with different sampling strategies to see how much it affects variance in common scenarios and then making a call whether that's worth the downsides (e.g. less predictable storage usage).

But you've convinced me that this is a good enough default that it's worth starting with this.

@umanwizard You bring up a valid point, in some tests I did for an unrelated project, calling bpf_ktime_get_ns too much can add up to significant overhead but that was at frequencies far higher than 500-1000Hz which I would assume is the worst-case of what we'd be dealing with here. It also seems simple enough implementation-wise to experiment with and compare. If that's something you'd still like to pursue, I would definitely be interested in the output.

jemalloc considers small allocations as “cheaper” whereas larger allocations result in more effort.

I'll admit that I need to find some more time to fully digest the highly interesting jemalloc doc. But my initial impression is that this aspect was not the deciding factor for their choice of sampling strategy and more of a "nice to have". I could be wrong tho.

The storage requirements are linearly correlated to configuration of the CLI flag -off-cpu-threshold. Changing the sampling strategy will make the storage predictions more complex and harder.

~~Will it? Assuming we want to sample from N events that are produced of some time period t, a Bernoulli strategy using a time rate R will produce data at a rate of t/R (independent of N)~~

~~A strategy that relies on sampling 1 event for every R events will produce data at a rate of N/R (independent of t).~~

In both cases the choice of R allows linear control over the data rate, but the second strategy requires the user to know N in order to pick an initial rate R and predict the data volume it will produce. But N might change over time, so this strategy doesn't really allow a predictable data rate from a user perspective as far as I can tell.

Sorry I was interrupted by having to drop my daughter off to daycare, so I submitted this too hastily this morning. The analysis above only works when there is a only a single thread involved. For multiple threads the data rate for the Bernoulli strategy increases with the number of threads. I think this could be handled by dividing the event durations by the number of threads, but I need to think about this a bit more. But my point about the unpredictable data volumes for the strategy in this PR remains valid I think.

With #144 a general sampling approach was accepted, which is implemented in this current state of code proposal.

I think the merging of that PR signaled high-level consensus on implementing Off-CPU, but not necessary any consensus on the sampling strategy. I explicitly called that out in my comment here.

Switching to sampling on the off CPU time will also introduce a management overhead.

I don't have immediate comments on this. It definitely needs to be considered. But I think there is also some value in thinking about the strategy we would like if implementation issues were not prohibitive.

I thought I replied to this thread at the beginning of the week. But it looks like I didn’t. So, sorry for the late reply.

With #144 a general sampling approach was accepted, which is implemented in this current state of code proposal.

I think the merging of that PR signaled high-level consensus on implementing Off-CPU, but not necessary any consensus on the sampling strategy. I explicitly called that out in my comment here.

You are right, #144 did not focus on a sampling strategy. My motivation of #144 was to introduce the general idea of off-CPU profiling and keeping its overhead minimal, so that there is no performance impact.

I think there is a major difference between the sampling strategies, per-allocation and per-byte, as used for jemalloc and the strategies, event based and off-CPU-time based, discussed here for off-CPU profiling.
The sampling strategy for jemalloc comes with the major advantage that at the time of decision the number of requested bytes are known. So there is no management overhead in changing the sampling strategy.
Whereas for off-CPU profiling, the time a task will be off-CPU is not known at the point when a scheduler decides to take computing resources from a task. To guarantee a fair sampling strategy based on off-CPU-time, it needs to be guaranteed that all tasks are tracked. Which will result in significantly more eBPF map updates/lookups but also the eBPF maps need to be scaled (significantly larger and) correctly. Besides this, there will also be some additional overhead in calling helpers like bpf_ktime_get_ns more often than required.

[..] But my point about the unpredictable data volumes for the strategy in this PR remains valid I think.

Can you elaborate a bit more on this? With the suggested event based sampling strategy, the storage requirement should be linear in-/decreasing depending on the following major components

Number of CPU cores

CPU frequency

off-CPU threshold

As the number of CPU cores should be known, as well their CPU frequency, the required storage is depending directly on the configured off-CPU threshold and will increase and decrease accordingly. Of course, this is a simplified view as environments change, CPU cores use a dynamic frequency and power management can take down CPU cores. But these environmental beneficial features should only reduce the required storage and the configuration with the most impact should be the configured off-CPU threshold.

fabled · 2024-10-28T08:52:21Z

tracer/tracer.go

+		// All the tail call targets are perf event programs. To be able to tail call them
+		// from a kprobe, adjust their specification.
+		if !unwindProg.noTailCallTarget {
+			// Adjust program type
+			progSpec.Type = cebpf.Kprobe
+
+			// Adjust program name for easier debugging
+			progSpec.Name = "kp_" + progSpec.Name
 		}
-		if err := tailcallMap.Update(unsafe.Pointer(&unwindProg.progID), unsafe.Pointer(&fd),
-			cebpf.UpdateAny); err != nil {
-			// Every eBPF program that is loaded within loadUnwinders can be the
-			// destination of a tail call of another eBPF program. If we can not update
-			// the eBPF map that manages these destinations our unwinding will fail.
-			return fmt.Errorf("failed to update tailcall map: %v", err)
+		if err := loadProgram(ebpfProgs, tailcallMap, unwindProg.progID, progSpec,
+			programOptions, unwindProg.noTailCallTarget); err != nil {
+			return err


While this is clever, and avoids some function declarations. I think this might have some maintenance overhead. Especially if anything in the ebpf binary becomes more function type specific. And this also prevents to have differing probe context usage for perf_event vs. kprobe. Perhaps in future we would like the perf_event probe to also record perf event specific data from the context?

I would prefer a solution to do this in the eBPF C code instead. Just separate is tracepoint to main function that gets called from each ebpf helper entry point separately. This would be future proof without maintenance fear. Even if it likely doubles the eBPF ELF binary size.

For the moment, I will keep the proposed solution. Currently there is a high demand to also check in the debug eBPF blobs - which would add binary blobs of 1.4M (amd64) + 1.4M (arm64) to the respository.

With a macro like the following it is possible to generate perf event and kprobe programs from the same source, avoiding code duplication:

#define MULTI_USE_FUNC(func_name) \ SEC("perf_event/"#func_name) \ int func_name##_perf(struct pt_regs *ctx) { \ return func_name(ctx); \ } \ \ SEC("kprobe/"#func_name) \ int func_name##_kprobe(struct pt_regs *ctx) { \ return func_name(ctx); \ }

If it is decided to also check in debug eBPF blobs, then the checked in binary blobs will be the sum of 1.4M (amd64/perf events) + 1.4M (arm64/perf events) + 1.4M (amd64/kprobe) + 1.4M (arm64/kprobe) just for the debug binary eBPF blobs.
Without the discussion around also keeping the debug binary eBPF blobs checked into the repository, switching to the marco for code generation sounds like a viable option. But with this discussion, I would like to defer this implementation detail for a moment and wait for the discussion result.

While using a macro to generate these two different kinds of eBPF programs from the same code is possible and avoid relabeling the eBPF programs, the more significant part is updating the tail call. I didn't find a way to rewrite the tail call maps when generating code with the above marco. So dynamic rewriting of the tail call map would stay anyway.

Independent of the way it is done in the end, the relabeling or program generation using a macro only affects the programs that are tail call destinations and are used for unwinding the stack. The entry hooks, no matter whether it is a perf event type of program of kprobe, will stay native eBPF program types and not touched. This allows to do eBPF program type specific tasks - only the tail call destinations are generic and not eBPF program type specific. So there is no limitation in fetching and handling eBPF program type specific information.

opened #225 to bring this discussion forward.

With c1724aa I have rewritten the approach to generate perf event and kprobe programs at compile time as requested.
With this change the new size of tracer.ebpf.release.amd64 is 570 KBytes (before 291 KBytes) and for tracer.ebpf.release.arm64 it becomes 554 KBytes (before 283 KBytes).

Let me know what you think - @fabled @christos68k

tsint · 2024-10-27T11:49:32Z

support/ebpf/tracemgmt.h

+  trace->pid = pid;
+  trace->tid = tid;
+  trace->ktime = ktime;
+  trace->offtime = off_cpu_time;


I didn't find any handling of offtime by the reporter. If it still follows the previous method (merely counting the number of traceEvents), it won't accurately reflect how long a process has been scheduled out of the CPU. Some samples have a long duration of being scheduled out of the CPU, while others have a short duration, yet they are counted as the same count, which is incorrect.

It is suggested that count=offtime/sampling-interval, so that it can maintain the same meaning of samples as on-CPU.

Handling of off-CPU traces in the user space part is still named as outstanding work to be done in #196 (comment).

The current OTel Profiling signal is not well suited for data like off-CPU traces. This is a known problem and the OTel Profiling SIG is working on improvements for the OTel Profiling signal.
This fact is one reason, why the user space part for off-CPU profiling is not yet implemented (as mentioned in #196 (comment)) , as there are major changes coming up.

cli_flags.go

florianl · 2024-10-29T09:20:32Z

Rebased proposed changes on current main and resolved merge conflicts.

cli_flags.go

libpf/symbol.go

support/ebpf/off_cpu.ebpf.c

christos68k · 2024-10-31T17:09:54Z

support/ebpf/tracemgmt.h

+  if (pid == 0) {
+    return 0;
+  }


I think this doesn't belong here and is better placed at the source (we're already checking against pid == 0 in one of the call sites so can also restore the pid == 0 check in the other).

I did place this check here on purpose as safe guard. While this check is also done in

opentelemetry-ebpf-profiler/support/ebpf/off_cpu.ebpf.c

Line 69 in ed083e9

if (pid == 0 || tid == 0) {

it is not done in

opentelemetry-ebpf-profiler/support/ebpf/native_stack_trace.ebpf.c

Line 626 in ed083e9

int native_tracer_entry(struct bpf_perf_event_data *ctx) {

. In the former kprobe/finish_task_switch checks are performed based on the PID. While the later perf_event/native_tracer_entry is just adapted to the changes of collect_trace(). The idea is, that collect_trace() serves as entry point for stack unwinding for any kind of eBPF program and therefore it should do the basic checks to not run into stack unwinding issues.
Also moving this check to the caller of collect_trace() will duplicate code in the current situation, which I tried to avoid.

support/ebpf/tracemgmt.h

christos68k · 2024-10-31T17:16:00Z

support/ebpf/types.h

+} TraceOrigin;
+
+// OFF_CPU_THRESHOLD_MAX defines the maximum threshold.
+#define OFF_CPU_THRESHOLD_MAX 1000


This should probably be 100 which is intuitive percentage-wise and also consistent with the probabilistic profiling setting:

Suggested change

#define OFF_CPU_THRESHOLD_MAX 1000

#define OFF_CPU_THRESHOLD_MAX 100

1000 is chosen over 100 to provide a more fine grained possibility to configure the off-CPU threshold. On a single core system, using regular percentages makes sense, I think. But the more cores are in use, the more data will be generated. And so I think it makes sense to provide the option to reduce the storage and do off-CPU profiling only for a smaller amount than 1% of all scheduler task switches.
Let me know, if 1% of all scheduler task switches is the minimum that should be allowed to configure.

support/ebpf/types.h

florianl · 2024-11-01T12:51:05Z

With d4a09ad the propagation of off CPU profiles in user space and reporting was added. The PR was rebased to resolve recent API changes in the reporter package.

florianl · 2024-11-01T12:52:47Z

reporter/otlp_reporter.go

@@ -226,20 +240,25 @@ func (r *OTLPReporter) ReportTraceEvent(trace *libpf.Trace, meta *TraceEventMeta
 		containerID:    containerID,
 	}

-	if events, exists := (*traceEventsMap)[key]; exists {
+	traceEventsMap := r.traceEvents.WLock()


No need to hold the lock while looking up the Cgroupv2 ID and creating the key. So I moved fetching the lock to this later point.

florianl · 2024-11-01T12:55:20Z

reporter/otlp_reporter.go

-	// This is to ensure that AttributeTable does not contain duplicates.
-	attributeMap := make(map[string]uint64)
+// getProfile returns an OTLP profile containing all collected traces up to this moment.
+func (r *OTLPReporter) getProfile(origin int) (profile *profiles.Profile, startTS, endTS uint64) {


getProfile() used to be +200 LOC. I did split it into multiple logical independent functions. So most of the code just shifted and was not changed.

This is the code that backs open-telemetry#144. It can be reused to add features like requested in open-telemetry#33 and therefore can be an alternative to open-telemetry#192. The idea that enables off CPU profiling is, that perf event and kprobe eBPF programs are quite similar and can be converted. This allows, with the dynamic rewrite of tail call maps, the reuse of existing eBPF programs and concepts. This proposal adds the new flag '-off-cpu-threshold' that enables off CPU profiling and attaches the two additional hooks, as discussed in Option B in open-telemetry#144. Outstanding work: - [ ] Handle off CPU traces in the reporter package - [ ] Handle off CPU traces in the user space side Signed-off-by: Florian Lehner <[email protected]>

Signed-off-by: Florian Lehner <[email protected]>

florianl · 2024-11-18T14:00:17Z

Needed to force-push to the branch to resolve merge conflicts. Looking forward on getting feedback.

This was referenced Oct 20, 2024

Proposal: support for third party OSS to use otel-ebpf-profiler #192

Draft

[RFC] design doc: proposal for off-cpu profiling #144

Merged

florianl commented Oct 20, 2024

View reviewed changes

florianl commented Oct 21, 2024

View reviewed changes

tracer/tracer.go Outdated Show resolved Hide resolved

umanwizard reviewed Oct 23, 2024

View reviewed changes

fabled reviewed Oct 28, 2024

View reviewed changes

tsint reviewed Oct 28, 2024

View reviewed changes

liad-miggo reviewed Oct 28, 2024

View reviewed changes

cli_flags.go Outdated Show resolved Hide resolved

florianl force-pushed the off-cpu branch from 70e92b3 to 0de915c Compare October 29, 2024 09:18

florianl force-pushed the off-cpu branch from f27fa68 to ed083e9 Compare October 29, 2024 10:22

christos68k reviewed Oct 31, 2024

View reviewed changes

florianl force-pushed the off-cpu branch from 3cd36fe to d4a09ad Compare November 1, 2024 12:46

florianl marked this pull request as ready for review November 1, 2024 12:51

florianl requested review from a team as code owners November 1, 2024 12:51

florianl commented Nov 1, 2024

View reviewed changes

florianl force-pushed the off-cpu branch from d4a09ad to c1724aa Compare November 8, 2024 11:10

florianl and others added 7 commits November 18, 2024 14:53

fixup: provide arm64 tracer blob

353a483

Signed-off-by: Florian Lehner <[email protected]>

fixup: fix integration test for tracer

83b5268

Signed-off-by: Florian Lehner <[email protected]>

fixup: dynamic symbol prefix lookup for kprobe

2baf76f

Signed-off-by: Florian Lehner <[email protected]>

fixup: add timestamp as argument to collect_trace()

ccf5987

Signed-off-by: Florian Lehner <[email protected]>

fixup: apply feedback

06d0364

Signed-off-by: Florian Lehner <[email protected]>

fixup: propagate off-CPU profiling in user space

db56f5b

Signed-off-by: Florian Lehner <[email protected]>

fixup: generate perf_event and kprobe programs at compile time

3a5aca2

Signed-off-by: Florian Lehner <[email protected]>

florianl force-pushed the off-cpu branch from c1724aa to 3a5aca2 Compare November 18, 2024 13:59

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Off CPU profiling #196

Off CPU profiling #196

florianl commented Oct 20, 2024 •

edited

Loading

florianl Oct 20, 2024

florianl Oct 20, 2024

florianl Oct 20, 2024

florianl Oct 20, 2024

umanwizard left a comment

umanwizard Oct 23, 2024

florianl Oct 24, 2024

umanwizard Oct 24, 2024 •

edited

Loading

florianl Oct 31, 2024

umanwizard Oct 31, 2024

christos68k Oct 31, 2024 •

edited

Loading

felixge Nov 1, 2024 •

edited

Loading

florianl Nov 8, 2024

fabled Oct 28, 2024

florianl Nov 1, 2024 •

edited

Loading

florianl Nov 6, 2024

florianl Nov 8, 2024

tsint Oct 27, 2024

florianl Oct 28, 2024

florianl Oct 28, 2024

florianl commented Oct 29, 2024

christos68k Oct 31, 2024 •

edited

Loading

florianl Nov 1, 2024

christos68k Oct 31, 2024

florianl Nov 1, 2024

florianl commented Nov 1, 2024

florianl Nov 1, 2024

florianl Nov 1, 2024

florianl commented Nov 18, 2024

	#define OFF_CPU_THRESHOLD_MAX 1000
	#define OFF_CPU_THRESHOLD_MAX 100

Off CPU profiling #196

Are you sure you want to change the base?

Off CPU profiling #196

Conversation

florianl commented Oct 20, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

umanwizard left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

umanwizard Oct 24, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

christos68k Oct 31, 2024 • edited Loading

Choose a reason for hiding this comment

felixge Nov 1, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

florianl Nov 1, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

florianl commented Oct 29, 2024

christos68k Oct 31, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

florianl commented Nov 1, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

florianl commented Nov 18, 2024

florianl commented Oct 20, 2024 •

edited

Loading

umanwizard Oct 24, 2024 •

edited

Loading

christos68k Oct 31, 2024 •

edited

Loading

felixge Nov 1, 2024 •

edited

Loading

florianl Nov 1, 2024 •

edited

Loading

christos68k Oct 31, 2024 •

edited

Loading