Linux Plumbers Conference

Author: tomoakisato

392

@@ -1 +1,44 @@
-(Negotiations are underway with the YouTube channel concerned.)
+
+So, thank you for attending this talk. So, I'm SJ, and this time, last LSFMM, I presented some status updates and the future plans for this project, and then I failed at managing the time at the time, and therefore there were some questions or concerns that I was unable to address. So, I'd like to handle such things today again.
+
+So, first of all, as always, the opinions are my own, and this talk is more focused on discussion rather than presentation, so it will not touch much of the internals of the DAMON itself.
+
+So, how many of you know about DAMON? How many of you have heard about DAMON? I mean, not the movie star. Okay. So, DAMON has merged into the Linux kernel for a while. It's from 5.15, so it's a little bit of a young and small subsystem. So, to introduce it again, DAMON is a data access pattern snapshot generator, which informs you about which address range is how frequently accessed and for how long. And it currently supports virtual space and physical space, but it can be expanded for some more special types of other spaces. And it is designed for low and controllable monitoring overhead, and the each extensions. And it supports an access-aware memory optimization engine feature called DAMON. It allows users to specify which memory regions of which access pattern they are interested in, and what kind of memory action to take. And it is currently being used in some real-world products, including AWS Aurora Serverless v2 and SK Hynix HMSDK.
+
+So, my two long-term goals for DAMON are making it help the kernel be more extensible and also just work. So, for making DAMON help the kernel be more extensible, we are trying... My plan is to expose the DAMON functionalities as much as possible using some stable but extensible user interface or ABI. This is required by some of the big tech companies who have confidence in knowing more information—more than even the kernel—and have a very good understanding of their workloads. And to expose the DAMON functionality, we are currently using... We are using the sysfs interface, which is in the  kernel/mm/damon directory. And we are using the sysfs philosophy, which generates one file per one data, and in this way, I believe that it can be more extensible for future extensions. It therefore would have many, many files inside the directory, and it's already having many files, and it might make it not very easy to be used by users manually, but our intention is to provide a user-space tool for manual user control of DAMON. For that, we are directly maintaining and developing the DAMON interface tool, and we are striving to provide backward compatibility regardless of the version of the underlying kernel. And also, we are very open to some kind of integration of DAMON with BPF for more extensible control of DAMON’s internal logic.
+
+And why do we need to—why do we want to make DAMON help make the kernel ‘just work’? That is, as I just mentioned, sometimes some users know many things about their workloads and systems, even more than the kernel, but that is not always the case, and therefore we still need to make the kernel at least reasonably work well without such complex tunings or controls. For this, we will add DAMON modules. That is, DAMON is currently implemented as an in-kernel framework; that is, it provides an API to other kernel components like modules or subsystems. And therefore, for some kind of more concrete and simple but common use cases, like proactive reclamation or proactive LRU-list sorting, we are currently providing specific kernel modules for specific use cases. Currently, we are providing modules for proactive reclamation and modules for proactive LRU-list sorting. And also, we will continue adding more as we find more specific use cases. Especially in the case of SK Hynix HMSDK, the CXL memory tuning could also be such a use case. And also, we will continue simplifying the user space ABI by adding more auto-tuning features that can replace the old sysfs files. That is, we will add more files, but the old files will continuously be rarely used and replaced by the use of the new files. And eventually, in the long term—you know, far future—we want to reduce the knobs one by one in this way, and eventually it should be one single knob, that is, one single kernel configuration. I have no idea what the name of the configuration could be, but say, ‘CONFIG_DAMON_X’. By setting this as ‘yes’, the system will utilize all the DAMON-based system optimizations at once by default.
+
+Of course, the configuration will be ‘no’ by default, but as time goes by, we can think about it. So, that’s it. And any questions or comments would be very welcomed. If you have no idea what question can be raised, just use this. If I presented it too perfectly?
+
+Could we use this with the BPF thing to get, like, a trace of exactly where—like, you generate a heat map and you tell us what is, like, very much accessed a lot or whatever, and then we could figure out the call path as to where it’s getting accessed or where it was created? Kind of like tying into what Surin said earlier and Surin mentioned.
+
+Yeah, it is available. Currently, the DAMON’s monitoring research can be collected using perf because it provides a trace point. So, it exposes the monitoring research using a trace point, and if you can use perf, you can collect the data and then visualize the research in a heatmap. That’s exactly what DAMON’s tool is doing from user space. And also, as you just mentioned, we can also make some mappings from the memory regions to specific code using some additional information. I believe that that is not necessarily needed to be done by DAMON itself, but we can use some different tracing tools like BPF trace, perf, or whatever, and actually, that’s already integrated inside the DAMON’s tool. The DAMON’s tool collects not only DAMON monitoring research but also some additional information such as memory usage and memory mapping information and the hotspot of the code using perf and vmstat or mapinfo. And therefore, I think such information can be generated, and we are providing the feature with the DAMON user space tool because the DAMON can… So, my basic intention with DAMON is making a system that ‘just works’ but is also extensible in some easy way. But in some cases, some people would want to do some kind of profiling-guided optimization-like thing. For that, the DAMON user space tool is currently being developed in a way to support that, you know, in an easy way.
+
+So, maybe I have a question about the sysfs interface. So, I believe the rule is it’s a single value per file. If you are providing more complex data, then it should be debugfs. So maybe that’s the answer to that question.
+
+Yeah, as you just mentioned, for more complex but human-friendly output representation, debugfs or tracefs would be the right approach. But I also have no real good idea about how DAMON will evolve. We are trying to use just random solutions, more than intellectual design. And yeah, so I think that could be extended much more than I can imagine at the moment. And therefore, I believe that the sysfs interface might be a better approach for long-term stability.
+
+Does DAMON actually support working in a VM right now, or does it only support working on the host?
+
+Yeah, you can run DAMON for monitoring access patterns of virtual machines from the host side. And of course, you can run DAMON inside the guest—whatever you want.
+
+By the way, we didn’t say to the remote speakers: if you want to ask a question, then either turn your camera on or use the ‘raise hand’ feature, so we know to tell you to speak.
+
+So, SJ, for the “kernel that just works” idea, is this basically doing proactive reclaim—like, what is DAMON actually going to be doing in the “just works”? Is it proactive reclaim to reduce unused memory and all that kind of thing?
+
+So, it’s an open question. For now, we are providing proactive reclamation, and we will continue adding more of that kind of purpose-optimized kernel modules that use DAMON for assisting the system. So maybe we can add memory tiering and also THP optimization, and some more things could be added there. And I was talking about access-continuity-aware memory autoscaling in the last LSFMM, and such things could be added. And by turning this ‘CONFIG_DAMON_X’ on, it will find the system’s configuration—if it has CXL memory or not—and based on such things, all the features that could reasonably be beneficial can be turned on.
+
+Okay, so it’ll look at your system and see where it can be useful, and then it will enable those bits, alright.
+
+And even after introducing the ‘CONFIG_DAMON_X’, we might add some more such features.
+
+I have another question. So, is there any plan for DAMON to integrate with PSI, for example, to conduct some PSI-driven reclamation? On your slides, I saw it’s fixed for two minutes right now. So, I was wondering if you can have some fine control of how many amount of CPU cycles you spend on scanning. That would probably be more ideal.
+
+You’re correct. And it’s already providing a kind of auto-tuning that adjusts the aggressiveness of DAMON-based operations based on PSI. Currently, some memory pressure store information value can be used as a target goal of the auto-tuning. If you just set that goal, then the auto-tuning will adjust DAMON on its own to meet the targeted value of the PSI.
+
+Yeah, I have a question, but more about the downstream part of the DAMON. Like, we have the DAMON with all the data there, and some are using it for decision-making. Do you think, like, if we could add some plug-in there to analyze this data somehow? I’m just wondering, like, from your perspective, should this be something from DAMON, or is it more of a consumer-side thing, or have you actually thought about that?
+
+Yeah, currently we provide trace point; it can be used. It also provides the snapshot in some efficient way using sysfs. And also, in the future, we might be able to integrate BPF so that you can run your analysis code written in BPF inside the DAMON.
+
+Thank you.