Background house-keeping & pre-existing thread

[vlovich]

In our use-case of Cloudflare Workers, we already have a GC thread that's responsible
for house-keeping activities. To minimize having a different thread, because
ReleasePerCpuMemoryToOS is an internal API, & to be protected against future book
keeping improvements, I've split up the loop into an init & tick phase so that it can be
integrated into an existing book keeping thread (ours has a 0.2 Hz tick rate).

I'm not 100% sure about what to do about the marking of the thread as idle. For now I made
it the callers responsibility to determine if the thread is actually idle but the rules feel squishy
(e.g. should it basically be invoked before every invocation of the Tick method?)

[google-cla]

Thanks for your pull request. It looks like this may be your first contribution to a Google open source project (if not, look below for help). Before we can look at your pull request, you'll need to sign a Contributor License Agreement (CLA).

📝 Please visit https://cla.developers.google.com/ to sign.

Once you've signed (or fixed any issues), please reply here with @googlebot I signed it! and we'll verify it.

---

What to do if you already signed the CLA

Individual signers

• It's possible we don't have your GitHub username or you're using a different email address on your commit. Check your existing CLA data and verify that your email is set on your git commits.

Corporate signers

• Your company has a Point of Contact who decides which employees are authorized to participate. Ask your POC to be added to the group of authorized contributors. If you don't know who your Point of Contact is, direct the Google project maintainer to go/cla#troubleshoot (Public version).
• The email used to register you as an authorized contributor must be the email used for the Git commit. Check your existing CLA data and verify that your email is set on your git commits.
• The email used to register you as an authorized contributor must also be attached to your GitHub account.

ℹ️ Googlers: Go here for more info.

[google-cla]

Thanks for your pull request. It looks like this may be your first contribution to a Google open source project (if not, look below for help). Before we can look at your pull request, you'll need to sign a Contributor License Agreement (CLA).

📝 Please visit https://cla.developers.google.com/ to sign.

Once you've signed (or fixed any issues), please reply here with @googlebot I signed it! and we'll verify it.

---

What to do if you already signed the CLA

Individual signers

• It's possible we don't have your GitHub username or you're using a different email address on your commit. Check your existing CLA data and verify that your email is set on your git commits.

Corporate signers

• Your company has a Point of Contact who decides which employees are authorized to participate. Ask your POC to be added to the group of authorized contributors. If you don't know who your Point of Contact is, direct the Google project maintainer to go/cla#troubleshoot (Public version).
• The email used to register you as an authorized contributor must be the email used for the Git commit. Check your existing CLA data and verify that your email is set on your git commits.
• The email used to register you as an authorized contributor must also be attached to your GitHub account.

ℹ️ Googlers: Go here for more info.

[vlovich]

@googlebot I signed it!

[ckennelly]

re marking as idle: This is primarily a consideration for turning down the ThreadCache (if one's present from before the pre-CPU cache started up). It can likely be done during init, although it'd be inexpensive to check on every tick if there weren't any intervening allocations (and no per-CPU cache).

In terms of tick rates: As you correctly surmised, there's more logic that can be added to the background processes (such as rebalancing caches, etc.). If more work did happen during a tick, would that be a problem in any way for the other house keeping activities?

[vlovich]

@ckennelly great questions. Our book keeping does allocations on every tick as we need to do some RPC to get the memory usage of some forked children (+ for keeping track of resources we need to release from our book keeping thread, there's promises being allocated, etc). I think it might be good to let the caller be responsible for marking the thread as free/busy as needed. On that note, we sleep for 5s when we don't think we're under memory pressure before checking again - would it be appropriate to mark thread idle/busy across that sleep or should I just not even bother with idleness because the book keeping thread isn't idle from malloc's perspective & a 5s sleep isn't materially important for evicting this stuff just to bring it back in?

If more work did happen during a tick, would that be a problem in any way for the other house keeping activities?

If the work is guaranteeing that we're freeing more memory, I can't imagine a scenario where that would be a problem. Our house keeping is basically saying "am I under memory pressure? if yes, release application memory until we're not". So TCmalloc doing more work on that behalf seems harmless to me as it might help prevent us needing to release application memory.

When not under memory pressure, we're probably only primarily interesting in releasing memory to the OS so that we're not hogging memory we're not using & are good citizens, but I don't know if it makes sense to do much beyond that (unless it's perhaps work to optimize the malloc performance of all other cpus/threads, which is a potential tradeoff consideration). Not sure if the background tasks are intended to do more than that. If that's the case, maybe we could have an enum to describe the kind of tick work we'd like.

Where it does get tricky if there's a lot of CPU time spent but no memory is reclaimed or there's CPU time spent reorganizing data for a marginal workload-specific impact. That would be true though for the current mechanism I think since which thread CPU usage happens doesn't really matter to us as it's stealing cycles from customer workloads.