L1 required for stampede protection, and validation of setup

[mderriey]

Hi there 👋

Thanks a lot for this package, we're currently evaluating it, and it looks great.

We're currently using raw IDistributedCache, and want to evaluate options that can provide stampede protection. It's worth noting that we're not interested in caching data in memory.

L1 required for stampede protection

I saw in this discussion and in the XML comments of the SetSkipMemoryCache function that L1 is required for stampede protection.

I tried to use SetSkipMemoryCache(true), and the outcome indicates that stampede protection was not enabled in that case, where multiple factories were running at the same time.

The discussion linked above mentioned to use L1 and set a short duration, so stampede protection is enabled, while most of the time the data will be fetched with L2.

We think we achieved this with the following:

var fusionCacheBuilder = builder.Services.AddFusionCache()
    .WithSystemTextJsonSerializer()
    .WithRegisteredDistributedCache(ignoreMemoryDistributedCache: !builder.Environment.IsDevelopment())
    .WithDefaultEntryOptions(new FusionCacheEntryOptions()
    {
        // Memory cache duration
        Duration = TimeSpan.FromSeconds(1),
    });

Then, when interacting with IFusionCache, we set the distributed cache duration to the appropriate value:

return await fusionCache.GetOrSetAsync(
    "MyCacheKey",
    async _ => await RunFactory(),
    options => options.SetDistributedCacheDuration(TimeSpan.FromMinutes(5)));

My understanding of the following setup is:

1. L1 is enabled, so this provides stampede protection.
2. Data will be cached in L1 for up to 1 second.
3. Between second 1 and minute 5, if that cache key is requested, data will be fetched from L2, and L1 will be updated for another second.

Could you please confirm whether that's accurate?

Special case during local dev with no real distributed cache

During local dev, we register the in-memory distributed cache, as we want cached data to be purged across application restarts.

We seemed to face an issue initially where because both L1 and L2 used the same MemoryCache instance, when L1 data expired after 1 second, it also expired that data for L2. Does that sounds right?

One workaround we found is to register a separate IMemoryCache/MemoryCache instance during local dev to keep L1 and L2 separate:

if (builder.Environment.IsDevelopment())
{
    // Use a different memory cache implementation when in local dev so the shorter
    // memory cache duration doesn't wipe out the distributed cache.
    const string localDevFusionCacheMemoryCacheKey = "LocalDevFusionCacheMemoryCache";
    builder.Services.AddKeyedSingleton<IMemoryCache, MemoryCache>(localDevFusionCacheMemoryCacheKey);
    fusionCacheBuilder.WithRegisteredKeyedMemoryCache(localDevFusionCacheMemoryCacheKey);
}

Does that look OK to you? Is there a better way to do this?

[jodydonetti]

Hi @mderriey

Thanks a lot for this package, we're currently evaluating it, and it looks great.

Great, happy you're liking it!

We're currently using raw IDistributedCache, and want to evaluate options that can provide stampede protection. It's worth noting that we're not interested in caching data in memory.

Got it.
Out of curiosity, if I may, why is that? Memory allocation concerns? Fear of updates not being reflected instantly? Something else?

L1 required for stampede protection

I saw in this discussion and in the XML comments of the SetSkipMemoryCache function that L1 is required for stampede protection.

Yes, this is a different design from some potential alternatives mostly because it allows for extra features like fail-safe and similar to work seamlessly.

The discussion linked above mentioned to use L1 and set a short duration, so stampede protection is enabled, while most of the time the data will be fetched with L2.

Correct.

We think we achieved this with the following:

var fusionCacheBuilder = builder.Services.AddFusionCache()
    .WithSystemTextJsonSerializer()
    .WithRegisteredDistributedCache(ignoreMemoryDistributedCache: !builder.Environment.IsDevelopment())
    .WithDefaultEntryOptions(new FusionCacheEntryOptions()
    {
        // Memory cache duration
        Duration = TimeSpan.FromSeconds(1),
    });

Then, when interacting with IFusionCache, we set the distributed cache duration to the appropriate value:

return await fusionCache.GetOrSetAsync(
    "MyCacheKey",
    async _ => await RunFactory(),
    options => options.SetDistributedCacheDuration(TimeSpan.FromMinutes(5)));

Interesting... why use a fake ditributed cache (the in-memory one) while in dev env instead of not using an L2 at all?
One of the good thing about an hybrid cache is that the api surface area is transparent in respect of it being L1 or L1+L2.
You may as well use only L1 locally, and L1+L2 when in prod env.

If it is about the difference in how to set up the Duration/DistributedCacheDuration, you can just specify them differently based on the env, and even make a custom ext method that does that based on the env.

Out of curiosity: I'm thinking from some time about adding a TimeSpan? MemoryCacheDuration, that will be used for the L1 with of course a fallback to the nominal Duration. This could be a way to just work with the Duration from a logical point of view and tweak/override the one you want based on the need.
Eg: you can have the nominal Duration as "the duration for the entry" and override either the L1 or the L2 specific in case that's needed.
This though may add additional complexity and I'm still not sure it's the right way.

Thoughts?

My understanding of the following setup is:

1. L1 is enabled, so this provides stampede protection.
2. Data will be cached in L1 for up to 1 second.
3. Between second 1 and minute 5, if that cache key is requested, data will be fetched from L2, and L1 will be updated for another second.

Could you please confirm whether that's accurate?

Correct.

Special case during local dev with no real distributed cache

During local dev, we register the in-memory distributed cache, as we want cached data to be purged across application restarts.

Mmmh, help me understand: if the goal is to have clean cold starts (eg: on restarts the cache is empty), then why use an L2 at all? As stated above you can just use L1 and call it a day.

We seemed to face an issue initially where because both L1 and L2 used the same MemoryCache instance

I don't think this is how it works: if you don't specify anything in particular, FusionCache will create a private new MemoryCache(...) instance, not shared with anything else (including the fake distributed cache in-memory).

when L1 data expired after 1 second, it also expired that data for L2. Does that sounds right?

Nope, very strange.
Can you pass me an MRE or your current setup?

One workaround we found is to register a separate IMemoryCache/MemoryCache instance during local dev to keep L1 and L2 separate:

if (builder.Environment.IsDevelopment())
{
    // Use a different memory cache implementation when in local dev so the shorter
    // memory cache duration doesn't wipe out the distributed cache.
    const string localDevFusionCacheMemoryCacheKey = "LocalDevFusionCacheMemoryCache";
    builder.Services.AddKeyedSingleton<IMemoryCache, MemoryCache>(localDevFusionCacheMemoryCacheKey);
    fusionCacheBuilder.WithRegisteredKeyedMemoryCache(localDevFusionCacheMemoryCacheKey);
}

Does that look OK to you? Is there a better way to do this?

If the need is not to share the MemoryCache with anything else, instead of doing WithRegisteredKeyedMemoryCache(...) or WithRegisteredMemoryCache() or else, you can just avoid specifying anything a all, like this:

// THIS WILL HAVE ITS OWN PRIVATE MemoryCache
services.AddFusionCache();

Hope this helps, let me know.

[mderriey]

We're currently using raw IDistributedCache, and want to evaluate options that can provide stampede protection. It's worth noting that we're not interested in caching data in memory.

Got it. Out of curiosity, if I may, why is that? Memory allocation concerns? Fear of updates not being reflected instantly? Something else?

Some more context is probably worth explaining why that is.

This project started 6 months ago, and we're only considering FusionCache now.

We started caching data in memory only, because we didn't need to invalidate cache entries at first.
When we started needing to invalidate, we wanted the invalidation to happen across instances, so we switched from IMemoryCache to IDistributedCache, using the in-memory distributed cache in local dev.

We're close to delivery, and while we want to minimze risk associated with the overhaul of our caching strategy, we also realize that we're subject to cache stampede with some operations that are both taking a long time and will be called often.

In this context, we're evaluating using a caching library that can provide stampede protection, but only for those operations that are "at risk". We're also at the moment not interested in other features like fail-safe or backplane. We do see the benefits they could bring, but it's a decision we're not willing to make just yet.
This means the code base, for now, will be using a mix of raw IDistributedCache and IFusionCache.

I'll refer back to this first long explanation for most of the questions you have.

Interesting... why use a fake ditributed cache (the in-memory one) while in dev env instead of not using an L2 at all?
One of the good thing about an hybrid cache is that the api surface area is transparent in respect of it being L1 or L1+L2.
You may as well use only L1 locally, and L1+L2 when in prod env.

Back to the context piece above, we're still using raw IDitributedCache, so we need an implementation available at runtime.

Out of curiosity: I'm thinking from some time about adding a TimeSpan? MemoryCacheDuration, that will be used for the L1 with of course a fallback to the nominal Duration. This could be a way to just work with the Duration from a logical point of view and tweak/override the one you want based on the need.
Eg: you can have the nominal Duration as "the duration for the entry" and override either the L1 or the L2 specific in case that's needed.
This though may add additional complexity and I'm still not sure it's the right way.

Personally, I found the Duration and DistributedCacheDuration a bit confusing. The names suggest that Duration is the common case, and DistributedCacheDuration is specific to distributed cache, but I had to look at the docs to confirm that's the case.

I'm not sure how I'd feel about MemoryCacheDuration. I think I'd then prefer to rename Duration to DefaultDuration or FallbackDuration, to indicate that it's used only if the specific ones don't have a value, but that's a breaking change.

During local dev, we register the in-memory distributed cache, as we want cached data to be purged across application restarts.

Mmmh, help me understand: if the goal is to have clean cold starts (eg: on restarts the cache is empty), then why use an L2 at all? As stated above you can just use L1 and call it a day.

I understand the question. Hopefully the additional context I provided above answers that.

when L1 data expired after 1 second, it also expired that data for L2. Does that sounds right?

Nope, very strange. Can you pass me an MRE or your current setup?

If the need is not to share the MemoryCache with anything else, instead of doing WithRegisteredKeyedMemoryCache(...) or WithRegisteredMemoryCache() or else, you can just avoid specifying anything a all, like this:

// THIS WILL HAVE ITS OWN PRIVATE MemoryCache
services.AddFusionCache();

Riiiight, OK, that makes sense.

I think my first attempt didn't have distributed cache enabled, as I thought it was enabled by default.
What I thought was L1 and L2 sharing the same memory cache was in fact the app was just using L1, effectively causing more cache misses than I expected.

I removed the FusionCache-specific memory cache implementation and it all works a treat.
No issues in the library, just me holding it wrong.

Thanks a ton for the long and detailed reply, it means a lot.

[jodydonetti]

Some more context is probably worth explaining why that is.

This project started 6 months ago, and we're only considering FusionCache now.

We started caching data in memory only, because we didn't need to invalidate cache entries at first. When we started needing to invalidate, we wanted the invalidation to happen across instances, so we switched from IMemoryCache to IDistributedCache, using the in-memory distributed cache in local dev.

Makes sense, and a pretty common approach.

We're close to delivery, and while we want to minimze risk associated with the overhaul of our caching strategy, we also realize that we're subject to cache stampede with some operations that are both taking a long time and will be called often.

Apropos of "operations that are both taking a long time and will be called often", although we already talked about cache stampede protection, I would suggest you also take a look at both soft/hard timeouts and eager refresh: they are different techniques to tackle the same problem (meaning: slow factories).
The end result is the best of both worlds: almost always super fast response + fresh data as soon as possible.

If you try to use them, usually the next question is "ok, which one should I use?": the answer is "one, the other, or both, up to you".
Personally I use them both all the times, because eager refresh help me by not having to wait for the exact expiration to occur by anticipating it a little bit in a non-blocking way, whereas soft/hard timeouts help me when the request that kickstart the request arrives after the expiration.

Hope this helps.

In this context, we're evaluating using a caching library that can provide stampede protection, but only for those operations that are "at risk". We're also at the moment not interested in other features like fail-safe or backplane. We do see the benefits they could bring, but it's a decision we're not willing to make just yet.

Got it, makes sense.

This means the code base, for now, will be using a mix of raw IDistributedCache and IFusionCache.

Ok, just watch out that they are not binary compatible on the distributed cache: what I mean is that if you save in the distributed cache something via FusionCache, you should not try to read it via a raw IDistributedCache, because the internal binary format is not so "plain to read".
Maybe it was already clear, maybe not, just wanted to highlight this.

Personally, I found the Duration and DistributedCacheDuration a bit confusing. The names suggest that Duration is the common case, and DistributedCacheDuration is specific to distributed cache, but I had to look at the docs to confirm that's the case.

Thanks for the insights into your experience, it is really valuable to me as it helps me put into other people's shoes!
So in the end you got it right, but felt the need to read the docs to confirm it, got it.

I'm not sure how I'd feel about MemoryCacheDuration. I think I'd then prefer to rename Duration to DefaultDuration or FallbackDuration, to indicate that it's used only if the specific ones don't have a value, but that's a breaking change.

Note taken.

I removed the FusionCache-specific memory cache implementation and it all works a treat. No issues in the library, just me holding it wrong.

Great 💪

Thanks a ton for the long and detailed reply, it means a lot.

Are you kidding me? Again, being able to have these conversations helps me incredibly, so thanks to you!

[mderriey]

Apropos of "operations that are both taking a long time and will be called often", although we already talked about cache stampede protection, I would suggest you also take a look at both soft/hard timeouts and eager refresh [...]

Thanks, we'll definitely take a look at these in the future.

Ok, just watch out that they are not binary compatible on the distributed cache: what I mean is that if you save in the distributed cache something via FusionCache, you should not try to read it via a raw IDistributedCache, because the internal binary format is not so "plain to read".
Maybe it was already clear, maybe not, just wanted to highlight this.

Phew, thanks a lot for that! We've added a custom cache key prefix through FusionCache, so the existing cache entries will naturally expire, and we'll create new ones when we deploy FusionCache.