Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Issue-6642 Add http.send request attribute to ignore headers for caching key #6675

Open
wants to merge 2 commits into
base: main
Choose a base branch
from

Conversation

rudrakhp
Copy link
Contributor

@rudrakhp rudrakhp commented Apr 4, 2024

Why the changes in this PR are needed?

Need to support exclusion of certain headers from http.send query cache key.

What are the changes in this PR?

  • A new attribute cache_ignored_headers in the http.send built in request object. Enables policy authors to define an exclusion list for headers to ignore while caching.
  • Modify key used by the HTTP request executor when building the cache key.
  • Unit tests for these changes.

Notes to assist PR review:

  • Note that if request attribute cache_ignored_headers changes but does not affect the final computed cache key, the results are still served from cache. This attribute is always excluded from the cache key.
  • If there is a change in the exclusion list that leads to change in the cache key (addition/removal of a header), a new cache key is generated which leads to a cache miss.

Further comments:

Resolves #6642

Copy link
Member

@ashutosh-narkar ashutosh-narkar left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for working on this. Here we are excluding certain headers from the key calculation. Is there a reason for that level of granularity? Why not exclude certain request object params themselves from the key calculation?

@rudrakhp
Copy link
Contributor Author

rudrakhp commented Apr 5, 2024

Thanks for working on this. Here we are excluding certain headers from the key calculation. Is there a reason for that level of granularity? Why not exclude certain request object params themselves from the key calculation?

I would have to explore if there is any straight forward way to delete a value by path reference in the Request ast.Object, since we might require that to ignore any generic request attribute from the cache key. Let me get back to you on this next week. Meanwhile please do share if you have any pointers. Thanks!

@ashutosh-narkar
Copy link
Member

@rudrakhp just checking to see if you got a chance to explore the option of excluding certain request object params.

@rudrakhp
Copy link
Contributor Author

@ashutosh-narkar was out last week, will raise an updated PR this week.

Copy link

stale bot commented May 17, 2024

This pull request has been automatically marked as stale because it has not had any activity in the last 30 days.

@stale stale bot added the inactive label May 17, 2024
Copy link
Member

@ashutosh-narkar ashutosh-narkar left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@rudrakhp the changes seem fine. More testing would be helpful. Also we need to update the docs for the builtin.

func getKeyFromRequest(req ast.Object) (ast.Object, error) {
var cacheIgnoredHeaders []string
var allHeaders map[string]interface{}
cacheIgnoredHeadersTerm := req.Get(ast.StringTerm("cache_ignored_headers"))
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nit: We can do an early exit here

if cacheIgnoredHeadersTerm == nil {
    return nil, nil
}

expectedReqCount: 1,
},
{
note: "http.send GET cache miss different headers (force_cache enabled)",
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Not sure what scenario in the changes is this test case trying to exercise.

note: "http.send GET different cache_ignored_headers but still cached (force_cache enabled)",
ruleTemplate: `p = x {
r1 = http.send({"method": "get", "url": "%URL%", "force_json_decode": true, "headers": {"h1": "v1", "h2": "v2"}, "force_cache": true, "force_cache_duration_seconds": 300, "cache_ignored_headers": ["h2"]})
r2 = http.send({"method": "get", "url": "%URL%", "force_json_decode": true, "headers": {"h1": "v1", "h2": "v2"}, "force_cache": true, "force_cache_duration_seconds": 300, "cache_ignored_headers": ["h2", "h3"]}) # cached
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Just for testing we can actually have a h3 header in the headers object.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Also a test for the scenario when the value of cache_ignored_headers and headers differs and we get a cache miss would be helpful

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

One more case I can think of:

R1: {"headers": {"h1": "v1"}}
R2: {"headers": {"h1": "v1", "h2": "v2"}, "cache_ignored_headers": ["h2"]}

So here R1 and R2 are equivalent, correct?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Another one:

R1: {"headers": {"h1": "v1"}, "cache_ignored_headers": []}
R2: {"headers": {"h1": "v1", "h2": "v2"}, "cache_ignored_headers": ["h2"]}

So here R1 and R2 are equivalent, correct?

@stale stale bot removed the inactive label May 18, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Interquery http.send cache - Add provision to exclude/include headers from cache key
2 participants