Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Introduce graphql-cache #411

Merged
merged 9 commits into from
Apr 22, 2020
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
5 changes: 3 additions & 2 deletions Gemfile
Original file line number Diff line number Diff line change
Expand Up @@ -11,7 +11,7 @@ gem 'puma', '~> 3.12'
# Build JSON APIs with ease. Read more: https://github.com/rails/jbuilder
# gem 'jbuilder', '~> 2.5'
# Use Redis adapter to run Action Cable in production
gem 'redis', '~> 4.0'
gem 'hiredis'
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is just a more performant version of the redis gem

# Use ActiveModel has_secure_password
# gem 'bcrypt', '~> 3.1.7'

Expand Down Expand Up @@ -68,13 +68,14 @@ gem 'prometheus_exporter'

gem 'activerecord-import'
gem 'oj'
gem 'newrelic_rpm'
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We are simply not using it and it adds overhead to every API call

gem 'gitlab-sidekiq-fetcher', require: 'sidekiq-reliable-fetch'

gem 'uuid'

gem 'openscap_parser', '~> 1.0.0'

gem 'graphql-cache'

group :development, :test do
gem 'brakeman'
gem 'byebug', platforms: %i[mri mingw x64_mingw]
Expand Down
8 changes: 5 additions & 3 deletions Gemfile.lock
Original file line number Diff line number Diff line change
Expand Up @@ -145,7 +145,10 @@ GEM
graphql-batch (0.4.2)
graphql (>= 1.3, < 2)
promise.rb (~> 0.7.2)
graphql-cache (0.6.0)
graphql (~> 1, > 1.8)
hashdiff (1.0.0)
hiredis (0.6.3)
i18n (1.8.2)
concurrent-ruby (~> 1.0)
io-console (0.5.4)
Expand Down Expand Up @@ -183,7 +186,6 @@ GEM
msgpack (1.3.1)
multi_json (1.14.1)
multipart-post (2.1.1)
newrelic_rpm (6.8.0.360)
nio4r (2.5.2)
nokogiri (1.10.9)
mini_portile2 (~> 2.4.0)
Expand Down Expand Up @@ -370,11 +372,12 @@ DEPENDENCIES
graphiql-rails
graphql
graphql-batch
graphql-cache
hiredis
irb (>= 1.2)
listen (>= 3.0.5, < 3.2)
minitest-reporters
mocha
newrelic_rpm
oj
openscap_parser (~> 1.0.0)
pg
Expand All @@ -387,7 +390,6 @@ DEPENDENCIES
racecar
rack-cors
rails (~> 5.2.1)
redis (~> 4.0)
rspec-rails
rswag-api
rswag-specs
Expand Down
3 changes: 1 addition & 2 deletions app/graphql/schema.rb
Original file line number Diff line number Diff line change
Expand Up @@ -15,6 +15,5 @@ class Schema < GraphQL::Schema
mutation Types::Mutation
lazy_resolve(Promise, :sync)
use GraphQL::Batch
use GraphQL::Execution::Interpreter
use GraphQL::Analysis::AST
use GraphQL::Cache
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Unfortunately the new interpreter is incompatible with gql-cache (https://graphql-ruby.org/queries/interpreter.html) (stackshareio/graphql-cache#71)

end
1 change: 1 addition & 0 deletions app/graphql/types/base_object.rb
Original file line number Diff line number Diff line change
Expand Up @@ -9,6 +9,7 @@ class BaseObject < GraphQL::Types::Relay::BaseObject
global_id_field :global_id
field :node, field: GraphQL::Relay::Node.field
field :nodes, field: GraphQL::Relay::Node.plural_field
field_class ::GraphQL::Cache::Field

class << self
def model_class(new_model_class = nil)
Expand Down
14 changes: 7 additions & 7 deletions app/graphql/types/benchmark.rb
Original file line number Diff line number Diff line change
Expand Up @@ -6,13 +6,13 @@ class Benchmark < Types::BaseObject
graphql_name 'Benchmark'
description 'A representation of a SCAP Security Guide version'

field :id, ID, null: false
field :description, String, null: true
field :title, String, null: false
field :ref_id, String, null: false
field :version, String, null: false
field :profiles, [::Types::Profile], null: true
field :rules, [::Types::Rule], null: true
field :id, ID, null: false, cache: true
field :description, String, null: true, cache: true
field :title, String, null: false, cache: true
field :ref_id, String, null: false, cache: true
field :version, String, null: false, cache: true
field :profiles, [::Types::Profile], null: true, cache: true
field :rules, [::Types::Rule], null: true, cache: true

def profiles
object.profiles.canonical
Expand Down
4 changes: 2 additions & 2 deletions app/graphql/types/business_objective.rb
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,7 @@ class BusinessObjective < Types::BaseObject
graphql_name 'BusinessObjective'
description 'A Business Objective registered in Insights Compliance'

field :id, ID, null: false
field :title, String, null: false
field :id, ID, null: false, cache: true
field :title, String, null: false, cache: true
end
end
1 change: 1 addition & 0 deletions app/graphql/types/interfaces/rules_preload.rb
Original file line number Diff line number Diff line change
Expand Up @@ -4,6 +4,7 @@
# rule results, compliant,and references
module RulesPreload
include GraphQL::Schema::Interface
field_class GraphQL::Cache::Field
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is this required for some reason? This module is implemented from classes which inherit from BaseObject, so I thought the field_class would be set already.


def rules(args = {})
context[:parent_profile_id] ||= {}
Expand Down
48 changes: 25 additions & 23 deletions app/graphql/types/profile.rb
Original file line number Diff line number Diff line change
Expand Up @@ -13,59 +13,61 @@ class Profile < Types::BaseObject
graphql_name 'Profile'
description 'A Profile registered in Insights Compliance'

field :id, ID, null: false
field :name, String, null: false
field :description, String, null: true
field :ref_id, String, null: false
field :compliance_threshold, Float, null: false
field :benchmark_id, ID, null: false
field :account_id, ID, null: false
field :policy, Types::Profile, null: true
field :rules, [::Types::Rule], null: true, extras: [:lookahead] do
field :id, ID, null: false, cache: true
field :name, String, null: false, cache: true
field :description, String, null: true, cache: true
field :ref_id, String, null: false, cache: true
field :compliance_threshold, Float, null: false, cache: true
field :benchmark_id, ID, null: false, cache: true
field :account_id, ID, null: false, cache: true
field :policy, Types::Profile, null: true, cache: true
field :rules, [::Types::Rule], null: true, cache: true,
extras: [:lookahead] do
argument :system_id, String,
'System ID to filter by', required: false
argument :identifier, String,
'Rule identifier to filter by', required: false
argument :references, [String],
'Rule references to filter by', required: false
end
field :hosts, [::Types::System], null: true
field :benchmark, ::Types::Benchmark, null: true
field :business_objective, ::Types::BusinessObjective, null: true
field :business_objective_id, ID, null: true
field :total_host_count, Int, null: false
field :external, Boolean, null: false

field :score, Float, null: false do
field :hosts, [::Types::System], null: true, cache: true
field :benchmark, ::Types::Benchmark, null: true, cache: true
field :business_objective, ::Types::BusinessObjective, null: true,
cache: true
field :business_objective_id, ID, null: true, cache: true
field :total_host_count, Int, null: false, cache: true
field :external, Boolean, null: false, cache: true

field :score, Float, null: false, cache: true do
argument :system_id, String,
'Latest TestResult score for this system and profile',
required: false
end

field :compliant, Boolean, null: false do
field :compliant, Boolean, null: false, cache: true do
argument :system_id, String, 'Is a system compliant with this profile?',
required: false
end

field :rules_passed, Int, null: false do
field :rules_passed, Int, null: false, cache: true do
argument :system_id, String,
'Rules passed for a system and a profile', required: false
end

field :rules_failed, Int, null: false do
field :rules_failed, Int, null: false, cache: true do
argument :system_id, String,
'Rules failed for a system and a profile', required: false
end

field :last_scanned, String, null: false do
field :last_scanned, String, null: false, cache: true do
argument :system_id, String,
'Last time this profile was scanned for a system',
required: false
end

field :compliant_host_count, Int, null: false
field :compliant_host_count, Int, null: false, cache: true

field :major_os_version, String, null: false
field :major_os_version, String, null: false, cache: true

def compliant_host_count
::CollectionLoader.for(object.class, :hosts).load(object).then do |hosts|
Expand Down
24 changes: 12 additions & 12 deletions app/graphql/types/rule.rb
Original file line number Diff line number Diff line change
Expand Up @@ -6,17 +6,17 @@ class Rule < Types::BaseObject
graphql_name 'Rule'
description 'A Rule registered in Insights Compliance'

field :id, ID, null: false
field :title, String, null: false
field :ref_id, String, null: false
field :rationale, String, null: true
field :description, String, null: true
field :severity, String, null: false
field :remediation_available, Boolean, null: false
field :profiles, [::Types::Profile], null: true
field :identifier, String, null: true
field :references, String, null: true
field :compliant, Boolean, null: false do
field :id, ID, null: false, cache: true
field :title, String, null: false, cache: true
field :ref_id, String, null: false, cache: true
field :rationale, String, null: true, cache: true
field :description, String, null: true, cache: true
field :severity, String, null: false, cache: true
field :remediation_available, Boolean, null: false, cache: true
field :profiles, [::Types::Profile], null: true, cache: true
field :identifier, String, null: true, cache: true
field :references, String, null: true, cache: true
field :compliant, Boolean, null: false, cache: true do
argument :system_id, String, 'Is a system compliant?',
required: false
argument :profile_id, String, 'Is a system compliant with this profile?',
Expand All @@ -35,7 +35,7 @@ def references
if context[:"rule_references_#{object.id}"].nil?
::CollectionLoader.for(::Rule, :rule_references)
.load(object).then do |references|
references.map { |ref| [ref.href, ref.label] }.to_json
references.map { |ref| { href: ref.href, label: ref.label } }.to_json
end
else
references_from_context
Expand Down
18 changes: 9 additions & 9 deletions app/graphql/types/system.rb
Original file line number Diff line number Diff line change
Expand Up @@ -7,26 +7,26 @@ class System < Types::BaseObject
graphql_name 'System'
description 'A System registered in Insights Compliance'

field :id, ID, null: false
field :name, String, null: false
field :profiles, [::Types::Profile], null: true
field :compliant, Boolean, null: false do
field :id, ID, null: false, cache: true
field :name, String, null: false, cache: true
field :profiles, [::Types::Profile], null: true, cache: true
field :compliant, Boolean, null: false, cache: true do
argument :profile_id, String, 'Filter results by profile ID',
required: false
end
field :profile_names, String, null: false
field :rules_passed, Int, null: false do
field :profile_names, String, null: false, cache: true
field :rules_passed, Int, null: false, cache: true do
argument :profile_id, String, 'Filter results by profile ID',
required: false
end
field :rules_failed, Int, null: false do
field :rules_failed, Int, null: false, cache: true do
argument :profile_id, String, 'Filter results by profile ID',
required: false
end
field :rule_objects_failed, [::Types::Rule], null: true do
field :rule_objects_failed, [::Types::Rule], null: true, cache: true do
description 'Rules failed by a system'
end
field :last_scanned, String, null: true do
field :last_scanned, String, null: true, cache: true do
argument :profile_id, String, 'Filter results by profile ID',
required: false
end
Expand Down
12 changes: 6 additions & 6 deletions app/graphql/types/test_result.rb
Original file line number Diff line number Diff line change
Expand Up @@ -7,11 +7,11 @@ class TestResult < Types::BaseObject
graphql_name 'TestResult'
description 'A TestResult as recorded in Insights Compliance'

field :id, ID, null: false
field :start_time, String, null: true
field :end_time, String, null: false
field :score, Float, null: false
field :profile, ::Types::Profile, null: false
field :host, ::Types::System, null: false
field :id, ID, null: false, cache: true
field :start_time, String, null: true, cache: true
field :end_time, String, null: false, cache: true
field :score, Float, null: false, cache: true
field :profile, ::Types::Profile, null: false, cache: true
field :host, ::Types::System, null: false, cache: true
end
end
36 changes: 36 additions & 0 deletions config/initializers/graphql_cache.rb
Original file line number Diff line number Diff line change
@@ -0,0 +1,36 @@
GraphQL::Cache.configure do |config|
config.namespace = 'compliance' # Cache key prefix for keys generated by graphql-cache
config.cache = Rails.cache # The cache object to use for caching
config.logger = Rails.logger # Logger to receive cache-related log messages
config.expiry = 15552000 # 6 months (in seconds)
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We should probably discuss this and even think more seriously about introducing an LRU cache redis instance directly. Fields are cached alongside its updated_at, e.g: compliance:Rule:rules/30b9d4c7-4924-4996-bd89-7e29117fccf7-20200227230834209106:title to force us to realize whether the key can be used or not. It's a sort of 'automatic invalidation' as any time a rule is updated, the old entries will be useless.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

So the old entries with older updated_at in the key are kept in redis?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Correct, that's why I'd want to go with an instance like https://redis.io/topics/lru-cache

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Correct, that's why I'd want to go with an instance like https://redis.io/topics/lru-cache

end

module GraphQL
module Cache
module DeconstructorExtensions
def perform
return raw.value if method == 'lazy'
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

GraphQL::Cache::Deconstructor figures out which sort of value each field is returning. Based on that, the current methods cannot handle lazy methods, or Promises. This is just a way to handle those.

if raw.is_a?(Promise)
raw.wait
raw.value
end
super
end
end

class Deconstructor
prepend DeconstructorExtensions
end

class Key
def to_s
@to_s ||= [
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This overrides the way the keys are set - I'll comment more on this in a few

GraphQL::Cache.namespace,
object_clause,
arguments_clause,
field_clause,
].flatten.compact.join(':')
end
end
end
end
4 changes: 2 additions & 2 deletions test/graphql/queries/rule_query_test.rb
Original file line number Diff line number Diff line change
Expand Up @@ -101,8 +101,8 @@ class RuleQueryTest < ActiveSupport::TestCase
assert_not result.dig('errors'),
"Query was unsuccessful: #{result.dig('errors')}"
assert result.dig('data', 'profile', 'rules').any?, 'No rules returned!'
assert_equal [[rule_references(:one).href,
rule_references(:one).label]].to_json,
assert_equal [{ href: rule_references(:one).href,
label: rule_references(:one).label }].to_json,
result.dig('data', 'profile', 'rules',
0, 'references')
end
Expand Down