Implement lookup_citation model to deal w/ lambda

** Why are these changes being introduced:

We need a way for TACOS to communicate with the new detector lambda that
serves the next iteration of our citation detector.

** Relevant ticket(s):

* https://mitlibraries.atlassian.net/browse/tco-183

** How does this address that need:

* The big change is that we add a lookup_citation model that handles
communication with the lambda. The patterns used are similar to our
other external integrations (the other lookup_ classes) - although this
is not consistently implemented.

* A secondary change is to define a fourth instance variable on the
citation model, for @features. This is compiled from the existing work,
reformatting and joining two other instance variables.

* There is a mismatch between the features generated by the existing
citation class and those expected by the new detector. Two features
have different names, while one feature is not used at all. That change
is handled via the extract_features method on lookup_citation, in order
to make these changes as close as possible to the external call.

** Document any side effects to this change:

* Using a lookup_* pattern for this is a departure from the way the
other classes are used, and I'm tempted to refactor this PR in order
to adopt a gateway pattern instead. The other lookup classes are used
by GraphQL to look up external information after a detector has fired -
while this lookup_citation is much earlier in the application flow,
being part of the detector itself.

Author: matt-bernhardt

.env.test

@@ -3,3 +3,6 @@ LINKRESOLVER_BASEURL=https://mit.primo.exlibrisgroup.com/discovery/openurl?insti
 TACOS_EMAIL=[email protected]
 LIBKEY_KEY=FAKE_LIBKEY_KEY
 LIBKEY_ID=FAKE_LIBKEY_ID
+DETECTOR_LAMBDA_URL=http://localhost:3000
+DETECTOR_LAMBDA_PATH=/foo
+DETECTOR_LAMBDA_CHALLENGE_SECRET=secret_phrase

app/models/detector/citation.rb

@@ -10,7 +10,7 @@ class Detector
   # hallmarks of being a citation.
   # Phrases whose score is higher than the REQUIRED_SCORE value can be registered as a Detection.
   class Citation
-    attr_reader :score, :subpatterns, :summary
+    attr_reader :features, :score, :subpatterns, :summary
 
     # shared singleton methods
     extend Detector::BulkChecker
@@ -67,10 +67,13 @@ def detection?
     # @return Nothing intentional. Data is written to Hashes `@subpatterns`, `@summary`,
     #   and `@score` during processing.
     def initialize(phrase)
+      @features = {}
       @subpatterns = {}
       @summary = {}
       pattern_checker(phrase)
       summarize(phrase)
+      @features = @subpatterns.deep_dup.transform_values(&:length).merge(summary)
+      @subpatterns.delete_if { |_, v| v == [] }
       @score = calculate_score
     end
 
@@ -141,7 +144,7 @@ def commas(phrase)
     # @return hash
     def pattern_checker(phrase)
       CITATION_PATTERNS.each_pair do |type, pattern|
-        @subpatterns[type.to_sym] = scan(pattern, phrase) if scan(pattern, phrase).present?
+        @subpatterns[type.to_sym] = scan(pattern, phrase)
       end
     end
 

app/models/lookup_citation.rb

@@ -0,0 +1,97 @@
+# frozen_string_literal: true
+
+class LookupCitation
+  # The info method is the way to return information about whether a given phrase is a citation. It consults an
+  # external lambda service (address in env) and returns either a true or a false. The default if anything goes wrong
+  # is to return false.
+  #
+  # @return Boolean or nil
+  def info(phrase)
+    return unless expected_env?
+
+    external_data = fetch(phrase)
+    return if external_data == 'Error'
+
+    external_data
+  end
+
+  private
+
+  def lambda_path
+    ENV.fetch('DETECTOR_LAMBDA_PATH', nil)
+  end
+
+  def lambda_secret
+    ENV.fetch('DETECTOR_LAMBDA_CHALLENGE_SECRET', nil)
+  end
+
+  def lambda_url
+    ENV.fetch('DETECTOR_LAMBDA_URL', nil)
+  end
+
+  # define_lambda connects to the detector lambda.
+  #
+  # @return Faraday connection
+  def define_lambda
+    Faraday.new(
+      url: lambda_url,
+      params: {}
+    )
+  end
+
+  # define_payload defines the Hash that will be sent to the lambda.
+  #
+  # @return Hash
+  def define_payload(phrase)
+    {
+      action: 'predict',
+      features: extract_features(phrase),
+      challenge_secret: lambda_secret
+    }
+  end
+
+  # expected_env? confirms that all three required environment variables are defined.
+  #
+  # @return Boolean
+  def expected_env?
+    Rails.logger.error('No lambda URL defined') if lambda_url.nil?
+
+    Rails.logger.error('No lambda path defined') if lambda_path.nil?
+
+    Rails.logger.error('No lambda secret defined') if lambda_secret.nil?
+
+    [lambda_url, lambda_path, lambda_secret].all?(&:present?)
+  end
+
+  # extract_features passes the search phrase through the citation detector, and massages the resulting features object
+  # to correspond with what the lambda expects.
+  #
+  # @return Hash
+  def extract_features(phrase)
+    features = Detector::Citation.new(phrase).features
+    features[:apa] = features.delete :apa_volume_issue
+    features[:year] = features.delete :year_parens
+    features.delete :characters
+    features
+  end
+
+  # Fetch handles the communication with the detector lambda: defining the connection, building the payload, and any
+  # error handling with the response.
+  #
+  # @return Boolean or 'Error'
+  def fetch(phrase)
+    lambda = define_lambda
+    payload = define_payload(phrase)
+
+    response = lambda.post(lambda_path, payload.to_json)
+
+    if response.status == 200
+      JSON.parse(response.body)['response'] == 'true'
+    else
+      Rails.logger.error(response.body)
+      Rails.logger.error(response.body['error'])
+
+      'Error'
+    end
+  end
+end

test/models/detector/citation_test.rb

@@ -4,10 +4,12 @@
 
 class Detector
   class CitationTest < ActiveSupport::TestCase
-    test 'detector::citation exposes three instance variables' do
+    test 'detector::citation exposes four instance variables' do
       t = terms('citation')
       result = Detector::Citation.new(t.phrase)
 
+      assert_predicate result.features, :present?
+
       assert_predicate result.score, :present?
 
       assert_predicate result.summary, :present?
@@ -196,6 +198,29 @@ class CitationTest < ActiveSupport::TestCase
       assert_operator 0, :<, result.score
     end
 
+    test 'features instance method is a hash of integers' do
+      result = Detector::Citation.new('simple search phrase')
+
+      assert_instance_of(Hash, result.features)
+
+      assert(result.features.all? { |_, v| v.integer? })
+    end
+
+    test 'features instance method includes all elements of citation detector regardless of search string' do
+      result_simple = Detector::Citation.new('simple')
+      result_complex = Detector::Citation.new('Science Education and Cultural Diversity: Mapping the Field. Studies in Science Education, 24(1), 49–73.')
+
+      assert_equal result_simple.features.length, result_complex.features.length
+    end
+
+    test 'features instance method should include all elements of citation patterns and summary thresholds' do
+      patterns = Detector::Citation.const_get :CITATION_PATTERNS
+      summary = Detector::Citation.const_get :SUMMARY_THRESHOLDS
+      result = Detector::Citation.new('simple')
+
+      assert_equal (patterns.length + summary.length), result.features.length
+    end
+
     test 'detection? convenience method returns true for obvious citations' do
       result = Detector::Citation.new(terms('citation').phrase)
 

test/models/lookup_citation_test.rb

@@ -0,0 +1,43 @@
+# frozen_string_literal: true
+
+require 'test_helper'
+
+class LookupCitationTest < ActiveSupport::TestCase
+  test 'DETECTOR_LAMBDA_CHALLENGE_SECRET is required' do
+    ClimateControl.modify DETECTOR_LAMBDA_CHALLENGE_SECRET: nil do
+      assert_nil(LookupCitation.new.info('ping'))
+    end
+  end
+
+  test 'DETECTOR_LAMBDA_PATH is required' do
+    ClimateControl.modify DETECTOR_LAMBDA_PATH: nil do
+      assert_nil(LookupCitation.new.info('ping'))
+    end
+  end
+
+  test 'DETECTOR_LAMBDA_URL is required' do
+    ClimateControl.modify DETECTOR_LAMBDA_URL: nil do
+      assert_nil(LookupCitation.new.info('ping'))
+    end
+  end
+
+  test 'lookup returns true when lambda running' do
+    # These cassettes should be regenerated once the lambda is running in AWS. For now it will need to be running
+    # on localhost should the cassettes need to be regenerated.
+    VCR.use_cassette('lambda running') do
+      prediction = LookupCitation.new.info('ping')
+
+      assert(prediction)
+    end
+  end
+
+  test 'lookup returns nil when challenge_secret is wrong' do
+    ClimateControl.modify DETECTOR_LAMBDA_CHALLENGE_SECRET: 'something wrong' do
+      VCR.use_cassette('lambda with wrong secret') do
+        prediction = LookupCitation.new.info('oops')
+
+        assert_nil(prediction)
+      end
+    end
+  end
+end

test/test_helper.rb

@@ -15,6 +15,7 @@
 require 'rails/test_help'
 
 VCR.configure do |config|
+  config.ignore_localhost = false
   config.cassette_library_dir = 'test/vcr_cassettes'
   config.hook_into :webmock