Add dig feature. (crayfishx#66)

* added dig features

* added docs

* typo

* updated README with more about digging

* [skip ci] fixed example error in README

Author: crayfishx

README.md

@@ -64,39 +64,47 @@ The following mandatory Hiera 5 options must be set for each level of the hierar
 
 The following are optional configuration parameters supported in the `options` hash of the Hiera 5 config
 
-`:output: ` : Specify what handler to use for the output of the request.  Currently supported outputs are plain, which will just return the whole document, or YAML and JSON which parse the data and try to look up the key
+#### Lookup options
 
-`:http_connect_timeout: ` : Timeout in seconds for the HTTP connect (default 10)
+`output: ` : Specify what handler to use for the output of the request.  Currently supported outputs are plain, which will just return the whole document, or YAML and JSON which parse the data and try to look up the key
 
-`:http_read_timeout: ` : Timeout in seconds for waiting for a HTTP response (default 10)
+`http_connect_timeout: ` : Timeout in seconds for the HTTP connect (default 10)
 
-`:confine_to_keys: ` : Only use this backend if the key matches one of the regexes in the array
+`http_read_timeout: ` : Timeout in seconds for waiting for a HTTP response (default 10)
+
+`confine_to_keys: ` : Only use this backend if the key matches one of the regexes in the array
 
       confine_to_keys:
         - "application.*"
         - "apache::.*"
 
-`:failure: ` : When set to `graceful` will stop hiera-http from throwing an exception in the event of a connection error, timeout or invalid HTTP response and move on.  Without this option set hiera-http will throw an exception in such circumstances
+`failure: ` : When set to `graceful` will stop hiera-http from throwing an exception in the event of a connection error, timeout or invalid HTTP response and move on.  Without this option set hiera-http will throw an exception in such circumstances
+
+`ignore_404: ` : If `failure` is _not_ set to `graceful` then any error code received from the HTTP response will throw an exception.  This option makes 404 responses exempt from exceptions.  This is useful if you expect to get 404's for data items not in a certain part of the hierarchy and need to fall back to the next level in the hierarchy, but you still want to bomb out on other errors.
+
+`dig:` : (true or false)  When the output is parsed YAML or JSON, whether or not to dig into the hash and return the value defined by the `dig_key` option below.  This option defaults to `true`
+
+`dig_key` : When the `dig` option is true (default), this option specifies what key is looked up from the results hash returned by the HTTP endpoint.  See [Digging values](#digging-values) below for more information
 
-`:ignore_404: ` : If `failure` is _not_ set to `graceful` then any error code received from the HTTP response will throw an exception.  This option makes 404 responses exempt from exceptions.  This is useful if you expect to get 404's for data items not in a certain part of the hierarchy and need to fall back to the next level in the hierarchy, but you still want to bomb out on other errors.
+#### HTTP options
 
-`:use_ssl:`: When set to true, enable SSL (default: false)
+`use_ssl:`: When set to true, enable SSL (default: false)
 
-`:ssl_ca_cert`: Specify a CA cert for use with SSL
+`ssl_ca_cert`: Specify a CA cert for use with SSL
 
-`:ssl_cert`: Specify location of SSL certificate
+`ssl_cert`: Specify location of SSL certificate
 
-`:ssl_key`: Specify location of SSL key
+`ssl_key`: Specify location of SSL key
 
-`:ssl_verify`: Specify whether to verify SSL certificates (default: true)
+`ssl_verify`: Specify whether to verify SSL certificates (default: true)
 
-`:use_auth:`: When set to true, enable basic auth (default: false)
+`use_auth:`: When set to true, enable basic auth (default: false)
 
-`:auth_user:`: The user for basic auth
+`auth_user:`: The user for basic auth
 
-`:auth_pass:`: The password for basic auth
+`auth_pass:`: The password for basic auth
 
-`:headers:`: Hash of headers to send in the request
+`headers:`: Hash of headers to send in the request
 
 ### Interpolating special tags
 
@@ -130,6 +138,71 @@ hierarchy:
       failure: graceful
 ```
 
+### Digging values
+
+Hiera-HTTP supports options to automatically dig into the returned data structure to find a corresponding key.  Puppet lookup itself supports similar dig functionality but being able to specify it at the backend means that where an API wraps the required data up in a different way, we can always lookup the desired value before passing it to Puppet to ensure that class parameter lookups work without having to hard code the `lookup` function and dig down into the data for each request.   The dig functionality in Puppet is intended to enable you to parse your data more effectivly, the dig functionality in hiera-http is intended to make the API of the endpoint you are talking to compatible.
+
+By default, when a hash is returned by the HTTP endpoint (eg: JSON) then hiera-http will attempt to lookup the key corresponding with the lookup key.  For example, when looking up a key `apache::port` we would expect the HTTP endpoint to return something like;
+
+```json
+{
+  "apache::port": 80
+}
+```
+
+Returned value would be `80`
+
+Depending on what HTTP endpoint we are hitting, the returned output may contain other data with the key that we want to look up nested below it. This behaviour can be overriden by using the options `dig` and `dig_key`.
+
+The `dig_key` option can be used to change the key that is looked up, it also supports a dot-notation for digging values in nested hashes. [Special tags](#interpolating-special-tags) can also be used in the `dig_key` option.  Consider the following example output from our HTTP endpoint;
+
+```json
+{
+  "document": {
+    "settings": {
+      "apache::port": 80
+    }
+  }
+}
+```
+
+
+In this scenario we wouldn't be able to use class parameter lookups out-of-the-box, even if we just returned the whole structure, because we always need to drill down into `document.settings` to get the correct value, so In order to map the lookup to find the correct value, we can interpolate the __KEY__ tag into `lookup_key` and tell hiera-http to always dig into the hash with the following option;
+
+```yaml
+  options:
+    dig_key: document.settings.__KEY__
+```
+
+A more complicated example;
+
+```json
+{
+  "document": {
+    "settings": {
+      "apache": {
+        "port": 80
+      }
+    }
+  }
+}
+```
+
+Can be looked up with;
+
+```
+  options:
+    dig_key: document.settings.__MODULE__.__PARAMETER__
+```
+
+In both examples, the returned value to Puppet will be `80`
+
+### Returning the entire data structure
+
+The `dig` option can be used to disable digging altogether and the entire data hash will be returned with no attempt to resolve a key
+
+
+
 ### Author
 
 * Craig Dunn <[email protected]>

lib/puppet/functions/hiera_http.rb

@@ -30,15 +30,57 @@ def lookup_key(key, options, context)
     options['uri'] = parse_tags(key, options['uri'])
     result = http_get(context, options)
 
-    answer = result.is_a?(Hash) ? result[key] : result
-    context.not_found if answer.nil?
-    return answer
+    answer = return_answer(result, key, options)
+    if answer == :not_found
+      context.not_found
+      return nil
+    else
+      return answer
+    end
+
+  end
+
+  def return_answer(result, key, options)
+
+    # dig defaults to true, dig_key defaults to the value of the 
+    # lookup key.
+    #
+    dig = options.has_key?('dig') ? options['dig'] : true
+    dig_key = options.has_key?('dig_key') ? options['dig_key'] : key
+
+    # Interpolate values such as __KEY__ into each element of the
+    # dig path, eg: dig_key: document.data.__MODULE__
+    #
+    dig_path = dig_key.split(/\./).map { |p| parse_tags(key, p) }
+
+
+    if result.is_a?(String)
+      return result
+    else
+      return dig ? hash_dig(result, dig_path) : result
+    end
+
+  end
+
+
+  def hash_dig(data, dig_path)
+    key = dig_path.shift
+    if dig_path.empty?
+      if data.has_key?(key)
+        return data[key]
+      else
+        return :not_found
+      end
+    else
+      return :not_found unless data[key].is_a?(Hash)
+      return hash_dig(data[key], dig_path)
+    end
   end
 
-  def parse_tags(key,uri)
+  def parse_tags(key,str)
     key_parts = key.split(/::/)
 
-    parsed_uri = uri.gsub(/__(\w+)__/i) do
+    parsed_str = str.gsub(/__(\w+)__/i) do
       case $1
       when 'KEY'
         key
@@ -51,7 +93,7 @@ def parse_tags(key,uri)
       end
     end
 
-    return parsed_uri
+    return parsed_str
   end
 
 

spec/functions/hiera_http_spec.rb

@@ -9,6 +9,7 @@
     @lookuphttp = instance_double("LookupHttp")
     @context = instance_double("Puppet::LookupContext")
     allow(LookupHttp).to receive(:new).and_return(@lookuphttp)
+    allow(@lookuphttp).to receive(:get_parsed).and_return('value')
     allow(@context).to receive(:cache_has_key)
     allow(@context).to receive(:explain)
     allow(@context).to receive(:interpolate)
@@ -98,6 +99,7 @@
     context "Output of lookup_http" do
       let(:options) { {
         'uri' => 'http://localhost/path',
+        'output' => 'json'
       } }
       it "should dig if the value is a hash" do
         expect(@lookuphttp).to receive(:get_parsed).with('/path').and_return({ 'tango' => 'delta' })
@@ -111,7 +113,7 @@
 
     context "cached values" do
       let(:options) { {
-        'uri' => 'http://localhost/path'
+        'uri' => 'http://localhost/path',
       } }
 
       it "should used cached value when available" do
@@ -121,5 +123,70 @@
         expect(function.lookup_key('tango', options, @context)).to eq('bar')
       end
     end
+
+    context "nil versus undefined" do
+      let(:options) { {
+        'uri' => 'http://localhost/path',
+      } }
+
+      it "should return nil when a value is set to nil" do
+        response = { "config" => nil }
+        expect(@lookuphttp).to receive(:get_parsed).and_return(response)
+        expect(@context).not_to receive(:not_found)
+        expect(function.lookup_key('config', options, @context)).to eq(nil)
+      end
+      it "should return call not_found when the key doesn't exist" do
+        response = { "foo" => "bar"  }
+        expect(@lookuphttp).to receive(:get_parsed).and_return(response)
+        expect(@context).to receive(:not_found)
+        expect(function.lookup_key('config', options, @context)).to eq(nil)
+      end
+    end
+        
+
+      
+
+    context "Digging values" do
+      let(:options) { {
+       'uri' => 'http://localhost/path',
+      } }
+
+      let(:response) { {
+        "document" => {
+          "settings" => {
+            "docroot" => "/www"
+           }
+        }
+      } }
+      before(:each) do
+        expect(@lookuphttp).to receive(:get_parsed).with('/path').and_return(response)
+      end
+
+      it "should default to digging for the lookup key" do
+        expect(function.lookup_key('document', options, @context)).to eq({ "settings" => { "docroot" => "/www" } } )
+      end
+
+      it "should be able to dig for other keys" do
+        extra_opts = { "dig_key" => "document" }
+        expect(function.lookup_key('bar', options.merge(extra_opts), @context)).to eq({ "settings" => { "docroot" => "/www" } } )
+      end
+
+      it "should dig nested values using the dot notation" do
+        extra_opts = { "dig_key" => "document.settings" }
+        expect(function.lookup_key('bar', options.merge(extra_opts), @context)).to eq({ "docroot" => "/www" } )
+      end
+
+      it "should interpolate tags in the key path" do
+        extra_opts = { "dig_key" => "document.__MODULE__.__PARAMETER__" }
+        expect(function.lookup_key('settings::docroot', options.merge(extra_opts), @context)).to eq("/www")
+      end
+
+      it "should return the whole data structure if dig is disabled" do
+        extra_opts = { "dig" => false }
+        expect(function.lookup_key('bar', options.merge(extra_opts), @context)).to eq(response)
+      end
+
+
+    end
   end
 end