Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: 🦠 Metrics prototype for Rack instrumentation #1129

Open
wants to merge 13 commits into
base: main
Choose a base branch
from
Open
1 change: 1 addition & 0 deletions instrumentation/base/lib/opentelemetry/instrumentation.rb
Original file line number Diff line number Diff line change
Expand Up @@ -7,6 +7,7 @@
require 'opentelemetry'
require 'opentelemetry-registry'
require 'opentelemetry/instrumentation/base'
require 'opentelemetry/instrumentation/metrics_patch' if defined?(OpenTelemetry::Metrics) # maybe also add Env var check?
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We talked about requiring "two keys to be turned", to enable metrics on instrumentation:

  1. Detect the metrics api is installed by the application (don't list it as a dependency)
  2. Add a configuration option to send the metrics (turned off by default)

Since we do some of our setup in base, base checks to see if the library is installed. If it is installed, we require the patch. The config isn't available at this time, so we don't use it in this context. The instrumentation for a particular library, however, checks for both the installation of the API and that the config option is true.

Is this sufficient to protect non-metrics users from any potential problems? Are there other approaches we'd like to explore?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think these two safeguards are enough for users who don’t want metrics in instrumentation or wish to create custom metrics themselves (e.g., include the metrics SDK but not use it in instrumentation).

Is the patch (e.g., prepend) only because metrics are still experimental features? I was thinking of including them in the base but not installing if OpenTelemetry::Metrics is missing (similar to line 25 in metrics_patch.rb).

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you! I think that could work! Yes, the patch is just because the metrics are experimental features. I thought this approach might feel "safer" (since all our instrumentation runs through base), but I don't know if truly provides a benefit.

It also tries to follow the configurator patches for metrics in the SDK:

I'm happy to cut out the patch and just put everything in base. I sorta tried that with the other prototype.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Whoops, forgot to follow up here. I've changed the structure to use conditions instead of the patches.


module OpenTelemetry
# The instrumentation module contains functionality to register and install
Expand Down
18 changes: 17 additions & 1 deletion instrumentation/base/lib/opentelemetry/instrumentation/base.rb
Original file line number Diff line number Diff line change
Expand Up @@ -4,6 +4,9 @@
#
# SPDX-License-Identifier: Apache-2.0

# TODO: Maybe add an env var for globally enabling metrics as second switch?
require_relative 'metrics_patch' if defined?(OpenTelemetry::Metrics)

kaylareopelle marked this conversation as resolved.
Show resolved Hide resolved
module OpenTelemetry
module Instrumentation
# The Base class holds all metadata and configuration for an
Expand Down Expand Up @@ -189,7 +192,7 @@ def infer_version
end
end

attr_reader :name, :version, :config, :installed, :tracer
attr_reader :name, :version, :config, :installed, :tracer, :meter

alias installed? installed

Expand All @@ -205,9 +208,13 @@ def initialize(name, version, install_blk, present_blk,
@installed = false
@options = options
@tracer = OpenTelemetry::Trace::Tracer.new
create_meter
end
# rubocop:enable Metrics/ParameterLists

# no-op, overridden in metrics patch
def create_meter; end

# Install instrumentation with the given config. The present? and compatible?
# will be run first, and install will return false if either fail. Will
# return true if install was completed successfully.
Expand All @@ -221,9 +228,18 @@ def install(config = {})

instance_exec(@config, &@install_blk)
@tracer = OpenTelemetry.tracer_provider.tracer(name, version)
install_meter
@installed = true
end

# no-op, defined in metrics patch to use if metrics installed
def install_meter; end

# Metrics should not be enabled if we're trying to load them directly from base
def metrics_enabled?
false
end

# Whether or not this instrumentation is installable in the current process. Will
# be true when the instrumentation defines an install block, is not disabled
# by environment or config, and the target library present and compatible.
Expand Down
Original file line number Diff line number Diff line change
@@ -0,0 +1,31 @@
# frozen_string_literal: true

# Copyright The OpenTelemetry Authors
#
# SPDX-License-Identifier: Apache-2.0

module OpenTelemetry
module Instrumentation
# MetricsPatch is a module that provides functionality to create a meter
# and record metrics if both the opentelemetry-metrics-api is present
# and the instrumentation to emit metrics has enabled metrics by setting
# :send_metrics to true
module MetricsPatch
def create_meter
@meter = OpenTelemetry::Metrics::Meter.new
end

def install_meter
@meter = OpenTelemetry.meter_provider.meter(name, version: version) if metrics_enabled?
end

def metrics_enabled?
return @metrics_enabled if defined?(@metrics_enabled)

@metrics_enabled ||= defined?(OpenTelemetry::Metrics) && @config[:send_metrics]
end
end
end
end

OpenTelemetry::Instrumentation::Base.prepend(OpenTelemetry::Instrumentation::MetricsPatch)
Original file line number Diff line number Diff line change
Expand Up @@ -29,8 +29,9 @@ class Instrumentation < OpenTelemetry::Instrumentation::Base
option :untraced_requests, default: nil, validate: :callable
option :response_propagators, default: [], validate: :array
# This option is only valid for applications using Rack 2.0 or greater
option :use_rack_events, default: true, validate: :boolean

option :use_rack_events, default: true, validate: :boolean
# TODO: This option currently exclusively uses the event handler, should we support old and new Rack?
option :send_metrics, default: false, validate: :boolean
kaylareopelle marked this conversation as resolved.
Show resolved Hide resolved
# Temporary Helper for Sinatra and ActionPack middleware to use during installation
#
# @example Default usage
Expand All @@ -47,10 +48,16 @@ def middleware_args
end
end

# def metrics_enabled?
# super
# # other conditions unique to Rack? Like Events also being available?
# end

kaylareopelle marked this conversation as resolved.
Show resolved Hide resolved
private

def require_dependencies
require_relative 'middlewares/event_handler' if defined?(::Rack::Events)
require_relative 'middlewares/metrics_patch' if metrics_enabled?
require_relative 'middlewares/tracer_middleware'
end

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -158,6 +158,9 @@ def untraced_request?(env)
false
end

# no-op, defined in MetricsPatch and required if metrics enabled
def record_http_server_request_duration_metric(span); end

kaylareopelle marked this conversation as resolved.
Show resolved Hide resolved
# https://github.com/open-telemetry/opentelemetry-specification/blob/master/specification/data-http.md#name
#
# recommendation: span.name(s) should be low-cardinality (e.g.,
Expand Down Expand Up @@ -203,6 +206,7 @@ def detach_context(request)
token, span = request.env[OTEL_TOKEN_AND_SPAN]
span.finish
OpenTelemetry::Context.detach(token)
record_http_server_request_duration_metric(span)
rescue StandardError => e
OpenTelemetry.handle_error(exception: e)
end
Expand Down Expand Up @@ -247,6 +251,12 @@ def tracer
OpenTelemetry::Instrumentation::Rack::Instrumentation.instance.tracer
end

# no-op, overwritten by metrics patch if metrics is enabled
def meter; end

# no-op, overwritten by metrics patch if metrics is enabled
def http_server_request_duration_histogram; end

def config
OpenTelemetry::Instrumentation::Rack::Instrumentation.instance.config
end
Expand Down
Original file line number Diff line number Diff line change
@@ -0,0 +1,68 @@
# frozen_string_literal: true

# Copyright The OpenTelemetry Authors
#
# SPDX-License-Identifier: Apache-2.0

module OpenTelemetry
module Instrumentation
module Rack
module Middlewares
# MetricsPatch is a module that provides functionality to record metrics
# if both the opentelemetry-metrics-api is present and the rack
# instrumentation is configured to emit metrics by setting
# :send_metrics to true
module MetricsPatch
def meter
OpenTelemetry::Instrumentation::Rack::Instrumentation.instance.meter
end

def http_server_request_duration_histogram
# TODO: Add Advice to set small explicit histogram bucket boundaries
# TODO: Does this need to be memoized?
@http_server_request_duration_histogram ||= meter.create_histogram('http.server.request.duration', unit: 's', description: 'Duration of HTTP server requests.')
end

# TODO: Update this to define attributes based on SEMCONV_STABILITY_OPT_IN once available
def record_http_server_request_duration_metric(span)
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This method is very long because it's leaving all the options open for attribute names. The attribute names we currently use are not the ones marked as stable.

I think we could use OTEL_SEMCONV_STABILITY_OPT_IN as a way to determine what attributes to send. Or, we could only emit the metric if that env var is set to http, which would mean only the new attributes would be sent. I'd love feedback here.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If the metrics attribute only cares about HTTP-related attributes, can it emit metrics with the attributes that are based on the existing span.attributes? I'm thinking something like:

# Assume only one of url.scheme and http.scheme will appear
span.attributes.select { |key, value| key.match(/http\..*/) || key.match(/url\..*/) }

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I like where you're headed with this! There's a lot of duplication in the current code. However, there are some http.* and url.* attributes present on the span that should not be attached to metrics (ex. url.full is on spans, but not metrics).

I wonder if we could use a key-based allowlist to select things instead? Maybe add a constant with all the HTTP metrics keys, and we search for those?

Things might need to shift again when the OTEL_SEMCONV_STABILITY_OPT_IN variable is out in the world, but we can address that later.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the updates. Yes, I like key-based allowlist HTTP_SERVER_REQUEST_DURATION_ATTRS_FROM_SPAN.
In the future, I think we can even have something like HTTP_SERVER_REQUEST_DURATION_ATTRS_FROM_SPAN_V1_2 (V1_2 = semantic version 1.2)

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ooh, nice idea!

# find span duration
# end - start / a billion to convert nanoseconds to seconds
duration = (span.end_timestamp - span.start_timestamp) / Float(10**9)
# Create attributes
#
attrs = {}
# pattern below goes:
# # stable convention
# current span convention

# attrs['http.request.method']
attrs['http.method'] = span.attributes['http.method']

# attrs['url.scheme']
attrs['http.scheme'] = span.attributes['http.scheme']

# same in stable semconv
attrs['http.route'] = span.attributes['http.route']

# attrs['http.response.status.code']
attrs['http.status_code'] = span.attributes['http.status_code']

# attrs['server.address'] ???
# attrs['server.port'] ???
# span includes host and port
attrs['http.host'] = span.attributes['http.host']

# attrs not currently in span payload
# attrs['network.protocol.version']
# attrs['network.protocol.name']
attrs['error.type'] = span.status.description if span.status.code == OpenTelemetry::Trace::Status::ERROR

http_server_request_duration_histogram.record(duration, attributes: attrs)
end
end
end
end
end
end

OpenTelemetry::Instrumentation::Rack::Middlewares::EventHandler.prepend(OpenTelemetry::Instrumentation::Rack::Middlewares::MetricsPatch)
Loading