Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[datasetexporter]: Upgrade library to 0.17.0 #29446

Merged
merged 17 commits into from
Nov 28, 2023
Merged

[datasetexporter]: Upgrade library to 0.17.0 #29446

Changes from 1 commit
Commits
Show all changes
17 commits
Select commit Hold shift + click to select a range
cd6d51d
[datasetexporter]: Upgrade library to 0.16.0
martin-majlis-s1 Nov 22, 2023
ea6bec5
Add changelog
martin-majlis-s1 Nov 22, 2023
de9282c
Make `go mod tidy -compat=1.20` happy
martin-majlis-s1 Nov 22, 2023
3ea0348
Update documentation
martin-majlis-s1 Nov 22, 2023
5b6e880
Merge branch 'main' into datasetexporter-update-to-0.16.0
martin-majlis-s1 Nov 23, 2023
2cf16e5
Fix typos
martin-majlis-s1 Nov 23, 2023
9423b19
Update exporter/datasetexporter/README.md
martin-majlis-s1 Nov 27, 2023
1b4d15f
Update .chloggen/datasetexporter-update-to-0.16.0.yaml
martin-majlis-s1 Nov 27, 2023
78e874b
Update exporter/datasetexporter/README.md
martin-majlis-s1 Nov 27, 2023
70554bf
Merge branch 'main' into datasetexporter-update-to-0.16.0
martin-majlis-s1 Nov 27, 2023
c45046e
Update library to enable attributes
martin-majlis-s1 Nov 27, 2023
d1b8922
Run `make gotidy` one more time
martin-majlis-s1 Nov 27, 2023
c60935e
Fix imports order
martin-majlis-s1 Nov 27, 2023
6c5f7a8
Merge branch 'main' into datasetexporter-update-to-0.16.0
martin-majlis-s1 Nov 27, 2023
23b34d5
Merge branch 'main' into datasetexporter-update-to-0.16.0
martin-majlis-s1 Nov 28, 2023
ecadde1
Incorporate comments from PR
martin-majlis-s1 Nov 28, 2023
d555330
Merge branch 'main' into datasetexporter-update-to-0.16.0
martin-majlis-s1 Nov 28, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Next Next commit
[datasetexporter]: Upgrade library to 0.16.0
  • Loading branch information
@martin-majlis-s1
martin-majlis-s1 committed Nov 22, 2023
commit cd6d51dd0343016ea4fa91a13d82207e512a7b35
2 changes: 1 addition & 1 deletion cmd/configschema/go.mod
View file
Original file line number Diff line number Diff line change
@@ -568,7 +568,7 @@ require (
github.com/relvacode/iso8601 v1.3.0 // indirect
github.com/rs/cors v1.10.1 // indirect
github.com/scaleway/scaleway-sdk-go v1.0.0-beta.21 // indirect
github.com/scalyr/dataset-go v0.14.0 // indirect
github.com/scalyr/dataset-go v0.16.0 // indirect
github.com/seccomp/libseccomp-golang v0.9.2-0.20220502022130-f33da4d89646 // indirect
github.com/secure-systems-lab/go-securesystemslib v0.7.0 // indirect
github.com/segmentio/asm v1.2.0 // indirect
4 changes: 2 additions & 2 deletions cmd/configschema/go.sum
View file

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

2 changes: 1 addition & 1 deletion cmd/otelcontribcol/go.mod
View file
Original file line number Diff line number Diff line change
@@ -586,7 +586,7 @@ require (
github.com/rs/cors v1.10.1 // indirect
github.com/samber/lo v1.37.0 // indirect
github.com/scaleway/scaleway-sdk-go v1.0.0-beta.21 // indirect
github.com/scalyr/dataset-go v0.14.0 // indirect
github.com/scalyr/dataset-go v0.16.0 // indirect
github.com/seccomp/libseccomp-golang v0.9.2-0.20220502022130-f33da4d89646 // indirect
github.com/secure-systems-lab/go-securesystemslib v0.7.0 // indirect
github.com/segmentio/asm v1.2.0 // indirect
4 changes: 2 additions & 2 deletions cmd/otelcontribcol/go.sum
View file

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

8 changes: 7 additions & 1 deletion exporter/datasetexporter/config.go
View file
Original file line number Diff line number Diff line change
@@ -5,6 +5,7 @@ package datasetexporter // import "github.com/open-telemetry/opentelemetry-colle

import (
"fmt"
"strings"
"time"

"github.com/cenkalti/backoff/v4"
@@ -144,9 +145,12 @@ func newDefaultServerHostSettings() ServerHostSettings {
}
}

const debugDefault = false

type Config struct {
DatasetURL string `mapstructure:"dataset_url"`
APIKey configopaque.String `mapstructure:"api_key"`
Debug bool `mapstructure:"debug"`
BufferSettings `mapstructure:"buffer"`
TracesSettings `mapstructure:"traces"`
LogsSettings `mapstructure:"logs"`
@@ -182,14 +186,15 @@ func (c *Config) Validate() error {
func (c *Config) String() string {
s := ""
s += fmt.Sprintf("%s: %s; ", "DatasetURL", c.DatasetURL)
s += fmt.Sprintf("%s: %s (%d); ", "APIKey", strings.Repeat("*", len(c.APIKey)), len(c.APIKey))
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

a configopaque.String will by default be replaced with [Redacted], is there a reason to override the behaviour here?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@codeboten : Thanks. But when I have tried it out - it's returning the real value. I have to use apiKey, _ := c.APIKey.MarshalText() - which is not too much better. I will replace it, so it's more consistent with other places.

--- FAIL: TestConfigString (0.00s)
    config_test.go:143:
                Error Trace:    /Users/martin.majlis/development/scalyr-org/opentelemetry-collector-contrib/exporter/datasetexporter/config_test.go:143
                Error:          Not equal:
                                expected: "DatasetURL: https://example.com; APIKey: ****** (6); Debug: true; BufferSettings: {MaxLifetime:123ns GroupBy:[field1 field2] RetryInitialInterval:0s RetryMaxInterval:0s RetryMaxElapsedTime:0s RetryShutdownTimeout:0s}; LogsSettings: {ExportResourceInfo:true ExportResourcePrefix:AAA ExportScopeInfo:true ExportScopePrefix:BBB DecomposeComplexMessageField:true DecomposedComplexMessagePrefix:EEE exportSettings:{ExportSeparator:CCC ExportDistinguishingSuffix:DDD}}; TracesSettings: {exportSettings:{ExportSeparator:TTT ExportDistinguishingSuffix:UUU}}; ServerHostSettings: {UseHostName:false ServerHost:foo-bar}; RetrySettings: {Enabled:true InitialInterval:5s RandomizationFactor:0.5 Multiplier:1.5 MaxInterval:30s MaxElapsedTime:5m0s}; QueueSettings: {Enabled:true NumConsumers:10 QueueSize:1000 StorageID:<nil>}; TimeoutSettings: {Timeout:5s}"
                                actual  : "DatasetURL: https://example.com; APIKey: secret (6); Debug: true; BufferSettings: {MaxLifetime:123ns GroupBy:[field1 field2] RetryInitialInterval:0s RetryMaxInterval:0s RetryMaxElapsedTime:0s RetryShutdownTimeout:0s}; LogsSettings: {ExportResourceInfo:true ExportResourcePrefix:AAA ExportScopeInfo:true ExportScopePrefix:BBB DecomposeComplexMessageField:true DecomposedComplexMessagePrefix:EEE exportSettings:{ExportSeparator:CCC ExportDistinguishingSuffix:DDD}}; TracesSettings: {exportSettings:{ExportSeparator:TTT ExportDistinguishingSuffix:UUU}}; ServerHostSettings: {UseHostName:false ServerHost:foo-bar}; RetrySettings: {Enabled:true InitialInterval:5s RandomizationFactor:0.5 Multiplier:1.5 MaxInterval:30s MaxElapsedTime:5m0s}; QueueSettings: {Enabled:true NumConsumers:10 QueueSize:1000 StorageID:<nil>}; TimeoutSettings: {Timeout:5s}"

                                Diff:
                                --- Expected
                                +++ Actual
                                @@ -1 +1 @@
                                -DatasetURL: https://example.com; APIKey: ****** (6); Debug: true; BufferSettings: {MaxLifetime:123ns GroupBy:[field1 field2] RetryInitialInterval:0s RetryMaxInterval:0s RetryMaxElapsedTime:0s RetryShutdownTimeout:0s}; LogsSettings: {ExportResourceInfo:true ExportResourcePrefix:AAA ExportScopeInfo:true ExportScopePrefix:BBB DecomposeComplexMessageField:true DecomposedComplexMessagePrefix:EEE exportSettings:{ExportSeparator:CCC ExportDistinguishingSuffix:DDD}}; TracesSettings: {exportSettings:{ExportSeparator:TTT ExportDistinguishingSuffix:UUU}}; ServerHostSettings: {UseHostName:false ServerHost:foo-bar}; RetrySettings: {Enabled:true InitialInterval:5s RandomizationFactor:0.5 Multiplier:1.5 MaxInterval:30s MaxElapsedTime:5m0s}; QueueSettings: {Enabled:true NumConsumers:10 QueueSize:1000 StorageID:<nil>}; TimeoutSettings: {Timeout:5s}
                                +DatasetURL: https://example.com; APIKey: secret (6); Debug: true; BufferSettings: {MaxLifetime:123ns GroupBy:[field1 field2] RetryInitialInterval:0s RetryMaxInterval:0s RetryMaxElapsedTime:0s RetryShutdownTimeout:0s}; LogsSettings: {ExportResourceInfo:true ExportResourcePrefix:AAA ExportScopeInfo:true ExportScopePrefix:BBB DecomposeComplexMessageField:true DecomposedComplexMessagePrefix:EEE exportSettings:{ExportSeparator:CCC ExportDistinguishingSuffix:DDD}}; TracesSettings: {exportSettings:{ExportSeparator:TTT ExportDistinguishingSuffix:UUU}}; ServerHostSettings: {UseHostName:false ServerHost:foo-bar}; RetrySettings: {Enabled:true InitialInterval:5s RandomizationFactor:0.5 Multiplier:1.5 MaxInterval:30s MaxElapsedTime:5m0s}; QueueSettings: {Enabled:true NumConsumers:10 QueueSize:1000 StorageID:<nil>}; TimeoutSettings: {Timeout:5s}
                Test:           TestConfigString

Copy link
Contributor

@codeboten codeboten Nov 28, 2023 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

But when I have tried it out - it's returning the real value.

This is surprising to me... maybe @mx-psi can help here. Will not block this PR on this though.

s += fmt.Sprintf("%s: %t; ", "Debug", c.Debug)
s += fmt.Sprintf("%s: %+v; ", "BufferSettings", c.BufferSettings)
s += fmt.Sprintf("%s: %+v; ", "LogsSettings", c.LogsSettings)
s += fmt.Sprintf("%s: %+v; ", "TracesSettings", c.TracesSettings)
s += fmt.Sprintf("%s: %+v; ", "ServerHostSettings", c.ServerHostSettings)
s += fmt.Sprintf("%s: %+v; ", "RetrySettings", c.RetrySettings)
s += fmt.Sprintf("%s: %+v; ", "QueueSettings", c.QueueSettings)
s += fmt.Sprintf("%s: %+v", "TimeoutSettings", c.TimeoutSettings)

return s
}

@@ -218,6 +223,7 @@ func (c *Config) convert() (*ExporterConfig, error) {
UseHostName: c.ServerHostSettings.UseHostName,
ServerHost: c.ServerHostSettings.ServerHost,
},
Debug: c.Debug,
},
tracesSettings: c.TracesSettings,
logsSettings: c.LogsSettings,
3 changes: 2 additions & 1 deletion exporter/datasetexporter/config_test.go
View file
Original file line number Diff line number Diff line change
@@ -108,6 +108,7 @@ func TestConfigString(t *testing.T) {
config := Config{
DatasetURL: "https://example.com",
APIKey: "secret",
Debug: true,
BufferSettings: BufferSettings{
MaxLifetime: 123,
GroupBy: []string{"field1", "field2"},
@@ -140,7 +141,7 @@ func TestConfigString(t *testing.T) {
}

assert.Equal(t,
"DatasetURL: https://example.com; BufferSettings: {MaxLifetime:123ns GroupBy:[field1 field2] RetryInitialInterval:0s RetryMaxInterval:0s RetryMaxElapsedTime:0s RetryShutdownTimeout:0s}; LogsSettings: {ExportResourceInfo:true ExportResourcePrefix:AAA ExportScopeInfo:true ExportScopePrefix:BBB DecomposeComplexMessageField:true DecomposedComplexMessagePrefix:EEE exportSettings:{ExportSeparator:CCC ExportDistinguishingSuffix:DDD}}; TracesSettings: {exportSettings:{ExportSeparator:TTT ExportDistinguishingSuffix:UUU}}; ServerHostSettings: {UseHostName:false ServerHost:foo-bar}; RetrySettings: {Enabled:true InitialInterval:5s RandomizationFactor:0.5 Multiplier:1.5 MaxInterval:30s MaxElapsedTime:5m0s}; QueueSettings: {Enabled:true NumConsumers:10 QueueSize:1000 StorageID:<nil>}; TimeoutSettings: {Timeout:5s}",
"DatasetURL: https://example.com; APIKey: ****** (6); Debug: true; BufferSettings: {MaxLifetime:123ns GroupBy:[field1 field2] RetryInitialInterval:0s RetryMaxInterval:0s RetryMaxElapsedTime:0s RetryShutdownTimeout:0s}; LogsSettings: {ExportResourceInfo:true ExportResourcePrefix:AAA ExportScopeInfo:true ExportScopePrefix:BBB DecomposeComplexMessageField:true DecomposedComplexMessagePrefix:EEE exportSettings:{ExportSeparator:CCC ExportDistinguishingSuffix:DDD}}; TracesSettings: {exportSettings:{ExportSeparator:TTT ExportDistinguishingSuffix:UUU}}; ServerHostSettings: {UseHostName:false ServerHost:foo-bar}; RetrySettings: {Enabled:true InitialInterval:5s RandomizationFactor:0.5 Multiplier:1.5 MaxInterval:30s MaxElapsedTime:5m0s}; QueueSettings: {Enabled:true NumConsumers:10 QueueSize:1000 StorageID:<nil>}; TimeoutSettings: {Timeout:5s}",
config.String(),
)
}
7 changes: 4 additions & 3 deletions exporter/datasetexporter/datasetexporter.go
View file
Original file line number Diff line number Diff line change
@@ -17,12 +17,10 @@ import (
"go.opentelemetry.io/collector/exporter"
"go.opentelemetry.io/collector/pdata/pcommon"
"go.uber.org/zap"
"golang.org/x/time/rate"
)

type DatasetExporter struct {
client *client.DataSetClient
limiter *rate.Limiter
logger *zap.Logger
session string
exporterCfg *ExporterConfig
@@ -48,11 +46,15 @@ func newDatasetExporter(entity string, config *Config, set exporter.CreateSettin
set.BuildInfo.Version,
entity,
)

meter := set.MeterProvider.Meter("datasetexporter")

client, err := client.NewClient(
exporterCfg.datasetConfig,
&http.Client{Timeout: time.Second * 60},
logger,
&userAgent,
&meter,
)
if err != nil {
logger.Error("Cannot create DataSetClient: ", zap.Error(err))
@@ -61,7 +63,6 @@ func newDatasetExporter(entity string, config *Config, set exporter.CreateSettin

return &DatasetExporter{
client: client,
limiter: rate.NewLimiter(100*rate.Every(1*time.Minute), 100), // 100 requests / minute
session: uuid.New().String(),
logger: logger,
exporterCfg: exporterCfg,
1 change: 1 addition & 0 deletions exporter/datasetexporter/factory.go
View file
Original file line number Diff line number Diff line change
@@ -32,6 +32,7 @@ func createDefaultConfig() component.Config {
RetrySettings: exporterhelper.NewDefaultRetrySettings(),
QueueSettings: exporterhelper.NewDefaultQueueSettings(),
TimeoutSettings: exporterhelper.NewDefaultTimeoutSettings(),
Debug: debugDefault,
}
}

3 changes: 2 additions & 1 deletion exporter/datasetexporter/factory_test.go
View file
Original file line number Diff line number Diff line change
@@ -84,6 +84,7 @@ func TestLoadConfig(t *testing.T) {
expected: &Config{
DatasetURL: "https://app.scalyr.com",
APIKey: "key-full",
Debug: true,
BufferSettings: BufferSettings{
MaxLifetime: 3456 * time.Millisecond,
GroupBy: []string{"body.map.kubernetes.pod_id", "body.map.kubernetes.docker_id", "body.map.stream"},
@@ -160,7 +161,7 @@ func createExporterTests() []CreateTest {
{
name: "broken",
config: &Config{},
expectedError: fmt.Errorf("cannot get DataSetExpoter: cannot convert config: DatasetURL: ; BufferSettings: {MaxLifetime:0s GroupBy:[] RetryInitialInterval:0s RetryMaxInterval:0s RetryMaxElapsedTime:0s RetryShutdownTimeout:0s}; LogsSettings: {ExportResourceInfo:false ExportResourcePrefix: ExportScopeInfo:false ExportScopePrefix: DecomposeComplexMessageField:false DecomposedComplexMessagePrefix: exportSettings:{ExportSeparator: ExportDistinguishingSuffix:}}; TracesSettings: {exportSettings:{ExportSeparator: ExportDistinguishingSuffix:}}; ServerHostSettings: {UseHostName:false ServerHost:}; RetrySettings: {Enabled:false InitialInterval:0s RandomizationFactor:0 Multiplier:0 MaxInterval:0s MaxElapsedTime:0s}; QueueSettings: {Enabled:false NumConsumers:0 QueueSize:0 StorageID:<nil>}; TimeoutSettings: {Timeout:0s}; config is not valid: api_key is required"),
expectedError: fmt.Errorf("cannot get DataSetExpoter: cannot convert config: DatasetURL: ; APIKey: (0); Debug: false; BufferSettings: {MaxLifetime:0s GroupBy:[] RetryInitialInterval:0s RetryMaxInterval:0s RetryMaxElapsedTime:0s RetryShutdownTimeout:0s}; LogsSettings: {ExportResourceInfo:false ExportResourcePrefix: ExportScopeInfo:false ExportScopePrefix: DecomposeComplexMessageField:false DecomposedComplexMessagePrefix: exportSettings:{ExportSeparator: ExportDistinguishingSuffix:}}; TracesSettings: {exportSettings:{ExportSeparator: ExportDistinguishingSuffix:}}; ServerHostSettings: {UseHostName:false ServerHost:}; RetrySettings: {Enabled:false InitialInterval:0s RandomizationFactor:0 Multiplier:0 MaxInterval:0s MaxElapsedTime:0s}; QueueSettings: {Enabled:false NumConsumers:0 QueueSize:0 StorageID:<nil>}; TimeoutSettings: {Timeout:0s}; config is not valid: api_key is required"),
},
{
name: "valid",
2 changes: 1 addition & 1 deletion exporter/datasetexporter/go.mod
View file
Original file line number Diff line number Diff line change
@@ -6,7 +6,7 @@ require (
github.com/google/uuid v1.4.0
// github.com/open-telemetry/opentelemetry-collector-contrib/extension/storage/filestorage v0.77.0
github.com/open-telemetry/opentelemetry-collector-contrib/internal/coreinternal v0.89.0
github.com/scalyr/dataset-go v0.14.0
github.com/scalyr/dataset-go v0.16.0
github.com/stretchr/testify v1.8.4
go.opentelemetry.io/collector/component v0.89.0
go.opentelemetry.io/collector/confmap v0.89.0
4 changes: 2 additions & 2 deletions exporter/datasetexporter/go.sum
View file

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

Loading