feat: implement backoff for snowpipe streaming authorization errors

[shekhar-rudder]

Description

This PR introduces a backoff mechanism when creating channels in the Snowpipe streaming destination. If an authorization error is encountered at the schema, table, or column level, the backoff ensures that we avoid repeatedly attempting to connect to Snowflake unnecessarily.

• Added a new struct authzBackoff in manager to store backoff related state
• Refactoring: Added managerCreator as a field in the manager for making the code testable.
• Until now, any error encountered while creating the channel for the discards table resulted in the job being marked as aborted. In this PR, the behavior has been updated to mark the job as failed, allowing it to be retried.
• Backoff is applied whenever an authorization error is encountered.
• Backoff is being cleared whenver channels for all the tables are being created without any authz error

Security

• The code changed/added as part of this pull request won't create any security issues with how the software is being used.

[codecov]

Codecov Report

Attention: Patch coverage is 89.02439% with 9 lines in your changes missing coverage. Please review.

Project coverage is 74.80%. Comparing base (8a186e5) to head (b5a1417).
Report is 1 commits behind head on master.

Files with missing lines	Patch %	Lines
...tionmanager/snowpipestreaming/snowpipestreaming.go	86.36%	4 Missing and 2 partials ⚠️
...yncdestinationmanager/snowpipestreaming/channel.go	80.00%	3 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##           master    #5399      +/-   ##
==========================================
- Coverage   74.82%   74.80%   -0.03%     
==========================================
  Files         438      438              
  Lines       61331    61404      +73     
==========================================
+ Hits        45893    45935      +42     
- Misses      12895    12931      +36     
+ Partials     2543     2538       -5     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[RanjeetMishra]

1. Should we set MaxInterval as say 1 hr? This would mean once issue is resolved around snowflake permissions it will resume working by next 1 hour? Also from correctness perspective, overflow in backoff package is handled as maxInterval.
2. Are we using cenkalti/backoff/v4 elsewhere as well. I see in v5 pattern of initializing has been changed.

CC: @achettyiitr

[achettyiitr]

Are we using cenkalti/backoff/v4 elsewhere as well. I see in v5 pattern of initializing has been changed.

In rudder-server, we are still using v4. We haven't upgraded to v5 yet.

[shekhar-rudder]

1. Made MaxInterval as 1hr

[RanjeetMishra]

Would this mean we end up invoking .NextBackoff as many times as size of uploadInfos?

[shekhar-rudder]

Updated the code to set backoff only once

[achettyiitr]

We can probably simplify this by defining a sentinel error since the requirement is just to classify the errors. There is no requirement to expose these error codes to other clients. wdyt?

// Define sentinel error
var errAuth = errors.New("authorization error")

// when sending the error you can do something like this
if err := snowflakeManager.CreateSchema(ctx); err != nil {
  return nil, fmt.Errorf("creating schema: %w, %w", errAuth, err)
}

// For classification
if err != nil {
  if errors.Is(err, errAuth) {
    m.authzBackoff.set()
    return m.failedJobs(asyncDest, err.Error())
  } else {
     return m.abortJobs(asyncDest, fmt.Errorf("failed to prepare discards channel: %w", err).Error())
  }
}

[achettyiitr]

Suggested change

	managerCreator func(mCtx context.Context, modelWarehouse whutils.ModelWarehouse, conf *config.Config, mLogger logger.Logger, stats stats.Stats) (manager.Manager, error)
	managerCreator func(ctx context.Context, modelWarehouse whutils.ModelWarehouse, conf *config.Config, logger logger.Logger, statsFactory stats.Stats) (manager.Manager, error)

[achettyiitr]

Suggested change

	m.managerCreator = func(mCtx context.Context, modelWarehouse whutils.ModelWarehouse, conf *config.Config, logger logger.Logger, stats stats.Stats) (manager.Manager, error) {
	sf, err := manager.New(whutils.SnowpipeStreaming, conf, logger, stats)
	if err != nil {
	return nil, fmt.Errorf("creating snowflake manager: %w", err)
	}
	err = sf.Setup(mCtx, modelWarehouse, whutils.NewNoOpUploader())
	if err != nil {
	return nil, fmt.Errorf("setting up snowflake manager: %w", err)
	}
	return sf, nil
	}
	m.managerCreator = func(ctx context.Context, modelWarehouse whutils.ModelWarehouse, conf *config.Config, logger logger.Logger, statsFactory stats.Stats) (manager.Manager, error) {
	sf, err := manager.New(whutils.SnowpipeStreaming, conf, logger, statsFactory)
	if err != nil {
	return nil, fmt.Errorf("creating snowflake manager: %w", err)
	}
	err = sf.Setup(ctx, modelWarehouse, whutils.NewNoOpUploader())
	if err != nil {
	return nil, fmt.Errorf("setting up snowflake manager: %w", err)
	}
	return sf, nil
	}

[achettyiitr]

Can we override this on Manager struct instead of defining a new struct here.

func (m *Manager) Now() time.Time {
	return m.now()
}

// when using, u can do something like this.
newAuthzBackoff(conf.GetDuration("SnowpipeStreaming.backoffDuration", 1, time.Second), m)

[achettyiitr]

We can probably simplify this logic since we are required to only store nextBackoff time.Time with the Manager struct and use it to decide whether we are in Backoff.

Created a short draft PR since it's difficult to explain within here.

[achettyiitr]

Suggested change

	func (m *mockManager) CreateSchema(ctx context.Context) (err error) {
	return m.createSchemaErr
	}
	func (m *mockManager) CreateTable(ctx context.Context, tableName string, columnMap whutils.ModelTableSchema) (err error) {
	return nil
	}
	func (m *mockManager) CreateSchema(context.Context) error {
	return m.createSchemaErr
	}
	func (m *mockManager) CreateTable(context.Context, string, whutils.ModelTableSchema) error {
	return nil
	}

[achettyiitr]

Can we make these configurable instead of hard coding it?

[achettyiitr]

Since here managerCreator will return an error, sm.authzBackoff.isInBackoff() should return True right because Post upload, backoff will be set.