Manager API does not match singleton design

[MrAlias]

The Manager is designed in such a way that it expects Run is only ever called once. However, it is a method on the Manager and can be called multiple times (i.e. multi-process instrumentation).

• Only the first target is ever applied:
  

  opentelemetry-go-instrumentation/internal/pkg/instrumentation/manager.go

  Lines 313 to 315 in 098f594

  	if m.td == nil {
  	m.td = target
  	}

• Subsequent Runs will overwrite the config with the initial config:
  

  opentelemetry-go-instrumentation/internal/pkg/instrumentation/manager.go

  Line 260 in 098f594

  m.currentConfig = m.cp.InitialConfig(ctx)

• If run with a load indicator, it will be closed twice and panic:
  

  opentelemetry-go-instrumentation/internal/pkg/instrumentation/manager.go

  Line 274 in 098f594

  close(m.loadedIndicator)

• The ConfigLoop of one Run will receive updates while another will not due to the lack of channel multiplexing:
  

  opentelemetry-go-instrumentation/internal/pkg/instrumentation/manager.go

  Lines 231 to 249 in 098f594

  	func (m *Manager) ConfigLoop(ctx context.Context) {
  	for {
  	select {
  	case <-ctx.Done():
  	return
  	case c, ok := <-m.cp.Watch():
  	if !ok {
  	m.logger.Info("Configuration provider closed, configuration updates will no longer be received")
  	return
  	}
  	err := m.applyConfig(c)
  	if err != nil {
  	m.logger.Error("Failed to apply config", "error", err)
  	continue
  	}
  	m.currentConfig = c
  	}
  	}
  	}

There seems a strong need to refactor this type. Ideally, the refactored code will have independent Run functionality or unify so there can only ever be one "Run".

[MrAlias]

Additionally, from #1218:

It would be best to adhere to the idiom: "the writer should close the chan". This would mean that a probe's Run func needs to return the chan it writes to and only closes it when it returns. This would also mean the Manager will need to dynamically multiplex chan selection.

[RonFed]

We have this check when Instrumentation.Run is called:

opentelemetry-go-instrumentation/instrumentation.go

Lines 151 to 153 in 098f594

	if i.stop != nil {
	return parent, errors.New("instrumentation already running")
	}

So at least for the current implementation, it prevents from Run being called more than once (so maybe it should not be classified as a bug but as a feature request)

Regarding future multi-process support: do auto-instrumentations for other languages contain this kind of support or is it out of scope for them? (i.e detecting all the relevant processes for the language, filtering the processes the user wants, and managing the state for each process) - I guess I'm wondering whether it should be in the scope of this repo.

If the answer is yes, then maybe the Instrumentation object should exist per process - which means the manager will still be per process, and a higher level object will need to be created to manage all the Instrumentations and it will be responsible to handle stuff in the process level. The motivation for this is the manager already does quite a lot, and will probably have more jobs once we have support for more configuration options for each probe. Adding the multi-process handling to it can make it more complicated.

If the answer is no, we can keep the current structure and a higher-level implementation can make use of multiple Instrumentation objects to manage multi-process instrumentation.