Querier panic when marshaling frontendv2pb.(*QueryResultRequest)

[SungJin1212]

Describe the bug
The querier get panics when marshaling frontendv2pb.(*QueryResultRequest) in my test cluster.

To Reproduce
I reproduce it in my local branch test code

The panic scenario I guess is:

1. scheduler processor call querier handler
2. querier handler create go-routines to handle queries. In go-routines, it increments query-stats.
3. scheduler processor call QueryResult
4. the panic could happen when the query-stats values are incremented in fetchSeriesFromStores (by not yet exited go routines) while progressing (*QueryResultRequest).MarshalToSizedBuffer because the same query-stat objects are used.

Expected behavior
Not panic

Environment:

• Infrastructure: kubernetes

Additional Context

logs:

panic: runtime error: index out of range [-3]

goroutine 7055 [running]:
github.com/cortexproject/cortex/pkg/frontend/v2/frontendv2pb.encodeVarintFrontend(...)
	/go/src/github.com/cortexproject/cortex/pkg/frontend/v2/frontendv2pb/frontend.pb.go:402
github.com/cortexproject/cortex/pkg/frontend/v2/frontendv2pb.(*QueryResultRequest).MarshalToSizedBuffer(0xc0027eac90, {0xc002be8000, 0xa5, 0xa5?})
	/go/src/github.com/cortexproject/cortex/pkg/frontend/v2/frontendv2pb/frontend.pb.go:368 +0x2c7
github.com/cortexproject/cortex/pkg/frontend/v2/frontendv2pb.(*QueryResultRequest).Marshal(0xc0027eac90)
	/go/src/github.com/cortexproject/cortex/pkg/frontend/v2/frontendv2pb/frontend.pb.go:326 +0x4d
google.golang.org/protobuf/internal/impl.legacyMarshal({{}, {0x341faf0, 0xc002bd2250}, {0x0, 0x0, 0x0}, 0x0})
	/go/src/github.com/cortexproject/cortex/vendor/google.golang.org/protobuf/internal/impl/legacy_message.go:411 +0xf2
google.golang.org/protobuf/proto.MarshalOptions.marshal({{}, 0x90?, 0xac?, 0x7e?}, {0x0, 0x0, 0x0}, {0x341faf0, 0xc002bd2250})
	/go/src/github.com/cortexproject/cortex/vendor/google.golang.org/protobuf/proto/encode.go:195 +0x35d
google.golang.org/protobuf/proto.Marshal({0x33db240, 0xc002bd2250})
	/go/src/github.com/cortexproject/cortex/vendor/google.golang.org/protobuf/proto/encode.go:111 +0x50
google.golang.org/grpc/encoding/proto.codec.Marshal({}, {0x2cbe5e0?, 0xc0027eac90?})
	/go/src/github.com/cortexproject/cortex/vendor/google.golang.org/grpc/encoding/proto/proto.go:47 +0x52
google.golang.org/grpc.encode({0x7fd8963b7528?, 0x4c9f480?}, {0x2cbe5e0?, 0xc0027eac90?})
	/go/src/github.com/cortexproject/cortex/vendor/google.golang.org/grpc/rpc_util.go:647 +0x3e
google.golang.org/grpc.prepareMsg({0x2cbe5e0?, 0xc0027eac90?}, {0x7fd8963b7528?, 0x4c9f480?}, {0x0, 0x0}, {0x0, 0x0})
	/go/src/github.com/cortexproject/cortex/vendor/google.golang.org/grpc/stream.go:1772 +0xc7
google.golang.org/grpc.(*clientStream).SendMsg(0xc0020ceea0, {0x2cbe5e0, 0xc0027eac90})
	/go/src/github.com/cortexproject/cortex/vendor/google.golang.org/grpc/stream.go:894 +0xd7
google.golang.org/grpc.invoke({0x33f9868?, 0xc002baacc0?}, {0x2e3dc84?, 0x0?}, {0x2cbe5e0, 0xc0027eac90}, {0x2c917c0, 0x4c9f480}, 0x2b8b940?, {0xc002ef85e0, ...})
	/go/src/github.com/cortexproject/cortex/vendor/google.golang.org/grpc/call.go:70 +0x9f
github.com/cortexproject/cortex/pkg/querier/worker.(*schedulerProcessor).createFrontendClient.PrometheusGRPCUnaryInstrumentation.func1({0x33f9868, 0xc002baacc0}, {0x2e3dc84, 0x2c}, {0x2cbe5e0, 0xc0027eac90}, {0x2c917c0, 0x4c9f480}, 0xc00261cc08, 0x2fa02d0, ...)
	/go/src/github.com/cortexproject/cortex/pkg/util/middleware/grpc.go:20 +0x10b
github.com/cortexproject/cortex/pkg/util/grpcclient.(*Config).DialOption.ChainUnaryClient.func6.1({0x33f9868?, 0xc002baacc0?}, {0x2e3dc84?, 0x0?}, {0x2cbe5e0?, 0xc0027eac90?}, {0x2c917c0?, 0x4c9f480?}, 0xc002baac00?, {0xc002ef85e0, ...})
	/go/src/github.com/cortexproject/cortex/vendor/github.com/grpc-ecosystem/go-grpc-middleware/chain.go:113 +0x91
github.com/weaveworks/common/middleware.ClientUserHeaderInterceptor({0x33f9868?, 0xc002baac30?}, {0x2e3dc84, 0x2c}, {0x2cbe5e0, 0xc0027eac90}, {0x2c917c0, 0x4c9f480}, 0xc00261cc08, 0xc002baab40, ...)
	/go/src/github.com/cortexproject/cortex/vendor/github.com/weaveworks/common/middleware/grpc_auth.go:17 +0xd7
github.com/cortexproject/cortex/pkg/util/grpcclient.(*Config).DialOption.ChainUnaryClient.func6.1({0x33f9868?, 0xc002baac30?}, {0x2e3dc84?, 0xc002babf80?}, {0x2cbe5e0?, 0xc0027eac90?}, {0x2c917c0?, 0x4c9f480?}, 0xc000092030?, {0xc002ef85e0, ...})
	/go/src/github.com/cortexproject/cortex/vendor/github.com/grpc-ecosystem/go-grpc-middleware/chain.go:113 +0x91
github.com/opentracing-contrib/go-grpc.OpenTracingClientInterceptor.func1({0x33f9868, 0xc002baa060}, {0x2e3dc84, 0x2c}, {0x2cbe5e0, 0xc0027eac90}, {0x2c917c0, 0x4c9f480}, 0xc00261cc08, 0xc002baab70, ...)
	/go/src/github.com/cortexproject/cortex/vendor/github.com/opentracing-contrib/go-grpc/client.go:62 +0x44e
github.com/cortexproject/cortex/pkg/util/grpcclient.(*Config).DialOption.ChainUnaryClient.func6({0x33f9868, 0xc002baa060}, {0x2e3dc84, 0x2c}, {0x2cbe5e0, 0xc0027eac90}, {0x2c917c0, 0x4c9f480}, 0xc00261cc08, 0x2fa02d0, ...)
	/go/src/github.com/cortexproject/cortex/vendor/github.com/grpc-ecosystem/go-grpc-middleware/chain.go:116 +0x1d3
google.golang.org/grpc.(*ClientConn).Invoke(0xc00261cc08, {0x33f9868?, 0xc002baa060?}, {0x2e3dc84?, 0x7fd8dd280f18?}, {0x2cbe5e0?, 0xc0027eac90?}, {0x2c917c0?, 0x4c9f480?}, {0x0, ...})
	/go/src/github.com/cortexproject/cortex/vendor/google.golang.org/grpc/call.go:35 +0x205
github.com/cortexproject/cortex/pkg/frontend/v2/frontendv2pb.(*frontendForQuerierClient).QueryResult(0x40e81f?, {0x33f9868?, 0xc002baa060?}, 0x410b00?, {0x0?, 0x2a571a0?, 0x1?})
	/go/src/github.com/cortexproject/cortex/pkg/frontend/v2/frontendv2pb/frontend.pb.go:268 +0x5d
github.com/cortexproject/cortex/pkg/querier/worker.(*schedulerProcessor).runRequest(0xc001072c60, {0x33f9868?, 0xc002b87f50?}, {0x33d53e0, 0xc0020bceb0}, 0x5c6537c06c010d36, {0xc00291b170, 0x11}, 0x1, 0xc0020bcd20)
	/go/src/github.com/cortexproject/cortex/pkg/querier/worker/scheduler_processor.go:215 +0x924
github.com/cortexproject/cortex/pkg/querier/worker.(*schedulerProcessor).querierLoop.func1()
	/go/src/github.com/cortexproject/cortex/pkg/querier/worker/scheduler_processor.go:162 +0x628
created by github.com/cortexproject/cortex/pkg/querier/worker.(*schedulerProcessor).querierLoop in goroutine 4815
	/go/src/github.com/cortexproject/cortex/pkg/querier/worker/scheduler_processor.go:132 +0x96

[SungJin1212]

Should we use a forked query stat before calling the QueryResult? 🤔