OpenTelemetry-Go Contrib (https://github.com/open-telemetry/opentelemetry-go-contrib) is a collection of third-party packages for OpenTelemetry-Go. It offers a wide range of features to developers including metrics, tracing, and logging capabilities. However, a critical issue has been discovered in versions prior to .46., affecting the grpc Unary Server Interceptor and potentially causing memory exhaustion due to unbound cardinality. In this post, we will explore the details of CVE-2023-47108, the code snippets, and how to mitigate the vulnerability.

Exploit Details

The grpc Unary Server Interceptor, in OpenTelemetry-Go Contrib versions prior to .46., inadvertently adds two labels with unbound cardinality - net.peer.sock.addr and net.peer.sock.port. This can lead to memory exhaustion on the server when many malicious requests are sent.

An attacker can easily flood the peer address and port for requests, causing increased memory usage and a potential Denial of Service (DoS) attack. Below is a code snippet illustrating the problematic implementation:

func grpcLabels(ctx context.Context, info *grpc.UnaryServerInfo) []attribute.KeyValue {
	attrs := []attribute.KeyValue{
		attribute.String("rpc.system", "grpc"),
		attribute.String("rpc.service", serviceName(info.FullName)),
	}

	peer, ok := peer.FromContext(ctx)
	if ok {
		attrs = append(attrs,
			attribute.String("net.peer.sock.addr", peer.Addr.String()),
			attribute.Int64("net.peer.sock.port", peer.Addr.Port()),
		)
	}
	return attrs
}

Mitigation

To resolve this issue, OpenTelemetry-Go Contrib version .46. contains a fix limiting the cardinality of the added labels. Upgrading to this version or later is highly recommended. If upgrading is not possible, there are two workarounds available:

Use a view to remove the problematic attributes

import (
	"go.opentelemetry.io/contrib/instrumentation/google.golang.org/grpc/otelgrpc"
	"go.opentelemetry.io/otel/label"
)

// Create a view removing the problematic attributes
otelgrpc.UnaryServerInterceptor(
	otelgrpc.WithAllowedLabels([]label.Key{
		label.Key("rpc.system"),
		label.Key("rpc.service"),
	}),
)

2. Disable grpc metrics instrumentation by passing the otelgrpc.WithMeterProvider option with noop.NewMeterProvider:

import (
	"go.opentelemetry.io/contrib/instrumentation/google.golang.org/grpc/otelgrpc"
	"go.opentelemetry.io/otel/metric/noop"
)

// Disable grpc metrics instrumentation
otelgrpc.UnaryServerInterceptor(
	otelgrpc.WithMeterProvider(noop.NewMeterProvider()),
)

Conclusion

CVE-2023-47108 is a critical vulnerability affecting OpenTelemetry-Go Contrib prior to version .46., potentially leading to memory exhaustion and DoS attacks. Users are advised to upgrade their OpenTelemetry-Go Contrib version to .46. or later or apply one of the provided workarounds to mitigate the issue.

For more details and updates on this vulnerability, refer to the GitHub issue (https://github.com/open-telemetry/opentelemetry-go-contrib/issues/1326) and the associated pull request (https://github.com/open-telemetry/opentelemetry-go-contrib/pull/1327). Stay safe and keep your dependencies updated!

Timeline

Published on: 11/10/2023 19:15:16 UTC
Last modified on: 11/20/2023 19:34:26 UTC