CVE-2022-29224: Envoy Proxy Vulnerability Involving Segmentation Fault in GrpcHealthCheckerImpl

Envoy is a popular and widely used cloud-native high-performance proxy designed to handle large-scale, dynamic service meshes efficiently. However, recently, a security flaw has been identified in versions of Envoy prior to 1.22.1, which can cause a segmentation fault in the GrpcHealthCheckerImpl. This article explores the details of this vulnerability, CVE-2022-29224, and provides guidance on how to address or mitigate the issue.

Exploit Details

Envoy enables various types of upstream health checking to ensure that discovered services are functioning correctly. One such health checking method is through gRPC. Additionally, Envoy features the ability to "hold" or prevent the removal of upstream hosts obtained via service discovery until configured active health checks fail.

In the vulnerability under discussion, if an attacker gains control over an upstream host and its service discovery (achieved through DNS manipulation, the EDS API, or other means), they can crash Envoy by removing the host from the service discovery list and simultaneously causing the gRPC health check request to fail. Doing so results in a null pointer dereference, resulting in a segmentation fault and a crash of the Envoy proxy.

Below is a code snippet illustrating the issue

// Affected code in GrpcHealthCheckerImpl

// [1] Removing the host from service discovery list
void GrpcHealthCheckerImpl::Cluster::removeHost(const HostSharedPtr& host) {
  const auto host_address_health_check_iter = host_address_health_check_map_.find(host);
  
  // [2] Failing the gRPC health check request
  if (host_address_health_check_iter != host_address_health_check_map_.end()) {
    host_address_health_check_iter->second->setUnhealthy();
  }
}

// ...
void GrpcHealthCheckerImpl::setUnhealthy() {
  // This runs after [2], which ultimately results in a null pointer dereference crashing Envoy.
  event_logger_->logEjectUnhealthy(*host_.lock());
}

Mitigation Recommendations

The most effective solution to address this vulnerability is by upgrading Envoy to version 1.22.1 or later, which contains patches for CVE-2022-29224. Upgrading ensures that your Envoy deployment is no longer affected by the security issue.

However, if upgrading is not feasible, users can consider the following alternative mitigations

1. Disable gRPC health checking in your Envoy configuration. This prevents Envoy from using the affected functionality.

2. Replace gRPC health checking with a different type of health checking, such as HTTP or TCP. This can also prevent the exploitation of the vulnerability but may require reconfiguring your services to support alternative health checking methods.

Refer to Envoy's official documentation for guidance on configuring alternative health checks and other service mesh configurations: https://www.envoyproxy.io/docs/envoy/latest/intro/arch_overview/upstream/health_checking

Conclusion

CVE-2022-29224 is a critical vulnerability affecting Envoy versions prior to 1.22.1 due to a segmentation fault in GrpcHealthCheckerImpl. By gaining control of an upstream host and its service discovery, an attacker can crash Envoy and negatively impact the availability and reliability of your service mesh.

To safeguard against this vulnerability, it is highly recommended to upgrade Envoy to version 1.22.1 or later. If upgrading is not possible, consider disabling gRPC health checking or replacing it with another type of health check.

Timeline

Published on: 06/09/2022 19:15:00 UTC
Last modified on: 06/16/2022 17:46:00 UTC