Kubernetes: Kubelet resource exhaustion attack via metric label cardinality explosion from unauthenticated requests

Report Submission Form

Summary:

Malicious clients can potentially DOS a kubelet by sending a high amount of specially crafted requests to the kubelet’s HTTP server.

For each request the kubelet updates/sets 3 metrics:

• kubelet_http_requests_total (Counter)
• kubelet_http_requests_duration_seconds (Histogram with 7 buckets)
• kubelet_http_inflight_requests (Counter)

Each metric has the label path which will contain the path of each request.
It does not matter if the request is authenticated or not - The metrics will be set/updated regardless.
With each unique path, the kubelet creates 16 new time series.
By sending a high amount of requests with random path values, the kubelet’s memory usage will grow and eventually the kubelet will get OOM killed.

It’s also possible that the kubelet evicts all workloads before being OOM killed (Which might be worse than an OOM kill)

The corresponding kubelet server code: https://github.com/kubernetes/kubernetes/blob/v1.17.0/pkg/kubelet/server/server.go#L859-L865

Kubernetes Version:

• v1.17.0 kubeadm (tested)
• v1.16.4 kubeadm (tested)
• v1.15.7 kubeadm (tested)

Component Version:

Kubelet

Steps To Reproduce:

NODE_NAME="my-poor-node"
NODE_IP="192.168.1.100"

# Perform random requests from an unauthenticated client
curl --insecure https://${NODE_IP}:10250/foo
curl --insecure https://${NODE_IP}:10250/bar
curl --insecure https://${NODE_IP}:10250/baz

# Run in a dedicated shell to be able to get the metrics
kubectl proxy

# Load metrics from node
# For each path (foo, bar, baz) 16 time series got created
curl http://127.0.0.1:8001/api/v1/nodes/${NODE_NAME}/proxy/metrics 2>&1 | grep 'kubelet_http_requests_total\|kubelet_http_requests_duration_seconds\|kubelet_http_inflight_requests'

# Perform more random requests & see the output of the metrics endpoint to grow.

Supporting Material/References:

A gist with an additional go tool which spams the kubelet
https://gist.github.com/mrIncompetent/c6cbe483298c36668374363baf52a35d

Impact

Kill the kubelet / Make the kubelet consume all resources so it starts to evict pods.