Denial of service in Tendermint

Description

Denial of Service 1

Tendermint 0.33.2 and earlier does not limit the number of P2P connection requests. For each p2p connection, Tendermint allocates XXX bytes. Even though this memory is garbage collected once the connection is terminated (due to duplicate IP or reaching a maximum number of inbound peers), temporary memory spikes can lead to OOM (Out-Of-Memory) exceptions.

Tendermint 0.33.3 (and 0.32.10) limits the total number of P2P incoming connection requests to to p2p.max_num_inbound_peers + len(p2p.unconditional_peer_ids).

Notes:

• Tendermint does not rate limit P2P connection requests per IP (an attacker can saturate all the inbound slots);
• Tendermint does not rate limit HTTP(S) requests. If you expose any RPC endpoints to the public, please make sure to put in place some protection (https://www.nginx.com/blog/rate-limiting-nginx/). We may implement this in the future (https://github.com/tendermint/tendermint/issues/1696).

Denial of Service 2

Tendermint 0.33.2 and earlier does not reclaim activeID of a peer after it’s removed in Mempool reactor. This does not happen all the time. It only happens when a connection fails (for any reason) before the Peer is created and added to all reactors. RemovePeer is therefore called before AddPeer, which leads to always growing memory (activeIDs map). The activeIDs map has a maximum size of 65535 and the node will panic if this map reaches the maximum. An attacker can create a lot of connection attempts (exploiting Denial of Service 1), which ultimately will lead to the node panicking.

Tendermint 0.33.3 (and 0.32.10) claims activeID for a peer in InitPeer, which is executed before MConnection is started.

Notes:

• InitPeer function was added to all reactors to combat a similar issue - https://github.com/tendermint/tendermint/issues/3338;
• Denial of Service 2 is independent of Denial of Service 1 and can be executed without it.

Specific Go Packages Affected

github.com/tendermint/tendermint/p2p

Impact

• All full nodes (except for validators who are behind closed networks)
• Node’s memory usage increases, then it panics either in the mempool or due to OOM.

Patches

• v0.33.3
• v0.32.10
• v0.31.12

Workarounds

No workarounds.

References

• https://hackerone.com/reports/820317 (not disclosed yet)
• https://github.com/tendermint/tendermint/issues/3338
• https://github.com/tendermint/tendermint/issues/1696

For more information

If you have any questions or comments about this advisory:

• Open an issue in tendermint/tendermint
• Email us at [email protected]

More information can be found here.

Credits

• fudongbai for discovering and reporting Denial of Service 2
• Ethan Buchman (@ebuchman) for writing a test case for Denial of Service 2 and Tess Rinearson (@tessr) for fixing it
• Anton Kaliaev (@melekes) for fixing Denial of Service 1