Security Bulletin: Log4j vulnerability CVE-2021-44228 affects IBM Cloud Pak for Data System 1.0

Summary

Log4j is used by IBM Cloud Pak for Data System 1.0 in openshift-logging. This bulletin provides a remediation and mitigation for the reported Apache Log4j vulnerability, CVE-2021-44228.

Vulnerability Details

CVEID:CVE-2021-44228
**DESCRIPTION:**Apache Log4j could allow a remote attacker to execute arbitrary code on the system, caused by the failure to protect against attacker controlled LDAP and other JNDI related endpoints by JNDI features. By sending a specially crafted code string, an attacker could exploit this vulnerability to load arbitrary Java code on the server and take complete control of the system. Note: The vulnerability is also called Log4Shell or LogJam.
CVSS Base score: 10
CVSS Temporal Score: See: https://exchange.xforce.ibmcloud.com/vulnerabilities/214921 for the current score.
CVSS Vector: (CVSS:3.0/AV:N/AC:L/PR:N/UI:N/S:C/C:H/I:H/A:H)

Affected Products and Versions

IBM Cloud Pak for Data System 1.0 - Openshift Container Platform 3.11

| 1.0.0.0- 1.0.7.7

Remediation/Fixes

IBM strongly recommends addressing the vulnerability now by applying below patch. The remediation is applicable to ICPDS v1.0.7.6 - 1.0.7.7 releases.

IBM Cloud Pak for Data System 1.0 - Openshift Container Platform 3.11

| 1.0.0.0-openshift-3.11.log4j-WS-ICPDS-fp136| Link to Fix Central

• Please follow the steps given in release notes to apply above remediation.

Workarounds and Mitigations

Customers on ICPDS v1.0.0.0- 1.0.7.5 should apply the mitigation below

Mitigation For OpenShift Container Platform 3.11

Note: Below mitigation is needed and applicable if openshift-logging is enabled on system.

Follow below steps to identify if the system has openshift-logging enabled:

1. Login to control vm e1n1-1-control : ssh e1n1-1-control

2. Run below command:

   oc get dc -n openshift-logging

Example:

When openshift-logging is enabled:

$ oc get dc -n openshift-logging
NAME                              REVISION   DESIRED   CURRENT   TRIGGERED BY
logging-es-data-master-76ovaz98   2          1         1
logging-kibana                    1          1         1         config  

When openshift-logging is NOT enabled:

$ oc get dc -n openshift-logging
No resources found.

**Follow the below steps to mitigate the reported CVE-2021-44228 for Openshift Container Platform 3.11 **

1. Login to control vm e1n1-1-control : ssh e1n1-1-control

Run below commands as apadmin user:

2. Change to project where Logging stack deployed (by default “openshift-logging” project)

   $ oc project openshift-logging

3. Find the ‘elasticsearch’ deploymentConfigs deployed for passing later to oc set env command

   $ oc get dc -l component=es

   NAME REVISION DESIRED CURRENT TRIGGERED BY
   logging-es-data-master-kfity61t 9 1 1
   logging-es-data-master-o68rc18y 4 1 1
   logging-es-data-master-u6hh29n4 3 1 1

4. Set environment variable ‘ES_JAVA_OPTS’ in ‘elasticsearch’ for system property log4j2.formatMsgNoLookups to true

   $ oc set env -c elasticsearch dc/<elasticsearch_deploymentConfig_name> ES_JAVA_OPTS=“-Dlog4j2.formatMsgNoLookups=true”

Note:
Please check if there are already some custom environment variables set for ES_JAVA_OPTS and append them if needed.

5. Confirm before rolling out the variable is present:

   $ oc set env -c elasticsearch dc -l component=es --list | grep ES_JAVA_OPTS

6. Rollout new replicationControllers for ‘pods’ to start with the new values:. Do this for all deploymentConfigs:

   $ oc rollout latest dc/<deploymentConfig_name>

7. Check new ES pod has been spawned automatically after the rollout:

   $ oc get pods -l component=es

   NAME READY STATUS RESTARTS AGE
   elasticsearch-cdm-ba9c6evk-1-796f6cfdbc-4dqc6 2/2 Running 0 27m
   elasticsearch-cdm-ba9c6evk-2-7959d4d857-z5km9 2/2 Running 0 2d9h
   elasticsearch-cdm-ba9c6evk-3-5f9c5d668c-cr8lj 2/2 Running 0 2d9h

8. Open a shell into the newly-spawned ‘ES pods’ to check Java command-line arguments passed correctly including “-Dlog4j2.formatMsgNoLookups=true”

   $ for es_pod in $(oc get pods -l component=es --no-headers -o jsonpath=‘{range .items[?(@.status.phase==“Running”)]}{.metadata.name}{“\n”}{end}’);
   do echo “Confirm changes on $es_pod” ; sleep 1 ;
   oc rsh -Tc elasticsearch $es_pod ps auxwww | grep log4j2.formatMsgNoLookups ; sleep 3;
   done

-Dlog4j2.formatMsgNoLookups=true should be visible in above output

9. The pods should also have this variable set:

   $ for es_pod in $(oc get pods -l component=es --no-headers -o jsonpath=‘{range .items[?(@.status.phase==“Running”)]}{.metadata.name}{“\n”}{end}’);
   do echo “Confirm changes on $es_pod” ; sleep 1 ;
   oc rsh -Tc elasticsearch $es_pod printenv | grep ES_JAVA_OPTS ; sleep 3;
   done

You should see something like:

ES_JAVA_OPTS=“-Dlog4j2.formatMsgNoLookups=true”
ES_JAVA_OPTS=“-Dlog4j2.formatMsgNoLookups=true”
ES_JAVA_OPTS=“-Dlog4j2.formatMsgNoLookups=true”