Impact of CVE-2021-44228, CVE-2021-45046, CVE-2021-45105 Apache Log4j Vulnerability Mitigation for Flex Scale Appliance
About CVE-2021-44228, CVE-2021-45046, CVE-2021-45105 Apache Log4j Vulnerabilities
Apache Log4j is an open-source, Java-based logging utility widely used by enterprise applications and cloud services.
The Apache Software Foundation has released a security advisory to address a remote code execution vulnerability (CVE-2021-44228) and a denial of service vulnerability (CVE-2021-45046) affecting Log4j versions 2.0-beta9 to 2.15. A remote attacker could exploit these vulnerabilities to take control of an affected system.
More information is available from the Apache Announcement and recommends upgrading to the latest Log4j 2.16.0 or applying recommended mitigations immediately.
Issue
CVE-2021-44228: Apache Log4j2 JNDI features do not protect against attacker controlled LDAP and other JNDI related endpoints.
Severity: Critical
Base CVSS Score: 10.0
CVSS:3.0/AV:N/AC:L/PR:N/UI:N/S:C/C:H/I:H/A:H
CVE-2021-45046: Apache Log4j2 JNDI features do not protect against malicious input data using a JNDI Lookup pattern resulting in a denial of service (DOS) attack.
Severity: Critical
Base CVSS Score: 9.0
CVSS:3.1/AV:N/AC:H/PR:N/UI:N/S:C/C:H/I:H/A:H
CVE-2021-45105: Apache Log4j2 does not always protect from infinite recursion in lookup evaluation. Fixed in Log 4j 2.17.0. Apache recommends upgrading to Log4j 2.17.0. NetBackup Flex Scale is not impacted by this vulnerability
JMSAppender in Log4j 1.2 is vulnerable to deserialization of untrusted data when the attacker has write access to the Log4j configuration. Fixed in Log 4j 2. Apache recommends upgrading to Log4j 2. NetBackup Flex Scale Appliances are not exploitable by this vulnerability.
Apache Log4j2 versions 2.0-beta7 through 2.17.0 (excluding security fix releases 2.3.2 and 2.12.4) are vulnerable to a remote code execution (RCE) attack where an attacker with permission to modify the logging configuration file can construct a malicious configuration using a JDBC Appender with a data source referencing a JNDI URI which can execute remote code. Fixed in Log 4j 2.17.1. Apache recommends upgrading to Log4j 2.17.1. NetBackup Flex Scale Appliances are not exploitable by this vulnerability.
Affected NetBackup Appliance Versions:
Product / Component |
Version |
Mitigation Steps |
Download EEB from |
NetBackup Flex Scale |
1.3 or 1.3.1 |
Veritas strongly recommends customers upgrade to NetBackup Flex Scale 2.1 release in order to be able to perform the mitigation steps below. |
|
NetBackup Flex Scale |
2.1 |
Set parameter log4j2.formatMsgNoLookups to true |
Mitigation for Flex Scale 2.1
Pre-requisite: You should have nbfs 2.1 GA iso deployed on all nodes of your cluster. After cluster configuration follow below mentioned steps to install the EEBs..
1. Download the VRTSnbfsapp_EEB_ET4059862-2.1.0.0-1.x86_64.rpm eeb from UPD195687
2. Login to Appliance Management GUI (https://<Management_Server_FQDN>:14161)
3. Navigate to Settings → Software Management page
4. Click on "Add-ons" tab
5. Click on "Add" link and then click "choose add-ons" to upload the downloaded EEB from your local machine
6. Once the EEB rpm package is uploaded successfully, it will list there in the GUI table on that page
7. Click on the radio button corresponding to your EEB which you want to install, and then click on "Install" button
8. It will start the EEB installation, keep checking the progress of install from Recent Activity → View All activities task table
9. Once EEB installation completes, it will automatically start full discovery to sync all the tasks/processes
10. Wait for full discovery to complete
11. Elevate to root. If lockdown mode is configured, please contact Veritas Support to get access to root shell.
12. Verify that log4j class is deleted from asc-apiserver and asc-globalapiservice containers using the below steps :
Login into the node through ssh using an Appliance Administrator account
Run “Support elevate” and elevate to root shell
Run hacli -cmd "docker exec asc-globalapiservice grep -r JndiLookup.class /opt"
Run hacli -cmd "docker exec asc-apiserver grep -r JndiLookup.class /opt"
Ensure the output of above commands doesn’t show any references to the effected JndiLookup class
Verify asc-apiserver and asc-globalapiservice is running properly ‘docker ps -a’.
Verify one of the nodes has globalapiservice running. You can check that from this command ‘hastatus -sum | grep GLOBAL_API_SERVER | grep ONLINE’.
13. Above steps can also be run on newly added node or replaced node to verify EEB fixes are available on these nodes.
Manual mitigation steps for Flex Scale 2.1, if unable to apply the EEB:
Please run all the steps on all the nodes in the cluster unless stated otherwise.
Elevate to root. If lockdown mode is configured, please contact Veritas Support to get access to root shell.
Note tag details for asc-apiserver and asc-globalapiservice using ‘docker images’ these two images will be available on all the nodes and one can note them down from any one node.
Note down Entrypoint’s value section from ‘docker inspect asc-apiserver’ and ‘docker inspect asc-globalapiservice’ [NOTE: asc-globalapiservice will only be running on one node and one can find that node using this command ‘hastatus -sum | grep GLOBAL_API_SERVER | grep ONLINE’]
Create a file /tmp/asc-apiserver/Dockerfile and navigate to /tmp/asc-apiserver.
mkdir -p /tmp/asc-apiserver
cd /tmp/asc-apiserver
vi Dockerfile
Paste below contents in this Docker file. Update the Entry point details with details which you note down in step 2 removing tabs and new line characters.
FROM asc-apiserver:latest
RUN /bin/bash -c 'for log4jcore in `find /opt -name \*log4j\*core\*.jar 2> /dev/null`;do echo $log4jcore; zip -q -d $log4jcore org/apache/logging/log4j/core/lookup/JndiLookup.class;done'
ENTRYPOINT ["/usr/bin/java", "--module-path=/opt/autosupport/apiserver/lib/ccj-3.0.1.jar", "--add-modules=ccj", "--add-exports", "java.base/sun.security.provider=ccj", "--add-exports", "java.base/sun.security.internal.spec=ccj", "-Dcom.safelogic.cryptocomply.fips.approved_only=true", "-Xmx1024m", "-XX:MaxMetaspaceSize=256m", "-cp", "/opt/autosupport/apiserver/ApiServer-2.4.0.jar:/opt/autosupport/apiserver/lib/*", "com.veritas.appliance.autosupport.client.apiserver.ApiServerMain"]
Run below command with the noted tag number in step 1. In this case it would be 2.4.0.
“docker build -t asc-apiserver:<tag_number> .”
Run ‘systemctl restart autosupportclient’.
Create a file /tmp/asc-globalapiservice/Dockerfile and navigate to /tmp/asc- globalapiservice.
mkdir -p /tmp/asc-globalapiservice
cd /tmp/asc- globalapiservice
vi Dockerfile
Paste below contents in this Docker file. Update the Entry point details with details which you note down in step 2 removing tabs and new line characters.
FROM asc-globalapiservice:latest
RUN /bin/bash -c 'for log4jcore in `find /opt -name \*log4j\*core\*.jar 2> /dev/null`;do echo $log4jcore; zip -q -d $log4jcore org/apache/logging/log4j/core/lookup/JndiLookup.class;done'
ENTRYPOINT ["./bin/globalapiservice"]
Run below command with the noted tag number in step 1. In this case it would be 2.4.0.
“docker build -t asc-globalapiservice:<tag_number> .”
. Only perform the below steps on the machine that has globalapiservice running. You can check that from this command ‘hastatus -sum | grep GLOBAL_API_SERVER | grep ONLINE’.
/opt/VRTS/bin/hagrp -offline GLOBAL_API_SERVER -sys <node-name>
/opt/VRTS/bin/hagrp -state GLOBAL_API_SERVER (Make sure all globalapiserver are offline)
/opt/VRTS/bin/hagrp -online GLOBAL_API_SERVER -sys <node-name>
Verify asc-apiserver and asc-globalapiservice is running properly ‘docker ps -a’.
Verify if test email is successful from webgui.
Also, verify if you can see hardware health information on webgui.
Steps need to be repeated on the new node which is added to the cluster using the NetBackup Flex Scale Web interface.
Similarly, steps also need to be repeated on the node which is replaced in the cluster using the NetBackup Flex Scale Web interface.
Flex Scale primary server container instructions (updated 2021-12-13)
NOTE: The steps outlined in this article will require connecting to the NetBackup Primary/Master Application server with a NetBackup administrator user, usually with `ssh`. For example: `ssh nbadmin@nbuprimary.example.com`.
Mitigation for Flex Scale 2.1
Steps for testing Primary server container Log4j Version:
************************************************************
Pre-requisite : You should have nbfs 2.1 GA iso deployed on all nodes of your cluster. After cluster configuration follow below mentioned steps to install the eebs
Mitigation using EEB for NetBackup Flex Scale 2.1
1. Download the VRTSnbfsapp_nb_EEB_ET4058560-9.1.0.1-2.x86_64.rpm eeb from UPD195687
2. Login to Appliance Management GUI (https://<Management_Server_FQDN>:14161)
3. Navigate to Settings → Software Management page
4. Click on "Add-ons" tab
5. Click on "Add" link and then click "choose add-ons" to upload the downloaded EEB from your local machine
6. Once the EEB rpm package is uploaded successfully, it will list there in the GUI table on that page
7. Click on the radio button corresponding to your EEB which you want to install, and then click on "Install" button
8. It will start the EEB installation, keep checking the progress of install from Recent Activity → View All activities task table
9. Once EEB installation completes, it will automatically start full discovery to sync all the tasks/processes
10. Wait for full discovery to complete
11. Find the node where NBU primary server is running and then login Inside Netbackup Primary container on that node using below command :
command : docker exec -it nb_master /bin/sh
12. Once you are inside the container, find the version of log4j using below command :
command : sudo find / -name 'log4j*'
13. Output of above command should show the log4j related jar files version as 2.16
Manual mitigation steps for Flex Scale 2.1, if unable to apply the EEB:
Section 1
STEPS IF ATTEMPTING REMEDIATION FOR THE FIRST TIME
NOTE: There is example output in the steps below. Please note that your output may differ and that important information has been called out where applicable.
Step 1: Add mitigation to `/etc/environment` file
For NetBackup Flex Scale versions 2.1
Append the mitigation environment variable to `/etc/environment`
$ sudo bash -c 'sudo echo LOG4J_FORMAT_MSG_NO_LOOKUPS="true" >> /etc/environment'
$ cat /etc/environment
LOG4J_FORMAT_MSG_NO_LOOKUPS=true
NOTE: There may be more entries in the `/etc/environment` file.
Create an `environment` file in the persistent location so the mitigation will be in place after instance restart:
$ sudo bash -c 'echo LOG4J_FORMAT_MSG_NO_LOOKUPS="true" > /mnt/nbdata/vxos/etc/environment'
$ cat /mnt/nbdata/vxos/etc/environment
LOG4J_FORMAT_MSG_NO_LOOKUPS=true
Restart nbwmc to activate the mitigation:
$ sudo /usr/openv/netbackup/bin/nbwmc stop
$ sudo /usr/openv/netbackup/bin/nbwmc start
Verification that the environment variable is in effect:
Run bpps to find the PID of the NetBackup webservice application:
$ sudo /usr/openv/netbackup/bin/bpps -a |grep java
nbwebsvc 13138 1 64 22:41 ? 00:03:27 /usr/openv/java/jre/bin/java -Dnop -Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager -Dnbwmc -d64 -Xms4096m -Xmx4096m -XX:MetaspaceSize=192m -XX:NewRatio=1 -XX:SurvivorRatio=4 -XX:-UseAdaptiveSizePolicy -Djava.library.path=/usr/openv/lib -DNB_HOSTNAME=deecl01vm054p5.engba.veritas.com -DNB_INSTALL_DIR=/usr/openv -DVAR_GLOBAL=/usr/openv/var/global -Djava.util.logging.config.file=/usr/openv/wmc/config/tomcat.logging.properties -Dlog.conf.path=/usr/openv/netbackup/nblog.conf -Dvrtsat.home=/usr/openv/netbackup/sec/at -Djdk.tls.ephemeralDHKeySize=2048 -Djdk.tls.rejectClientInitiatedRenegotiation=true -Dcom.safelogic.cryptocomply.fips.approved_only=false -verbose:jni -Djava.security.egd=file:/dev/./urandom -XX:ErrorFile=/usr/openv/wmc/webserver/logs/hs_err_pid%p.log -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/usr/openv/wmc/webserver/logs -XX:+PrintGCDetails -verbose:gc -XX:+PrintGCDateStamps -XX:+PrintGCApplicationStoppedTime -Xloggc:/usr/openv/wmc/webserver/logs/gc.log -XX:+UseGCLogFileRotation -XX:NumberOfGCLogFiles=5 -XX:GCLogFileSize=1M -Djdk.tls.ephemeralDHKeySize=2048 -Djava.protocol.handler.pkgs=org.apache.catalina.webresources -Dorg.apache.catalina.security.SecurityListener.UMASK=0027 -Dignore.endorsed.dirs= -classpath /usr/openv/wmc/webserver/bin/bootstrap.jar:/usr/openv/wmc/webserver/bin/tomcat-juli.jar -Dcatalina.base=/usr/openv/var/global/wsl/webserver -Dcatalina.home=/usr/openv/wmc/webserver -Djava.io.tmpdir=/usr/openv/wmc/webserver/temp org.apache.catalina.startup.Bootstrap start
Using PID 13138 from the example output, grep for the environment variable in the running process environ.
$ sudo -u nbwebsvc cat /proc/13138/environ | strings | grep LOG4J_FORMAT_MSG_NO_LOOKUPS
LOG4J_FORMAT_MSG_NO_LOOKUPS=true
Step 2: Create mitigation for systemd service
Create an override directory:
$ sudo mkdir -pv /etc/systemd/system/netbackup.service.d/
Create an override file for the log4j2 mitigation:
$ sudo bash -c 'cat <<END_CONTENT > /etc/systemd/system/netbackup.service.d/log4j2.conf
[Service]
Environment="LOG4J_FORMAT_MSG_NO_LOOKUPS=true"
END_CONTENT
'
NOTE: Please pay special attention to the quoting and the ending END_CONTENT on a line by itself.
Verify file content:
$ sudo cat /etc/systemd/system/netbackup.service.d/log4j2.conf
[Service]
Environment="LOG4J_FORMAT_MSG_NO_LOOKUPS=true"
Check the netbackup service:
$ systemctl status netbackup
● netbackup.service - LSB: NetBackup
Loaded: loaded (/etc/rc.d/init.d/netbackup; static; vendor preset: disabled)
Drop-In: /etc/systemd/system/netbackup.service.d
└─orchestration.conf
Active: active (running) since Mon 2021-12-13 23:21:22 UTC; 5h 19min ago
Docs: man:systemd-sysv-generator(8)
Process: 7652 ExecStart=/etc/rc.d/init.d/netbackup start (code=exited, status=0/SUCCESS)
CGroup: /system.slice/docker-f4203999fe84af43b18be0bff7bb773e1923da1880628a56bb9cdf6ffbb54629.scope/system.slice/netbackup.service
├─ 7660 /usr/openv/netbackup/bin/vnetd -standalone
-->8 snip 8<--
└─25306 /usr/openv/netbackup/bin/bpdbm
Warning: netbackup.service changed on disk. Run 'systemctl daemon-reload' to reload units.
We expect to see the "Warning: netbackup.service changed on disk. Run 'systemctl daemon-reload' to reload units." message indicating the new log4j2.conf has been noticed.
Execute 'systemctl daemon-reload' as suggested:
$ sudo systemctl daemon-reload
Check the service again and the "Warning" should not be observed anymore. Additionally, the log4j2.conf file will be listed in the "Drop-In" section.
$ systemctl status netbackup
● netbackup.service - LSB: NetBackup
Loaded: loaded (/etc/rc.d/init.d/netbackup; static; vendor preset: disabled)
Drop-In: /etc/systemd/system/netbackup.service.d
└─log4j2.conf, orchestration.conf
Active: active (running) since Mon 2021-12-13 23:21:22 UTC; 5h 21min ago
Docs: man:systemd-sysv-generator(8)
CGroup: /system.slice/docker-f4203999fe84af43b18be0bff7bb773e1923da1880628a56bb9cdf6ffbb54629.scope/system.slice/netbackup.service
├─ 7660 /usr/openv/netbackup/bin/vnetd -standalone
-->8 snip 8<--
└─25306 /usr/openv/netbackup/bin/bpdbm
Finally, use `systemctl cat netbackup` to view the unit file contents:
$ systemctl cat netbackup
# /etc/systemd/system/netbackup.service
# Automatically generated by systemd-sysv-generator
# Copied here to update from 5min default timeout to 15min
[Unit]
Documentation=man:systemd-sysv-generator(8)
SourcePath=/etc/rc.d/init.d/netbackup
Description=LSB: NetBackup
Before=shutdown.target
After=network-online.target
After=vxpbx_exchanged.service
Wants=network-online.target
Conflicts=shutdown.target
[Service]
Type=forking
Restart=no
TimeoutSec=15min
IgnoreSIGPIPE=no
KillMode=process
GuessMainPID=no
RemainAfterExit=yes
ExecStart=/etc/rc.d/init.d/netbackup start
ExecStop=/etc/rc.d/init.d/netbackup stop
# /etc/systemd/system/netbackup.service.d/log4j2.conf
[Service]
Environment=”LOG4J_FORMAT_MSG_NO_LOOKUPS=true"
Notice the new section with the log4j2.conf file:
# /etc/systemd/system/netbackup.service.d/log4j2.conf
[Service]
Environment=”LOG4J_FORMAT_MSG_NO_LOOKUPS=true"
Note: the file log4j2.conf in the location above WILL NOT persist across restarts.
To persist the file, follow these steps:
$ sudo mkdir -pv /mnt/nbdata/vxos/etc/systemd/system
mkdir: created directory '/mnt/nbdata/vxos/etc/systemd'
mkdir: created directory '/mnt/nbdata/vxos/etc/systemd/system'
$ sudo cp -prv /etc/systemd/system/netbackup.service.d/ /mnt/nbdata/vxos/etc/systemd/system
‘/etc/systemd/system/netbackup.service.d/’ -> ‘/mnt/nbdata/vxos/etc/systemd/system/netbackup.service.d’
‘/etc/systemd/system/netbackup.service.d/log4j2.conf’ -> ‘/mnt/nbdata/vxos/etc/systemd/system/netbackup.service.d/log4j2.conf’
With those steps completed, the override IS persisted.
Section 2: To revert the environment variable changes, follow these instructions.
Remove the 'LOG4J_FORMAT_MSG_NO_LOOKUPS="true"' entry from /etc/environment
$ cat /etc/environment
LOG4J_FORMAT_MSG_NO_LOOKUPS=true
$ sudo sed -i.bak -e 's|LOG4J_FORMAT_MSG_NO_LOOKUPS=true||g' /etc/environment
$ cat /etc/environment
Remove the /mnt/nbdata/vxos/etc/environment file
$ sudo rm -v /mnt/nbdata/vxos/etc/environment
removed ‘/mnt/nbdata/vxos/etc/environment’
Remove the systemd service override
$ ls -l {'',/mnt/nbdata/vxos}/etc/systemd/system/netbackup.service.d/
/etc/systemd/system/netbackup.service.d/:
total 8
-rw-r--r--. 1 root root 57 Dec 14 04:37 log4j2.conf
-rw-r--r--. 1 root root 50 Dec 13 23:15 orchestration.conf
/mnt/nbdata/vxos/etc/systemd/system/netbackup.service.d/:
total 16
-rw-r--r--. 1 root root 57 Dec 14 04:37 log4j2.conf
-rw-r--r--. 1 root root 50 Dec 13 23:15 orchestration.conf
$ sudo rm -v /etc/systemd/system/netbackup.service.d/log4j2.conf
removed ‘/etc/systemd/system/netbackup.service.d/log4j2.conf’
Remove the file from persistent storage:
$ sudo rm -v /mnt/nbdata/vxos/etc/systemd/system/netbackup.service.d/log4j2.conf
Reload the unit files:
$ sudo systemctl daemon-reload
If desired, stop/start nbwmc:
$ sudo /usr/openv/netbackup/bin/nbwmc stop
$ sudo /usr/openv/netbackup/bin/nbwmc start
Please note that permanent fix for CVE-2021-44228 and CVE-2021-45046 is on the way.
Was this content helpful?
Translated Content
Please note that this document is a translation from English, and may have been machine-translated. It is possible that updates have been made to the original version after this document was translated and published. Veritas does not guarantee the accuracy regarding the completeness of the translation. You may also refer to the English Version of this knowledge base article for up-to-date information.