CVE-2021-44228 Apache Log4j Vulnerability Mitigation for NetBackup Master/Primary server container on Flex Appliance
Description
Veritas NetBackup Primary/Master Server container on Flex Appliance is subject to CVE-2021-44228 Apache Log4j Vulnerability.
For Flex Appliance Platform, refer to KB #100052106
Affected Versions
NetBackup Primary/Master Server 8.1.2 - 9.1.0.1 containers running on Flex Appliance
Veritas is aware of the recently announced zero-day vulnerability in Apache’s Log4j component (CVE-2021-44228), as well as the related vulnerability described here (CVE-2021-45046). All Veritas Product Security and Development teams are actively reviewing our software to determine if these vulnerabilities exist in any of our products.
If we determine a particular product is impacted by these issues, Veritas will provide temporary mitigation guidance while we work to quickly provide a patch to permanently address the problem. This is an urgent issue, and we are working aggressively to help keep our customers secure. We will provide updates and guidance as soon as possible.
Mitigation Steps for NetBackup Master/Primary server container on Flex Appliance
IMPORTANT
On Dec 12, 2021, Veritas Engineering provided Apache log4j vulnerability mitigation steps for the NetBackup container on a Flex Appliance. Since then, Veritas has an updated procedure to ensure these changes are not impacted by future upgrades.
Which Sections of this document should you follow?
If you followed the steps provided on Dec 12, follow Sections 1 and 2.
If you are implementing the mitigation for the first time, follow Section 2.
What does each section contain?
Section 1 - Steps to rollback procedure from Dec 12, 2021 [Status - vulnerable state]
Section 2 - Steps to apply mitigation for NB Primary/Master in Flex Container [Status - Vulnerability mitigated]
Section 3 - Steps to rollback procedure in Section 2 [Status - vulnerable state]
Mitigation Steps for NetBackup Master/Primary server container on Flex Appliance
Instructions for Flex Appliance Instance (Last Updated 2021-12-14)
Section 1 - Steps to rollback previous procedure (2021-12-12)
Log in to the NetBackup application instance.
Stop nbwmc:
$ sudo /usr/openv/netbackup/bin/nbwmc stop
Remove the '/usr/openv/netbackup/bin/nbwmc' entry from the addon-symlink file:
$ cat /mnt/nbdata/usr/openv/.addon-symlink.donotremove
/usr/openv/netbackup/bin/nbwmc
$ sudo sed -i.bak -e 's|/usr/openv/netbackup/bin/nbwmc||g' /mnt/nbdata/usr/openv/.addon-symlink.donotremove
OR if '/usr/openv/netbackup/bin/nbwmc' is the only entry, the file can be removed
(e.g. sudo rm /mnt/nbdata/usr/openv/.addon-symlink.donotremove)
Move the persisted 'nbwmc' file back into it’s original location.
$ sudo mv -vf /mnt/nbdata/usr/openv/netbackup/bin/nbwmc /usr/openv/netbackup/bin/nbwmc
Section 1 - END
Section 2
Follow these steps if this is the first time you are remediating NB container on the Flex Appliance.
Step 1: Add mitigation to `/etc/environment` file
For NetBackup versions 8.1.2+
Append the mitigation environment variable to `/etc/environment`
$ sudo bash -c 'sudo echo LOG4J_FORMAT_MSG_NO_LOOKUPS="true" >> /etc/environment'
$ cat /etc/environment
LOG4J_FORMAT_MSG_NO_LOOKUPS=true
NOTE: There may be more entries in the `/etc/environment` file.
Create an `environment` file in the persistent location so the mitigation will be in place after instance restart:
$ sudo bash -c 'echo LOG4J_FORMAT_MSG_NO_LOOKUPS="true" > /mnt/nbdata/vxos/etc/environment'
$ cat /mnt/nbdata/vxos/etc/environment
LOG4J_FORMAT_MSG_NO_LOOKUPS=true
Restart nbwmc to activate the mitigation:
$ sudo /usr/openv/netbackup/bin/nbwmc stop
$ sudo /usr/openv/netbackup/bin/nbwmc start
Step 2: Create mitigation for systemd service
Create an override directory:
$ sudo mkdir -pv /etc/systemd/system/netbackup.service.d/
Create an override file for the log4j2 mitigation:
$ sudo bash -c 'cat <<END_CONTENT > /etc/systemd/system/netbackup.service.d/log4j2.conf
[Service]
Environment=LOG4J_FORMAT_MSG_NO_LOOKUPS="true"
END_CONTENT
'
NOTE: Please pay special attention to the quoting and the ending END_CONTENT on a line by itself.
Verify file content:
$ sudo cat /etc/systemd/system/netbackup.service.d/log4j2.conf
[Service]
Environment=LOG4J_FORMAT_MSG_NO_LOOKUPS="true"
Check the netbackup service:
$ systemctl status netbackup
● netbackup.service - LSB: NetBackup
Loaded: loaded (/etc/rc.d/init.d/netbackup; static; vendor preset: disabled)
Drop-In: /etc/systemd/system/netbackup.service.d
└─orchestration.conf
Active: active (running) since Mon 2021-12-13 23:21:22 UTC; 5h 19min ago
Docs: man:systemd-sysv-generator(8)
Process: 7652 ExecStart=/etc/rc.d/init.d/netbackup start (code=exited, status=0/SUCCESS)
CGroup: /system.slice/docker-f4203999fe84af43b18be0bff7bb773e1923da1880628a56bb9cdf6ffbb54629.scope/system.slice/netbackup.service
├─ 7660 /usr/openv/netbackup/bin/vnetd -standalone
<output has been snipped for brevity>
└─25306 /usr/openv/netbackup/bin/bpdbm
Warning: netbackup.service changed on disk. Run 'systemctl daemon-reload' to reload units.
We expect to see the "Warning: netbackup.service changed on disk. Run 'systemctl daemon-reload' to reload units." message indicating the new log4j2.conf has been noticed.
Execute 'systemctl daemon-reload' as suggested:
$ sudo systemctl daemon-reload
Check the service again and the "Warning" should not be observed anymore. Additionally, the log4j2.conf file will be listed in the "Drop-In" section.
$ systemctl status netbackup
● netbackup.service - LSB: NetBackup
Loaded: loaded (/etc/rc.d/init.d/netbackup; static; vendor preset: disabled)
Drop-In: /etc/systemd/system/netbackup.service.d
└─log4j2.conf, orchestration.conf
Active: active (running) since Mon 2021-12-13 23:21:22 UTC; 5h 21min ago
Docs: man:systemd-sysv-generator(8)
CGroup: /system.slice/docker-f4203999fe84af43b18be0bff7bb773e1923da1880628a56bb9cdf6ffbb54629.scope/system.slice/netbackup.service
├─ 7660 /usr/openv/netbackup/bin/vnetd -standalone
<output has been snipped for brevity>
└─25306 /usr/openv/netbackup/bin/bpdbm
Finally, use `systemctl cat netbackup` to view the unit file contents:
$ systemctl cat netbackup
# /etc/systemd/system/netbackup.service
# Automatically generated by systemd-sysv-generator
# Copied here to update from 5min default timeout to 15min
[Unit]
Documentation=man:systemd-sysv-generator(8)
SourcePath=/etc/rc.d/init.d/netbackup
Description=LSB: NetBackup
Before=shutdown.target
After=network-online.target
After=vxpbx_exchanged.service
Wants=network-online.target
Conflicts=shutdown.target
[Service]
Type=forking
Restart=no
TimeoutSec=15min
IgnoreSIGPIPE=no
KillMode=process
GuessMainPID=no
RemainAfterExit=yes
ExecStart=/etc/rc.d/init.d/netbackup start
ExecStop=/etc/rc.d/init.d/netbackup stop
# /etc/systemd/system/netbackup.service.d/log4j2.conf
[Service]
Environment=LOG4J_FORMAT_MSG_NO_LOOKUPS="true"
# /etc/systemd/system/netbackup.service.d/orchestration.conf
[Service]
Environment="ENV_NB_ORCHESTRATION=Flex"
Notice the new section with the log4j2.conf file:
# /etc/systemd/system/netbackup.service.d/log4j2.conf
[Service]
Environment=LOG4J_FORMAT_MSG_NO_LOOKUPS="true"
For NetBackup version 9.0 and later, this file will now be automatically persisted and will be present after the instance is restarted. This is the last step for NetBackup 9.0 and later versions
However, for pre-9.0 NetBackup versions, at this point, the file WILL NOT persist across restarts.
To persist the file, follow these steps:
$ sudo mkdir -pv /mnt/nbdata/vxos/etc/systemd/system
mkdir: created directory '/mnt/nbdata/vxos/etc/systemd'
mkdir: created directory '/mnt/nbdata/vxos/etc/systemd/system'
$ sudo cp -prv /etc/systemd/system/netbackup.service.d/ /mnt/nbdata/vxos/etc/systemd/system
‘/etc/systemd/system/netbackup.service.d/’ -> ‘/mnt/nbdata/vxos/etc/systemd/system/netbackup.service.d’
‘/etc/systemd/system/netbackup.service.d/log4j2.conf’ -> ‘/mnt/nbdata/vxos/etc/systemd/system/netbackup.service.d/log4j2.conf’
With those steps completed, the override IS persisted.
Section 2 - END
Section 3 - Steps to rollback vulnerability mitigation procedure in Section 2
Remove the 'LOG4J_FORMAT_MSG_NO_LOOKUPS="true"' entry from /etc/environment. Doing so will revert the environment back to the pre-mitigated state. Please engage Veritas Technical Support if you are having problems with mitigation.
$ cat /etc/environment
ENV_NB_ORCHESTRATION=Flex
LOG4J_FORMAT_MSG_NO_LOOKUPS=true
$ sudo sed -i.bak -e 's|LOG4J_FORMAT_MSG_NO_LOOKUPS=true||g' /etc/environment
$ cat /etc/environment
ENV_NB_ORCHESTRATION=Flex
Remove the /mnt/nbdata/vxos/etc/environment file
$ sudo rm -v /mnt/nbdata/vxos/etc/environment
removed ‘/mnt/nbdata/vxos/etc/environment’
Remove the systemd service override
$ ls -l {'',/mnt/nbdata/vxos}/etc/systemd/system/netbackup.service.d/
/etc/systemd/system/netbackup.service.d/:
total 8
-rw-r--r--. 1 root root 57 Dec 14 04:37 log4j2.conf
-rw-r--r--. 1 root root 50 Dec 13 23:15 orchestration.conf
/mnt/nbdata/vxos/etc/systemd/system/netbackup.service.d/:
total 16
-rw-r--r--. 1 root root 57 Dec 14 04:37 log4j2.conf
-rw-r--r--. 1 root root 50 Dec 13 23:15 orchestration.conf
$ sudo rm -v /etc/systemd/system/netbackup.service.d/log4j2.conf
removed ‘/etc/systemd/system/netbackup.service.d/log4j2.conf’
For pre-9.0 NetBackup versions, also remove the file from persistent storage:
$ sudo rm -v /mnt/nbdata/vxos/etc/systemd/system/netbackup.service.d/log4j2.conf
Reload the unit files:
$ sudo systemctl daemon-reload
If desired, stop/start nbwmc:
$ sudo /usr/openv/netbackup/bin/nbwmc stop
$ sudo /usr/openv/netbackup/bin/nbwmc start
Section 3 - END
Disclaimer
THE SECURITY ADVISORY IS PROVIDED "AS IS" AND ALL EXPRESS OR IMPLIED CONDITIONS, REPRESENTATIONS AND WARRANTIES, INCLUDING ANY IMPLIED WARRANTY OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE OR NON-INFRINGEMENT, ARE DISCLAIMED, EXCEPT TO THE EXTENT THAT SUCH DISCLAIMERS ARE HELD TO BE LEGALLY INVALID. VERITAS TECHNOLOGIES LLC SHALL NOT BE LIABLE FOR INCIDENTAL OR CONSEQUENTIAL DAMAGES IN CONNECTION WITH THE FURNISHING, PERFORMANCE, OR USE OF THIS DOCUMENTATION. THE INFORMATION CONTAINED IN THIS DOCUMENTATION IS SUBJECT TO CHANGE WITHOUT NOTICE.
Was this content helpful?
Translated Content
Please note that this document is a translation from English, and may have been machine-translated. It is possible that updates have been made to the original version after this document was translated and published. Veritas does not guarantee the accuracy regarding the completeness of the translation. You may also refer to the English Version of this knowledge base article for up-to-date information.