Microsoft’s Response to CVE-2021-44228 Apache Log4j 2

Microsoft’s Response to CVE-2021-44228 Apache Log4j 2

Microsoft’s Response to CVE-2021-44228 Apache Log4j 2

SUMMARY

Microsoft continues our analysis of the remote code execution vulnerabilities related to Apache Log4j (a logging tool used in many Java-based applications) disclosed on 9 Dec 2021. Currently, Microsoft is not aware of any impact, outside of the initial disclosure involving Minecraft: Java Edition, to the security of our enterprise services and has not experienced any degradation in availability of those services as a result of this vulnerability.

Our security teams have been analyzing our products and services to identify and mitigate any instances of CVE-2021-4428 and CVE-2021-45046 in Apache Log4j 2.

Affected Microsoft products requiring customer action have been released in our Security Update Guide – CVE-2021-44228. Customers are encouraged to apply these updates as quickly as possible. If you are using any Microsoft services other than those explicitly listed in the CVE, no action is required by you at this time. As we continue our investigation, we will notify affected parties if we identify any impact to customer data.

To help customers protect themselves, we are also providing the following product specific guidance to help customers improve their security posture. Links are provided to jump to the content below:

Apply the Latest Security Updates

To address this vulnerability, Microsoft recommends customers apply the latest security updates to remediate this vulnerability. Please review the Apache CVEs and the Apache security advisory for further details: 

All systems, including those that are not internet facing, are potentially vulnerable to these vulnerabilities, so backend systems and microservices should also be upgraded.  No Java version can mitigate these vulnerabilities. The recommended action is to update Apache Log4j 2. An application restart will be required.  

  • Java 8 or newer: update Log4j to 2.16.0
  • Java 7: update to Log4j 2.12.2 when it becomes available

Systems that have already been updated to 2.15.0, should plan to move to 2.16.0 as soon as possible for extra protection against a potential DoS attack under specific conditions as described in CVE-2021-45046.  

Systems running on Log4j 1.x are not impacted by these vulnerabilities. In 2015, Apache announced Log4j 1.x has reached end-of-life. Microsoft recommends customers to upgrade to Log4j 2.16.0 for the latest security updates.  

Workarounds

To help mitigate the risk of these vulnerabilities in Log4j 2.x until the more complete security update can be applied, customers should consider the following mitigations steps. These workarounds should not be considered a complete solution to resolve these vulnerabilities: 

  • In case the Log4j 2 vulnerable component cannot be updated, Log4j versions 2.10 to 2.14.1 support the parameter log4j2.formatMsgNoLookups to be set to ‘true’, to disable the vulnerable feature. Ensure this parameter is configured in the startup scripts of the Java Virtual Machine:  
    -Dlog4j2.formatMsgNoLookups=true. 
  • Alternatively, customers using Log4j 2.10 to 2.14.1 may set the LOG4J_FORMAT_MSG_NO_LOOKUPS=”true” environment variable to force this change. 
  • Kubernetes administrators may use “kubectl set env” to set the LOG4J_FORMAT_MSG_NO_LOOKUPS=”true” environment variable to apply the mitigation across Kubernetes clusters where the Java applications are running Log4j 2.10 to 2.14.1, effectively reflecting on all pods and containers automatically. 
  • For all releases between 2.0-beta9 and 2.15.0, a better mitigation approach is to prevent the JndiLookup.class file from being loaded in the applications’s classpath. Customers can do this by deleting the class from affected JAR files. For example: 
    $ zip -q -d log4j-core-*.jar org/apache/logging/log4j/core/lookup/JndiLookup.class 
  • Log4j may also be present in other files as a bundle or as a shaded library. Microsoft advises customers to do an extensive search beyond log4j-core-*.jar files.
  • An application restart will be required for these changes to take effect. 

Background of Log4j

The vulnerability, tracked as CVE-2021-44228 and referred to as “Log4Shell,” affects Java-based applications that use Log4j 2 versions 2.0 through 2.14.1. Log4j 2 is a Java-based logging library that is widely used in business system development, included in various open-source libraries, and directly embedded in major software applications. The scope of impact has expanded to thousands of products and devices, including Apache products such as Struts 2, Solr, Druid, Flink, Swift, Karaf, and others.

Because this vulnerability is in a Java library, the cross-platform nature of Java means the vulnerability is exploitable on many platforms, including Windows, macOS, and Linux. As many Java-based applications can leverage Log4j 2 directly or indirectly, organizations should contact application vendors or ensure their Java applications are running the latest up-to-date version. Developers using Log4j 2 should ensure that they are incorporating the latest version of Log4j into their applications as soon as possible to protect users and organizations.

Analysis of the CVE-2021-44228 vulnerability

The vulnerability is a remote code execution vulnerability that can allow an unauthenticated attacker to gain complete access to a target system. It can be triggered when a specially crafted string is parsed and processed by the vulnerable Log4j 2 component. This could happen through any user provided input.

Successful exploitation allows for arbitrary code execution in the targeted application. Attackers do not need prior access to the system to log the string and can remotely cause the logging event by using commands like curl against a target system to log the malicious string in the application log. When processing the log, the vulnerable system reads the string and executes it, which in current attacks is used to execute the code from the malicious domain. Doing so can grant the attacker full access and control of the affected application.

Given the fact that logging code and functionalities in applications and services are typically designed to process a variety of external input data coming from upper layers and from many possible vectors, the biggest risk factor of this vulnerability is predicting whether an application has a viable attack vector path that will allow the malformed exploit string to reach the vulnerable Log4j 2 code and trigger the attack.

A common pattern of exploitation risk, for example, is a web application with code designed to process usernames, referrer, or user-agent strings in logs. These strings are provided as external input (e.g., a web application built with Apache Struts). An attacker can send a malformed username or set user-agent with the crafted exploit string hoping that this external input will be processed at some point by the vulnerable Log4j 2 code and trigger code execution.

Figure 1. CVE-2021-44228 exploit vectors and attack chain 

Mitigation Guidance for Microsoft Services

After further analysis of our services and products, below are a few mitigation strategies given by various Microsoft services.

Azure App Service (Windows, Linux and Containers)

Azure App Service and Functions does not distribute Log4J in the managed runtimes such as Tomcat, Java SE, JBoss EAP, or the Functions Runtime. However, your applications may use Log4J and be susceptible to this vulnerability.

Customers are recommended to apply the latest Log4j security updates and re-deploy applications.

If you are not able to re-package your application with a newer version of Log4j and you are using Log4j versions 2.10 to 2.14, you can mitigate this behavior by creating an application setting for the environment variable LOG4J_FORMAT_MSG_NO_LOOKUPS with value true, with the Azure CLI as follows:

 
$ az webapp config appsettings set \
 --resource-group <group-name> \ 
 --name <app-name> \ 
 --settings LOG4J_FORMAT_MSG_NO_LOOKUPS=true

Note that this command will also restart your App Service. If you are using Log4J version 2.9 or lower, this configuration mitigation will not work, and you must apply the latest security updates or consider other mitigation steps described previously in this article.

Azure Application Gateway, Azure Front Door, and Azure WAF 

In our investigation so far, we have not found any evidence that these services are vulnerable however customer applications running behind these services might be vulnerable to this exploit. We highly recommend customers to follow mitigations and workarounds mentioned in this blog to protect their applications. Additional guidance for Azure WAF is located here.

Azure Functions

Customers are recommended to apply the latest Log4j security updates and re-deploy applications. If you are not able to and you are using Log4j versions 2.10 to 2.14.1, configuring the environment variable or system property will depend on your choice of hosting option: dedicated, premium or consumption.

  • Dedicated and Premium Functions: Create two application settings:
    1. LOG4J_FORMAT_MSG_NO_LOOKUPS with value =true
    2. WEBSITE_USE_PLACEHOLDER with value =0
  • This can be done with the following Azure CLI command:

$ az functionapp config appsettings set \
--subscription \
--name \
--resource-group \
--settings "LOG4J_FORMAT_MSG_NO_LOOKUPS=true" "WEBSITE_USE_PLACEHOLDER=0"

  • Consumption Functions:
    • Linux: Create an application setting named “languageWorkers__java__arguments” with a value of “-Dlog4j2.formatMsgNoLookups=true”.
    • Windows: Create an application setting named “languageWorkers:java:arguments” with a value of “-Dlog4j2.formatMsgNoLookups=true”. 

Note that updating application settings will restart your Function apps, which could affect cold start performance. If you are using Log4j version 2.9 or lower, these mitigations will not work and you must apply the latest security updates for Log4j or follow other mitigations as described earlier in this article.

Azure HDInsights

Automatic updates are enabled by default for existing clusters and customers do not need to take any action. Customers that have turned off automatic updates must apply the updates by running the following script on every cluster node.    

Azure Spring Cloud

Azure Spring Cloud does NOT directly distribute Log4j. However, your applications deployed to Azure Spring Cloud may use Log4j and be susceptible to this vulnerability. Log4j usage may originate from:  

  • Your application sources. 
  • Application Performance Monitoring tools activated for the application. 

Spring Boot Applications

Spring Boot applications are only affected if they have switched the default logging framework to Log4j 2. The log4j-to-slf4j and log4j-api jar files that are included in spring-boot-starter-logging cannot be exploited on their own. Only applications using log4j-core are vulnerable. If your application is impacted and you can redeploy the application, we recommend that you upgrade your application with the latest security updates for Log4j, and redeploy to Azure Spring Cloud – see more details at Log4j 2 vulnerability and Spring Boot

If you are not able to re-deploy, you may mitigate impacted applications that are using Log4j 2.10 to 2.14.1 by setting the log4j2.formatMsgNoLookups system property to true OR by setting the environment variable LOG4J_FORMAT_MSG_NO_LOOKUPS to true. You can set the system property or environment variable using: 

  • Azure Portal 
  • Azure CLI 
  • ARM Template 
  • Bicep or  
  • Terraform.  
For Example – set the system property log4j2.formatMsgNoLookups via the Azure Portal or CLI 

In the Azure Portal, navigate to your application in Azure Spring Cloud and change the configuration as illustrated below: 

You can set the log4j2.formatMsgNoLookups system property to true using the Azure CLI: 

$ az spring-cloud app update -s ${SERVICE_NAME} \
      -n ${APP_NAME} -d ${DEPLOYMENT_NAME} -g ${RESOURCE_GROUP} \
      --jvm-options='-Dlog4j2.formatMsgNoLookups=true' 

For Example – set the LOG4J_FORMAT_MSG_NO_LOOKUPS environment variable via the Azure Portal or CLI 

In the Azure Portal, navigate to your application in Azure Spring Cloud and change the configuration as illustrated below: 

You can set the LOG4J_FORMAT_MSG_NO_LOOKUPS environment variable to true using the Azure CLI: 

$ az spring-cloud app update -s ${SERVICE_NAME} \
      -n ${APP_NAME} -d ${DEPLOYMENT_NAME} -g ${RESOURCE_GROUP} \
      --env 'LOG4J_FORMAT_MSG_NO_LOOKUPS=true' 

Application Performance Monitoring tools activated by your application 

Applications in Azure Spring Cloud are only impacted by the Log4j vulnerability if users activated New Relic and AppDynamics Java Agents. Applications monitored by Application Insights or Dynatrace Java Agents do not carry any potential risk associated with the Log4j vulnerability.  

We already patched-in updated New Relic and AppDynamics Java Agents. If you activated New Relic or AppDynamics Agents for your applications, we recommend that you restart your applications. Azure Spring Cloud will take steps to automatically protect customers and auto-restart any application with activated New Relic or AppDynamics Java Agents by Tuesday, December 21st, 2021 to ensure the latest fixes take effect.   

Microsoft Azure AD

While the industry is determining and mitigating overall exposure, attackers are probing all endpoints for vulnerability. Applying rigorous least privilege access policies to all resources in your environment is critical. If you use Azure Active Directory for single-sign on in your environment, we recommend you do the following with a special focus on applications you deploy or manage directly (SaaS apps, including those deployed by Microsoft, must be secured by their vendors). Note that log4j2 usage may be pre-auth for some of your applications, but these steps will help prevent post-authentication exploitation. Templates and examples for these policies are built in to facilitate deployment:

For key guidance on securing your identity deployment, see https://aka.ms/securitysteps.  

Minecraft: Java Edition

We’ve taken steps to keep our Minecraft customers safe and protected, which included rolling out a fix that blocks this issue for Minecraft Java Edition 1.18.1. Minecraft customers running their own servers are encouraged to deploy the latest Minecraft server update to protect their users. More information is available at Security Vulnerability in Minecraft: Java Edition.

Information for Security Operations and Hunters

Microsoft security teams have put together the following guidance and resources to help customers understand this vulnerability and to help detect and hunt for exploits:

We will further update this guidance as we continue to learn from our investigation.

The MSRC Team

Revision History:
12/16/2021 – Clarified customer guidance in the summary and linked to affected software in the security update guide.
12/15/2021 – Clarified guidance on Azure service. Added guidance for Java 7. Added guidance on Azure libraries for Java.
12/14/2021 – Added HDInsights guidance, updated guidance for CVE-2021-45046, and updated workaround guidance.
12/13/2021 – Added table to index guidance, added additional guidance for Azure WAF and Azure Spring Cloud.
12/11/2021 – Initial publish.



    • Related Articles

    • Apache Log4j Vulnerability Guidance

      Immediate Actions to Protect Against Log4j Exploitation • Discover all internet facing assets that allow data inputs and use Log4j Java library anywhere in the stack. • Discover all assets that use the Log4j library. • Update or isolate affected ...
    • Guidance for preventing, detecting, and hunting for CVE-2021-44228 Log4j 2 exploitation

      Guidance for preventing, detecting, and hunting for CVE-2021-44228 Log4j 2 exploitation Microsoft 365 Defender Threat Intelligence Team Microsoft Threat Intelligence Center (MSTIC) Updates: [12/16/2021] New Microsoft Sentinel solution and additional ...
    • Log4j Overview: Related Software

      This page contains an overview of any related software regarding the Log4j vulnerability. On this page NCSC-NL and partners will maintain a list of all known vulnerable and not vulnerable software. Furthermore, references to software will contain ...
    • Log4J Affected Apps/Vendors

      Lists of affected components and affected apps/vendors by CVE-2021-44228 (aka Log4shell or Log4j RCE) for security responders. We believe it is important to classify the vendors and products between: Internal risk - what you need to patch first to ...
    • Log4j Advisories, notices, patches, or updates

      Given the severity of the vulnerability and how easy it is to exploit it, CISA today released guidance for companies to set up defenses against Log4Shell attacks. The agency's recommendation is to "apply available patches immediately" and to ...