windows

Windows Event Forwarding for Network Defense

Dane Stuckey

11 Sep 2017 • 15 min read

This was a post I co-authored which originally published on the Palantir Blog. It is copied here for posterity.

Incident detection and response across thousands of hosts requires a deep understanding of actions and behavior across users, applications, and devices. While endpoint detection and protection tools can provide some lift out-of-the-box, deep insight and analysis of security-relevant events is crucial to detecting advanced threats. Over the past few years, Palantir has maintained an internal Windows Event Forwarding (WEF) pipeline for generating and centrally collecting logs of forensic and security value from Microsoft Windows hosts. Once these events are collected and indexed, alerting and detection strategies (ADS) can be constructed not only on high-fidelity security events (e.g. log deletion), but also for deviations from normalcy, such as unusual service account access, access to sensitive filesystem or registry locations, or installation of malware persistence.

The goal of this blog post is twofold: First, to share our learnings and step-by-step instructions with WEF configuration and management workflows, and second to introduce our recently open-sourced library of consolidated WEF configurations, subscriptions, and group policy objects: http://github.com/palantir/windows-event-forwarding. The GitHub project provides the necessary building blocks for organizations to rapidly evaluate and deploy WEF to a production environment, and centralize public efforts to improve WEF subscriptions and encourage adoption. While WEF has become more popular in recent years, there are few comprehensive deployment guides. As such, WEF is — in our opinion — still dramatically underrepresented in the community, and it is our hope that this project may encourage others to adopt it for incident detection and response purposes. As we work with customers across the globe to help secure their environments, we believe that our configuration represents a solid security standard that can be applied in organizations of any size and maturity to deliver immediate security outcomes for detection and response.

We acknowledge the efforts that Microsoft, IAD, and other contributors have made to this space — and wish to thank them for providing many of the subscriptions, ideas, and techniques that are covered in this post.

WEF Basics

Windows Event Forwarding (WEF) is a powerful log forwarding solution integrated within modern versions of Microsoft Windows and has excellent documentation on its Microsoft Docs page. In summary:

Windows Event Forwarding allows for event logs to be sent, either via a push or pull mechanism, to one or more centralized Windows Event Collector (WEC) servers.
WEF is agent-free, and relies on native components integrated into the operating system. WEF is supported for both workstation and server builds of Windows.
WEF supports mutual authentication and encryption through Kerberos (in a domain), or can be extended through the usage of TLS (additional authentication or for non-domain-joined machines).
WEF has a rich XML-based language to control which event IDs are submitted, suppress noisy events, batch events together, and configure submission frequency. Subscription XML supports a subset of XPath, which simplifies the process of writing expressions to select the events you’re interested in.

Mechanics

While WEF can be configured as either a source or a collector-based model, we will be focusing on a source-initiated model, where each device forwards their logs to a centralized collector. This allows mobile devices (e.g., laptops) to connect back to the network and forward logs on their own schedule.

A WEF connection requires a few basic components:

Group Policy Objects (GPOs) to control security auditing and event logging.
One or more servers with a configured Windows Event Log Collector service (often referred to as the “WEF Server” or “WEF Collector”).
Functional Kerberos for all endpoints (domain) or a valid TLS certificate (non-domain) for the Event Log Collector servers.
Windows Remote Management (WinRM) enabled on all workstations and servers that will forward events.
Firewall rules permitting WinRM connectivity between the devices.
GPOs to specify the URL of the WEF subscription manager(s).
One or more event log subscriptions. A subscription is a collection of events based on Event IDs or other criteria to tell the endpoints which event logs to forward.

The following actions occur upon first receiving appropriate GPOs on a workstation:

The workstation configures security auditing and starts writing to the local event log.
The workstation connects to the subscription manager(s) using WinRM, authenticated either via Kerberos or TLS. In both cases, transport-layer encryption is applied.
The workstation registers itself in the registry of the Event Log Collector, and downloads a list of all relevant WEF Subscriptions.
The workstation periodically sends events to the Event Log Collector(s) as defined in the subscription files. Additionally, the workstation connects on a periodic heartbeat.

As new devices are added to the domain and receive the appropriate security logging and WEF subscription GPOs, they will automatically begin forwarding events, reducing the administrative burden of ensuring log coverage and quality.

The following is a visual depiction of a deployment scenario:

A WEF server (e.g., WEST-WEF-01) is deployed for each Active Directory site (e.g., US-WEST) and runs the Event Log Collector service. A group policy object instructs all clients in the site to communicate with the WEF server, which provides a copy of the subscriptions that the workstation should use. The subscriptions are managed in Git, and are deployed via a continuous integration (CI) pipeline. Lastly, the forwarded events are written to custom Event Channels, and are then tagged and indexed into a Security Information and Event Management (SIEM) system.

Limitations

While WEF provides immense value, it is not without limitations. These limitations should be considered when evaluating a WEF deployment for your organization.

Load balancing is difficult. When using Kerberos, it is difficult — if not impossible — to effectively load balance the forwarded events between multiple nodes. While events can be forwarded to multiple WEF servers, traditional methods of load balancing the traffic do not work as the Service Principle Name (SPN) cannot be duplicated.
Active diagnosis and troubleshooting is limited. When WEF fails, it is often difficult to diagnose why. There are limited tools for troubleshooting client issues, or validating the health of a given node. We have dedicated a section to addressing these issues later in this article.
WEF has a steep learning curve. Out of the box, WEF requires multiple components of server and client infrastructure, GPOs, subscriptions, and functional Kerberos. For many organizations, it may be daunting to ensure that all of the prerequisite components are functional, making a WEF deployment difficult.

WEF Subscriptions and Channels

Subscriptions are the central WEF configuration mechanism and determine which events should be forwarded, how they should be stored, and at what cadence and batch size they are sent.

Defining WEF Subscriptions

The meat of a WEF subscription ruleset is defined by a collection of XML documents. The below picture annotates a representative subscription and its core components:

The XML schema is explained in the Microsoft MSDN documentation. The central configuration points are:

MaxItems. The maximum number of items that should be batched before being sent. It is recommended to use a lower MaxItems value for security-critical events such as log deletion.
MaxLatencyTime. The maximum time (in milliseconds) that can elapse before event logs are forwarded, provided the MaxItems value is not exceeded. This should be treated as the maximum latency you are willing to tolerate waiting for any given event.
Query. The query is the heart of the subscription and defines which event IDs should be sent and which should be suppressed. XPath queries can be used to filtering even further, see below.
ContentFormat. Determines whether the entire text of the event entry is sent (larger, but more verbose) or just the binary data. For human readability and parsing, we use RenderedText for all subscriptions.
LogFile. The log destination on the remote Event Log Collector. While Windows Event Forwarding will use the native ‘Forwarded Logs’ event channel, we will rely on custom log channels, see below.

As the subscriptions are XML files, they can be source-controlled in a Git repository. When combined with a continuous integration (CI) pipeline, WEF subscription modification, revision, collaboration, and deployment become reliable and repeatable. An example workflow could allow the InfoSec team to directly modify the WEF subscriptions to collect security-critical event logs, while allowing the Support and Operations team to file a pull request with a subscription to collect crash logs for their own usage.

Filtering Events with XPath

In order to maintain a high-fidelity event database or SIEM, it is important to filter down events to the subset relevant to an organization’s infrastructure. WEF supports XPath as a query language to implement such filters. The following is only a short primer on WEF XPath filtering, but hopefully will get you started with producing and testing your own custom filtering to make the most out of WEF.

XPath is essentially a method for selecting specific XML nodes from an XML document, and WEF supports a subset of XPath 1.0. The primary restriction is that only XML elements that represent events can be selected by an event selector. All valid selectors start with Event or *.

For simple queries that select events from a single source, using an XPath expression is fine. If the XPath expression is a compound expression that contains more than 20 expressions or you are querying for events from multiple sources, then you must use a structured XML query, see Consuming Events. For details on the elements of a structured XML query, see Query Schema. There are 4 main selectors:

QueryList. The root node of the query
Query (QueryListType). Defines the set of selectors and suppressors that are used to include and exclude events from the result set
Select. An XPath query that identifies what to include in the result set
Suppress. An XPath query that identifies what to exclude in the result set

The following shows a structured XML query that specifies a set of selectors and suppressors. This filter will grab all of the specified event IDs except those matching the SECURITY_LOCAL_SYSTEM_RID.

<Query><![CDATA[
    <QueryList>
      <Query Id="0" Path="Security">
        <!-- 4624: An account was successfully logged on. -->
        <!-- 4625: An account failed to log on. -->
        <!-- 4634: An account was successfully logged off. -->
        <!-- 4672: Special privileges assigned to a new logon, administrative logins -sa, -ada, etc. -->
        <!-- 4775: An account could not be mapped for logon. -->
        <!-- 4777: The domain controller failed to validate the credentials for an account. -->
        <!-- Suppress SECURITY_LOCAL_SYSTEM_RID A special account used by the OS, noisy -->
        <Select Path="Security">*[System[(EventID=4624 or EventID=4625 or EventID=4634 or EventID=4672 or EventID=4775 or EventID=4777)]]</Select>
        <Suppress Path="Security">*[EventData[Data[1]="S-1-5-18"]]</Suppress>
      </Query>
    </QueryList>]]></Query>

You can use the Event Schema Elements as a starting point for your selector and suppressor queries. A few quick tips:

You can always use the Event Viewer UI to build a filter, then switch to the XML view and see how it’s represented in XPath.
If a selector and suppressor select the same event, the suppressor takes precedence.
WEF supports standard XML concepts such as <!— comments --> or CDATA literal text data.

Custom Windows Event Channels

As described in the blog post Creating Custom Windows Event Forwarding Logs, WEF can be extended with additional custom event channels. Extending the number of event channels available provides a few primary benefits:

Each event channel can have an independent maximum size and rotation strategy.
Each event channel can be used as a unique identifier for tagging data for ingestion into a SIEM.
Event channels may be placed on different disks or storage devices for improving disk I/O.

See below for instructions for defining and deploying custom event channels.

Deploying WEF

The following is an enumeration of installation and configuration steps for a typical WEF deployment.

Configure Auditing

In addition to the default Security, System, and Application logs generated by Windows, there are additional auditing settings available that are not enabled by default. These settings enable Windows to generate events that can be invaluable during the course of an investigation. For example, you can enable detailed process, registry, and file auditing among many others.

The most straightforward way to configure these settings is by creating Group Policy Objects (GPOs) for them. You might consider creating separate policies for your Domain Controllers, servers, and workstations depending on your log capacity and risk profile. Auditing recommendations can be found at Microsoft — Security Auditing.

It is highly recommended that you also account for PowerShell logging, and enable Script Block, Module, and Transcript logging.

Deploy GPOs

The easiest way to manage WEF at scale is to create a series of GPOs that will configure subcomponents of the pipeline. For instance, with a multi-site network, you may wish for the following:

Workstation Auditing GPO: Configures all of the Workstation-appropriate security and PowerShell logging settings, adjusts event log size and retention strategy, adjusts firewall rules, enables WinRM, etc.
Member Server Auditing GPO: Configures all of the Server-appropriate security and PowerShell logging settings, adjusts event log size and retention strategy, adjusts firewall rules, enables WinRM, etc.
Domain Controller Auditing GPO: Configures all of the DC-appropriate security and PowerShell logging settings, adjusts event log size and retention strategy, adjusts firewall rules, enables WinRM, etc.
Windows Event Collector GPO: Configures the WEC servers to have the WEC service automatically start, enables WinRM, adjust firewall rules, etc.
US-WEST WEF Management GPO: Configures the target subscription manager setting to point to your West Coast WEC server.
US-EAST WEF Management GPO: Configures the target subscription manager setting to point to your East Coast WEC server.

Examples of how to configure the GPOs can be found on the “Use Windows Event Forwarding to help with intrusion detection” post by Microsoft, or can be viewed in our GitHub Repository.

Define and Deploy Subscriptions

After configuring auditing settings across your fleet, the next step is to determine which of those logs you would like to collect and centralize via the WEF infrastructure. Palantir’s WEF library contains a curated series of subscriptions for you to adopt or modify to suit your needs, see below for a more detailed description.

Subscriptions can be defined and deployed from XML definitions or can be manually created in the Event Viewer GUI. To deploy descriptions from XML files, place them all in a single directory and navigate to that directory using PowerShell. The following script can be used to create and enable the Subscriptions that we provide in this repo.

NOTE: This script is specifically designed to work with the Subscription filenames we provide in our WEF library. You may have to modify it if your XML filenames don’t match your actual Subscription names.Write-Output "Starting the Windows Event Collector Service"
Start-Service wecsvc
Set-Service wecsvc -StartupType AutomaticWrite-Output "Creating custom event subscriptions"
cd c:\folder-containing-xml-subscriptions\
cmd /c "for /r %i in (*.xml) do wecutil cs %i"Write-Output "Enabling custom event subscriptions"
cmd /c "for /r %i in (*.xml) do wecutil ss %~ni /e:true"

Configure Collectors

After you’ve defined the events you would like to collect via Subscriptions, it’s time to configure one or more servers to act as event collectors (also commonly referred to as Subscription Managers). This is also generally accomplished via a GPO, as described in the “Deploy GPOs” section above.

If configuring WEC servers by hand, it is important to start the “Windows Event Collector” service and to configure it to start at boot. You will additionally need to enable WinRM and allow inbound connectivity on TCP/5985 (Kerberos) or TCP/5986 (HTTPS).

Deploy Channels

Custom event channels are not a requirement in a WEF setup, but they do provide additional benefits outlined above in the “Windows Event Channels” section. Generally, this process involves creating a manifest file and building a resulting DLL from it, see Creating Custom Windows Event Forwarding Logs for details. Palantir’s WEF library provides a manifest and a pre-compiled DLL. The following steps to deploy the DLL have to be executed on each Subscription Manager:

Unload the existing manifest via wevtutil um C:\windows\system32\CustomEventChannels.man
Copy your newly created CustomEventChannels.man and CustomEventChannels.dll files into c:\windows\system32.
Import the new manifest via wevtutil im C:\windows\system32\CustomEventChannels.man. This creates the defined channels and log files on the WEC servers.
We also recommend increasing the size of each channel to 4GB:

$xml = wevtutil el | select-string -pattern “WEC”
foreach ($subscription in $xml) { 
  wevtutil sl $subscription /ms:4194304
}

Deploy PowerShell Transcript Logging

Once you have functional WEC servers, you should consider turning them into dual-purpose PowerShell transcript servers. As PowerShell transcription logging creates files over an SMB share, it will not use the native WEF pipeline.

We can take advantage of the existing infrastructure and simplify aggregation and collection through an SMB share, a security script, and additional GPOs. An overview of how to do so can be found in the blog post Microsoft ❤s the Blue Team.

Extend WEF

Once a WEF pipeline has been created, it can be leveraged for reporting custom events. With strong assurances that any given event will eventually be indexed in a SIEM, both security and productivity tools can be deployed and write to a unified log facility. Our WEF library contains two such extensions, Autoruns-To-WineventLog and EMETDiag, see below.

Shoot the Trouble

It can be frustrating to stand up logging infrastructure, only to discover that it’s not sending any of the logs you expected it to. Although sometimes unintuitive, there are a few key tools you can use to gain deeper insight into where a breakdown exists.

Start by reviewing the necessary components described in the “WEF Overview” section above. Ensure all of the required components exist in your environment and are configured correctly. If you’re in the testing phase, consider setting the Subscription Manager refresh interval to a small value such as 60 seconds. This will ensure that logs are offloaded from your clients in a timely basis and reduce the amount of time you need to wait for logs to arrive. If you ever need to force push logs to the Subscription manager, running gpupdate /force from the client will also force a check-in.

Additionally, information about errors or misconfigurations can be found in the Microsoft-Windows-Eventlog-ForwardingPlugin Event Log Channel on each of your clients. This event log is helpful for determining when ACLs are misconfigured on event logs, Subscriptions are somehow invalid, or when logging channels are missing from a host.

On a subscription manager, the Event Viewer tool can help you gain insight into the status of each subscription by clicking on the “Subscriptions” option in the left hand column, selecting a Subscription, and clicking “Runtime status”.

Palantir’s WEF Library

The following sections describe the configuration and tools provided in Palantir WEF library on GitHub: https://github.com/palantir/windows-event-forwarding/. It is important to note that the subscriptions and configurations provided will not solve all security use cases and may not run in all environments. It is highly recommended that the configurations are tested and tweaked for each organization. While we have done our best to find a good trade-off between signal and noise, we greatly welcome and encourage community participation in this project by filing a pull request or opening a GitHub issue.

Subscriptions

In addition to our own definitions, our subscription library contains subscriptions from IADGOV’s excellent Event Forwarding Guidance repository as well as Microsoft’s documentation. The exhaustive list of WEF subscriptions can be found in the GitHub project; they cover the following scenarios:

Account management, lockout, kerberos, and authentication events
Process execution and termination events
Powershell module, script block, and operational events
Advanced security events (e.g. autoruns, code integrity, EMET, etc.)
Operating system events (e.g. package installation, driver loading, DNS events, etc.)
Anti-forensic events (e.g. log deletion)

Note that our WEF subscriptions assume you will be using the provided event channels, see below. If you do not choose to do so, you will need to change the target log file in each subscription.

Event Channels

The Palantir WEF library contains a collection of custom Windows Event Channels, please refer to the README for a complete list. The channels are provided as a standard manifest file as well as a precompiled DLL. If you’re like us and don’t trust random DLLs, feel free to use our manifest file and build your own DLL.

The Event Channel manifest provided in this project consists of 16 individual providers, each with 7 channels. Channels follow a standard naming scheme of WEC[#], where the number is related to the provider.

Autoruns-To-WineventLog

Sysinternal’s Autoruns tool is used to search across different components of the Windows operating system to enumerate areas that are commonly used for persistence by malware and attack tools. While it’s commonly used on an ad hoc basis by forensic investigators, we wanted a steady stream of this information from our entire fleet. Having the ability to search through freshly reported persistence indicators is an invaluable dataset when hunting for badness.

To accomplish this, we wrote a PowerShell script that we call Autoruns-to-Wineventlog. This script downloads the latest version of Autoruns onto the host, sets up a scheduled task to kick off Autoruns on a daily basis, and then parses the resulting CSV data into the “Autoruns” Windows Event Log channel. The resulting data is then ingested into our SIEM via the WEF pipeline.

The code for Autoruns-to-Wineventlog serves as a representative sample of how easy it is to plug custom security tooling into a WEF pipeline.

EMETDiag

While officially sunset and slated to be replaced with Defender Exploit Guard, the Enhanced Mitigation Experience Toolset (EMET) provides an invaluable defense-in-depth layer against memory corruption and other exploit techniques. While configuring and using EMET is out of the scope of this article, there are instances where an application may not play well with the protection mechanisms offered, causing instability and crashes.

When troubleshooting EMET crashes internally, we rely on a custom PowerShell script called EMETDiag that can be remotely pushed via our systems management suite, or deployed by hand. Once deployed, it automatically queries the EMET configuration, pulls back the most recent EMET events, application crashes, and other related data, and then summarizes and writes the data to a custom event that is indexed by WEF.

Leveraging WEF allows for near-instant generation, forwarding, and indexing of data that can be used for troubleshooting purposes. Once indexed in the SIEM, it is available for immediate use by the Desktop Engineering team for analysis.

The code for EMETDiag is slated for release in the GitHub project in the near future.