Full Report
Microsoft Defender for Endpoint Internal 0x06 — Custom CollectionMicrosoft recently published a new feature for Defender for Endpoint (MDE) called Custom Collection. At FalconForce, we've had the great privilege over the last year to work with the development team as a design partner, testing this feature and providing feedback.While MDE has been a great product with a valuable data set for threat hunting, detection engineering, and alert analysis, one thing has limited teams' ability to gain deep visibility – up until now: the capping of events. We have discussed this topic several times in previous blog posts, so feel free to read up on it at the provided links.Sysmon vs Microsoft Defender for Endpoint, MDE Internals 0x01Microsoft Defender for Endpoint Internals 0x02 — Audit Settings and TelemetryMicrosoft Defender for Endpoint Internals 0x03 — MDE telemetry unreliability and log augmentationMicrosoft Defender for Endpoint Internals 0x04 — Timeline telemetryMicrosoft Defender for Endpoint Internals 0x05 — Telemetry for sensitive actionsWhat is itCustom Collection, in short, is what the name implies: it's an MDE feature that lets users define a custom set of rules to be collected alongside regular MDE telemetry, but without a hard cap on the number of events we're collecting. That last part isn't entirely accurate; there *is* a cap of 25000 events per day per configured rule, per device, primarily to protect you from very high costs. This cap should not be an issue; you shouldn't have configurations that generate that many events per day.These custom collection rules can be applied to the same data sources MDE collects using the same schema. Currently, the following tables are supported:DeviceProcessEventsDeviceNetworkEventsDeviceFileEventsDeviceImageloadEventsDeviceScriptEvents (AMSI events not available in Advanced Hunting)The collected data is not stored in the MDE tables, which are included with the license at a fixed fee, but is instead sent directly to Sentinel tables. This means that this additional data will incur an extra cost, which is understandable, but important to be aware of. The tables that will be stored in Sentinel will follow the same schema with the addition of some fields: RuleName, RuleLastModificationTime and Type. These make it easy to understand, for example, which rule may require tuning.The tables in Sentinel will also have a slightly different name to avoid conflicts with the native MDE data you might be ingesting there. Essentially, the ”custom” word gets added to all of them, resulting in the following names:DeviceCustomProcessEventsDeviceCustomNetworkEventsDeviceCustomFileEventsDeviceCustomImageloadEventsDeviceCustomScriptEventsWhy should you careThere are many reasons why Custom Collection is a great feature to us. Some of the biggest are:Uncapped telemetry for the events you care about, without needing additional agent(s).Central management of these rules, without additional overhead.Flexibility per device group.As we discussed earlier, there are two main limitations to the default MDE telemetry: event capping and event filtering. This leads to an incomplete picture of what might be important to you. We see many organizations augment the MDE telemetry with additional solutions, like WDAC in audit mode or, more commonly, Sysmon. While these are perfectly fine options, they also introduce significantly more administrative overhead, as well as two additional agents that need to be installed and maintained: Sysmon and a log collection agent. And we are not even discussing the data structure, which is different, making querying the data less convenient.The new Custom Collection feature might be the perfect replacement, assuming you have MDE and the current table set covers your primary needs.Custom Collection essentially allows you to create a set of rules for data collection, similar to Sysmon, but with more fine-grained control over what to include and exclude. Additionally, these rules can be assigned to tagged device groups, allowing you to get specific events only from those groups. You can also assign rules to the whole organization.These custom collection rules are a separate part of the MDE configuration and will be updated by the agent more frequently than the default configuration. In my experience, newly added custom collection rules or tags become active within 20 minutes to an hour, as long as the targeted machines are online, which allows for a fair level of flexibility.You can, for example, imagine creating a “verbose” set of logging for machines impacted by a high alert and automatically assign them to the device group that will receive those additional custom collection rules for a certain period of time. Don't forget to also remove it, as these additional events could become costly.Other examples include monitoring your web servers for potential web shells being added to their folder structure. Since MDE doesn't monitor the creation of all file types by default, and certainly not in all locations, this feature allows you to create specific monitoring for those files and folders. These are just a couple of small examples; I'm confident you have many others in mind.How to use itTo start using the Custom Collection feature, you will need a Defender P2 license, a connected Sentinel workspace, and the Defender for Endpoint client version 10.8805 or higher — which will automatically update in most organizations.The administration of custom collection rules is located in the Defender XDR portal under Settings > Endpoints > Custom Collection. Here, you'll find a overview of all your implemented custom collection rules, as well as the option to create a new rule using a wizard.Example custom collection rule overview.This whole process is well-documented on the Microsoft Learn page, so I won’t repeat that here. When I was getting started on implementing the Custom Collection feature and testing it thoroughly, I quickly realized I needed a more structured and automatable approach. Additionally, the implementation of the rule builder did not feel very pleasant to me personally, especially when building more complex rules. Moreover, there is no quick way to analyze all your rules, besides opening them one by one and going through the wizard for each one.Automation — TelemetryCollectionManagerSadly, at this moment, there isn’t an API endpoint that allows us to use a service principal, like a managed identity or app registration, to manage the custom collection configuration. However, by spending some time in the developer tools, I learned that at least the Defender API has a set of endpoints that you can access with a bearer token to manage these rules.At FalconForce, we've built a toolkit for testing and deploying our detection logic, which takes a YAML input. Since this works really well for us and our clients, I also wanted to create a similar workflow for creating and maintaining these custom collection rules.We’ve designed a YAML schema that will allow the creation of any rule type available in the Defender portal, with several quality-of-life improvements. For example, it allows providing multiple values in a single expression, compared to the multi-line approach in the Defender portal.Example YAML custom detection rule with multiple values, where the MDE UI only allows 1 per line.The same detection in the Defender portal, where each statement is an individual line item.The schema supports multiple values for easier editing, and the repository includes an examples folder with more complex nested rules and combinations. If you use a VSCode-like editor and have installed a YAML language server, you can also benefit from autocomplete options, as it can use the included rule schema.Schema-based Intellisense example in VSCode.To enable the deployment and maintenance of these YAML files, we’ve built a Go application that can:Validate all YAML files to work with the Defender model.Create and update custom collection rules using these YAML files in the Defender portal.List or export all custom collection rules in the Defender portal.Convert exported rules to YAML files, in case you’ve built several in the portal already and want to transition to maintaining them in YAML.Delete individual or all custom collection rules in the Defender portalTelemetry CollectionManager command-line optionsSadly, at this moment, the application will need a user-based token to access the APIs that expose this functionality. The most convenient way to facilitate this, is to have the Azure CLI logged in with a user that has the proper MDE permissions to make these changes. If this is available, the application will use this session to request the required access token seamlessly. If this isn’t possible in your environment, you’ll have to provide the application with a valid access token each time, which you can retrieve via the browser developer tools.Some example command outputs from TelemetryCollectionManager.By default, the command output is very minimal by design, as the tool is intended for future pipeline implementation once service principal authentication is supported.The application is available in our GitHub repository.When you decide to use these rules, it's highly recommended to store them in a Git repository, which allows for proper reviews and versioning, ensuring your Git repository is your single source of truth.With this release, we've included several converted Sysmon rules, along with some other rules to kickstart this repository. Our intention for this project is to make it a community effort, where people can contribute rules and help other defenders improve their visibility. We're welcoming pull requests to this project!We'll definitely add more rules over time. This initial release is primarily intended to get people started with a rules-as-code approach.All rules are added in a non-enabled state by design, as it's up to the user to decide whether they should log into their environment. We want this repository to be a starting point, like the Sysmon-modular project, where each individual can determine what's appropriate for their organization.Happy logging!Microsoft Defender for Endpoint Internal 0x06 — Custom Collection was originally published in FalconForce on Medium, where people are continuing the conversation by highlighting and responding to this story.
Analysis Summary
# Best Practices: Extending MDE Visibility with Custom Collection
## Overview
These practices focus on leveraging the Microsoft Defender for Endpoint (MDE) Custom Collection feature to overcome inherent limitations in default telemetry (event capping and filtering). Custom Collection allows defining targeted, high-fidelity event collection rules that bypass standard caps and stream data directly to Azure Sentinel for enhanced threat hunting and detection engineering, potentially replacing the need for auxiliary agents like Sysmon for specific data needs.
## Key Recommendations
### Immediate Actions
1. **Determine Custom Collection Prerequisites:** Ensure all target devices are running MDE client version 10.8805 or higher and that you have a Microsoft Defender P2 license and an active Azure Sentinel workspace connected to MDE.
2. **Map Current Gaps to Data Sources:** Identify security monitoring blind spots (e.g., specific file creations, detailed process activity) currently missed due to default MDE event caps and map these requirements to the available Custom Collection schemas: `DeviceProcessEvents`, `DeviceNetworkEvents`, `DeviceFileEvents`, `DeviceImageloadEvents`, and `DeviceScriptEvents`.
3. **Enable Initial Test Rule:** Configure a single, high-value custom collection rule (e.g., verbose logging for a pilot group of servers) and deploy it to a small device group to validate data flow to the corresponding Sentinel tables (e.g., `DeviceCustomProcessEvents`).
### Short-term Improvements (1-3 months)
1. **Migrate Critical Sysmon Use Cases:** Convert existing high-value Sysmon monitoring rules (if applicable) into MDE Custom Collection rules to reduce administrative overhead associated with maintaining a separate agent, utilizing the custom schema flexibility.
2. **Implement Targeted Auditing with Device Groups:** Use custom collection rules to define "verbose" logging profiles for devices undergoing active incident investigation or those belonging to high-risk groups (e.g., Web Servers). Assign these rules dynamically to the relevant device groups.
3. **Establish Cost Monitoring:** Since custom data streams to Sentinel and incurs cost, monitor the daily event volume against the 25,000 events/day/rule/device cap. Tune high-volume rules immediately if they approach this threshold, as this indicates potential inefficiency or unexpected behavior.
### Long-term Strategy (3+ months)
1. **Adopt a Rules-as-Code Workflow:** Implement a structured process (like using the recommended YAML tooling or similar approach) to define, manage, and version all custom collection rules in a Git repository. This creates a documented Single Source of Truth (SSOT) for advanced telemetry configuration.
2. **Standardize Telemetry Augmentation:** Formally deprecate or standardize alternative logging solutions (like Sysmon) by confirming that Custom Collection provides sufficient, more conveniently structured data for your most critical detection engineering needs.
3. **Integrate Rule Management into CI/CD:** Once API authentication matures (or using token management), integrate the deployment of validated Custom Collection YAML files into automated security deployment pipelines to ensure consistent, version-controlled governance across the environment.
## Implementation Guidance
### For Small Organizations
- **Focus on Essential Gaps:** Prioritize filling the most critical visibility gaps, such as monitoring sensitive file share access (`DeviceFileEvents`) or specific command-line arguments not logged by default.
- **Use the Portal Wizard Initially:** Start by manually configuring rules via the Defender XDR portal (`Settings > Endpoints > Custom Collection`) to gain proficiency before introducing automation tools.
### For Medium Organizations
- **Pilot Automation:** Begin testing automation tools (like the referenced TelemetryCollectionManager) for creating and exporting rules to establish a rules-as-code foundation early.
- **Group-Based Control:** Utilize device tagging and grouping effectively to ensure verbose logging is only applied temporarily or permanently to specific tiers of assets (e.g., Domain Controllers, high-value endpoints).
### For Large Enterprises
- **Mandate Rules-as-Code:** Enforce the use of a centralized, version-controlled system (Git) for all custom rule definitions to maintain compliance, auditability, and scalability.
- **Leverage Sentinel Schema Benefits:** Design querying practices in Sentinel to seamlessly join native MDE data with custom data using the `RuleName` and custom table structure for comprehensive analysis.
- **Implement Token Management for Automation:** Establish secure processes for obtaining or leveraging existing user tokens required by current management tools to enable automated rule deployment and auditing lifecycle management.
## Configuration Examples
**Schema Supported Data Sources (Stored in Sentinel Tables):**
* `DeviceProcessEvents` (Stored in `DeviceCustomProcessEvents`)
* `DeviceNetworkEvents` (Stored in `DeviceCustomNetworkEvents`)
* `DeviceFileEvents` (Stored in `DeviceCustomFileEvents`)
* `DeviceImageloadEvents` (Stored in `DeviceCustomImageloadEvents`)
* `DeviceScriptEvents` (Stored in `DeviceCustomScriptEvents`)
**Important Field Addition in Sentinel:**
* `RuleName`
* `RuleLastModificationTime`
* `Type`
**Monitoring Specific Server Types:**
Create rules targeting specific paths or processes on web servers to monitor for web shell creation, leveraging `DeviceFileEvents` where default MDE monitoring might be too limited or lacks sufficient detail. Target specific directories or file extensions not typically covered by default MDE configuration.
## Compliance Alignment
* **NIST SP 800-53 (AC, AU, RA):** Enhances Audit (AU) logging capabilities beyond baseline, improves Risk Assessment (RA) data availability, and supports Access Control (AC) monitoring needs by providing detailed event context.
* **ISO/IEC 27001 (A.12.4):** Supports the requirement for logging and monitoring by providing high-fidelity, centrally manageable event data streams essential for detecting policy violations and security incidents.
* **CIS Benchmarks:** Provides necessary granular detail needed to enforce and verify configuration baselines across specific device groups, often surpassing the capabilities of standard agent logging.
## Common Pitfalls to Avoid
1. **Ignoring Sentinel Costs:** Deploying overly broad custom rules without volume testing can lead to significant, unexpected Azure consumption costs due to the direct stream into Sentinel tables.
2. **Manual Rule Management Drift:** Relying solely on the Portal Wizard without version control will result in unreviewable, undocumented configuration changes across the environment.
3. **Assuming Zero Cap:** Understand that a cap of 25,000 events per rule per day per device *does* exist; configurations generating more than this will truncate data, necessitating rule tuning.
4. **Using Only Native Tables:** Failing to query the custom prefixed tables (e.g., `DeviceCustomProcessEvents`) in Sentinel will lead to missing the data collected via this feature.
## Resources
* **Prerequisites Check:** Confirm MDE Client Version (10.8805 or higher).
* **Configuration Portal:** Defender XDR portal under `Settings > Endpoints > Custom Collection`.
* **Advanced Management Tools:** Review FalconForce's open-source tools (e.g., TelemetryCollectionManager) for **Rules-as-Code** implementation using YAML schemas, enabling automation and versioning.
* **Microsoft Learn Documentation:** Consult official Microsoft Learn pages for the precise technical specifications on rule creation wizards and schema definitions.