Improving Long Tail Analysis Using Neighborhood Keeper

This is the third blog in our series covering the foundational elements that make up Neighborhood Keeper, how the information-sharing program fits into the broader context of Cyber Threat Intelligence (CTI), and how to operationalize this resource to augment network defense. In this instalment, we focus on the concept of long tail analysis and how to use this methodology to identify anomalous activity within participant environments for additional investigation.

Identifying Anomalies With Long Tail Analysis

At its heart, long tail analysis seeks to identify potentially significant, but ultimately anomalous, activity. Much of the activity on a given network will fall within a predictable range of common occurrences, which can often be overwhelming in terms of volume for analysis, and which will be comprised of predominately benign user interaction.

The long tail in this context are those low occurrence events which fall significantly outside the core of routine user behavior.  These events aren’t always malicious and can often be the result of misconfigurations, network maintenance, or efforts to troubleshoot related issues; however, they are a place to begin in efforts to sort signal from noise in network communications and events.

Baseline Understanding

One of the most important starting points when considering long tail analysis is a baseline understanding of what normal looks like for a given network. Many of these events will be things that happen rarely, but perhaps predictably at regular intervals or coinciding with specific tasks or operations.

Understanding the “what” and “why” of these periodic but low incidence events in a network environment over time makes it much easier to sift through the variety of activity in the long tail of the distribution to quickly zero in on things that are truly anomalous and therefore more likely to correlate to malicious activity.

Developing an effective baseline requires purposeful observation of a network over time and a persistent curiosity surrounding uncommon occurrences. In some ways, these occurrences can be considered part of the idiosyncrasies which make any given network environment unique. Once this baseline understanding is established, the data in Neighborhood Keeper can help augment understanding of activity identified through long tail analysis, roughly following this line of questioning:

• What low incidence events are observed within a specific network environment?
• Are these low incidence events also present in Neighborhood Keeper’s Least Common Analytics over a given time frame (and thus representing activity which isn’t just anomalous to a single network, but rather to the Neighborhood Keeper Community as a whole)?
• Can this activity be immediately explained through a baseline understanding of the specific network environment in which it was observed?
• If not, this activity warrants further investigation.

Visualization in the Neighborhood Keeper Portal

Contextual data on long tail events can be found within the Neighborhood Keeper Portal under “Least Common Threat Analytic.”

As with the other views in the portal, this information can be viewed for a defined timeframe, which should match the timeframe of investigation for an individual network. Alternatively, this information can be viewed in the “Trends” tab to contextualize anomalous activity over a longer time horizon. Lastly, it is possible to pivot from a defined timeframe to a trends view by clicking the “Compare” button available in each time-bound view.

This is yet another example of Neighborhood Keeper’s potential use cases when integrated as a tool in a larger CTI and threat hunting program. Neighborhood Keeper data is uniquely positioned to provide context surrounding both common and uncommon events in specific networks, particularly how these events relate to those observed across the broader Neighborhood Keeper community.

The next blog in this series will focus on Strategic Analysis use cases for Neighborhood Keeper data, and ways to leverage Neighborhood Keeper’s insights to craft content and answer questions for an executive audience. We’re excited to provide Neighborhood Keeper participants additional approaches to maximize the benefits from participation in the program. If you’d like to catch up on the first two posts in this series, we introduced using Neighborhood Keeper in the Broader Context of operationalizing threat intelligence, followed by how to use it for Trend Analysis.