ThreatPipes 101

ThreatPipes sounds cool, but how to I actually use it?

Use-cases

ThreatPipes can be used offensively, i.e. as part of a black-box penetration test to gather information about the target or defensively to identify what information your organisation is freely providing for attackers to use against you.

In either case, ThreatPipes works the same way for both attacker and defender.

Read more about how others use ThreatPipes here.

How ThreatPipes works

All data collection by scans in ThreatPipes modularised.

Each module typically queries a third-party service for intelligence on a target (e.g. haveibeenpwned).

When a module discovers a piece of data, that data is transmitted to all other modules that are watching for that data type for processing. Those modules will then act on that piece of data to identify new data, and in turn generate new events for other modules which may be interested, and so on.

Visually it looks like this:

ThreatPipes network graph

The red node is the seed target and black nodes are intelligence generated from the target by modules.

By building up a complete threat graph, you discover both directly related intelligence on the target (original threat) but also any affiliates to that potential threat.

A working example

Sometimes it's easier to explain this concept with a worked example, so here's one for you.

The DNS Resolver module identifies an IP address associated with a target, notifying all interested modules watching for IP related events.

One of those interested modules is the RIPE module, that takes the IP address and identifies the netblock it is a part of, the BGP ASN and so on.

Meanwhile, as each event is generated to a module, it is also recorded in the ThreatPipes database for reporting and viewing.

How modules are constructed

Use-case

Each module is classified into one or more of the following use cases:

  • footprint: Understand what information this target exposes to the Internet.

  • investigate: Best for when you suspect the target to be malicious but need more information.

  • passive: When you don't want the target to even suspect they are being investigated.

Modules might have multiple classifications.

For example, the abuse.ch module checks if a host/domain, IP or netblock is malicious according to abuse.ch is classified into two use cases; "investigate" and "passive".

Categories

All modules are classified into categories that describe the type of service they relate to (e.g. Reputation Systems, DNS, Public Registries).

For example, the Whois module that searches Whois servers for domain names and netblocks is classified into the category; "Public Registries".

Data elements

ThreatPipes modules are driven by events. Each module listen to and consumes data elements to generate new events (intelligence) about these data elements, which in turn other modules listen to and consume, and so on.

For example, the Name Extractor module listens and consumers the data element types TARGET_WEB_CONTENT and EMAILADDR events for identifying human names (HUMAN_NAME data elements). In turn, the Accounts module, and many others, listen for HUMAN_NAME data elements to gather more intelligence.

Data element types

The data elements are classified into one of the following types:

  1. entities: e.g. IP addresses, hostnames, sub-domains, domains

  2. sub-entities: e.g. port numbers, URLs, software installed

  3. descriptors: (of entities) e.g. malicious, physical location information

  4. data: (of entities) e.g. web page content, port banners, raw DNS records

To help explain the differences between data elements, here an example; the data element "Email Address" is an entity, the data element "Hacked Email Address" is a descriptor, and the "Search Engine's Web Content" that might have uncovered the email is "data".

Default Modules

ThreatPipes ships with over 150 default modules . It is also possible to install custom modules developed by third parties to work with ThreatPipes.

Extending ThreatPipes Modules

Read this document for more information on how to add custom modules or to get started developing new modules for ThreatPipes.