Developing New Modules

Extend the power of ThreatPipes by building your own modules.

Introduction modules

If you're new to how ThreatPipes modules work, read this document first.

For the technically minded, module functionally is best illustrated by looking at the module code itself.

For example, the Name Extractor module looks for TARGET_WEB_CONTENT and EMAILADDR events for identifying human names (HUMAN_NAME).

# What events is this module interested in for input
# * = be notified about all events.
def watchedEvents(self):
return ["TARGET_WEB_CONTENT", "EMAILADDR"]
# What events this module produces
# This is to support the end user in selecting modules based on events
# produced.
def producedEvents(self):
return ["HUMAN_NAME"]

Writing a module

1. The template module

A great place to start is by looking at the template module tpp_template.py.

2. Clone the template

You can create a copy of the template module as a base for your new module.

Name it based on what you're going to build. Try and make it a descriptive name so others can quickly understand what it is for. See examples of appropriate names by looking at other modules.

Open up the file for the new module you've just created. Replace references to XXX with the name of your module.

It is also very helpful to include descriptive information (purpose, author, date, etc) about your module in the header.

3. Setting the module classification

The comment for the class is used by the ThreatPipes UI to correctly categorise modules.

There are five variables you can enter separated by a semi-colon (":") see the following 5 variable

  1. Name

  2. Use Cases

  3. Category

  4. Labels

  5. Description

4. Defining what the module does

Modules have:

  1. watchedEvents() - data elements the module consumes to perform an action on

  2. producedEvents() - data elements the module produces as a result of the watched event

If you are producing a new data element that does not already exist in ThreatPipes, you must add it to the database first.

If the database already exists (i.e. you have started ThreatPipes) you can add it to the database like so:

$ sqlite3 $THREATPIPES_HOME/db/threatpipes.db
$ sqlite> INSERT INTO tbl_event_types (event, event_descr, event_raw) VALUES ('NAME_OF_NEW_DATA_ELEMENT_TYPE_HERE', 'DESCRIPTION_OF_NEW_DATA_ELEMENT_TYPE_HERE', 0, 'DESCRIPTOR or DATA or ENTITY or SUBENTITY', IS RISKY);`

5. Adding the logic for the module

Put the logic for the module in handleEvent().

Each call to handleEvent() is provided a SpiderFootEvent object. The most important values within this object are:

  • eventType: The data element ID (e.g. IP_ADDRESS)

  • data: The actual data (e.g. IP address)

  • module: The name of the module that produced the event (e.g. sfp_dnsresolve)

When it is time to generate your event, create an instance of SpiderFootEvent:

e = SpiderFootEvent("IP_ADDRESS", ipaddr, self.__name__, event)

Note: the event passed as the last variable is the event that your module received. This is what builds a relationship between data elements in the ThreatPipes database.

Finally, notify all modules that may be interested in the event:

self.notifyListeners(e)

6. Test and deploy

Copy the module into the ThreatPipes /modules directory.

7. A working example

Use any of the 100+ ThreatPipes modules to see how they work.