If you're new to how ThreatPipes modules work, read this document first.
For the technically minded, module functionally is best illustrated by looking at the module code itself.
For example, the Name Extractor module looks for
EMAILADDR events for identifying human names (
# What events is this module interested in for input# * = be notified about all events.def watchedEvents(self):return ["TARGET_WEB_CONTENT", "EMAILADDR"]# What events this module produces# This is to support the end user in selecting modules based on events# produced.def producedEvents(self):return ["HUMAN_NAME"]
A great place to start is by looking at the template module
You can create a copy of the template module as a base for your new module.
Name it based on what you're going to build. Try and make it a descriptive name so others can quickly understand what it is for. See examples of appropriate names by looking at other modules.
Open up the file for the new module you've just created. Replace references to
XXX with the name of your module.
It is also very helpful to include descriptive information (purpose, author, date, etc) about your module in the header.
The comment for the class is used by the ThreatPipes UI to correctly categorise modules.
There are five variables you can enter separated by a semi-colon (":") see the following 5 variable
watchedEvents() - data elements the module consumes to perform an action on
producedEvents() - data elements the module produces as a result of the watched event
If you are producing a new data element that does not already exist in ThreatPipes, you must add it to the database first.
If the database already exists (i.e. you have started ThreatPipes) you can add it to the database like so:
$ sqlite3 $THREATPIPES_HOME/db/threatpipes.db$ sqlite> INSERT INTO tbl_event_types (event, event_descr, event_raw) VALUES ('NAME_OF_NEW_DATA_ELEMENT_TYPE_HERE', 'DESCRIPTION_OF_NEW_DATA_ELEMENT_TYPE_HERE', 0, 'DESCRIPTOR or DATA or ENTITY or SUBENTITY', IS RISKY);`
Put the logic for the module in
Each call to handleEvent() is provided a SpiderFootEvent object. The most important values within this object are:
eventType: The data element ID (e.g. IP_ADDRESS)
data: The actual data (e.g. IP address)
module: The name of the module that produced the event (e.g. sfp_dnsresolve)
When it is time to generate your event, create an instance of SpiderFootEvent:
e = SpiderFootEvent("IP_ADDRESS", ipaddr, self.__name__, event)
Note: the event passed as the last variable is the event that your module received. This is what builds a relationship between data elements in the ThreatPipes database.
Finally, notify all modules that may be interested in the event:
Copy the module into the ThreatPipes