Stoq

Overview

The Stoq class is the core of the framework. It must be instantiated in order for all other modules to function properly. This class is meant to be called from stoq.py.

Upon instantiation, default configuration options are defined within __init__. These are overridden if there is identical configuration option in stoq.cfg.

The StoqPluginManager will also be instantiated as a child class automatically. This allows for the ability to globally access the API for plugins and easily grant the ability for plugins to load other plugins.

Examples

Instantiate the Stoq class:

from stoq.core import Stoq
stoq = Stoq()

Retrieve a file from a url:

content = stoq.get_file("http://google.com")

Write content to disk:

stoq.write("example content", path="/tmp", filename="example.txt")

Note

If no filename is given, Stoq.get_uuid will be called and a random filename will be defined automatically. Additionally, if the filename already exists, the file will not be overwritten. However, if Stoq.write() is called with overwrite=True, the file will be overwritten. If the content to be written is binary, one may add binary=True when calling Stoq.write().

In many cases, you may wish to define plugin options. This is especially so if you are not using stoQ from the command line. You may provide the parameter plugin_options when instantiating the Stoq() class.

Instantiate Stoq class, and set attributes for plugins:

from stoq.core import Stoq

plugin_options = {
    'worker': {
        'yara': {
            'yararules': '/data/yara/rules.yar'
            }
        }
    }
stoq = Stoq(plugin_options=plugin_options)

The plugin options will be available within the plugin object itself. For instance, in the above example the yara worker plugin will now have the attribute yararules defined as /data/yara/rules.yar.

API

class stoq.core.Stoq(argv=None, base_dir=None, log_dir=None, results_dir=None, temp_dir=None, plugin_dir_list=None, archive_base=None, config_file=None, dispatch_rules=None, useragent=None, plugin_options=None, log_level=None, log_maxbytes=None, log_backup_count=None, default_connector=None, default_source=None, filename_suffix=None, max_recursion=None, max_queue=None, source_base_tuple=None, url_prefix_tuple=None, log_syntax=None, sentry_url=None, sentry_ignore_list=None, default_tlp=None)

Core stoQ Framework Class

dumps(data, indent=4, compactly=False)

Wrapper for json library. Dump dict to a json string

Parameters:
  • data (dict) – Python dict to convert to json
  • indent (int) – Indent level for return value
  • compactly – set to True to return unindented JSON (no newlines between key/values),
Returns:

Converted json string

Return type:

str

force_unicode(payload)

Force a string to be properly encoded in unicode using BeautifulSoup4

Parameters:payload (bytes) – String to be forced into unicode
Returns:Unicode bytes
Return type:bytes
get_file(source, params=None, verify=True, auth=None, timeout=30, **kwargs)

Obtain contents of file from disk or URL.

Note

A file will only be opened from disk if the path of the file matches the regex defined by source_base_tuple in stoq.cfg.

Parameters:
  • source (bytes) – Path or URL of file to read.
  • params (bytes) – Additional parameters to pass if requesting a URL
  • verify (bool) – Ensure SSL Certification Verification
  • auth – Authentication methods supported by python-requests
  • timeout (int) – Time to wait for a server response
  • **kwargs – Additional HTTP headers
Returns:

Content of file retrieved

Return type:

bytes or None

get_time

Get the current time, in ISO format

Returns:Current time in ISO Format
Return type:str
get_uuid

Generate a random uuid

Returns:Random uuid
Return type:str
hashpath(sha1)

Generate a path based on the first five chars of a SHA1 hash

example: The SHA1 4caa16eba080d3d4937b095fb68999f3dbabd99d would return a path similar to: /opt/malware/4/c/a/a/1

Parameters:sha1 (str) – SHA1 hash of a payload
Returns:Path
Return type:str
load_config()

Load configuration file. Defaults to stoq.cfg.

loads(data)

Wrapper for json library. Load json string as a python dict

Parameters:data (str) – json string to load into dict
Returns:Converted dict
Return type:dict
logger_init()

Initialize the logger globally.

Returns:True
normalize_json(obj)
Normalize json blobs:
  • If a key’s value is a dict:
    • Make the value a list
    • Iterate over sub keys and do the same
  • If a key’s value is a list:
    • Iterate over the values to ensure they are a string
  • If the key’s value is anything else:
    • Force the value to be a string
Parameters:obj (dict) – dict object to normalize
Returns:Normalized dict object
Return type:dict
post_file(url, params=None, files=None, data=None, auth=None, verify=True, timeout=30, **kwargs)

Handles POST request to specified URL

Parameters:
  • url (bytes) – URL to for POST request
  • params (bytes) – Additional parameters to pass if requesting a URL
  • files (tuple) – Tuple of file data to POST
  • data (bytes) – Content to POST
  • auth – Authentication methods supported by python-requests
  • verify (bool) – Ensure SSL Certification Verification
  • timeout (int) – Time to wait for a server response
  • **kwargs – Additional HTTP headers
Returns:

Content returned from POST request

Return type:

bytes or None

put_file(url, params=None, data=None, auth=None, verify=True, timeout=30, **kwargs)

Handles PUT request to specified URL

Parameters:
  • url (bytes) – URL to for PUT request
  • params (bytes) – Additional parameters to pass if requesting a URL
  • data (bytes) – Content to PUT
  • auth – Authentication methods supported by python-requests
  • verify (bool) – Ensure SSL Certification Verification
  • timeout (int) – Time to wait for a server response
  • **kwargs – Additional HTTP headers
Returns:

Content returned from PUT request

Return type:

bytes or None

sanitize_json(obj)

Sanitize json so keys do not contain ‘.’ or ‘ ‘. Required for compaitibility with databases such as mongodb and elasticsearch

Parameters:obj (dict) – dict object
Returns:Sanitized dict object
Return type:dict
write(payload, filename=None, path=None, binary=False, overwrite=False, append=False)

Write content to disk

Parameters:
  • payload (str) – Data to be written to disk
  • filename (str) – Filename, if none is provided, a random filename will be used
  • path (str) – Path for output file
  • binary (bool) – Define whether content is binary or not
  • overwrite (bool) – Define whether output file should be overwritten
  • append (bool) – Define whether output file should be appended to
Returns:

Full path of file that was written

Return type:

str or False