Stoq¶
Overview¶
The Stoq class is the core of the framework. It must be instantiated in order for all other modules to function properly. This class is meant to be called from stoq.py.
Upon instantiation, default configuration options are defined within __init__. These are overridden if there is identical configuration option in stoq.cfg.
The StoqPluginManager will also be instantiated as a child class automatically. This allows for the ability to globally access the API for plugins and easily grant the ability for plugins to load other plugins.
Examples¶
Instantiate the Stoq class:
from stoq.core import Stoq
stoq = Stoq()
Retrieve a file from a url:
content = stoq.get_file("http://google.com")
Write content to disk:
stoq.write("example content", path="/tmp", filename="example.txt")
Note
If no filename is given, Stoq.get_uuid
will be called and a
random filename will be defined automatically. Additionally, if
the filename already exists, the file will not be overwritten.
However, if Stoq.write()
is called with overwrite=True
,
the file will be overwritten. If the content to be written is
binary, one may add binary=True
when calling Stoq.write()
.
In many cases, you may wish to define plugin options. This is especially so if you are not using stoQ from the command line. You may provide the parameter plugin_options when instantiating the Stoq() class.
Instantiate Stoq class, and set attributes for plugins:
from stoq.core import Stoq
plugin_options = {
'worker': {
'yara': {
'yararules': '/data/yara/rules.yar'
}
}
}
stoq = Stoq(plugin_options=plugin_options)
The plugin options will be available within the plugin object itself. For instance, in the above example the yara worker plugin will now have the attribute yararules defined as /data/yara/rules.yar.
API¶
-
class
stoq.core.
Stoq
(argv=None, base_dir=None, log_dir=None, results_dir=None, temp_dir=None, plugin_dir_list=None, archive_base=None, config_file=None, dispatch_rules=None, useragent=None, plugin_options=None, log_level=None, log_maxbytes=None, log_backup_count=None, default_connector=None, default_source=None, filename_suffix=None, max_recursion=None, max_queue=None, source_base_tuple=None, url_prefix_tuple=None, log_syntax=None, sentry_url=None, sentry_ignore_list=None, default_tlp=None)¶ Core stoQ Framework Class
-
dumps
(data, indent=4, compactly=False)¶ Wrapper for json library. Dump dict to a json string
Parameters: - data (dict) – Python dict to convert to json
- indent (int) – Indent level for return value
- compactly – set to True to return unindented JSON (no newlines between key/values),
Returns: Converted json string
Return type: str
-
force_unicode
(payload)¶ Force a string to be properly encoded in unicode using BeautifulSoup4
Parameters: payload (bytes) – String to be forced into unicode Returns: Unicode bytes Return type: bytes
-
get_file
(source, params=None, verify=True, auth=None, timeout=30, **kwargs)¶ Obtain contents of file from disk or URL.
Note
A file will only be opened from disk if the path of the file matches the regex defined by source_base_tuple in stoq.cfg.
Parameters: - source (bytes) – Path or URL of file to read.
- params (bytes) – Additional parameters to pass if requesting a URL
- verify (bool) – Ensure SSL Certification Verification
- auth – Authentication methods supported by python-requests
- timeout (int) – Time to wait for a server response
- **kwargs – Additional HTTP headers
Returns: Content of file retrieved
Return type: bytes or None
-
get_time
¶ Get the current time, in ISO format
Returns: Current time in ISO Format Return type: str
-
get_uuid
¶ Generate a random uuid
Returns: Random uuid Return type: str
-
hashpath
(sha1)¶ Generate a path based on the first five chars of a SHA1 hash
example: The SHA1 4caa16eba080d3d4937b095fb68999f3dbabd99d would return a path similar to: /opt/malware/4/c/a/a/1
Parameters: sha1 (str) – SHA1 hash of a payload Returns: Path Return type: str
-
load_config
()¶ Load configuration file. Defaults to stoq.cfg.
-
loads
(data)¶ Wrapper for json library. Load json string as a python dict
Parameters: data (str) – json string to load into dict Returns: Converted dict Return type: dict
-
logger_init
()¶ Initialize the logger globally.
Returns: True
-
normalize_json
(obj)¶ - Normalize json blobs:
- If a key’s value is a dict:
- Make the value a list
- Iterate over sub keys and do the same
- If a key’s value is a list:
- Iterate over the values to ensure they are a string
- If the key’s value is anything else:
- Force the value to be a string
Parameters: obj (dict) – dict object to normalize Returns: Normalized dict object Return type: dict
-
post_file
(url, params=None, files=None, data=None, auth=None, verify=True, timeout=30, **kwargs)¶ Handles POST request to specified URL
Parameters: - url (bytes) – URL to for POST request
- params (bytes) – Additional parameters to pass if requesting a URL
- files (tuple) – Tuple of file data to POST
- data (bytes) – Content to POST
- auth – Authentication methods supported by python-requests
- verify (bool) – Ensure SSL Certification Verification
- timeout (int) – Time to wait for a server response
- **kwargs – Additional HTTP headers
Returns: Content returned from POST request
Return type: bytes or None
-
put_file
(url, params=None, data=None, auth=None, verify=True, timeout=30, **kwargs)¶ Handles PUT request to specified URL
Parameters: - url (bytes) – URL to for PUT request
- params (bytes) – Additional parameters to pass if requesting a URL
- data (bytes) – Content to PUT
- auth – Authentication methods supported by python-requests
- verify (bool) – Ensure SSL Certification Verification
- timeout (int) – Time to wait for a server response
- **kwargs – Additional HTTP headers
Returns: Content returned from PUT request
Return type: bytes or None
-
sanitize_json
(obj)¶ Sanitize json so keys do not contain ‘.’ or ‘ ‘. Required for compaitibility with databases such as mongodb and elasticsearch
Parameters: obj (dict) – dict object Returns: Sanitized dict object Return type: dict
-
write
(payload, filename=None, path=None, binary=False, overwrite=False, append=False)¶ Write content to disk
Parameters: - payload (str) – Data to be written to disk
- filename (str) – Filename, if none is provided, a random filename will be used
- path (str) – Path for output file
- binary (bool) – Define whether content is binary or not
- overwrite (bool) – Define whether output file should be overwritten
- append (bool) – Define whether output file should be appended to
Returns: Full path of file that was written
Return type: str or False
-