Archiver Plugins¶
Overview¶
Archiver plugins are used for retrieving or saving scanned payloads. A payload can be anything from the initial payload scanned, or extracted payloads from previous scans. There are two types of archivers, source and destination.
destination¶
Archiver plugins used as a destination useful for saving payloads, be it the original scanned payload or any extracted payloads. Multiple destination archivers can be defined, allowing for a payload to be saved in either a single or multiple locations. The results from this plugin method may be used to subsequently load the payload again.
Destination archiver plugins can be defined multiple ways. In these examples, we will
use the filedir
archiver plugin.
From stoq.cfg
:
[core]
dest_archivers = filedir
Note
Multiple plugins can be defined separated by a comma
From the command line:
$ stoq run -A filedir [...]
Note
Multiple plugins can be defined by simply adding the plugin name
Or, when instantiating the Stoq()
class:
>>> import stoq
>>> dest_archivers = ['filedir']
>>> s = Stoq(dest_archivers=dest_archivers)
source¶
Archiver plugins used as a source retrieve payloads for scanning. This is useful in several use cases, such as when using a provider plugin that isn’t able to pass a payload to stoQ. For example, if the provider plugin being used leverages a queueing system, such as RabbitMQ, there may be problems placing multiple payloads onto a queue as it is inefficient, prone to failure, and does not scale well. With archiver plugins as a source, the queuing system can be leveraged by sending a message with a payload location, and the archiver plugin can then retrieve the payload for scanning. The ArchiverResponse results returned from ArchiverPlugin.archive() is used to load the payload.
Source archiver plugins can be defined multiple ways. In these examples, we will
use the filedir
archiver plugin.
From stoq.cfg
:
[core]
source_archivers = filedir
Note
Multiple plugins can be defined separated by a comma
From the command line:
$ stoq run -S filedir [...]
Note
Multiple plugins can be defined by simply adding the plugin name
Or, when instantiating the Stoq()
class:
>>> import stoq
>>> source_archivers = ['filedir']
>>> s = Stoq(source_archivers=source_archivers)
Writing a plugin¶
Unlike most other stoQ plugins, archiver plugins have two core methods, of which at least one of the below is required.
archive
get
The archive
method is used to archive payloads that are passed to stoQ or extracted
from other plugins. In order for a payload to be archived, that attribute should_archive
must be set to True
in the payloads PayloadMeta
object. If set to False
, the
payload will not be archived.
An archiver plugin must be a subclass of the ArchiverPlugin
class.
As with any plugin, a configuration file must also exist and be properly configured.
Example¶
from typing import Dict, Optional
from stoq.plugins import ArchiverPlugin
from stoq.helpers import StoqConfigParser
from stoq.data_classes import ArchiverResponse, Payload, Request, PayloadMeta
class ExampleArchiver(ArchiverPlugin):
def __init__(self, config: StoqConfigParser) -> None:
super().__init__(config)
self.archive_path = config.get(
'options', 'archive_path', fallback='/tmp/archive_payload')
async def archive(
self, payload: Payload, request: Request
) -> Optional[ArchiverResponse]:
with open(f'{self.archive_path}', 'wb) as out:
out.write(payload.content)
ar = ArchiverResponse({'path': f'{self.archive_path}'})
return ar
async def get(self, task: ArchiverResponse) -> Optional[Payload]:
with open(task.results['path'], 'rb') as infile:
return Payload(
infile.read(),
PayloadMeta(
extra_data={'path': task.results['path']}))
Note
ArchiverPlugin.archive() returns an ArchiverResponse object, which contains metadata that is later used by ArchiverPlugin.get() to load the payload.
API¶
-
class
stoq.plugins.archiver.
ArchiverPlugin
(config)[source]¶ -
async
archive
(payload, request)[source]¶ Archive payload
- Parameters
payload (
Payload
) – Payload object to archiverequest (
Request
) – Originating Request object
- Return type
Optional
[ArchiverResponse
]- Returns
ArchiverResponse object. Results are used to retrieve payload.
>>> import asyncio >>> from stoq import Stoq, Payload >>> payload = Payload(b'this is going to be saved') >>> s = Stoq() >>> loop = asyncio.get_event_loop() >>> archiver = s.load_plugin('filedir') >>> loop.run_until_complete(archiver.archive(payload)) ... {'path': '/tmp/bad.exe'}
-
async
get
(task)[source]¶ Retrieve payload for processing
- Parameters
task (
ArchiverResponse
) – Task to be processed to load payload. Must contain ArchiverResponse results from ArchiverPlugin.archive()- Return type
Optional
[Payload
]- Returns
Payload object for scanning
>>> import asyncio >>> from stoq import Stoq, ArchiverResponse >>> s = Stoq() >>> loop = asyncio.get_event_loop() >>> archiver = s.load_plugin('filedir') >>> task = ArchiverResponse(results={'path': '/tmp/bad.exe'}) >>> payload = loop.run_until_complete(archiver.get(task))
-
async
Response¶
-
class
stoq.data_classes.
ArchiverResponse
(results=None, errors=None)[source]¶ Object containing response from archiver destination plugins
- Parameters
results (
Optional
[Dict
]) – Results from archiver pluginerrors (
Optional
[List
[Error
]]) – Errors that occurred
>>> from stoq import ArchiverResponse >>> results = {'file_id': '12345'} >>> archiver_response = ArchiverResponse(results=results)