Archiver Plugins

Overview

Archiver plugins are used for retrieving or saving scanned payloads. A payload can be anything from the initial payload scanned, or extracted payloads from previous scans. There are two types of archivers, source and destination.

destination

Archiver plugins used as a destination useful for saving payloads, be it the original scanned payload or any extracted payloads. Multiple destination archivers can be defined, allowing for a payload to be saved in either a single or multiple locations. The results from this plugin method may be used to subsequently load the payload again.

Destination archiver plugins can be defined multiple ways. In these examples, we will use the filedir archiver plugin.

From stoq.cfg:

[core]
dest_archivers = filedir

Note

Multiple plugins can be defined separated by a comma

From the command line:

$ stoq run -A filedir [...]

Note

Multiple plugins can be defined by simply adding the plugin name

Or, when instantiating the Stoq() class:

>>> import stoq
>>> dest_archivers = ['filedir']
>>> s = Stoq(dest_archivers=dest_archivers)

source

Archiver plugins used as a source retrieve payloads for scanning. This is useful in several use cases, such as when using a provider plugin that isn’t able to pass a payload to stoQ. For example, if the provider plugin being used leverages a queueing system, such as RabbitMQ, there may be problems placing multiple payloads onto a queue as it is inefficient, prone to failure, and does not scale well. With archiver plugins as a source, the queuing system can be leveraged by sending a message with a payload location, and the archiver plugin can then retrieve the payload for scanning. The ArchiverResponse results returned from ArchiverPlugin.archive() is used to load the payload.

Source archiver plugins can be defined multiple ways. In these examples, we will use the filedir archiver plugin.

From stoq.cfg:

[core]
source_archivers = filedir

Note

Multiple plugins can be defined separated by a comma

From the command line:

$ stoq run -S filedir [...]

Note

Multiple plugins can be defined by simply adding the plugin name

Or, when instantiating the Stoq() class:

>>> import stoq
>>> source_archivers = ['filedir']
>>> s = Stoq(source_archivers=source_archivers)

Writing a plugin

Unlike most other stoQ plugins, archiver plugins have two core methods, of which at least one of the below is required.

  • archive

  • get

The archive method is used to archive payloads that are passed to stoQ or extracted from other plugins. In order for a payload to be archived, that attribute should_archive must be set to True in the payloads PayloadMeta object. If set to False, the payload will not be archived.

An archiver plugin must be a subclass of the ArchiverPlugin class.

As with any plugin, a configuration file must also exist and be properly configured.

Example

from typing import Dict, Optional

from stoq.plugins import ArchiverPlugin
from stoq.helpers import StoqConfigParser
from stoq.data_classes import ArchiverResponse, Payload, Request, PayloadMeta


class ExampleArchiver(ArchiverPlugin):
    def __init__(self, config: StoqConfigParser) -> None:
        super().__init__(config)
        self.archive_path = config.get(
            'options', 'archive_path', fallback='/tmp/archive_payload')

    async def archive(
        self, payload: Payload, request: Request
    ) -> Optional[ArchiverResponse]:
        with open(f'{self.archive_path}', 'wb) as out:
            out.write(payload.content)
        ar = ArchiverResponse({'path': f'{self.archive_path}'})
        return ar

    async def get(self, task: ArchiverResponse) -> Optional[Payload]:
        with open(task.results['path'], 'rb') as infile:
            return Payload(
                infile.read(),
                PayloadMeta(
                    extra_data={'path': task.results['path']}))

Note

ArchiverPlugin.archive() returns an ArchiverResponse object, which contains metadata that is later used by ArchiverPlugin.get() to load the payload.

API

class stoq.plugins.archiver.ArchiverPlugin(config)[source]
async archive(payload, request)[source]

Archive payload

Parameters
  • payload (Payload) – Payload object to archive

  • request (Request) – Originating Request object

Return type

Optional[ArchiverResponse]

Returns

ArchiverResponse object. Results are used to retrieve payload.

>>> import asyncio
>>> from stoq import Stoq, Payload
>>> payload = Payload(b'this is going to be saved')
>>> s = Stoq()
>>> loop = asyncio.get_event_loop()
>>> archiver = s.load_plugin('filedir')
>>> loop.run_until_complete(archiver.archive(payload))
... {'path': '/tmp/bad.exe'}
async get(task)[source]

Retrieve payload for processing

Parameters

task (ArchiverResponse) – Task to be processed to load payload. Must contain ArchiverResponse results from ArchiverPlugin.archive()

Return type

Optional[Payload]

Returns

Payload object for scanning

>>> import asyncio
>>> from stoq import Stoq, ArchiverResponse
>>> s = Stoq()
>>> loop = asyncio.get_event_loop()
>>> archiver = s.load_plugin('filedir')
>>> task = ArchiverResponse(results={'path': '/tmp/bad.exe'})
>>> payload = loop.run_until_complete(archiver.get(task))

Response

class stoq.data_classes.ArchiverResponse(results=None, errors=None)[source]

Object containing response from archiver destination plugins

Parameters
  • results (Optional[Dict]) – Results from archiver plugin

  • errors (Optional[List[Error]]) – Errors that occurred

>>> from stoq import ArchiverResponse
>>> results = {'file_id': '12345'}
>>> archiver_response = ArchiverResponse(results=results)