Provider Plugins

Overview

Provider plugins are designed for passing multiple payloads, or locations of payloads, to stoQ. They allow for multiple payloads to be run against stoQ until the source is exhausted. As such, they are useful for monitoring directories for new files, subscribing to a queue (i.e., RabbitMQ, Google PubSub, ZeroMQ), or scanning entire directories recursively. Multiple provider plugins can be provided allowing for even more flexibility. Provider plugins may either send a payload to stoQ for scanning, or send a message that an Archiver plugin is able to handle for loading of a payload.

Note

Provider plugins are not available when using scan mode. This is due to scan mode being designed for individual scans, not multiple payloads.

Provider plugins can be defined multiple ways. In these examples, we will use the dirmon provider plugin.

From stoq.cfg:

[core]
providers = dirmon

Note

Multiple plugins can be defined separated by a comma

From the command line:

$ stoq run -P dirmon [...]

Note

Multiple plugins can be defined by simply adding the plugin name

Or, when instantiating the Stoq() class:

>>> import stoq
>>> providers = ['dirmon']
>>> s = Stoq(providers=providers, [...])

Writing a plugin

Provider plugins add either Payload objects to the stoQ queue, or a str. If a Payload object is added, stoQ will begin processing the payload. However, if a str is added, stoQ will pass it to Archiver plugins that were loaded when Stoq was instantiated with the source_archivers argument.

A provider plugin must be a subclass of the ProviderPlugin class.

As with any plugin, a configuration file must also exist and be properly configured.

Example

from queue import Queue

from stoq import Payload
from stoq.plugins import ProviderPlugin


class ExampleProvider(ProviderPlugin):
    def ingest(self, queue: Queue) -> None:
        queue.put(Payload(b'This is a payload'))

API

class stoq.plugins.provider.ProviderPlugin(config, plugin_opts)[source]
ingest(queue)[source]
Return type:None