Source code for stoq.plugins.provider

#!/usr/bin/env python3

#   Copyright 2014-2018 PUNCH Cyber Analytics Group
#   Licensed under the Apache License, Version 2.0 (the "License");
#   you may not use this file except in compliance with the License.
#   You may obtain a copy of the License at
#   Unless required by applicable law or agreed to in writing, software
#   distributed under the License is distributed on an "AS IS" BASIS,
#   WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
#   See the License for the specific language governing permissions and
#   limitations under the License.

    .. _provider:


    Provider plugins are designed for passing multiple payloads, or locations of payloads,
    to `stoQ`. They allow for multiple payloads to be run against `stoQ` until the source
    is exhausted. As such, they are useful for monitoring directories for new files,
    subscribing to a queue (i.e., RabbitMQ, Google PubSub, ZeroMQ), or scanning entire
    directories recursively. Multiple provider plugins can be provided allowing for even more
    flexibility. Provider plugins may either send a payload to `stoQ` for scanning, or send a
    message that an :ref:`Archiver plugin <archiver>` is able to handle for loading of a

    .. note:: Provider plugins are not available when using `scan mode`. This is due to
              `scan mode` being designed for individual scans, not multiple payloads.

    Provider plugins can be defined multiple ways. In these examples, we will use the
    ``dirmon`` provider plugin.

    From ``stoq.cfg``::

        providers = dirmon

    .. note:: Multiple plugins can be defined separated by a comma

    From the command line::

        $ stoq run -P dirmon [...]

    .. note:: Multiple plugins can be defined by simply adding the plugin name

    Or, when instantiating the ``Stoq()`` class::

        >>> import stoq
        >>> providers = ['dirmon']
        >>> s = Stoq(providers=providers, [...])

    Writing a plugin

    `Provider plugins` add either ``Payload`` objects to the `stoQ` queue, or a ``str``.
    If a ``Payload`` object is added, `stoQ` will begin processing the payload. However,
    if a ``str`` is added, `stoQ` will pass it to ``Archiver`` plugins that were
    loaded when ``Stoq`` was instantiated with the ``source_archivers`` argument.

    A `provider` plugin must be a subclass of the ``ProviderPlugin`` class.

    As with any plugin, a :ref:`configuration file <pluginconfig>` must also exist
    and be properly configured.



        from queue import Queue
        from typing import Dict, Optional
        from configparser import ConfigParser

        from stoq import Payload, PayloadMeta
        from stoq.plugins import ProviderPlugin

        class ExampleProvider(ProviderPlugin):
            def __init__(self, config: ConfigParser, plugin_opts: Optional[Dict]) -> None:
                super().__init__(config, plugin_opts)
                self.meta = config.get('options', 'meta', fallback='This msg will always be')

            def ingest(self, queue: Queue) -> None:
                payload_meta = PayloadMeta(extra_data={'msg': self.meta})
                queue.put(Payload(b'This is a payload', payload_meta=payload_meta))



from abc import abstractmethod
from queue import Queue

from stoq.plugins import BasePlugin

[docs]class ProviderPlugin(BasePlugin):
[docs] @abstractmethod def ingest(self, queue: Queue) -> None: pass