Source code for stoq.plugins.provider

#!/usr/bin/env python3

#   Copyright 2014-2018 PUNCH Cyber Analytics Group
#   Licensed under the Apache License, Version 2.0 (the "License");
#   you may not use this file except in compliance with the License.
#   You may obtain a copy of the License at
#   Unless required by applicable law or agreed to in writing, software
#   distributed under the License is distributed on an "AS IS" BASIS,
#   WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
#   See the License for the specific language governing permissions and
#   limitations under the License.

    .. _provider:


    Provider plugins are designed for passing multiple payloads, or locations of payloads,
    to `stoQ`. They allow for multiple payloads to be run against `stoQ` until the source
    is exhausted. As such, they are useful for monitoring directories for new files,
    subscribing to a queue (i.e., RabbitMQ, Google PubSub, ZeroMQ), or scanning entire
    directories recursively. Multiple provider plugins can be provided allowing for even more
    flexibility. Provider plugins may either send a payload to `stoQ` for scanning, or send a
    message that an :ref:`Archiver plugin <archiver>` is able to handle for loading of a

    .. note:: Provider plugins are not available when using `scan mode`. This is due to
              `scan mode` being designed for individual scans, not multiple payloads.

    Provider plugins can be defined multiple ways. In these examples, we will use the
    ``dirmon`` provider plugin.

    From ``stoq.cfg``::

        providers = dirmon

    .. note:: Multiple plugins can be defined separated by a comma

    From the command line::

        $ stoq run -P dirmon [...]

    .. note:: Multiple plugins can be defined by simply adding the plugin name

    Or, when instantiating the ``Stoq()`` class::

        >>> import stoq
        >>> providers = ['dirmon']
        >>> s = Stoq(providers=providers, [...])

    Writing a plugin

    `Provider plugins` add either ``Payload`` objects to the `stoQ` queue, or a ``str``.
    If a ``Payload`` object is added, `stoQ` will begin processing the payload. However,
    if a ``str`` is added, `stoQ` will pass it to ``Archiver`` plugins that were
    loaded when ``Stoq`` was instantiated with the ``source_archivers`` argument.

    A `provider` plugin must be a subclass of the ``ProviderPlugin`` class.

    As with any plugin, a :ref:`configuration file <pluginconfig>` must also exist
    and be properly configured.



        from queue import Queue

        from stoq import Payload
        from stoq.plugins import ProviderPlugin

        class ExampleProvider(ProviderPlugin):
            def ingest(self, queue: Queue) -> None:
                queue.put(Payload(b'This is a payload'))



from abc import abstractmethod
from queue import Queue

from stoq.plugins import BasePlugin

[docs]class ProviderPlugin(BasePlugin):
[docs] @abstractmethod def ingest(self, queue: Queue) -> None: pass