batch sink

class KafkaBatchSink(app, pipeline, connection, key_serializer=None, id=None, config=None)

Bases: KafkaSink

KafkaBatchSink is a sink processor that forwards the event to an Apache Kafka specified by a KafkaConnection object in batches.

It is a proof of concept sink, that allows faster processing of events in the pipeline, but does not guarantee processing of all events in situations when the pump is closed etc.

There is a work to be done with cooperation with aiokafka, so the send_and_wait method works properly and is able to send events in batches.

__init__(app, pipeline, connection, key_serializer=None, id=None, config=None)

Initializing parameters passed to the BatchSink class.

Parameters

appApplication

Name of the Application.

pipelinePipeline

Name of the Pipeline.

connectionConnection

Information needed to creates connection.

key_serializer : ,default None

id : , default = None

configJSON, default = None

Configuration file with additional information.

batch sink methods

KafkaBatchSink.process(context, event: Union[dict, str, bytes])

Starts the sink process.

Parameters

contexttype?

Additional information.

event: typing.Union[dict, str, bytes] : type?

async KafkaBatchSink._connection()

Description:

Returns