# read-kafka

Connect to brokers and consume messages from topic as consumer group. Messages are emitted as events with the value in the payload field. Will start from the beginning of the topic when replay is set. Will shut down the pipeline once max messages are consumed.

Committing consumer group offsets is still a work in progress.

# Config

Required:

brokers: Array of Kafka broker addresses
topic: Topic to consume from
group: Consumer group ID

Optional:

client: Kafka client ID (default: logbus)
max: Maximum messages to consume before stopping
replay: Start from beginning of topic

# Example

pipeline:
  read-kafka:
    config:
      brokers:
        - kafka1:9092
        - kafka2:9092
      topic: logs
      group: logbus-consumer
      client: logbus-instance-1