write-s3

Buffer events as JSONL and upload them as objects to an S3-compatible store. A new object is created when the buffer reaches max-bytes or max-seconds elapses, whichever comes first.

Config

Required:

bucket: S3 bucket name

Optional:

region: AWS region (default: us-east-1)
endpoint: Custom endpoint URL for S3-compatible stores (MinIO, R2, GCS)
path-style: Use path-style addressing instead of virtual-hosted buckets (default: false)
key-template: Object key template with placeholders (default: logbus/{year}/{month}/{day}/{hour}{minute}{second}-{uuid}.jsonl)
max-seconds: Seconds between time-based rotations (default: 60)
max-bytes: Byte threshold for size-based rotation (default: 10485760 / 10 MB)
compression: none (default) or gzip. When gzip, the buffer is compressed before upload and .gz is appended to the key.

Key Template Placeholders

Built-in (evaluated at flush time, UTC, zero-padded):

{year}, {month}, {day}, {hour}, {minute}, {second}
{uuid} — a new v4 UUID per object

Event-field placeholders: any other {name} is resolved from the event's top-level fields. When event-field placeholders are present, events are partitioned by their resolved values and each unique combination produces a separate object. Missing fields are replaced with _unknown_.

Credentials

Credentials are resolved by the AWS SDK's standard chain: environment variables, ~/.aws/credentials profile, ECS task role, EC2/EKS instance metadata. No Logbus-specific credential fields.

Examples

Basic usage

pipeline:
  stdin:
    config: {}
  s3:
    module: write-s3
    inputs: [stdin]
    outputs: []
    config:
      bucket: my-log-archive
      region: us-west-2

With gzip compression and custom key

pipeline:
  source:
    module: read-journal
  s3:
    module: write-s3
    inputs: [source]
    outputs: []
    config:
      bucket: my-log-archive
      key-template: "logs/{year}/{month}/{day}/{host}/{uuid}.jsonl"
      compression: gzip
      max-seconds: 120
      max-bytes: 52428800

MinIO / S3-compatible store

pipeline:
  source:
    module: stdin
  s3:
    module: write-s3
    inputs: [source]
    outputs: []
    config:
      bucket: local-logs
      endpoint: http://minio:9000
      path-style: true
      region: us-east-1

Notes

Objects are uploaded via single PutObject (no multipart). Keep max-bytes under 100 MB.
Upload failures are retried up to 3 times with exponential backoff. Persistent failures are reported to the ERRORS channel.
On graceful shutdown, pending buffered events are flushed as a final object.
Data in the in-flight buffer is lost on hard crash. Keep max-seconds low to bound the loss window.
Content headers are set automatically: content-type: application/x-ndjson, and content-encoding: gzip when compression is enabled.