Kafka

Overview

The Kafka source reads Kafka topics as scheduled, bounded Extract streams. Each topic becomes one stream, and each run reads a finite range of offsets from every partition.

Kafka partition offsets are the incremental checkpoint and determine where the next run resumes. Kafka message timestamps are used to calculate the run's upper boundary, but are not used as the durable checkpoint because timestamps are not guaranteed to be ordered within a partition.

Requirements

• Network access from Extract to every broker address advertised by the Kafka cluster.
• Permission to describe and read the selected topics.
• Permission to commit consumer-group offsets for the configured consumer group.
• Message values encoded as JSON objects. Other payload formats are not supported.

Kafka tombstones (messages with a key and no value) are supported. Extract skips the tombstone record while advancing the partition checkpoint.

Authentication

The connector supports these security protocols:

• PLAINTEXT
• SASL_PLAINTEXT
• SASL_SSL

For SASL connections, the supported mechanisms are:

• PLAIN
• SCRAM-SHA-256
• SCRAM-SHA-512

TLS client certificates are not currently supported.

Source Configuration

Field	Required	Description
Bootstrap Servers	Yes	Comma-separated Kafka bootstrap servers, such as broker1:9092,broker2:9092.
Consumer Group ID	Yes	Consumer group used to commit offsets after successful stream runs. Use a group dedicated to this Extract source.
Security Protocol	Yes	PLAINTEXT, SASL_PLAINTEXT, or SASL_SSL.
SASL Mechanism	For SASL	PLAIN, SCRAM-SHA-256, or SCRAM-SHA-512.
Username	For SASL	SASL username.
Password	For SASL	SASL password.
Topics	No	Explicit topics and schemas to expose. If omitted, Extract discovers readable non-internal topics.

Topics and Streams

Each Kafka topic is exposed as one Extract stream.

You can configure topics explicitly or leave Topics empty to discover readable, non-internal topics. An explicitly configured topic can define:

• Topic: Kafka topic name.
• Stream Name: optional Extract stream name. If omitted, Extract normalizes the topic name.
• Schema Fields: optional list of payload fields and their types.

Explicit stream names must be unique. Configured schema fields take precedence over schema inference.

Schema

Kafka message values must be JSON objects. Supported configured field types are:

• string
• number
• integer
• boolean
• json
• date
• date-time

When no schema is configured, Extract samples retained messages from the topic and infers the payload fields. Sampling skips tombstones. Because sampling may not observe sparse fields, configure the schema when a stable and complete schema is required.

Every record also includes:

Field	Description
_extract_kafka_timestamp	Kafka message timestamp in UTC, when present.
_extract_kafka_topic	Kafka topic name.
_extract_kafka_partition	Kafka partition number.
_extract_kafka_offset	Kafka record offset within the partition.
_extract_kafka_key	Kafka message key, when present.
_extract_kafka_headers	Kafka message headers, when present.

The record primary key is the combination of topic, partition, and offset.

Incremental Reads

Kafka auto commit is disabled. For each stream run, Extract:

1. Loads the saved next offset for every topic partition. A first run starts at each partition's earliest retained offset.
2. Captures the run time once and resolves it to a finite end offset for each partition.
3. Reads each partition in offset order until its end offset, then pauses that partition.
4. Writes JSON records to Extract. Tombstones are skipped but still advance the offset checkpoint.
5. Persists the next offset per partition and synchronously commits those offsets only after the stream completes successfully.

If a run fails, its final offsets are not committed and the next run can read the failed range again. The offset checkpoint stored by Extract is the source of truth; consumer-group commits are confirmation.

A Full Refresh ignores saved offsets and starts from the earliest offsets still retained by Kafka.

Operational Notes

• Kafka offsets are ordered only within a partition; the connector checkpoints every partition independently.
• Producer-created timestamps can be late or out of order. Offsets prevent those messages from being permanently skipped, while timestamps remain useful for choosing finite batch boundaries.
• Retention can remove records before Extract reads them. When a saved offset is below the current retained range, extraction resumes from the partition's earliest available offset.
• Use a dedicated consumer group so another consumer does not overwrite Extract's confirmation commits.