Pulsar
Reading from Apache Pulsar topics as a streaming source.
Pulsar is supported as a streaming source. The connector uses the standard Pulsar Consumer model with explicit subscription name, and is compatible with both single-tenant and multi-tenant clusters.
Registering a source
session.register_pulsar_source(
"orders",
schema,
"pulsar://broker:6650",
"persistent://public/default/orders",
"krishiv-app", // subscription name
"shared", // subscription type: exclusive | shared | failover | key_shared
)?;
Supported features
| Feature | Notes |
|---|---|
| Schemas | Bytes, string, JSON, Avro. Protobuf via schema registry (in flight). |
| Compression | LZ4, ZLIB, ZSTD, SNAPPY (server-side decompression is on by default). |
| Acknowledgement | Per-message cumulative ack bound to checkpoint epoch. |
| DLQ | Standard Pulsar retry topic. Krishiv surfaces the count in krishiv_streaming_rows_emitted_total with a topic label. |
Offsets and checkpointing
Per-subscription MessageId is stored in the source offset. On restart, the source seeks to the checkpointed message id.
Auth
Pulsar uses TLS + token auth. Configure via pulsar://broker:6650 with a token in the URL or via PULSAR_TOKEN env.
Preview: Source-only at this release. No Pulsar sink.