Skip to content

Releases

v2.16.0

Choose a tag to compare

@ojasvajain ojasvajain released this 06 Oct 16:31
c6c9973

librdkafka v2.16.0 is a feature release:

  • Fix re-bootstrap cases that never reached a bootstrap broker while the learned brokers were still connected, or kept an already connected bootstrap broker without re-resolving its address (#5560).
  • The ALL_BROKERS_DOWN error is now reported only once every reconnect.backoff.max.ms or when the outage restarts (#5600).
  • Avoid duplicate FETCH_STOP for the same toppar during assignment removal (#5574).
  • Fix rd_kafka_clusterid(), rd_kafka_query_watermark_offsets() and rd_kafka_offsets_for_times() waiting until their timeout and accessing the freed client instance when it's destroyed during the call (#5616).
  • Upgraded bundled OpenSSL to 3.5.8 and libcurl to 8.22.0 (#5598).

Security considerations

Bundled dependencies were further upgraded, beyond what v2.15.1 already
covers, for source/autoconf builds:
OpenSSL 3.5.7 → 3.5.8 (LTS); libcurl 8.21.0 → 8.22.0.

Upgrade considerations

  • Admin requests in flight on a decommissioned broker now fail with
    RD_KAFKA_RESP_ERR__TRANSPORT instead of RD_KAFKA_RESP_ERR__DESTROY_BROKER, so
    callers retry them instead of treating them as a hard failure.
  • The ALL_BROKERS_DOWN error is now reported only once every reconnect.backoff.max.ms. In case there are multiple re-bootstrap attempts, caused
    by no available broker connection, this reduces the amount of events while still signalling that the outage is ongoing.

Fixes

General fixes

  • Issues: #5600.
    Fix re-bootstrap cases that never reached a bootstrap broker while the learned brokers were still connected, or kept an already connected bootstrap broker.
    The client kept asking the very brokers that reported its metadata as stale. Learned and bootstrap brokers are now decommissioned when a re-bootstrap sequence starts, so the bootstrap servers are re-created and connected again, re-resolving their addresses, and only they are used until a Metadata response rebuilds the broker list. The transaction coordinator is reset too when its broker is decommissioned, so the coordinator connection is re-established and its address re-resolved, as already done for the group coordinator. Queued messages are handed back to their partitions and re-sent once new leaders are known. Admin requests in flight on a decommissioned broker now fail with RD_KAFKA_RESP_ERR__TRANSPORT (previously
    RD_KAFKA_RESP_ERR__DESTROY_BROKER) and should be retried.
    Happening since 2.11.0 (#5600).
  • Issues: #5546.
    ALL_BROKERS_DOWN was reported on every re-bootstrap cycle during a sustained
    outage. It is now reported once per outage, re-armed when a broker connection
    comes up, and at most once every reconnect.backoff.max.ms while it lasts.
    Happening since 2.11.1 (#5600).
  • rd_kafka_clusterid(), rd_kafka_query_watermark_offsets() and
    rd_kafka_offsets_for_times() now return immediately, with NULL or
    RD_KAFKA_RESP_ERR__DESTROY, when the client is destroyed during the call.
    Previously they kept waiting for metadata until their timeout and then
    accessed the client instance after it was freed. rd_kafka_destroy() now
    waits for these calls to return before freeing it.
    Happening since 0.11.3 (#5616).

Consumer fixes

  • Issues: #5585.
    A consumer with enable.auto.commit=true no longer sends an OffsetCommit
    for an assignment it has already lost. On a client-side session timeout, or
    when max.poll.interval.ms is exceeded, the member id is reset, the
    assignment is marked lost, and its partitions are revoked. The revoke-time
    auto-commit of those partitions was still being sent, because the
    assignment-lost flag was cleared inside rd_kafka_cgrp_unassign() and
    rd_kafka_cgrp_incremental_unassign() before the removed partitions were
    served, defeating the guard that skips commits for a lost assignment. The
    commit went out with an empty member id and the previous generation, which a
    broker rejects with UNKNOWN_MEMBER_ID, or, for a static member
    (group.instance.id set), with the fatal FENCED_INSTANCE_ID that stops the
    consumer. The flag is now kept set until the removal has been served and the
    unassign completes (rd_kafka_cgrp_unassign_done(),
    rd_kafka_cgrp_incr_unassign_done() and, for the KIP-848 consumer protocol,
    rd_kafka_cgrp_consumer_incr_unassign_done()), so the offsets of a lost
    assignment are never committed while the flag is still cleared as soon as
    the revoke is done, keeping commits, close() and unsubscribe() working
    for a member that retains other partitions, and rd_kafka_assignment_lost()
    reporting false again by the time the next assignment is delivered.
    Happening since 1.6.0 (#5585).
  • Issues: #5573.
    Prevents duplicate FETCH_STOP requests during assignment removal
    by introducing an assignment-owned rktp_wait_stop flag (#5574).

Checksums

Release asset checksums:

  • v2.16.0.zip SHA256 df41723517183306a004502d2ecbc05888e0f26d1dd75d5cb9f0cb750de9ce58
  • v2.16.0.tar.gz SHA256 e6b61de61d3282879a88e4ee3d9f634a8b05bf78a9198dfc25c4820c0c9ed231

v2.15.1

Choose a tag to compare

@emasab emasab released this 09 Sep 17:12
c58bbed

librdkafka v2.15.1 is a maintenance release:

  • Update bundled OpenSSL and libcurl dependencies, and refresh the Windows
    build toolchain (msys2, vcpkg) (#5579).
  • Fix int_latency metric calculation, that was reporting bogus values when
    message.timeout.ms is set to 0 (infinite) (#5335).
  • IPv6 addresses are wrapped in square brackets following RFC 3986, allowing to connect to those compressed IPv6 addresses (#5544).
  • IPv6 addresses are correctly passed to OpenSSL as IPs not as hostnames with brackets and the certificate is validated against its iPAddress entries instead of against dNSName entries (#5544).

Upgrade considerations

  • If you're parsing the nodename as received by connect_cb, stats_cb, ssl_cert_verify_cb or throttle_cb, make sure you're correctly parsing it following RFC 3986. Metadata API calls are unaffected: they return the host and port as separate fields.
  • Make sure the brokers' certificates carry the IP in an iPAddress entry, not a dNSName entry. Previously the certificate was validated against dNSName (falling back to the subject CN) whenever the address could not be parsed as an IP literal: with OpenSSL < 3.0 or BoringSSL for any IP address, and on all OpenSSL versions for an IPv6 address configured in brackets in bootstrap.servers ([::1]:9092), which never verified. A scoped address (fe80::1%eth0) was likewise dNSName-matched on every version and now has the zone stripped before matching. IPv6 addresses ending in :: could not be connected to at all, as the nodename became host:::port and failed name resolution. In the remaining cases, a bare IPv4 or IPv6 literal on OpenSSL >= 3.0, the iPAddress entry was already used, and behaviour is unchanged.

Security considerations

Bundled dependencies were further upgraded as follows:
OpenSSL 3.5.6 → 3.5.7 for source/autoconf builds, and 3.6.2 → 3.6.3 for
vcpkg-based packages; libcurl 8.20.0 → 8.21.0, now used by both
source/autoconf builds and vcpkg (previously pinned to 8.19.0 in vcpkg,
so vcpkg-based packages also pick up the fixes below that source/autoconf
builds already got from 8.20.0).

Fixes

Producer fixes

  • Issues: #5555.
    Fix int_latency metric calculation. It was derived from the message
    timeout timestamp (now + message.timeout.ms - rkm_ts_timeout), which
    yields a large negative value when message.timeout.ms is 0 (infinite)
    and rkm_ts_timeout is INT64_MAX. It's now computed directly as
    now - rkm_ts_enq, the actual time the message spent in the queue,
    regardless of the timeout setting.
    Happening since 0.11.0 (#5335).

Checksums

Release asset checksums:

  • v2.15.1.zip SHA256 1ca308c10c4234962695fbf9c7f1ab0649ab43c7ad03d384f1645db7b74c92ce
  • v2.15.1.tar.gz SHA256 23c8575c7d1ced07246cb9cf200c11325b72201fd4134a02414ca869fbdd8ed3

v2.15.0

Choose a tag to compare

@PratRanj07 PratRanj07 released this 30 Jun 19:23
9a94e11

KIP-932 Queues for Kafka

  • Queues for Kafka support is in preview.
  • Added a preview implementation of the share consumer (Queues for Kafka,
    KIP-932).
    Members of a share group cooperatively consume from the same partitions with
    per-record acquire/acknowledge semantics and redelivery, providing queue-like
    consumption on top of Kafka.
  • New rd_kafka_share_* public API, with a dedicated rd_kafka_share_t handle
    created via rd_kafka_share_consumer_new():
    • Subscription: rd_kafka_share_subscribe(), rd_kafka_share_unsubscribe(),
      rd_kafka_share_subscription().
    • Batch polling: rd_kafka_share_poll() returns an rd_kafka_messages_t
      batch (rd_kafka_messages_count() / rd_kafka_messages_get() /
      rd_kafka_messages_destroy()).
    • Acknowledgement: rd_kafka_share_acknowledge(),
      rd_kafka_share_acknowledge_type(), rd_kafka_share_acknowledge_offset()
      with ACCEPT / RELEASE / REJECT types, and
      rd_kafka_message_delivery_count().
    • Commit: rd_kafka_share_commit_sync(), rd_kafka_share_commit_async() and
      the acknowledgement-commit callback
      (rd_kafka_share_set_acknowledgement_commit_cb()).
    • Lifecycle: rd_kafka_share_consumer_close(),
      rd_kafka_share_consumer_close_queue(), rd_kafka_share_destroy().
  • Two acknowledgement modes selected by share.acknowledgement.mode
    (default implicit; explicit requires the application to acknowledge every
    record before the next poll).
  • New max.poll.records property (default 500) and adjusted defaults for
    several network properties for share consumers (receive.message.max.bytes,
    connections.max.idle.ms, reconnect.backoff.ms,
    reconnect.backoff.max.ms).
  • See the Share consumers (Queues for Kafka) section of
    INTRODUCTION.md, the
    Share consumer section in rdkafka.h, and the
    examples/share_consumer* programs.

Note

The KIP-932
share consumer is currently in Preview and should not be used in
production environments. The public interfaces may change before General
Availability, and known limitations apply (see
INTRODUCTION.md). The
share consumer is single-threaded and not thread-safe by design. It requires
a broker with share groups enabled (generally available in Apache Kafka
4.2.0).

Enhancements

  • Add aws_iam option to sasl.oauthbearer.metadata.authentication.type, with a defensive stub that fails when no token-refresh callback is registered.

Fixes

General fixes

  • Issues: #5135.
    Fix compilation with CMake when CURL is disabled.
    The OAuthBearer OIDC code included <curl/curl.h> under #ifdef WITH_OAUTHBEARER_OIDC, but
    CMake always defines that macro (to 0 or 1), so CURL was required even when it was turned off.
    Happening since 2.11.0 (#5136).
  • Issues: #5282.
    Fix rd_atomic32_set/rd_atomic64_set returning the new value instead of the previous one in CMake builds.
    CMake never defined HAVE_ATOMICS_{32,64}_ATOMIC, so the setters used a non-atomic fallback that
    returned the new value, which prevented the ALL_BROKERS_DOWN event from being raised under CMake.
    Happening since 2.11.1 (#5136).

Consumer fixes

  • Issues: #5541.
    Improve error handling in the KIP-848 consumer group protocol:
    • Defer the leave heartbeat until the assignment is revoked, so a member that
      exceeds max.poll.interval.ms rejoins cleanly instead of being rejected
      with a fatal INVALID_REQUEST.
    • Treat GROUP_ID_NOT_FOUND in the ConsumerGroupHeartbeat response as fatal,
      except while the member is already leaving.
    • Surface unexpected permanent broker-level heartbeat errors instead of
      retrying them in a loop; internal transport/timeout codes keep their
      existing retry/reconnect handling.
      Happening since 2.12.0 (#5488)

Checksums

Release asset checksums:

  • v2.15.0.zip SHA256 d0655c770f7edc5699855f65b391d1e20dc24fc03ecf2dd680665934dd4d1cc1
  • v2.15.0.tar.gz SHA256 259015220cdca708afe838b5aa79ebf1a5fb710fb4179cf918d390aed85d5dbc

v2.14.2

Choose a tag to compare

@Devarsh010 Devarsh010 released this 03 Jun 10:28
6516057

librdkafka v2.14.2 is a maintenance release:

  • Fix duplicate groups in ListConsumerGroups when multiple brokers
    return the same group (#5417).
  • Fix data race in timers (#5089).
  • Update bundled OpenSSL, libcurl, zstd, zlib and cJSON
    dependencies (#5346).

Security considerations

Bundled dependencies were upgraded as follows (see #5346):
OpenSSL 3.0.15 → 3.5.6 (LTS) for source/autoconf builds, and to 3.6.2 in
vcpkg-based packages (no LTS available in vcpkg); libcurl 8.10.1 → 8.20.0
for source/autoconf builds and to 8.19.0 in vcpkg; zlib 1.3.1 → 1.3.2;
zstd 1.5.6 → 1.5.7; cJSON 1.7.14 → 1.7.19.

Fixes

General fixes

  • Issues: #5082.
    Fix data race in timers. The callback and its argument could have been modified after the lock is released.
    Happening since 1.x (#5089).

Consumer fixes

  • Fix crash (SIGSEGV) in rd_kafka_cgrp_handle_LeaveGroup() when coordinator
    is unavailable during consumer close. The error logging path dereferenced
    a potentially NULL broker pointer. Happening since 1.x.

Admin client fixes

  • Issues: #5417.
    Fix duplicate groups in ListConsumerGroups when multiple brokers return the same group.
    Happening since 1.x (#5417).

Checksums

Release asset checksums:

  • v2.14.2.zip SHA256 2c0a563a39d5c1bc2e7b3ae81bbad9aca23c586ddab9f659b51983d4dc67cffb
  • v2.14.2.tar.gz SHA256 d7eec9c31c817fa44402f679c252dfbf97e4c338a849a25c3579a31fd127beb8

v2.14.1

Choose a tag to compare

@Devarsh010 Devarsh010 released this 15 Apr 05:22
e0da09c

librdkafka v2.14.1 is a maintenance release:

  • Bundle prebuilt binaries for linux-s390x (#5365).

Checksums

Release asset checksums:

  • v2.14.1.zip SHA256 b6f52ae7a743e504e416e620c94e1192906daf15eafe661e4e2e6a7b793efc06
  • v2.14.1.tar.gz SHA256 bb246e754dee3560e9b42bf4e844dc05de4b146a3cae937e36301ffacdc456e7

v2.14.0

Choose a tag to compare

@emasab emasab released this 01 Apr 13:57
f1b831e

librdkafka v2.14.0 is a feature release:

  • KIP-768 Extend SASL/OAUTHBEARER to support OIDC claim mapping beyond the default sub claim (#5336).

Checksums

Release asset checksums:

  • v2.14.0.zip SHA256 372589ac63b06f9cac5d9b50d4ed1998f46b1a6cca991b691527ccc62b7cb7dd
  • v2.14.0.tar.gz SHA256 c05c03ef00a13a8463fac3e8918c04843c416f11ced58c889d806a88ca92cf99

v2.13.2

Choose a tag to compare

@PratRanj07 PratRanj07 released this 02 Mar 18:32
901af7c

librdkafka v2.13.2 is a maintenance release:

  • The librdkafka.redist NuGet package now includes binary for alpine-arm64 (#5237, @mclayton7)
  • Remove CPU usage regression when a subscription matches no topics (#5324).
  • Fix rd_kafka_consume_batch_queue incorrectly updating the application position on EOF or error messages (#5213).
  • Fix compilation without getentropy (Oleg Babin (@olegrok), Leo Singer (@lpsinger), #5288).
  • Use a truly random seed for pseudo-random number generation whenever available (#5288).
  • Fix rd_list destroy callback type mismatch by changing rd_kafka_assignor_destroy to take a void * argument, as expected by rd_list_init() destroy callbacks, and casting internally to rd_kafka_assignor_t * (#5195) (#5278).

Fixes

General fixes

Consumer fixes

  • Issues: #5324. Remove CPU usage regression when a subscription matches no topics. The increased CPU usage (~30%) was seen in particular when there are many topics in the clusters and the given subscription regex doesn't match any. Happening since 2.10.0 (#5324).
  • Issues: #4844. Fix rd_kafka_consume_batch_queue incorrectly updating the application position when receiving EOF or error messages, causing the position to move forward and likely be stored and committed. When storing the application offset the leader epoch is also considered for correct offset ordering in case of log truncation. Happening since 2.2.0 (#5213).

Checksums

Release asset checksums:

  • v2.13.2.zip SHA256 1b71b01a33f54c5d3e359935f73445fde4010215efdb899e3a746475d6aa178a
  • v2.13.2.tar.gz SHA256 14972092e4115f6e99f798a7cb420cbf6daa0c73502b3c52ae42fb5b418eea8f

v2.13.0

Choose a tag to compare

@k-raina k-raina released this 05 Jan 12:59
59b2f66

librdkafka v2.13.0 is a feature release:

  • KIP-482 Upgrade CreateAcls, DescribeAcls, DeleteAcls to the first version supporting this KIP (#5081).
  • KIP-482 Upgrade DescribeGroups, DeleteTopics, DeleteRecords, CreatePartitions, DeleteGroups to the first version supporting this KIP (#5083).
  • Strip trailing dot of hostname to fix SSL certificate verification issue (#5253).
  • Fix memory management for interceptors in rd_kafka_conf to prevent
    double-free errors (#5240).
  • Fix for the pseudo-random generator seed on Windows involving as well
    the uniqueness of the new consumer group protocol member id (#5265).
  • Add secure random generation functionality used for UUID uniqueness
    and secure salt generation in rd_kafka_UserScramCredentialUpsertion
    using OpenSSL or the POSIX or WIN32 equivalent calls when it
    isn't available (#5265).

Fixes

General fixes

  • Issues: #4348.
    Strip trailing dot of hostname to fix SSL certificate verification issue.
    Happening since 1.x (#5253).
  • Issues: #4142.
    Fix memory management for interceptors in rd_kafka_conf to prevent double-free errors.
    In case the client instance fails the users needs to destroy the configuration
    data structure, it was causing a double-free because the interceptors were
    already freed in the constructor.
    Happening since 1.x (#5240).
  • Issues: #5263, #3929.
    Fix for the pseudo-random seed on Windows. The function rand_r isn't present
    on Windows and the global seed wasn't based on the current microseconds and thread
    id. Also it wasn't called on every thread as required on this platform but
    only once per process. The fix allows on this platform the uniqueness of client side
    member id generation in next-generation consumer group protocol.
    Happening since 1.x (#5265).

Checksums

Release asset checksums:

  • v2.13.0.zip SHA256 73d731322b34c59fb5245d27172c71824e9323acd934e214d70a77954749e79d
  • v2.13.0.tar.gz SHA256 3bd351601d8ebcbc99b9a1316cae1b83b00edbcf9411c34287edf1791c507600

New Contributors

Full Changelog: v2.12.1...v2.13.0

v2.12.1

Choose a tag to compare

@emasab emasab released this 21 Oct 15:59
e1db7ea

librdkafka v2.12.1 is a maintenance release:

  • Restored macOS binaries compatibility with macOS 13 and 14 (#5219).

Fixes

General fixes

  • Fix to restore macOS 13 and 14 compatibility in prebuilt binaries present in librdkafka.redist.
    Happening since 2.12.0 (#5219).

Checksums

Release asset checksums:

  • v2.12.1.zip SHA256 da7571a0c1dc374aabb18af6ca01411d4bc597d321977980c8d3211ec5adf696
  • v2.12.1.tar.gz SHA256 ec103fa05cb0f251e375f6ea0b6112cfc9d0acd977dc5b69fdc54242ba38a16f

v2.12.0

Choose a tag to compare

@emasab emasab released this 08 Oct 18:36
2f22088

librdkafka v2.12.0 is a feature release:

KIP-848 – General Availability

Starting with librdkafka 2.12.0, the next generation consumer group rebalance protocol defined in KIP-848 is production-ready. Please refer the following migration guide for moving from classic to consumer protocol.

Note: The new consumer group protocol defined in KIP-848 is not enabled by default. There are few contract change associated with the new protocol and might cause breaking changes. group.protocol configuration property dictates whether to use the new consumer protocol or older classic protocol. It defaults to classic if not provided.

Enhancements and Fixes

  • Support for OAUTHBEARER metadata based authentication types,
    starting with Azure IMDS. Introduction available (#5155).
  • Fix compression types read issue in GetTelemetrySubscriptions response
    for big-endian architectures (#5183, Faidon Liambotis (@paravoid)).
  • Fix for KIP-1102 time based re-bootstrap condition (#5177).
  • Fix for discarding the member epoch in a consumer group heartbeat response when leaving with an inflight HB (#4672).
  • Fix for an error being raised after a commit due to an existing error in the topic partition (#4672).
  • Fix double free of headers in rd_kafka_produceva method (blindspot (@blindspotbounty), #4628).
  • Fix to ensure rd_kafka_query_watermark_offsets enforces the specified timeout and does not continue beyond timeout expiry (#5201).
  • New walkthrough in the Wiki about configuring Kafka cross-realm authentication between Windows SSPI and MIT Kerberos.

Fixes

General fixes

  • Issues: #5178.
    Fix for KIP-1102 time based re-bootstrap condition.
    Re-bootstrap is now triggered only after metadata.recovery.rebootstrap.trigger.ms
    have passed since first metadata refresh request after last successful
    metadata response. The calculation was since last successful metadata response
    so it's possible it did overlap with the periodic topic.metadata.refresh.interval.ms
    and cause a re-bootstrap even if not needed.
    Happening since 2.11.0 (#5177).
  • Issues: #4878.
    Fix to ensure rd_kafka_query_watermark_offsets enforces the specified timeout and does not continue beyond timeout expiry.
    Happening since 2.3.0 (#5201).

Telemetry fixes

  • Issues: #5179 .
    Fix issue in GetTelemetrySubscriptions with big-endian
    architectures where wrong values are read as
    accepted compression types causing the metrics to be sent uncompressed.
    Happening since 2.5.0. Since 2.10.1 unit tests are failing when run on
    big-endian architectures (#5183, Faidon Liambotis (@paravoid)).

Consumer fixes

  • Issues: #5199
    Fixed an issue where topic partition errors were not cleared after a successful
    commit. Previously, a partition could retain a stale error state even though the
    most recent commit succeeded, causing misleading error reporting. Now, successful
    commits correctly clear the error state for the affected partitions
    Happening since 2.4.0 (#4672).

Producer fixes

Checksums

Release asset checksums:

  • v2.12.0.zip SHA256 9b2f373e03f3d5d87c2075b3ce07ee9ea3802eea00cea41b99d8351a68d8a062
  • v2.12.0.tar.gz SHA256 1355d81091d13643aed140ba0fe62437c02d9434b44e90975aaefab84c2bf237