Support for cluster metadata returning all topics
This pull request addresses issue #103.
The Kafka protocol supports returning all topics in the metadata info when a topic is not specified. This functionality has been added to the crate.
The protocol specifies that a null should be sent in the `topics` field of the `metadatarequest`. To achieve this, I changed the `MetadataRequest` struct so that the topic field becomes an option.
```rust
pub struct MetadataRequest<'a, T> {
pub header: HeaderRequest<'a>,
/// The topics to fetch metadata for.
/// Kafka expects the topic to be None for it to return all topics
pub topics: Option<&'a [T]>,
}
```
Downstream changes were made to ensure that if the topic vector is empty, the field is set to `None`, otherwise set to `Some(topics)`.
```rust
impl<'a, T: AsRef<str>> MetadataRequest<'a, T> {
pub fn new(correlation_id: i32, client_id: &'a str, topics: &'a [T]) -> MetadataRequest<'a, T> {
// better to make the topic into an option here than to change the whole code.
let topics = if topics.is_empty() {
None
} else {
Some(topics)
};
MetadataRequest {
header: HeaderRequest::new(API_KEY_METADATA, API_VERSION, correlation_id, client_id),
topics,
}
}
}
```
The Kafka protocol expects that a null value is encoded as -1, which is reflected as well in the code, as shown below
```rust
fn encode<W: BufMut>(&self, buffer: &mut W) -> Result<()> {
self.header.encode(buffer)?;
match self.topics {
Some(topics) => AsStrings(topics).encode(buffer)?,
None => {
// Kafka protocol uses -1 to signal a null array
buffer.put_i32(-1);
}
}
Ok(())
}
```
The decision to use `Options` instead of just checking if the vector is empty is because, in my opinion, the `None` variant is close to the null type and thus a more natural approach.
To ensure that the code works, I have added an integration test. The test involves programmatically creating topics in the broker, querying for the metadata without any topic field and ensuring that the created topics are in the metadata response.
I also made a change to the Docker readme to correct an error around creating a Docker network.
Let me know what you think.
PS: apologies for the complete silence on this issue.
合并状态:已合并 合并于 2025-08-31 关闭于 2025-08-31 4 条评论