Repository navigation
snappy: add Snappy compression - #26914
Conversation
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 56c2b4de35
ℹ️ About Codex in GitHub
Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".
| if dec.closed && dec.output.len == 0 { | ||
| return io.Eof{} |
There was a problem hiding this comment.
When StreamDecoder is closed, read() returns io.Eof{} solely based on output.len == 0, without checking whether buf still contains an incomplete header/body or whether the identifier was ever fully parsed. In practice, a truncated framed stream can be silently accepted in incremental mode: write() buffers the partial chunk, close() marks closed, and subsequent read() reports EOF instead of an error, which can hide data loss/corruption.
Useful? React with 👍 / 👎.
| val |= u32(b & 0x7f) << shift | ||
| if b & 0x80 == 0 { | ||
| return val, i, true |
There was a problem hiding this comment.
read_uvarint accepts any terminating 5th byte and ORs (b & 0x7f) << shift into a u32 without overflow validation. For headers where the 5th byte has bits above 0x0f set (e.g. 0x10), the value wraps modulo 32 bits and is treated as valid instead of rejecting the header. This lets malformed Snappy blocks bypass invalid header varint checks and can produce incorrect declared output lengths.
Useful? React with 👍 / 👎.
|
Thanks! Does it need to be in vlib or should it be installed via |
I noticed that Vlang supports some compression algorithms out of the box and I simply assumed that there might be interest in Snappy, as it is widely used to perform streaming compression due to its framing format. I would say it depends on how many do you wish to support out of the box, and would be interesting to hear the reasoning for the current already supported ones. |
|
Merged, thanks! |
This adds a new module called snappy with a pure V Snappy implementation. It's basically a copy of this.
The pre-computed CRC32C remainders were generated using this script.
Since I couldn't really find any pre-compressed Snappy files that are used to test if a given implementation is correct, I just used this in order to create alice29_block.snappy and alice29_framing.snappy.