Are you sure you want to delete this task? Once this task is deleted, it cannot be recovered.
Mura Li 9591185c8f | 4 years ago | |
---|---|---|
.. | ||
.gitignore | 5 years ago | |
LICENSE | 4 years ago | |
README.md | 4 years ago | |
binary.dat | 5 years ago | |
binary.dat.snappy | 5 years ago | |
rbuf.go | 6 years ago | |
snap.go | 6 years ago | |
unenc.txt | 5 years ago | |
unenc.txt.snappy | 5 years ago | |
unsnap.go | 4 years ago |
This is a small golang library for decoding and encoding the snappy streaming format, specified here: https://github.com/google/snappy/blob/master/framing_format.txt
Note that the streaming or framing format for snappy is different from snappy itself. Think of it as a train of boxcars: the streaming format breaks your data in chunks, applies snappy to each chunk alone, then puts a thin wrapper around the chunk, and sends it along in turn. You can begin decoding before receiving everything. And memory requirements for decoding are sane.
Strangely, though the streaming format was first proposed in Go[1][2], it was never upated, and I could not locate any other library for Go that would handle the streaming/framed snappy format. Hence this implementation of the spec. There is a command line tool[3] that has a C implementation, but this is the only Go implementation that I am aware of. The reference for the framing/streaming spec seems to be the python implementation[4].
Update to the previous paragraph: Horray! Good news: Thanks to @nigeltao, we have since learned that the github.com/golang/snappy package now provides the snappy streaming format too. Even though the type level descriptions are a little misleading because they don't mention that they are for the stream format, the snappy package header documentation points out that the snappy.Reader and snappy.Writer types do indeed provide stream (vs block) handling. Although I have not benchmarked, you should probably prefer that package as it will likely be maintained more than I have time to devote, and also perhaps better integrated with the underlying snappy as they share the same repo.
For binary compatibility with the python implementation in [4], one could use the C-snappy compressor/decompressor code directly; using github.com/dgryski/go-csnappy. In fact we did this for a while to verify byte-for-byte compatiblity, as the native Go implementation produces slightly different binary compression (still conformant with the standard of course), which made test-diffs harder, and some have complained about it being slower than the C.
However, while the c-snappy was useful for checking compatibility, it introduced dependencies on external C libraries (both the c-snappy library and the C standard library). Our go binary executable that used the go-unsnap-stream library was no longer standalone, and deployment was painful if not impossible if the target had a different C standard library. So we've gone back to using the snappy-go implementation (entirely in Go) for ease of deployment. See the comments at the top of unsnap.go if you wish to use c-snappy instead.
[1] https://groups.google.com/forum/#!msg/snappy-compression/qvLNe2cSH9s/R19oBC-p7g4J
[2] https://codereview.appspot.com/5167058
本项目是群体化方法与技术的开源实现案例,在基于Gitea的基础上,进一步支持社交化的协同开发、协同学习、协同研究等群体创新实践服务,特别是针对新一代人工智能技术特点,重点支持项目管理、git代码管理、大数据集存储管理与智能计算平台接入。
Go SVG Roff JavaScript Shell other
Dear OpenI User
Thank you for your continuous support to the Openl Qizhi Community AI Collaboration Platform. In order to protect your usage rights and ensure network security, we updated the Openl Qizhi Community AI Collaboration Platform Usage Agreement in January 2024. The updated agreement specifies that users are prohibited from using intranet penetration tools. After you click "Agree and continue", you can continue to use our services. Thank you for your cooperation and understanding.
For more agreement content, please refer to the《Openl Qizhi Community AI Collaboration Platform Usage Agreement》