This implementation was initially written in 2016 to satisfy the need of shuffling large amount
of bytes in Dropbox's Storage System (Magic Pocket).
Previously, we were using
rust-protobuf (and therefore generated APIs are exactly
the same to make migration easy) but serializing Rust structs to proto messages, and then serializing them again in
our RPC layer, meant multiple copies (and same thing in reverse on parsing stack). Taking control of this
implementation and integrating it in our RPC stack end-to-end helped avoid these extra copies.
Cargo.toml, or optionally
Bytesvia a proto extension
||Generates field type of
||Generates a custom field type||Field||
||Preserves unrecognized proto fields into an
||Generates non-nullable fields types||Field||
||Generates oneofs as non-nullable (fail on deserialization)||Oneof||
||Generates enums as non-zeroable (fail on deserialization)||Enum||
||Generates only a "closed" enum which will fail deserialization for unknown values, but is easier to work with in Rust||Enum||
||Generates serde serializable/deserializable messages||File||
pb-jellyin your project
Multiple crates, multiple languages, my oh my!
There are only two crates you'll need if you want to use this with you project
Contains all of the important traits and structs that power our generated code, e.g.
Lazy. Include this as a
[dependencies] pb-jelly = "0.0.7"
A framework for generating Rust structs and implementations for
In order to use pb-jelly, you need to add the pb-jelly-gen/codegen/codegen.py as a plugin to your protoc invocation.
We added some code here to handle the protoc invocation if you choose to use it.
You'll need to add a generation crate (see
examples_gen for an example)
pb-jelly-gen as a dependency of your generation crate, and
cargo run to invoke protoc for you.
[dependencies] pb-jelly-gen = "0.0.7"
Eventually, we hope to eliminate the need for a generation crate, and simply have generation occur
inside a build.rs with
pb-jelly-gen as a build dependency. However https://github.com/rust-lang/cargo/issues/8709
must be resolved first.
Note that you can always invoke protoc on your own (for example if you are already doing so to generate for multiple languages)
--rust_out=codegen.py as a plugin for rust.
protoc- The protobuf compiler, this can be downloaded or built from source
protobufor installed (mac) via
brew install protobuf.
python3- The codegen plugin used with
protocis written in Python3.
cargo runin the directory of the inner generation crate
python3 -m venv .pb_jelly_venv ; source .pb_jelly_venv/bin/activate
python3 -m pip install protobuf==[same_version_as_your protoc]
python3 -m pip install -e pb-jelly-gen/codegen(installs protoc-gen-rust into the venv)
protoc --rust_out=generated/ input.proto
Take a look at the
examples crate to see how we leverage
build.rs to get started using protobufs in Rust!
pb-testcontains integration tests and benchmarks. You don't need to worry about this one unless you want to contribute to this repository!
examplescontains some examples to help you get started
We mention "scalabilty" as a feature, what does that mean? We take an opinionated stance that every module should be a crate, as opposed to generating Rust files 1:1 with proto files. We take this stance because
rustc is parallel across crates, but not yet totally parallel within a crate. When we had all of our generated Rust code in a single crate, it was often that single crate that took the longest to compile. The solution to these long compile times, was creating many crates!
pb-jelly is a shoutout to the jellyfish known for its highly efficient locomotion. This library is capable of highly efficient locomotion of deserialized data. Also a shoutout to ability of the jellyfish to have substantial increases in population. This library handles generating a very large number of proto modules with complex dependencies, by generating to multiple crates.
We also like the popular sandwich.
First, contributions are greatly appreciated and highly encouraged. For legal reasons all outside contributors must agree to Dropbox's CLA. Thank you for your understanding.
Some of the features here require additional tooling to be useful, which are not yet public.
BUILD.in-gen-proto~to a BUILD file
Closed structs with public fields
brew install protobuf
brew install python3
rustup default nightly
( cd pb_test_gen ; cargo run ) ; cargo test
rust-protobuf - Rust implementation of Google protocol buffers
prost - PROST! a Protocol Buffers implementation for the Rust Language
quick-protobuf - A rust implementation of protobuf parser