/dropbox/pb-jelly

dropbox/pb-jelly

by

pb-jelly is a protobuf code generation framework for the Rust language developed at Dropbox.

History

This implementation was initially written in 2016 to satisfy the need of shuffling large amount
of bytes in Dropbox’s Storage System (Magic Pocket).
Previously, we were using rust-protobuf (and therefore generated APIs are exactly
the same to make migration easy) but serializing Rust structs to proto messages, and then serializing them again in
our RPC layer, meant multiple copies (and same thing in reverse on parsing stack). Taking control of this
implementation and integrating it in our RPC stack end-to-end helped avoid these extra copies.

Over the years, the implementation has grown and matured and is currently used in several parts of Dropbox, including
our Sync Engine, and the aforementioned Magic Pocket.

Other implementations exist in the Rust ecosystem (e.g. prost and rust-protobuf), we wanted to share ours as well.

Crates.io Documentation Crates.io

Features

  • Functional “Rust-minded” proto extensions, e.g. [(rust.box_it)=true]
  • Scalable – Generates separate crates per module, with option for crate-per-directory
    • Autogenerates Cargo.toml, or optionally Spec.toml / bazel BUILD files
  • Support for Serde
  • Zero-copy deserialization with Bytes via a proto extension [(rust.zero_copy)=true]
  • Automatically boxes messages if it finds a recursive message definition
  • Retains comments on proto fields
  • Supports proto2 and proto3

Extensions

Extension Description Type Example
(rust.zero_copy)=true Generates field type of Lazy<bytes::Bytes> for proto bytes fields to support zero-copy deserialization Field zero_copy
(rust.box_it)=true Generates a Box<Message> field type Field box_it
(rust.type)="type" Generates a custom field type Field custom_type
(rust.preserve_unrecognized)=true Preserves unrecognized proto fields into an _unrecognized struct field Field TODO
(gogoproto.nullable)=false Generates non-nullable fields types Field TODO
(rust.nullable)=false Generates oneofs as non-nullable (fail on deserialization) Oneof non_optional
(rust.err_if_default_or_unknown)=true Generates enums as non-zeroable (fail on deserialization) Enum non_optional
(rust.serde_derive)=true Generates serde serializable/deserializable messages File serde

Using pb-jelly in your project

Multiple crates, multiple languages, my oh my!

Essential Crates

There are only two crates you’ll need if you want to use this with you project pb-jelly and pb-jelly-gen.

pb-jelly

Contains all of the important traits and structs that power our generated code, e.g. Message and Lazy. Include this as a dependency, e.g.

[dependencies]
pb-jelly = "0.0.1"
pb-jelly-gen

A framework for generating Rust structs and implementations for proto2 and proto3 files. Include this as a build-dependency, e.g.

[build-dependencies]
pb-jelly-gen = "0.0.1"

Generating Rust Code

In order to generate Rust code from your proto definitions you’ll need three things

  1. pb-jelly-gen
  2. protoc – The protobuf compiler, this can be built from source protobuf or installed via brew install protobuf.
  3. python2 – The codegen plugin used with protoc is written in Python2. Before running it, you’ll need to install some packages, a requirements.txt is pending #18.

Take a look at the examples crate to see how we leverage pb-jelly-gen and build.rs to get started using protobufs in Rust!

Non-essential Crates

  • pb-test contains integration tests and benchmarks. You don’t need to worry about this one unless you want to contribute to this repository!
  • pb-types contains generated Rust types for well known proto types [TODO]: Might deprecate this?

A Note On Scalability 📝

We mention “scalabilty” as a feature, what does that mean? We take an opinionated stance that every module should be a crate, as opposed to generating Rust files 1:1 with proto files. We take this stance because rustc is parallel across crates, but not yet totally parallel within a crate. When we had all of our generated Rust code in a single crate, it was often that single crate that took the longest to compile. The solution to these long compile times, was creating many crates!

First, contributions are greatly appreciated and highly encouraged. For legal reasons all outside
contributors must agree to Dropbox’s CLA. Thank you for
your understanding.


Some of the features here require additional tooling to be useful, which are not yet public.

  • Spec.toml is a stripped down templated Cargo.toml – which you can script convert into
    Cargo.toml in order to get consistent dependency versions in a multi-crate project.
    Currently, the script to convert Spec.toml -> Cargo.toml isn’t yet available
  • Autogenerated BUILD files require additional tooling to convert BUILD.in-gen-proto~ to a BUILD file

Closed structs with public fields

  • Adding fields to a proto file will lead to compiler errors. This can be a benefit in that it allows the
    compiler to identify all callsites that may need to be visited. However, it can make updating protos with
    many callsites a bit tedious. We opted to go this route to make it easier to add a new field and update
    all callsites with assistance from the compiler.

Service Generation

  • Generating stubs for gPRC clients and servers
  1. Clone Repo.
  2. Install Dependencies / Testing Dependencies. These instructions are for OSX using the brew
    package manager. Use the appropriate package manager for your system.

    • protoc – part of Google’s protobuf tools
    • Install Go
    • Install gogoproto
      • go get github.com/gogo/protobuf/proto
    • Install Python & dependencies
      • brew install python3
      • pip install six
      • pip install protobuf
    • Generate test protos
      • On OSX: you’ll have to install coreutils for realpath brew install coreutils
      • ./gen_protos.sh
  3. pb-jelly currently uses an experimental test framework that requires a nightly build of rust.
  4. cargo test

TODO

Contributors

Dropboxers

Similar Projects

rust-protobuf – Rust implementation of Google protocol buffers
prost – PROST! a Protocol Buffers implementation for the Rust Language
quick-protobuf – A rust implementation of protobuf parser
serde-protobuf