Protobuf-like data (de)serialisation using D meta programming.
To use this package, run the following command in your project's root directory:
Put the following dependency into your project's dependences section:
D Message Buffers
I was pondering the use of either protobuf or flatbuffers for a private project of mine, and there are some good D modules out there that enable the use of those serialisation libraries from within D programs.
However, I was trying to find out if it would be possible to do a (de)serialisation without the use of external schema files that needed to be precompiled (i.e. .proto or .fbs) and instead use plain D data types directly.
This library is the result of this experiment.
It only uses D's brilliant meta programming capabilities to (de)serialise data from/to binary format. No external schema files or precompilation is required.
Tested with the following compilers:
- LDC2 1.26.0, 1.27.0, 1.28.0
- DMD64 v2.097.0, v2.097.2-beta.1, v2.098
LDC2 takes a bit longer to compile than DMD, but produces much faster code.
What works, what doesn't?
- most structure based types should be ok (as long as the members are public)
- types with private members can define toMsgBuf()/fromMsgBuf() methods for custom (de)serialization
- static arrays
- dynamic arrays
- associative arrays
- all basic D types
- enum (only integers at the moment)
- nested structures
- as long as new fields are appended to the end, old programs can read new message formats and vice versa
- oneof (types in a oneof message must be unique)
- CI/CD (built and tested on Ubuntu, macOS and Windows)
Doesn't work (yet)
- "any" (this should be easy to simulate using a ubyte member for the data and maybe a type / url / uuid field)
- adding/removing fields at arbitrary places (though adding new fields to the end is fine)
- D classes, unions (probably won't do)
Possible future improvements
- fixing stuff that doesn't work yet if sensible
- adding @attributes to D message structures and fields to specify e.g. field/version numbers
- adding a JSON encoding like e.g. proto3 does (see: https://developers.google.com/protocol-buffers/docs/proto3#json)
Some Microbenchmarks, take with a grain of salt.
Using Flat msgbuf (i.e. similar to Google Flatbuffers)
serialized size: 130072 (Flat) performed 1000 rounds in 39 ms (3335179 bytes / ms)
Using Variable msgbuf (i.e. similar to Google Protobuf)
serialized size: 109919 (Var) performed 1000 rounds in 76 ms (1446302 bytes / ms)
serialized size: 109921 performed 1000 rounds in 96 ms (1145010 bytes / ms)
Google Flatbuffers (note: these figures need to be verified!)
serialized size: 130096 performed 1000 ops in 107 ms (7650790 bytes / ms)
I have never worked with flatbuffers before, so I welcome all constructive criticism.
Please refer to the
- v0.0.9 Removed bidirectional control characters from unicode tests.
- v0.0.8 Added endianess (defaulting to little endian).
- Registered by Sinisa Susnjar
- 0.0.9 released 5 months ago
- Copyright © 2022, Sinisa Susnjar