cerealed ~master

cerealed

My DConf 2014 talk mentioning Cerealed.

Binary serialisation library for D. Minimal to no boilerplate necessary. Example usage:

    import cerealed;

    assert(cerealise(5) == [0, 0, 0, 5]); // returns ubyte[]
    cerealise!(a => assert(a == [0, 0, 0, 5]))(5); // faster than using the bytes directly

    assert(decerealise!int([0, 0, 0, 5]) == 5);

    struct Foo { int i; }
    const foo = Foo(5);
    // alternate spelling
    assert(foo.cerealize.decerealize!Foo == foo);

The example below shows off a few features. First and foremost, members are serialised automatically, but can be opted out via the @NoCereal attribute. Also importantly, members to be serialised in a certain number of bits (important for binary protocols) are signalled with the @Bits attribute with a compile-time integer specifying the number of bits to use.

    struct MyStruct {
        ubyte mybyte1;
        @NoCereal uint nocereal1; //won't be serialised
        @Bits!4 ubyte nibble;
        @Bits!1 ubyte bit;
        @Bits!3 ubyte bits3;
        ubyte mybyte2;
    }

    assert(MyStruct(3, 123, 14, 1, 2, 42).cerealise == [ 3, 0xea /*1110 1 010*/, 42]);

What if custom serialisation is needed and the default, even with opt-outs, won't work? If an aggregate type defines a member function void accept(C)(ref C cereal) it will be used instead. To get the usual automatic serialisation from within the custom accept, the grainAllMembers member function of Cereal can be called, as shown in the example below. This function takes a ref argument so rvalues need not apply.

The function to use on Cereal to marshall or unmarshall a particular value is grain. This is essentially what Cerealiser.~= and Decerealiser.value are calling behind the scenes (and therefore cerealise and decerealise).

    struct CustomStruct {
        ubyte mybyte;
        ushort myshort;
        void accept(C)(auto ref C cereal) {
             //do NOT call cereal.grain(this), that would cause an infinite loop
             cereal.grainAllMembers(this);
             ubyte otherbyte = 4; //make it an lvalue
             cereal.grain(otherbyte);
        }
    }

    assert(CustomStruct(1, 2).cerealise == [ 1, 0, 2, 4]);

    //because of the custom serialisation, passing in just [1, 0, 2] would throw
    assert([1, 0, 2, 4].decerealise!CustomStruct == CustomStruct(1, 2));

The other option when custom serialisation is needed that avoids boilerplate is to define a void postBlit(C)(ref C cereal) function instead of accept. The marshalling or unmarshalling is done as it would in the absence of customisation, and postBlit is called to fix things up. It is a compile-time error to define both accept and postBlit. Example below.

    struct CustomStruct {
        ubyte mybyte;
        ushort myshort;
        @NoCereal ubyte otherByte;
        void postBlit(C)(auto ref C cereal) {
             //no need to handle mybyte and myshort, already done
             if(mybyte == 1) {
                 cereal.grain(otherByte);
             }
        }
    }

    assert(CustomStruct(1, 2).cerealise == [ 1, 0, 2, 4]);
    assert(CustomStruct(3, 2).cerealise == [ 1, 0, 2]);

For more examples of how to serialise structs, check the tests directory or real-world usage in my MQTT broker also written in D.

Arrays are by default serialised with a ushort denoting array length followed by the array contents. It happens often enough that networking protocols have explicit length parameters for the whole packet and that array lengths are implicitly determined from this. For this use case, the @RestOfPacket attribute tells cerealed to not add the length parameter. As the name implies, it will "eat" all bytes until there aren't any left.

    private struct StringsStruct {
        ubyte mybyte;
        @RestOfPacket string[] strings;
    }

    //no length encoding for the array, but strings still get a length each
    const bytes = [ 5, 0, 3, 'f', 'o', 'o', 0, 6, 'f', 'o', 'o', 'b', 'a', 'r',
                    0, 6, 'o', 'h', 'w', 'e', 'l', 'l'];
    const strs = StringStruct(5, ["foo", "foobar", "ohwell"]);
    assert(strs.cerealise == bytes);
    assert(bytes.decerealise!StringsStruct ==  strs);

Derived classes can be serialised via a reference to the base class, but the child class must be registered first:

    class BaseClass  { int a; this(int a) { this.a = a; }}
    class ChildClass { int b; this(int b) { this.b = b; }}
    Cereal.registerChildClass!ChildClass;
    BaseClass obj = ChildClass(3, 7);
    assert(obj.cerealise == [0, 0, 0, 3, 0, 0, 0, 7]);

There is now support for InputRange and OutputRange objects. Examples can be found in the tests directory

Advanced Usage

Frequently in networking programming, the packets themselves encode the length of elements to follow. This happens often enough that Cerealed has two UDAs to automate this kind of serialisation: @ArrayLength and @LengthInBytes. The former specifies how to get the length of an array (usually a variable) The latter specifies how many bytes the array takes. Examples:

    struct Packet {
        ushort length;
        @ArrayLength("length") ushort[] array;
    }
    auto pkt = decerealise!Packet([
        0, 3, //length
        0, 1, 0, 2, 0, 3]); //array of 3 ushorts
    assert(pkt.length == 3);
    assert(pkt.array == [1, 2, 3]);

    struct Packet {
        static struct Header {
            ubyte ub;
            ubyte totalLength;
        }
        enum headerSize = unalignedSizeof!Header; //2 bytes

        Header header;
        @LengthInBytes("totalLength - headerSize") ushort[] array;
    }
    auto pkt = decerealise!Packet([
        7, //ub1
        6, //totalLength in bytes
        0, 1, 0, 2]); //array of 2 ushorts
    assert(pkt.ub1 == 7);
    assert(pkt.totalLength == 6);
    assert(pkt.array == [1, 2]);

Related Projects

• orange.
• msgpack-d.