asdf 0.0.8

A Simple Document Format


To use this package, run the following command in your project's root directory:

Manual usage
Put the following dependency into your project's dependences section:

Dub version License codecov.io Build Status

A Simple Document Format

ASDF is a cache oriented string based JSON representation. It allows to easily iterate over JSON arrays/objects multiple times without parsing them. ASDF does not parse numbers (they are represented as strings). ASDF values can be removed by setting deleted bit on.

For line separated JSON values see parseJsonByLine function. This function accepts a range of chunks instead of a range of lines.

Why ASDF?

ASDF is fast. It can be really helpful if you have gigabytes of JSON line separated values.

Specification

See ASDF Specification.

I/O Speed
  • Reading JSON line separated values and parsing them to ASDF - 300+ MB per second (SSD).
  • Writing ASDF range to JSON line separated values - 300+ MB per second (SSD).
current transformation functions
udafunction
@serializationKeys("bar_common", "bar")tries to read the data
@serializationKeysIn("a", "b")tries to read the data from b, then b
@serializationMultiKeysIn(["a", "b", "c"])tries to get the data from a sub object. this has not optimal performance yet if you are using more than 1 serializationMultiKeysIn in an object
@serializationIgnoreignore this property completely
@serializationIgnoreIndon't read this property
@serializationIgnoreOutdon't write this property
@serializationScopedDangerous! non allocating strings. this means data can vanish if the underlying buffer is removed.
@serializedAs!stringcall to!string
@serializationTransformIn!fincall function fin to transform the data
`@serializationTransformOut!`"str".repeat.take(a).joiner("_").to!string` int x`run function on serialization, different notation
ASDF Example
import std.algorithm;
import std.stdio;
import asdf;

void main()
{
	auto target = Asdf("red");
	File("input.jsonl")
		// Use at least 4096 bytes for real wolrd apps
		.byChunk(4096)
		// 32 is minimal value for internal buffer. Buffer can be realocated to get more memory.
		.parseJsonByLine(4096)
		.filter!(object => object
			// opIndex accepts array of keys: {"key0": {"key1": { ... {"keyN-1": <value>}... }}}
			["colors"]
			// iterates over an array
			.byElement
			// Comparison with ASDF is little bit faster
			//   then compression with a string.
			.canFind(target))
			//.canFind("tadmp5800"))
		// Formatting uses internal buffer to reduce system delegate and system function calls
		.each!writeln;
}
Input

Single object per line: 4th and 5th lines are broken.

null
{"colors": ["red"]}
{"a":"b", "colors": [4, "red", "string"]}
{"colors":["red"],
	"comment" : "this is broken (multiline) object"}
{"colors": "green"}
{"colors": "red"]}}
[]
Output
{"colors":["red"]}
{"a":"b","colors":[4,"red","string"]}
JSON and ASDF Serialization Examples
Simple struct or object
struct S
{
	string a;
	long b;
	private int c; // private feilds are ignored
	package int d; // package feilds are ignored
	// all other fields in JSON are ignored
}
Selection
struct S
{
	// ignored
	@serializationIgnore
	int temp;
	
	// can be formatted to json
	@serializationIgnoreIn
	int a;
	
	//can be parsed from json
	@serializationIgnoreOut
	int b;
}
Key overriding
struct S
{
	// key is overrided to "aaa"
	@serializationKeys("aaa")
	int a;

	// overloads multiple keys for parsing
	@serializationKeysIn("b", "_b")
	// overloads key for generation
	@serializationKeyOut("_b_")
	int b;
}
User-Defined Serialization
struct DateTimeProxy
{
	DateTime datetime;
	alias datetime this;

	static DateTimeProxy deserialize(Asdf data)
	{
		string val;
		deserializeScopedString(data, val);
		return DateTimeProxy(DateTime.fromISOString(val));
	}

	void serialize(S)(ref S serializer)
	{
		serializer.putValue(datetime.toISOString);
	}
}
//serialize a Doubly Linked list into an Array
struct SomeDoublyLinkedList
{
	@serializationIgnore DList!(SomeArr[]) myDll;
	alias myDll this;

	//no template but a function this time!
	void serialize(ref AsdfSerializer serializer)
    {
        auto state = serializer.arrayBegin();
        foreach (ref elem; myDll)
        {
            serializer.elemBegin;
            serializer.serializeValue(elem);
        }
        serializer.arrayEnd(state);
    }   
}
Serialization Proxy
struct S
{
	@serializedAs!DateTimeProxy
	DateTime time;
}
Finalizer
struct S
{
	string a;
	int b;

	@serializationIgnoreIn
	double sum;

	void finalizeDeserialization(Asdf data)
	{
		auto r = data["c", "d"];
		auto a = r["e"].get(0.0);
		auto b = r["g"].get(0.0);
		sum = a + b;
	}
}
assert(`{"a":"bar","b":3,"c":{"d":{"e":6,"g":7}}}`.deserialize!S == S("bar", 3, 13));
Authors:
  • Ilya Yaroshenko
Dependencies:
none
Versions:
0.7.17 2023-Feb-07
0.7.16 2023-Feb-01
0.7.15 2022-Jun-02
0.7.14 2022-Mar-24
0.7.13 2021-Nov-09
Show all 85 versions
Download Stats:
  • 25 downloads today

  • 114 downloads this week

  • 392 downloads this month

  • 298707 downloads total

Score:
3.6
Short URL:
asdf.dub.pm