README
data-record
Record type for Node.js.
Record Format
A record is consist of an array of field definitions, describing a physical data
structure in memory that can be mapped to a struct type in C or C++.
The following array defines a simple fixed size record with some nested records.
// A record definition
const def = [
{ name: 'value1', type: 'uint32_le' },
{ name: 'value2', type: 'int32_be' },
{ name: 'custom1', type: 'int_le', size: 3 },
{ name: 'custom2', type: 'int_le', size: 5 },
{ name: 'nested', type: 'record', def: [
{ name: 'a', type: 'uint32_le' },
{ name: 'b', type: 'uint32_le' }
]},
{ name: 'x', type: 'record', def: [
{ name: 'a', type: 'uint32_le' },
{ name: 'y', type: 'record', def: [
{ name: 'a', type: 'uint32_le' },
]}
]},
{ name: 'firstName', type: 'cstring', size: 15 },
]
const compiled = compile(def);
The compilation result contains the same information as the original definition but in an optimized data structure that can be accessed more efficiently than the human-readable record definition object.
Types
| Type | Description |
|---|---|
int8 |
8-bit signed integer |
int16 |
16-bit signed integer in host byte order |
int16_be |
16-bit signed integer in big-endian order |
int16_le |
16-bit signed integer in little-endian order |
int32 |
32-bit signed integer in host byte order |
int32_be |
32-bit signed integer in big-endian order |
int32_le |
32-bit signed integer in little-endian order |
int64 |
64-bit signed integer in host byte order |
int64_be |
64-bit signed integer in big-endian order |
int64_le |
64-bit signed integer in little-endian order |
uint8 |
8-bit unsigned integer |
uint16 |
16-bit unsigned integer in host byte order |
uint16_be |
16-bit unsigned integer in big-endian order |
uint16_le |
16-bit unsigned integer in little-endian order |
uint32 |
32-bit unsigned integer in host byte order |
uint32_be |
32-bit unsigned integer in big-endian order |
uint32_le |
32-bit unsigned integer in little-endian order |
uint64 |
64-bit unsigned integer in host byte order |
uint64_be |
64-bit unsigned integer in big-endian order |
uint64_le |
64-bit unsigned integer in little-endian order |
float |
32-bit single-precision floating-point in host byte order |
float_be |
32-bit single-precision floating-point in big-endian order |
float_le |
32-bit single-precision floating-point in little-endian order |
double |
64-bit double-precision floating-point in host byte order |
double_be |
64-bit double-precision floating-point in big-endian order |
double_le |
64-bit double-precision floating-point in little-endian order |
int_be |
0 to 48 bit variable size big-endian signed integer |
int_le |
0 to 48 bit variable size little-endian signed integer |
uint_be |
0 to 48 bit variable size big-endian unsigned integer |
uint_le |
0 to 48 bit variable size little-endian unsigned integer |
cstring |
null-terminated C-string (termination not enforced, same behavior as strcpy()) |
record |
A nested record |
int8_p |
A pointer to an array of 8-bit signed integers |
int16_p |
A pointer to an array of 16-bit signed integers in host byte order |
int16_be_p |
A pointer to an array of 16-bit signed integers in big-endian order |
int16_le_p |
A pointer to an array of 16-bit signed integers in little-endian order |
int32_p |
A pointer to an array of 32-bit signed integers in host byte order |
int32_be_p |
A pointer to an array of 32-bit signed integers in big-endian order |
int32_le_p |
A pointer to an array of 32-bit signed integers in little-endian order |
int64_p |
A pointer to an array of 64-bit signed integers in host byte order |
int64_be_p |
A pointer to an array of 64-bit signed integers in big-endian order |
int64_le_p |
A pointer to an array of 64-bit signed integers in little-endian order |
uint8_p |
A pointer to an array of 8-bit unsigned integers |
uint16_p |
A pointer to an array of 16-bit unsigned integers in host byte order |
uint16_be_p |
A pointer to an array of 16-bit unsigned integers in big-endian order |
uint16_le_p |
A pointer to an array of 16-bit unsigned integers in little-endian order |
uint32_p |
A pointer to an array of 32-bit unsigned integers in host byte order |
uint32_be_p |
A pointer to an array of 32-bit unsigned integers in big-endian order |
uint32_le_p |
A pointer to an array of 32-bit unsigned integers in little-endian order |
uint64_p |
A pointer to an array of 64-bit unsigned integers in host byte order |
uint64_be_p |
A pointer to an array of 64-bit unsigned integers in big-endian order |
uint64_le_p |
A pointer to an array of 64-bit unsigned integers in little-endian order |
float_p |
A pointer to an array of 32-bit single-precision floating-points in host byte order |
float_be_p |
A pointer to an array of 32-bit single-precision floating-points in big-endian order |
float_le_p |
A pointer to an array of 32-bit single-precision floating-points in little-endian order |
double_p |
A pointer to an array of 64-bit double-precision floating-points in host byte order |
double_be_p |
A pointer to an array of 64-bit double-precision floating-points in big-endian order |
double_le_p |
A pointer to an array of 64-bit double-precision floating-points in little-endian order |
cstring_p |
A pointer to a C-string |
Arrays
Any type can be used to create an array but there is a caveat, all the items
inside an must have the same fixed size. The size can be implicit from the type
or a variable size given in the field definition (int_be, int_le, uint_be,
cstring, and record).
The array notation is as follows:
// TYPE[SIZE]
{ name: 'intArr', type: 'int8[80]' }
Pointers
Pointer types can point to variable size data (at runtime) without need to
recompile the record definition. This is different from variable size field
types (int_be, int_le, uint_be, uint_l, cstring) as the size of those
fields is locked in compilation (fixed size array) and have a fixed position in
the data structure. Pointer types are marked with a _p suffix in the type name.
For example a cstring_p pointer can point to the string "Hello" during one
serialization call and to the string "world!!" on the next call. The string is
copied into the dynamic heap section of the resulting buffer which is reserved
for storing variable sized payloads.
Data Structure
In the following examples the data structure is represented in 32bit big-endian format, but all common architectures are supported 32-bit BE/LE, 64-bit BE/LE, or even mixed endianness is possible.
The serialize() function returns a Buffer object that contains a
record structure and a heap sections. The heap is only populated if the record
contains pointers to the data in heap.
The following example shows a record definition, what is stored in the buffer, and a matching C struct.
[
{ "name": "sport", "type": "uint16_be" },
{ "name": "dport", "type": "uint16_be" },
{ "name": "seqno", "type": "uint32_be" },
{ "name": "ackno", "type": "uint32_be" },
{ "name": "options", "type": "uint_be", "size": 3 },
{ "name": "data", "type": "cstring_p" }
]
0 1 2 3
0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ RECORD
| .sport | .dport |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| .seqno |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| .ackno |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| .options | PADDING |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| .data_offset |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| .data_size |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ HEAP
| DATA |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
Here data is inside the heap area.
struct frame {
uint16_t sport;
uint16_t dport;
uint32_t seqno;
uint32_t ackno;
struct {
unsigned int options : 24;
};
char * data;
size_t data_len;
};
API
Functions
compile(recordDef[, { align: true }])
allocRecord(compiledDef[, { unpool, heapSize }])
calcHeapSize(compiledDef, obj)
createRecord(compiledDef, obj)
generateRecordDef(obj)
generateCHeader(compiledDef)
serialize(compiledDef, buf, obj)
deserialize(compiledDef, buf)
readValue(compiledDef, buf, path)
readString(compiledDef, buf, path[, encoding])
writeValue(compiledDef, buf, path, value)
writeString(compiledDef, buf, path, value[, encoding])
createReader(compiledDef, buf, path)
createStringReader(compiledDef, buf, path[, encoding])
createWriter(compiledDef, buf, path)
generateRecordDef() makes a best effort guess on how an object could be
serialized. Strings will be serialized to the size they were seen in the
example object, and numbers will be stored using the same endianness as the
host architecture is currently using.
Record alignment
By default compile() aligns the resulting data for optimal access in C.
If align is set true for compile() then the resulting buffers will be
aligned to the expected C struct alignment on the underlying architecture.
if align is false, then the resulting data is packed as compact as
possible. generateCHeader() does not support unaligned mode.
However, currently subrecords/nested records are not aligned as C structures
even if align is set. Therefore, if nested records and especially record
arrays will be accessed in C care should be taking to ensure that all the
records are aligned to word size. This was a common manual task in
pre-ANSI C world.
Typically in C this manual alignment would look something like (assuming 32bit little-endian):
struct x {
struct {
int16_t value;
int16_t _spare;
} a;
uint32_t flags;
};
With the definition language here we can do the following:
[
{
"name": "x",
"type": "record",
"def": [
{ "name": "value", "type": "int16_le" },
{ "name": "_spare", "type": "int16_le" }
]
},
{ "name": "flags", "type": "uint32_le" }
]
This is the exact bitwise equivalent of the previous C struct.
Scripts
yarn build- run TS buildyarn lint- run ESlintyarn prettier- run Prettieryarn test- run testsyarn perf- run a perf test
Examples
const recordDefEx = [
{ name: 'a', type: 'uint32_le' },
{ name: 'b', type: 'int32_le' },
{ name: 'c', type: 'int_le', size: 3 },
{ name: 'd', type: 'int_le', size: 5 },
{ name: 'nested', type: 'record', def: [
{ name: 'a', type: 'uint32_le' },
{ name: 'b', type: 'uint32_le' }
]},
{ name: 'x', type: 'record', def: [
{ name: 'a', type: 'uint32_le' },
{ name: 'y', type: 'record', def: [
{ name: 'a', type: 'uint32_le' },
]}
]},
];
const obj = {
a: 4,
b: -128,
c: 10,
d: 5,
nested: {
a: 5,
b: 5,
},
x: {
a: 5,
y: {
a: 5
}
}
};
const compiled = compile(recordDefEx);
const buf = createRecord(compiled, obj);
const objSerialized = v8.serialize(obj);
const jsonStr = JSON.stringify(obj);
console.log(`buf.length = ${buf.length}, objSerialized.length = ${objSerialized.length}, jsonStr.length = ${jsonStr.length}`);
// buf.length = 32, objSerialized.length = 69, JSON.length = 76
Performance Testing
$ yarn perf
$ node --prof ./node_modules/.bin/ts-node __perf__/perf.ts
modify
======
nativeObjectTest 1.21 ms
nativeV8SerializerTest 18544.97 ms
jsonTest 171.72 ms
dataRecordTestSlow 55.11 ms
dataRecordTestFast 5.60 ms
serialization
=============
./data/simple.json
nativeV8SerializerTest 10953.32 ms
jsonTest 489.59 ms
dataRecordSerializeTest 370.34 ms
./data/nesting.json
nativeV8SerializerTest 5362.70 ms
jsonTest 2724.44 ms
dataRecordSerializeTest 2800.08 ms
./data/mega-flat.json
nativeV8SerializerTest 10268.37 ms
jsonTest 20080.70 ms
dataRecordSerializeTest 6110.32 ms
./data/numbers.json
nativeV8SerializerTest 5280.86 ms
jsonTest 7111.44 ms
dataRecordSerializeTest 769.95 ms
The performance tests are located under the __perf__ directory and can be executed with yarn perf.
The test modules can be run individually by giving one or more module names as an argument to the
comman, e.g. yarn perf serialization.
Each run will create a isolate file that can be parsed as follows:
node --prof-process isolate-0x5ecbef0-130826-v8.log > processed.txt