README
JSOX – JavaScript Object eXchange format.
100% Compatible reader for JSON. JSOX.stringify cannot generate JSON
compatible output; it would lose all the features anyway; use existing
JSON.stringify()
if required, all JSON(JSON3/JSON5/JSON6) is valid JSOX.
JSOX adds Map, BigInt, Date, and TypeArray transport support, cyclic objects, and includes keywords (5)'Infinity', (5)'NaN', (6)'undefined'.
JSOX adds optional processing of typed
data. Type names can be applied
to Objects, Arrays and Strings. Type names are defined and provided with
to/from JSOX handlers by users of this library. The data, decoded as
the object '{}', array '[]', or string '""' is passed to the fromJSOX
handler, and the resulting value returned as the decoded object.
Typed-objects may also be emitted as a class-defintition and then class-references. A class-defintition defines the fields in the object, and a class-reference would provide the values for each field respectively.
A typed-object example: v{ x, y } { a : v{1,2} }
, which decodes as { a : {x:1,y:2} }
.
It defines a template/class of object that has fields 'x', and 'y'. Then
defines an object with a field A what is a object of type 'v', with values
(1), and (2). This example does not gain any visible savings; savings
comes when you have a lot of the same sort of record with the same field
names repeated often.
- adds macro/class support for object field names.
- adds support for bigint numbers; indicated with an 'n' suffix.
- adds support for Date parsing and stringification; ISO dates as used are a sub-type of Number.
- adds support for circular references.
- typed-strings, typed-arrays, and typed-objects, for user defined types and to and from JSOX methods. more
- C style commants;
//
and/* */
. - string continuations using \ at the end of the line removes the newline; Otherwise strings continue until the next quote.
"
,'
,``
, are all valid quote pairings, with no differnce in meaning, other than the quotes they contain.- adds optional underscores in numbers, allowing user formatting of log numbers.
- fields are canonically ordered, so all objects that have the same field names will have their names in the same order. Keys in Map()s are not ordered.
- trailing commas are allowed, and silently ignored; however empty comma pairs in arrays will generate empty elements; and (throw an error in objects?).
- and of course
o === JSOX.parse(JSON.stringify(o))
should always be exactly true.
Example Encoding
r = JSOX.stringify( o = {
a: "simple object"
, b:3
, c:new Date()
, d:123n
, e:null
, f:undefined
, g:NaN
, h:Infinity
, i:-Infinity
, j:-0.302
, k:new Uint8Array(9)}, null, 3 );
cnsole.log( "pretty:", o, "=\n", r );
// -- output --
pretty:
{
a: "simple object",
b: 3,
c: 2018-09-14T02:55:27-07:00,
d: 123n,
e: null,
g: NaN,
h: Infinity,
i: -Infinity,
j: -0.302,
k: u8[AAAAAAAAAAA=]
}
JSOX is a proposed replacement to JSON that aims to make it easier for humans to write and maintain by hand, while also transporting the correct type of the data. Humans, for example, wouldn't hand-code a base64 encoding for a TypedArray; however sending a mesh from a server to a client already processed as a typed buffer ready for WebGL consumption may be of use.
The class/tag support is entirely optional, and while it's goal is to reduce redundancy, which for large datasets of similar records can benefit, it has been argued that gzip could just be used to reduce the size; However, this also reduces the size of the data to be parsed on input, which gzip does not do.
JSOX is a (super-sub)set of JavaScript, although adds no new data types, and works with all existing JSON content. Some features allowed in JSOX are not directly supported by Javascript; although all javascript parsable features can be used in JSOX, except functions or any other code construct, transporting only data save as JSON. Most ES6 structure can be parsed, with the extension of classes/macro-tags the reverse is not true. It was true for JSON6.
JSOX is a proprosal for an official successor to JSON, and JSOX stringified content will not work with existing JSON parsers. For this reason, JSOX files use a new .jsox extension. (TODO: new MIME type needed too.)
The code is a reference JavaScript implementation for both Node.js and all browsers. The code is derrived from JSON-6 sources.
Why
Beyond the existing reasons for JSON5/JSON6 for their modifications; this addresses the biggest shortcoming of JSON, which is the repetitive and redundant specification of field names; especially when lots of the same sort of object is represented.
This also aims to provide support for BigInt and Date format for less work at the application layer. A method for handling typed array object members should also be impelemented
(Historic Why below)
JSON isn’t the friendliest to write. Keys need to be quoted, objects and arrays can’t have trailing commas, and comments aren’t allowed — even though none of these are the case with regular JavaScript today.
That was fine when JSON’s goal was to be a great data format, but JSON’s usage has expanded beyond machines. JSON is now used for writing configs, manifests, even tests — all by humans.
There are other formats that are human-friendlier, like YAML, but changing from JSON to a completely different format is undesirable in many cases. JSON6’s aim is to remain close to JSON and JavaScript.
Features
The following is the exact list of additions to JSON’s syntax introduced by JSOX. All of these are optional.
Concise representation of dates and times including as much information as is available for the timestamp(timezone).
Supports encode and decode of BigInt numbers with no application overhead.
reduces overhead from none-requires quotes for identifiers.
can further reduce overall output size by using macro tags.
Caveats
JSOX.stringify will produce output that JSON.parse cannot handle; JSOX.parse can always handle JSON.stringify.
Summary of Changes from JSON6
- BigInt encoding
- ISO date/time Encoding/decoding (as part of Number format)
- Adds classes(revive user types) and macro tags to reduce redundant information.
Summary of Changes from JSON6/JSON
- Keyword undefined
- Objects/Strings back-tick quoted strings (no template support, just quotes); Object key names can be unquoted.
- Strings - generous multiline string definition; all javascript character escapes work. (\x##, \0###, \u####, \u{} )
- Numbers - underscore digit separation in numbers, octal and binary formats; all javascript number notations. Addtionally support leading 0 to interpret as octal as C, C++ and other languages support.
- Arrays - empty members
- Streaming reader interface
- (Twice the speed of JSON5; subjective)
- interprets non-breaking space as a space.
Additional support above JSON base
All items listed below are JSON5 additions if not specifed as JSON6.
ArrayBuffer/TypedArray
- (JSOX) Support transporting ArrayBuffer and TypedArray fields. This is implemented with constants as class user types applied prefixing and opening brace '[' and encoding the binary data as a base64 string(without quotes) before the closing ']'.
- these are prefix tags that can be applied. u8, u16, cu8, u32, s8,s16, s32, f32, f64, ab; the array is a base64 string without quotes.
- Base64 is as dense as is feasible; it's a 33% loss; where utf8 encoding of random bytes is 50% loss. Something like base127 would be 7 bytes to 8 encoded bytes; and potential length penalty of 5 bytes.
Objects
Object keys can be unquoted if they do not have ':', ']', '[', '{', '}', ',', any quote or whitespace (including non-breaking space, which on the human side of things looks the same), and do not begin like a number.
Object keys can be single-quoted, (JSON6) or back-tick quoted; any valid string
Object keys can be double-quoted (original JSON).
Objects can have a single trailing comma. Excessive commas in objects will cause an exception. '{ a:123,,b:456 }' is invalid.
Arrays
Arrays can have trailing commas. If more than 1 is found, additional empty elements will be added.
(JSON6) Arrays can have comma ( ['test',,,'one'] ), which will result with empty values in the empty places.
Strings
Strings can be double-quoted (as per original JSON).
Strings can be single-quoted.
Strings can be back-tick (`) (grave accent) -quoted.
Strings can be split across multiple lines; just prefix each newline with a backslash. [ES5 §7.8.4]
(JSON6) all strings will continue keeping every character between the start and end, this allows multi-line strings and keep the newlines in the string; if you do not want the newlines they can be escaped as previously mentioned.
Numbers
(JSOX) BitInt numbers are stringified with suffix of 'n' as in ES(?), and implemented in V8(google/chrome/node) 2018/09/12. BigInt number parsed with 'n' suffix.
(JSON6) Numbers can have underscores separating digits '_' these are treated as zero-width-non-breaking-space. (Proposal with the exception that _ can preceed or follow . and may be trailing.)
Numbers can be hexadecimal (base 16). ( 0x prefix )
(JSON6) Numbers can be binary (base 2). (0b prefix)
(JSON6) Numbers can be octal (base 8). (0o prefix)
(JSON6) Decimal numbers can have leading zeros. (0 prefix followed by more numbers, without a decimal)
Numbers can begin or end with a (leading or trailing) decimal point.
Numbers can include
Infinity
,-Infinity
,NaN
.Numbers can begin with an explicit plus sign.
Numbers can begin with multiple minus signs. For example '----123' === 123.
Dates
- (JSOX) Encodes date time with local timestamp information to recover as much information as the original date contained. Is treated as a subtype of Number parsing; and are stored without quotes.
Keyword Values
- (JSON6) supports 'undefined' in addition to 'true', 'false', 'null'.
Comments
- Both inline (single-line using '//' (todo:or '#'?) ) and block (multi-line using /* */ ) comments are allowed.
- (JSOX) single line comments using '#'; imposes required quoting for '#' as a field name.
// simple example, array buffer with 8 bytes
var ab = new ArrayBuffer([0,1,2,3,4,5,6,7]);
console.log( JSOX.stringify( {ab:new Float32Array(ab)} ) );
// example output
{ab:f32[AAECAwQFBgc=]}
Base64 Character Set
The following defines the characters used for base64 encoding and decoding.
const encodings = 'ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz0123456789$_'
// previous to 1.2.106
//const encodings = 'ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz0123456789+/'
const decodings = { '~':-1
,'=':-1
,'