JSON¶

The JSON specification defines 7 value types:

• string
• number
• object (container)
• array (container)
• true
• false
• null

Of those 7 values, 2 of them are types describing containers that hold the 5 basic values. We have a separate section below for looking at the 2 container types specifically, so for the time being let’s only consider the following 5 discrete value types:

• string
• number
• true
• false
• null

Most of these types have a 1:1 mapping to a primitive type in most popular programming languages (Java, C, Python, PHP, Erlang, etc.) except for number. This makes defining the types for the 4 easy, but let’s take a closer look at how we might deconstruct number into its core representations.

Universal Binary JSON¶

The Universal Binary JSON specification defines a total of 13 discrete value types (that we saw in the last section) all delimited in the binary file by a specific, 1-byte ASCII character (optionally) followed by a length and (optionally) a data payload containing the value data itself.

Some of the values (null, true` and false) are specific enough that the single 1-byte ASCII character is enough to represent the value in the format and they will have no length or data section.

All of the numeric values (except huge) automatically imply a length by virtue of the type of number they are. For example, a 4-byte int32 always has a length of 4-bytes; an 8-byte double always requires 8 bytes of data.

In these cases the ASCII marker for these types are immediately followed by the data representing the number with no length value in between.

Because string and huge are potentially variable length, they contain all 3 elements of the tuple: type-length-data.

This table shows the official definitions of the discrete value types:

With each field of the table described as:

• Type

  • The binary value data type defined by the spec.

• Size

  • The byte-size of the construct, as stored in the binary format. This is not the value of the length field, just an indicator to you (the reader) of approximately how much space writing out a value of this type will take.

• Marker

  • The single ASCII character marker used to delimit the different types of values in the binary format. When reading in bytes from a file stored in this format, you can simply check the decimal value of the byte (e.g. 'A' = 65) and switch on that value for processing.

• Length?

  • Indicates if the data type provides a length value between the ASCII marker and the data payload.
  • Many of the data types represented in the binary format either don’t have a length (null, true or false) or their types (e.g. the numeric values) are specific enough that the length is implied.
  • When specifying the length for a string or huge value (UTF-8 encoded), the length must represent the number of bytes of the UTF-8 string and not the number of characters in the string.

  Note

  For example, English typically uses 1-byte per character, so the string “hello” has a length of 5. The same string in Russian is “привет” with a byte length of 12 and in Arabic the text becomes “مرحبا” with a byte length of 10.

• Data?

  • Indicates if the data type provides a data payload representing the value.
  • Most types except for null, true and false provide a data payload indicating their value.
  • Variable-length types like string and huge do not provide a data payload when they are empty (i.e. length of 0).More specifically, if you are writing a parser for the Universal Binary JSON format and you encounter a string of length 0, you know the very next byte is an ASCII marker for another value since the string has no data payload.

To further clarify the binary layout of these data types, below are some visual examples of what the bytes would look like inside of a binary JSON file.

NOTE: [ ]-block notation is used for readability, the [ ] characters are not actually written out in the binary format.

Now that we have seen how the JSON data value types map to the binary format, in the next section we will see how we can combine these values together into the two container types (objects and arrays) to create complex object hierarchies using the Universal Binary JSON format.

JSON¶

The two JSON container types are described as follows:

• object
  • A construct containing 0 or more name-value pairings, where the name is always a string and the value can be any valid value type including container types themselves.
• array
  • A flat list of values only, where the values can by any valid value type including container types themselves.
  • The JSON specification does not make it a requirement that the values in an array are all of the same type and neither does the Universal Binary JSON specification.

Given these two constructs, let’s see how they are modeled in the Universal Binary JSON format.

Universal Binary JSON¶

The two container types defined by JSON are modeled using the same tuple that defines all of our other data structures in this specification so far with a minor modification: the length value is considered a count of the child elements the container holds. It does not mean the byte length of the child elements.

More specifically, the tuple stays exactly the same, it is just the meaning of the center length element that changes:

<type, 1-byte char>[<length, 1 or 4-byte integer>][<data>]

All the code used to process the constructs defined by this specification stays the same, but when an object or array construct are encountered, the code needs to be aware that the length value is the child element count so it can know when the scope of the container ends.

For example, if you have an object that contains 4 arrays of length 50, the length argument for the object is 4 (because it contains the four arrays) while the length argument for each array is 50 (because they each hold 50 elements).

Additionally, the only optional field in the tuple for container types is data, if the container is empty and contains no elements (i.e. the length is 0) then there is no data segment.

All together, the definitions for the object and array container types looks like this:

The details for each field are the same as described for the non-container values in the previous section with the one caveat that length is a count of child elements and not the number of bytes representing the contents of the container.

Let’s look at a quick example of encoding an object, again using the handy [ ]-notation we used before simply for readability (the [ ] chars are not written out or part of the file format).

Consider the following JSON (30-bytes compacted):

{
    "id": 1234567890,
    "name": "bob"
}

Storing that object in the Universal Binary JSON format would look like this (whitespace added for readability):

[o][2] 2 bytes
  [s][2][id][I][1234567890] 4 + 5 = 9 bytes
  [s][4][name][s][3][bob]   6 + 5 = 11 bytes

Our Universal Binary JSON format is 22 bytes, 27% smaller than our compacted JSON!

Walking through our example above, using a word-journey this is what a parser might see and do:

1. I see an o, so I know I am parsing an object and that the next byte is the length (or count) for this object.
2. I see a 2, so I know the object contains 2 elements that I must account for to know when the object scope is closed (because we don’t use the { } brackets like JSON).
3. I see an s, knowing how the name-value pairings inside of an object work, I know this is the name portion of some upcoming value.
4. I see an I, I know this is an int32 value and that it belongs to the name I parsed in the previous step.
5. I see another s, I know this is a new name-value pair and this is the name portion.
6. I see another s and know this is the value belonging to the name I just processed.
7. I have just parsed 2 values, so now I know the object scope is closed.

Encoding objects containing other objects would work identically except we would have encountered another o or O marker and descended a level further into a new object.

Let’s look at another example, this time a simple JSON array construct (remember, they only contain values and not name-value pairs like objects).

This array is 48-bytes in compacted JSON:

[
  null,
  true,
  false,
  4782345193,
  153.132417549,
  "ham"
]

Storing the array in the Universal Binary JSON format would look like this (whitespace added for readability):

[a][6] - 2 bytes
   [Z] - 1 byte
   [T] - 1 byte
   [F] - 1 byte
   [I][4782345193] - 5 bytes
   [D][153.132417549] - 9 bytes
   [s][3][ham] - 5 bytes

Our Universal Binary JSON format is 24 bytes or 50% smaller than the compacted JSON!

Because the container types specify their total child element count, it is easier and faster for parsers to know when the scope of a container has closed or is still open waiting for more children (e.g. in the case of streaming over the network). This is not unlike the high-performance Redis protocol.

This also has the added benefit of not needing any terminating values in the binary that need to be scanned for to know when a container-scope is closed. This way data can be read in chunks and not read-and-scanned byte-by-byte.

As was mentioned previously though, there are some cases where having an unbounded container are important (for example, streaming content from a server as it generates it on-the-fly).

In the next section we will take a look at the Universal Binary JSON constructs that are optimized for streaming. Fortunately, there are only 3 and they are just as easy as the constructs we have covered so far!

No-Op Type¶

The noop value stands for No Op or No Operation, it is a specific value (like Z for null, T for true and F for false) that is useful in streaming scenarios where an acknowledge of life needs to be sent between two end points, but the confirmation being sent cannot change the meaning of the data it is sent within.

The most common use for such a value type is as a keep-alive signal from a server to the client; letting the client know the server is possibly operating on a long-running job and is still alive, but just isn’t ready to send more data yet.

The noop type is defined as follows:

Any parser code written to load the Universal Binary Spec needs to be aware that encountering the N marker in files of any kind is valid and is merely useful as a signal mechanism from producer to consumer to say “Hey, I am still alive.”, the marker is intended to be safely ignored if the server or client doesn’t need the acknowledgement.

In order for this keep-alive-esque construct to work, the specification had to define a single byte value that had no meaning for the server and client to exchange if needed, but caused no modification to the meaning of the data that they are exchanging.

In code that handles streaming from a server, your handler for the noop type might just reset a disconnect timer. In code that handles UBJ files, you would simply ignore the noop marker when you encountered it in the file because it would mean nothing.

If your interaction with the Universal Binary JSON format is primarily as a file format, it is likely that you may never need to use the noop type; its value becomes more apparent in long-lived, client-server, data-streaming scenarios.

Unknown-Length Containers¶

The Universal Binary JSON specification supports containers (array and object) of unknown length to be specified when the producer of the binary data cannot (efficiently) know in advance how many elements it is going to write out.

In these cases, the lowercased, 1-byte-length versions of array or object must be used (a or o markers) with a length value of 0xFF (255) as well as specifying an E terminator character after the last element in the container.

The E type used to delimit the end of unknown-length containers is defined as follows:

An example would look like this:

[a][255]
   [S][3][bob]
   [I][1024]
   [T]
   [F]
   [S][4][ham!]
[E]

The three key elements being the lowercased a marker, the length of 0xFF (255) and the E marker at the end of the container.

Another example might look like this:

[o][255]
   [B][4]
   [D][21.786]
   [N]
   [Z]
   [h][27][131.098412283059e2371293452]
[E]

You might notice in the example above we injected a noop instruction right in the middle, before the null. As mentioned in the No-Op Type section, this is valid and can occur at any time while parsing the contents of an unknown-length container.

If your parser has no need for recognizing the noop code (e.g. listening for a keep-alive) then it can just be safely ignored and parsing continued (hence the name “no-op”). It is up to the implementation to decide what to do with the noop type.

You might be wondering how using a 1-byte E as a terminator to an unbounded container can work and not get confused with say another E inside of a string, the reason this works is because none of the discrete value types (numeric, string, etc.) are of unknown length.

The lengths of all the values contained inside of the container are known and must be read completely, doing so will guarantee that the E is only ever encountered by itself as an element marker which is easily handled by parsing code to know the scope of the container has been closed.

Size Reduction Tips¶

The amount of storage size reduction you’ll experience with the Universal Binary JSON format will depend heavily on the type of data you are encoding.

Some data shrinks considerably, some mildly and some not at all, but in every case your data will be stored in a much more efficient format that is faster to read and write.

Below are pointers to give you an idea of how certain data may shrink in this format:

• null, true and false values will compress 75% (80% in the case of false)
• large numeric values (> 5 digits < 20 digits) will compress on average 50%.
• string values * of length <= 254 stay the same size. * of length > 254 are 3-bytes bigger per string.
• object and array values compress 1-byte-per-element.

One of the great things about the Universal Binary JSON format is that even though most all your data will be represented in a smaller footprint, you still get two big wins:

1. A smaller data format means faster writes and smaller reads. It also means less data to process when parsing.
2. Binary format means no encoding/decoding primitive values to text and no parsing primitive values from text.

Handling huge Numbers¶

Not every language supports arbitrarily long numbers greater than 64-bits (represented by the huge data type), but many do.

If you are writing a library to read/write Universal Binary JSON and the platform you are working with does not support them, we recommend throwing an exception or returning an error to the caller, letting them know unsupported data is contained in the file they are trying to parse.

If the library you are writing is meant to be a general-purpose parser and needs to be more resilient than that, we recommend the following:

1. Make the default behavior to throw an exception or return an error when the unsupported huge data type is encountered.
2. Provide an optional behavior to the parser (that must be specifically enabled by the caller) that treats the huge value as a simple string and returns it to the caller to handle (e.g. insert in a database) if they need it.
3. Provide an optional behavior to the parser (again, that must be specifically enabled by the caller) to simply skip unsupported values.

This implementation should give the user the most functional experience working with your library and the Universal Binary JSON format while making it clear on their particular platform some data types could cause trouble; this is preferred to making the default operation to ignore the unsupported values.