Property-Based Testing for JSON Parser Components

Testing a JSON parser might seem straightforward at first. You provide some valid JSON strings and check if the output matches the expected JavaScript object or array. You also provide some invalid JSON and check if it throws the correct errors. This is called Example-Based Testing (EBT).

While necessary, EBT has a fundamental limitation: the number of possible JSON strings, especially complex and deeply nested ones, is practically infinite. How can you be confident your parser handles all the edge cases, combinations, and structures it might encounter in the wild?

This is where Property-Based Testing (PBT) shines.

What is Property-Based Testing?

Instead of testing with specific examples, PBT focuses on testing general properties that your code should satisfy for *all* valid inputs within a certain domain.

A PBT framework typically involves:

• Arbitraries (Generators): Tools to generate a wide variety of random, valid inputs for your function, based on a definition you provide (e.g., "generate any integer", "generate any list of strings", or "generate any valid JSON value").
• Properties: Functions that take the generated input(s) and return `true` if the property holds for that input, and `false` otherwise.
• Testing Engine: Runs the property function many times (hundreds or thousands) with randomly generated inputs. If a property fails for an input, it reports a counterexample.
• Shrinking: If a counterexample is found, the framework tries to find the *smallest* possible input that still fails the property. This makes debugging much easier.

Why PBT for JSON Parsers?

JSON has a clear, recursive, and potentially deep structure.

A JSON parser needs to handle:

• All primitive types: strings (with complex escapes), numbers (integers, floats, exponents, signs), booleans, null.
• Arbitrarily nested arrays of any value type.
• Arbitrarily nested objects with string keys and any value type.
• Combinations of objects and arrays nesting within each other.
• Empty objects and arrays.
• Various whitespace permutations (though typically ignored).

Generating a comprehensive set of EBT examples for this space is nearly impossible. PBT, by generating inputs based on the *structure* of JSON, can explore this space far more effectively and uncover bugs that hand-written examples might miss.

Defining Properties for a JSON Parser

What fundamental truths should always hold about a correct JSON parser and stringifier?

The Round Trip Property (Parse then Stringify)

The most classic PBT property for parsers/serializers. If you take a valid JavaScript value that *can* be represented as JSON, stringify it, and then parse the resulting string, you should get back the original value.

function propertyRoundTripValue(value): boolean {
  const jsonString = stringify(value);
  const parsedValue = parse(jsonString);
  return deepEquals(value, parsedValue); 
}

The Round Trip Property (Stringify then Parse)

If you take a valid JSON string, parse it, and then stringify the resulting value, you should get back a JSON string that parses to the *same* value as the original string.

function propertyRoundTripString(jsonString): boolean {
  if (!isValidJsonString(jsonString)) return true; 
  try {
    const parsedValue1 = parse(jsonString);
    const jsonString2 = stringify(parsedValue1);
    const parsedValue2 = parse(jsonString2);
    return deepEquals(parsedValue1, parsedValue2);
  } catch (e) {
    return false; 
  }
}

Type and Value Preservation

For any primitive JSON value (number, string, boolean, null), parsing it should result in the corresponding JavaScript primitive with the correct value.

function propertyNumberPreservation(num: number): boolean {
  const jsonString = stringify(num); 
  const parsedValue = parse(jsonString);
  return typeof parsedValue === 'number' && parsedValue === num;
}


function propertyStringPreservation(str: string): boolean {
  const jsonString = stringify(str); 
  const parsedValue = parse(jsonString);
  return typeof parsedValue === 'string' && parsedValue === str;
}

Structural Preservation (Arrays and Objects)

Parsing an array or object should result in a JavaScript array or object with the same number of elements/keys, and recursively, each element/value should also satisfy the properties.

function propertyArrayLengthPreservation(arr: any[]): boolean {
  const jsonString = stringify(arr);
  const parsedValue = parse(jsonString);
  return Array.isArray(parsedValue) && parsedValue.length === arr.length;
}

function propertyObjectKeyCountPreservation(obj: { [key: string]: any }): boolean {
  const jsonString = stringify(obj);
  const parsedValue = parse(jsonString);
  return typeof parsedValue === 'object' && parsedValue !== null
    && Object.keys(parsedValue).length === Object.keys(obj).length;
}

Handling Invalid JSON

A robust parser must correctly identify and reject invalid JSON. PBT can help generate invalid inputs.

function propertyInvalidInputThrows(invalidJsonString: string): boolean {
  
  try {
    parse(invalidJsonString);
    return false; 
  } catch (e) {
    return true; 
  }
}

Building Arbitraries for JSON

The power of PBT for JSON testing heavily relies on creating effective arbitraries that mimic the JSON structure. A good PBT library provides combinators to build complex arbitraries from simpler ones.

• Primitive Arbitraries: Generators for booleans, numbers (integers, floats, potentially NaN/Infinity if your parser handles them per standard or specific requirements), strings (important to include various characters and escape sequences), and the null value.
• Array Arbitrary: A generator that takes another arbitrary (for the element type) and generates arrays of random length containing elements generated by the inner arbitrary.
• Object Arbitrary: A generator that takes an arbitrary for keys (JSON keys must be strings) and an arbitrary for values, and generates objects with a random number of key-value pairs.
• Recursive Arbitrary (JSON Value): This is the core. An arbitrary that can generate *any* valid JSON value. It's defined recursively: a JSON value is *either* a primitive, *or* an array (where elements are JSON values), *or* an object (where values are JSON values). PBT frameworks handle the recursion depth to avoid infinite generation.

By combining these, you can generate highly complex and varied JSON structures that would be impractical to write manually.

Practical Considerations

When implementing PBT for a JSON parser:

• Comparison: Ensure your deep equality check for parsed values correctly handles object key order and potential floating-point inaccuracies.
• Test Coverage: While PBT is powerful, it complements, rather than replaces, EBT. Use EBT for specific, known edge cases and invalid syntax examples.
• Performance: Generating and testing thousands of complex structures can be slow. Tune your arbitraries (e.g., limit recursion depth or array/object size for some test runs) if needed.
• Shrinking: Pay attention to the shrunk counterexamples reported by the framework. They are often the most illuminating!

Conclusion

Property-Based Testing is an incredibly valuable technique for building confidence in complex components like JSON parsers. By shifting the focus from specific examples to general properties and using powerful data generation tools, you can explore the vast input space of JSON far more effectively than with traditional example-based tests. This leads to more robust parsers and fewer surprises in production when encountering unexpected, yet valid, JSON structures. Adopting PBT requires a shift in mindset, but the effort is often richly rewarded by the number and subtlety of bugs it can uncover.