Exploratory Testing Strategies for JSON Formatters

JSON (JavaScript Object Notation) is ubiquitous. It's the de facto standard for data exchange on the web and in countless applications. As developers, we often work with JSON data, and tools like JSON formatters, validators, and visualizers are essential for understanding, debugging, and working with this data effectively.

While automated testing is crucial, sometimes the most interesting and critical issues are found through exploratory testing. Exploratory testing is an approach where testers or developers actively design tests as they execute them, using their creativity, intuition, and understanding of the system to explore its behavior, particularly at its boundaries and under unusual conditions.

This article outlines exploratory testing strategies specifically for JSON formatters and related tools, aiming to uncover bugs or unexpected behaviors that automated tests might miss.

Why Explore JSON Formatters?

JSON formatters seem simple: they take JSON text and output the same text, but nicely indented and structured. What could go wrong? Plenty!

• Invalid Input Handling: How does it behave with malformed JSON? Does it crash, give a helpful error, or produce incorrect output?
• Edge Cases: What about empty objects/arrays, deeply nested structures, or strings with special characters?
• Performance: How does it handle extremely large JSON files?
• Formatting Options: If it has options (like indent size), do they work correctly?
• Character Encoding: Does it handle Unicode characters correctly?

Exploratory testing helps uncover these scenarios by encouraging "thinking outside the box" inputs and observations.

Core Strategies for Exploratory Testing

When exploring a JSON formatter, consider these angles:

1. Valid JSON Inputs

Start with valid JSON, but push the boundaries.

• Simple Structures: Test basic objects and arrays with different data types (strings, numbers, booleans, null).

  {
    "name": "Test",
    "version": 1.0,
    "enabled": true,
    "data": null,
    "emptyArray": [],
    "emptyObject": {}
  }

• Complex Nesting: Create deeply nested objects and arrays. Does the formatter handle indentation levels correctly? Are there depth limits?

  {
    "a": {
      "b": {
        "c": {
          "d": [
            [
              { "e": 1 }
            ]
          ]
        }
      }
    }
  }

• Large Objects/Arrays: Test objects with thousands of keys or arrays with thousands of elements. Does performance degrade significantly?
• Long Strings: Include strings that are very long. Does the formatter wrap them or handle them appropriately?
• Special Characters: Include strings with escaped characters (`\n`, `\t`, `\"`, `\\`, `\/`), Unicode characters (including astral plane characters like emojis), and characters that might interact with indentation (like newlines *within* a string).

  {
    "newlineString": "This has a\nnewline inside.",
    "quoteString": "Includes a \"quote\".",
    "unicodeString": "Hello world 👋",
    "escapedUnicode": "\u0048\u0065\u006c\u006c\u006f" // "Hello"
  }

• Numbers: Test integers, decimals, large numbers, small numbers, zero, negative numbers, and numbers using scientific notation (`1e+10`, `-2.5e-5`).

2. Invalid JSON Inputs

This is a prime area for finding robustness issues. Feed the formatter things that are *not* valid JSON.

• Syntax Errors: Missing commas, colons, brackets, braces, unquoted keys, trailing commas (if the formatter doesn't explicitly support them), incorrect escaping.

  Examples of Invalid JSON to try:

  { "key": "value" 

  { key: "value" }

  { "key": "value", }

  [1, 2 3]

  "unterminated string

• Incorrect Data Types: `NaN`, `Infinity`, `-Infinity`, `undefined` as values (not valid JSON primitives).
• Comments: JSON doesn't support comments (`//` or `/* */`). How does the formatter handle them? Does it produce valid JSON by stripping them, or does it error?

  {
    // This is a comment
    "key": "value" /* Another comment */
  }

• Non-JSON Content: HTML, XML, plain text, binary data. Does it fail gracefully?
• BOM (Byte Order Mark): Some files might start with a BOM. Does the formatter handle it?

3. Formatting Options

If the formatter offers options (indentation size, sort keys, compact output), test each option and combinations.

• Indent Size: Test indentation with 0 spaces (compact), 1 space, 2 spaces (common), 4 spaces (common), and perhaps very large numbers like 100 spaces. Does it handle tabs vs. spaces correctly if that's an option?
• Sorting Keys: If there's a "sort keys" option, test objects with keys in different orders (alphabetical, reverse alphabetical, mixed). Does the output consistently sort them? How does it handle keys with different character sets or cases?
• Compact Output: Test the option to remove all unnecessary whitespace. Does it produce valid, single-line JSON?

  {"a":1,"b":[2,3]}

4. Scale and Performance

How does the formatter cope with large amounts of data?

• Large Files: Find or generate JSON files that are MBs or GBs in size. Does the formatter become unresponsive, crash, or consume excessive memory/CPU?
• Deep Nesting: Test JSON with thousands of nested levels (e.g., `[[[[...]]]]`). Recursive algorithms might hit stack limits.
• Large Arrays/Objects: Test single arrays with hundreds of thousands or millions of elements, or objects with a similar number of keys.

5. Character Encoding

While JSON is specified to be UTF-8, UTF-16, or UTF-32, UTF-8 is the dominant encoding. Ensure the formatter handles various characters correctly.

• Non-ASCII Characters: Include characters from different languages (Cyrillic, Arabic, Chinese, etc.) in keys and values.
• Escaped vs. Literal Unicode: Test both literal Unicode characters (`"你好"`) and their escaped forms (`"\u4f60\u597d"`). A good formatter should preserve the original form or offer options.
• Invalid UTF-8 Sequences: Feed the formatter byte sequences that are not valid UTF-8. How does it react? Does it error or produce garbled output?

6. Comparison/Diffing

If the formatter is part of a tool that compares or diffs JSON, test how it handles formatting differences.

• Formatted vs. Compact: Compare a formatted version and a compact version of the *same* JSON. The diff tool should ideally ignore formatting whitespace.
• Different Formatting Styles: Compare the same JSON formatted with 2 spaces vs. 4 spaces.
• Sorted vs. Unsorted: If comparing objects with keys in different orders (but same content), a smart diff should recognize they are equivalent.

The "Oracle Problem"

A key challenge in testing, especially exploratory testing, is the "oracle problem": how do you know if the output is correct? For a JSON formatter, the output should be *semantically equivalent* to the input, just with different whitespace and potentially key order (if sorting).

To solve this, you can use a trusted, known-good JSON parser/serializer (like the built-in one in JavaScript or Python) as an oracle.

1. Take your input JSON (valid or invalid).
2. Pass it to the formatter under test.
3. If the formatter *claims* the input is invalid, check its error message. Is it helpful? Does it point to the correct location? Verify with an oracle parser.
4. If the formatter *claims* the input is valid and produces output:
   • Attempt to parse the formatter's output using your oracle parser. If the oracle parser fails, the formatter produced invalid JSON.
   • If the oracle parser succeeds, compare the parsed data structure from the formatter's output to the parsed data structure from the original input (if the original was valid). They should be identical.
   • Visually inspect the formatted output for correctness of indentation, line breaks, and character representation.

Using an oracle helps verify functional correctness beyond just visual appearance.

Tips for Effective Exploration

• Think like an Attacker: What kind of input would you provide to try and break the tool or make it behave unexpectedly?
• Vary the Input Source: Don't just paste text. If the tool accepts file uploads, test with different file sizes and encodings.
• Combine Strategies: Test large, deeply nested JSON with strings containing special characters and invalid syntax sprinkled in.
• Document Your Findings: Keep track of what you tested, the input, the output/behavior, and why you think it's a bug or an observation. This is crucial for reporting issues.
• Explore Around Bugs: If you find a bug with a specific input, try similar inputs to understand the boundaries of the bug.
• Don't Just Look at the Output: Observe how the tool behaves while processing. Does it freeze? Does it consume excessive resources?

Beyond Formatting: Related Tools

Many JSON tools combine formatting with other features. Apply similar exploratory strategies to:

• JSON Validators: How accurately do they identify invalid JSON? Do error messages help locate the issue?
• JSON Parsers/Serializers (Libraries): While not a "formatter" tool, the underlying parser/serializer logic is what formatters often use. Testing these directly with boundary inputs can reveal fundamental issues.
• JSON Visualizers/Tree Views: How do they render complex or large structures? Do they handle malformed JSON gracefully or crash?

Conclusion

Exploratory testing is a valuable complement to automated testing, especially for tools that process complex data formats like JSON. By creatively devising inputs, observing behavior, and using oracles to check correctness, you can uncover subtle bugs and robustness issues in JSON formatters and related tools that might otherwise go unnoticed. So, open up your favorite JSON formatter, put on your testing hat, and start exploring its limits!