Building JSON Formatter Test Automation Frameworks

JSON formatters are essential tools for developers working with structured data. They take unformatted or minified JSON strings and produce human-readable, indented output. Ensuring the correctness and consistency of a JSON formatter is crucial, especially if it's part of a larger system or library. While manual checks suffice for small changes, an automated test framework provides reliability, speed, and repeatability. This article explores how to build such a framework.

Why Automate Testing for JSON Formatters?

• Consistency: Ensure the formatter always produces the same output for the same input, regardless of execution environment or minor code changes.
• Correctness: Verify that the output is valid JSON and adheres to standard formatting rules (indentation, spacing).
• Regression Prevention: Catch unintended side effects when code is modified or updated.
• Efficiency: Quickly test a wide range of inputs and edge cases that would be time-consuming to check manually.
• Support for Variations: Easily test different formatting options (indentation size, sort keys, etc.).

Core Components of the Framework

A robust test automation framework for a JSON formatter typically consists of several key parts:

Input Generation Strategies

Creating varied inputs is key to comprehensive testing.

• Manual Samples: Create a directory of .json files representing typical and edge cases. This is simple and effective for specific scenarios.
• Generated Samples: Use libraries or custom code to generate random or structured JSON data, controlling depth, type distribution, and size.
• Real-world Samples: Use anonymized JSON data from actual applications or APIs.
• Mutation Testing Inputs: Take valid JSON and introduce small, controlled errors to test invalid input handling.

interface TestCase {
  name: string;
  input: string;
  expectedOutput: string;
  isExpectedToFail?: boolean; // For testing invalid inputs
}

const testCases: TestCase[] = [
  {
    name: "Simple Object",
    input: '{"a":1,"b":[2,3]}',
    expectedOutput: `{
  "a": 1,
  "b": [
    2,
    3
  ]
}`, // Assuming 2-space indentation
  },
  {
    name: "Empty Array",
    input: '[]',
    expectedOutput: '[]',
  },
  {
    name: "Invalid JSON",
    input: '{"a":',
    expectedOutput: "SyntaxError", // Or specific error message/type
    isExpectedToFail: true,
  },
  // ... more test cases
];

Comparison Logic: String vs. Structure

Comparing the actual and expected output strings directly might seem sufficient, but it has limitations. A more robust approach compares the underlying data structure (Abstract Syntax Tree or parsed object).

String Comparison

The simplest method: compare the actual output string character by character with the expected output string.

actualOutput === expectedOutput

• Pros: Easy to implement. Highlights *exact* differences, including whitespace and indentation.
• Cons: Extremely brittle. A single extra space or newline anywhere will cause a test failure, even if the JSON structure is correct. Requires the expected output string to be pixel-perfect.

Structural (Parsed Object) Comparison

Parse both the actual output string and the expected output string into their native language data structures (e.g., JavaScript objects/arrays) and compare these structures recursively.

JSON.parse(actualOutput) === JSON.parse(expectedOutput) (conceptually, requires deep comparison)

• Pros: Ignores differences in whitespace, indentation, and object key order (though JSON formatters often fix key order). Tests if the *content* and *structure* of the formatted JSON are correct. Much less brittle than string comparison.
• Cons: Doesn't verify the specific formatting (indentation, spacing). If the formatter outputs valid JSON but with incorrect indentation, this test might pass incorrectly. Requires a trusted JSON parser.

Recommended Approach: Combine Both

For comprehensive testing, use both methods:

1. First, parse both actual and expected outputs and compare the structures. If this fails, the formatter produced structurally invalid or incorrect JSON. Pass if structures match.
2. If the structures match, then compare the strings directly. This verifies the specific formatting. Pass only if strings also match.
3. For invalid inputs, verify that the formatter throws an error and potentially check the error type or message. Pass if expected error occurs.

Implementing the Test Runner

A simple test runner can iterate through test cases, execute the formatter, perform comparisons, and report results.

// Assuming 'formatJson' is the function being tested
// Assuming 'testCases' array exists from previous example

function runTests(formatJson: (jsonString: string, options?: any) => string): void {
  let passed = 0;
  let failed = 0;

  testCases.forEach(test => {
    console.log(`
Running test: ${test.name}`);
    try {
      const actualOutput = formatJson(test.input);

      if (test.isExpectedToFail) {
        console.error(`❌ FAILED: Test was expected to fail but succeeded.`);
        failed++;
        return; // Skip comparison for unexpected success
      }

      // 1. Structural Comparison
      let actualParsed, expectedParsed;
      try {
        actualParsed = JSON.parse(actualOutput);
        expectedParsed = JSON.parse(test.expectedOutput);
      } catch (parseError: any) {
        console.error(`❌ FAILED (Parse): Could not parse actual or expected output.`, parseError.message);
        console.log(`Actual output:
${actualOutput}`);
        console.log(`Expected output:
${test.expectedOutput}`);
        failed++;
        return;
      }

      // Simple deep comparison (requires helper or library for real world)
      const structuralMatch = JSON.stringify(actualParsed) === JSON.stringify(expectedParsed);

      if (!structuralMatch) {
         console.error(`❌ FAILED (Structural): Outputs have different structure.`);
         console.log(`Actual parsed:`, actualParsed);
         console.log(`Expected parsed:`, expectedParsed);
         failed++;
         return;
      }

      // 2. String Comparison (if structural match)
      if (actualOutput !== test.expectedOutput) {
        console.error(`❌ FAILED (String): Outputs match structurally but not exactly.`);
        // Implement a diffing utility here for better reporting
        console.log(`Actual output:
---
${actualOutput}
---`);
        console.log(`Expected output:
---
${test.expectedOutput}
---`);
        // A basic diff indicator (conceptual)
        // console.log(getDiff(test.expectedOutput, actualOutput));
        failed++;
      } else {
        console.log(`✅ PASSED.`);
        passed++;
      }

    } catch (error: any) {
      if (test.isExpectedToFail) {
        console.log(`✅ PASSED (Expected Error): Caught error as expected: ${error.message}`);
        passed++;
      } else {
        console.error(`❌ FAILED (Unexpected Error): ${error.message}`);
        failed++;
      }
    }
  });

  console.log(`
--- Test Summary ---`);
  console.log(`Passed: ${passed} ✅`);
  console.log(`Failed: ${failed} ❌`);
}

// To run:
// runTests(yourFormatterFunction);

This conceptual example uses basic comparisons. A real framework would use a testing library (like Jest, Mocha) for structure, assertions, and reporting, and potentially a dedicated diffing library for string comparison failures.

Handling Invalid JSON

A robust formatter should handle invalid JSON gracefully, typically by throwing a syntax error. Your tests should specifically cover these cases.

// Added to the testCases array
{
  name: "Unclosed Bracket",
  input: '[1, 2, 3',
  expectedOutput: 'SyntaxError', // Expecting an error
  isExpectedToFail: true,
},
{
  name: "Trailing Comma",
  input: '{"a": 1,}', // Invalid in strict JSON
  expectedOutput: 'SyntaxError',
  isExpectedToFail: true,
},
{
  name: "Invalid Escape Sequence",
  input: '"Hello \uZZZZ"',
  expectedOutput: 'SyntaxError',
  isExpectedToFail: true,
},
// ... more invalid cases

When testing invalid inputs, the assertion is that calling the formatter *throws an error* rather than returning a string. You might want to assert the type of error or check if the error message contains specific keywords.

Benefits Beyond Catching Bugs

Building this framework offers several advantages:

• Documentation: The test cases themselves serve as examples of how the formatter behaves with various inputs.
• Refactoring Confidence: You can make significant changes to the formatter's internal logic with confidence, knowing the tests will alert you to regressions.
• Feature Development: When adding new formatting options (e.g., sorting keys, compact output), you can add specific tests for those features.
• Performance Testing: The framework can be extended to measure formatting time for large inputs.

Potential Challenges

Be aware of potential hurdles:

• Defining Expected Output: Manually creating and maintaining expected output strings for complex cases can be tedious. Using a trusted reference formatter or a canonical generation function helps.
• Diffing Complexity: Simple string diffs can be hard to read for large outputs. Implementing or integrating a sophisticated diffing tool is beneficial.
• Floating Point Precision: Comparing numbers after parsing can sometimes hit floating-point issues if not handled carefully (though less common with standard JSON numbers).
• Performance: Parsing and comparing very large JSON structures can be slow; consider performance implications for your test suite.

Conclusion

Building an automated test framework for a JSON formatter is a valuable investment. It ensures the quality, consistency, and reliability of your formatter and provides confidence when making changes. By focusing on generating diverse inputs, defining clear expected outputs, and using appropriate comparison strategies (structural + string), you can create a robust system that saves time and prevents bugs in the long run. While the initial setup requires effort, the benefits for maintaining a high-quality formatter are significant.