Compatibility Testing with JSON Schemas

In modern software development, data exchange is often facilitated using structured formats like JSON. As applications evolve, so too do their data structures, frequently defined and enforced using JSON Schemas. Ensuring that different versions of a schema, or data produced against one schema version, remain compatible with systems expecting another version is critical for maintaining stability and preventing unexpected runtime errors. This is where JSON Schema compatibility testing comes in.

What is a JSON Schema?

JSON Schema is a powerful tool for defining the structure, content, and format of JSON data. Think of it as a blueprint or contract for your JSON documents. It allows you to specify:

• Which properties an object should have
• The data types of values (string, number, boolean, array, object, null)
• Required vs. optional properties
• Patterns for strings (regex)
• Ranges for numbers
• Minimum/maximum items for arrays
• And much more...

Using JSON Schema provides a standardized way to validate data, automatically document APIs, and generate code, but it also introduces the challenge of managing schema evolution.

Why Test Schema Compatibility?

Schema changes are inevitable as features are added or modified. Without proper compatibility testing, evolving your schema can break systems that rely on the old structure. Key reasons to test include:

• API Evolution: Ensure clients using an older API version can still understand responses from a newer version, and vice versa.
• Data Validation: Verify that data conforms to the expected structure before processing, catching errors early.
• Preventing Runtime Errors: Avoid application crashes caused by unexpected or missing data fields.
• Reliable Integrations: Guarantee smooth data exchange between different services or microservices using JSON.
• Clear Data Contracts: Reinforce the agreed-upon format between producers and consumers of data.

Types of Compatibility

When discussing schema changes, we often talk about two main types of compatibility:

Backward Compatibility

A new schema version is backward compatible with an old version if data that is valid according to the old schema is also valid according to the new schema.

This is crucial for evolving a producer (like an API) while ensuring existing consumers (using the old schema) can still process the data without breaking. Examples of backward-compatible changes include:

• Adding an optional property.
• Adding a new value to an enum.
• Making a required property optional.

Forward Compatibility

A new schema version is forward compatible with an old version if data that is valid according to the new schema is also valid according to the old schema.

This is less common in practice but can be relevant if you need to roll back a producer to an older version while newer consumers might still be sending data validated against the new schema. Examples of forward-compatible changes are often limited and stricter:

• Potentially, some very minor constraint relaxations, but it's tricky.
• Generally, forward compatibility is hard to guarantee with significant changes.

The most common and often required compatibility is backward compatibility.

Methods for Testing Compatibility

Testing compatibility involves verifying that data validated by one schema can be handled by a system designed for another. Here are common approaches:

1. Data Validation Against Schemas

The most practical way to test compatibility is to use JSON Schema validation libraries. You can take a set of JSON documents and validate them against different schema versions.

To check backward compatibility between Schema A (old) and Schema B (new):

• Take data known to be valid against Schema A.
• Validate this data against Schema B.
• If all such data is valid against Schema B, then Schema B is backward compatible with Schema A.

Similarly, for forward compatibility between Schema A (old) and Schema B (new):

• Take data known to be valid against Schema B.
• Validate this data against Schema A.
• If all such data is valid against Schema A, then Schema B is forward compatible with Schema A.

This requires having a comprehensive suite of test data that covers various valid cases for each schema version.

2. Automated Schema Comparison

Some tools and libraries exist specifically to compare two JSON Schemas and report on the types of changes made (e.g., field added, field removed, type changed, constraint added/removed). These tools can often analyze the changes structurally and determine compatibility types (backward, forward, or breaking) without needing test data.

While powerful, these tools might not catch all subtle issues that validation with real data might reveal, especially with complex schemas involving conditional logic or custom keywords.

Practical Example: Schema Evolution

Let's consider a simple user schema and how changes affect compatibility.

Schema V1:

{
  "type": "object",
  "properties": {
    "id": { "type": "integer" },
    "username": { "type": "string" },
    "email": { "type": "string", "format": "email" }
  },
  "required": ["id", "username", "email"],
  "additionalProperties": false
}

Schema V2 (Backward Compatible Change - Adding Optional Field):

We add an optional fullName field.

{
  "type": "object",
  "properties": {
    "id": { "type": "integer" },
    "username": { "type": "string" },
    "email": { "type": "string", "format": "email" },
    "fullName": { "type": "string" } // Added optional field
  },
  "required": ["id", "username", "email"], // 'required' list didn't change
  "additionalProperties": false
}

Data valid under V1 (without fullName) is still valid under V2 because the new field is optional. V2 is backward compatible with V1.

Data valid under V2 (with or without fullName) might *not* be valid under V1 if it includes fullName and V1 has "additionalProperties": false (as shown above). V2 is NOT forward compatible with V1 (due to additionalProperties: false). If additionalProperties were `true` or omitted, it would be forward compatible for adding optional fields.

Schema V3 (Breaking Change - Removing Required Field):

We remove the email field and make username optional.

{
  "type": "object",
  "properties": {
    "id": { "type": "integer" },
    "username": { "type": "string" },
    // "email" field removed
    "fullName": { "type": "string" }
  },
  "required": ["id"], // "username" and "email" removed from required
  "additionalProperties": false
}

Data valid under V1 (which requires email and username) will NOT be valid under V3 if email is missing or username is missing (though V1 data will have username). The removal of a required field is a breaking change. V3 is NOT backward compatible with V1.

Schema V4 (Breaking Change - Changing Type):

We change the type of id from integer to string.

{
  "type": "object",
  "properties": {
    "id": { "type": "string" }, // Changed type from integer
    "username": { "type": "string" },
    "email": { "type": "string", "format": "email" }
  },
  "required": ["id", "username", "email"],
  "additionalProperties": false
}

Data valid under V1 (where id is an integer) will NOT be valid under V4 (which expects a string). V4 is NOT backward compatible with V1.

Implementing the Tests

In a real-world scenario, you would automate these checks. This typically involves:

1. Defining your schemas (e.g., in .json or .yaml files).
2. Creating test data files for each schema version that cover various valid instances.
3. Using a JSON Schema validation library in your test suite (e.g., Jest, Mocha).
4. Writing tests that load old data and validate it against the new schema.

Conceptual Validation Test (TypeScript):

// Assume you have JSON schema objects loaded as constants
const schemaV1 = {...}; // JSON schema object for V1
const schemaV2 = {...}; // JSON schema object for V2 (e.g., adding optional field)

// Assume you have test data loaded as constants
const validDataV1 = {...}; // JSON object valid against V1
const anotherValidDataV1 = {...}; // Another JSON object valid against V1

// Assume you have a validation function (e.g., from a library like 'ajv')
// import Ajv from 'ajv';
// const ajv = new Ajv();
// const validateV2 = ajv.compile(schemaV2);
function isValid(schema: any, data: any): boolean {
  // This is where a real validation library would be used
  // For concept: return some_validation_library.validate(schema, data);
  // Placeholder logic (DO NOT USE IN PRODUCTION):
  try {
     // Simplified check: just stringify and compare basic structure/fields
     // This is NOT real schema validation!
     JSON.stringify(data); // Check if it's valid JSON
     if (schema.required && schema.required.some(field => data[field] === undefined)) return false;
     // ... more complex checks needed for types, formats, etc.
     console.warn("Using simplified placeholder validation.");
     return true; // Assume valid for conceptual example
  } catch (e) {
     return false;
  }
}


// --- Test for Backward Compatibility (V2 with V1) ---
// Test if data valid under V1 is still valid under V2
const test1Result = isValid(schemaV2, validDataV1);
const test2Result = isValid(schemaV2, anotherValidDataV1);

if (test1Result && test2Result) {
  console.log("Schema V2 is likely backward compatible with V1 (based on test data).");
  // In a test suite: expect(test1Result).toBe(true); expect(test2Result).toBe(true);
} else {
  console.error("Schema V2 is NOT backward compatible with V1 (at least one test failed).");
  // In a test suite: fail("Backward compatibility test failed");
}

// --- Test for Forward Compatibility (V2 with V1) ---
// This would require data valid under V2 (e.g., with the new optional field)
const validDataV2 = { ...validDataV1, fullName: "Test User" };

const test3Result = isValid(schemaV1, validDataV2);

if (test3Result) {
    console.log("Schema V2 is likely forward compatible with V1 (based on test data).");
    // In a test suite: expect(test3Result).toBe(true);
} else {
    console.error("Schema V2 is NOT forward compatible with V1 (at least one test failed).");
    // In a test suite: fail("Forward compatibility test failed");
}

// Note: Real tests use actual validation library methods like validate(data) or ajv.compile(schema)(data)
// which return boolean or include errors array.

A successful backward compatibility test means your data producer can safely upgrade its schema without breaking older consumers.

A failed backward compatibility test indicates a breaking change; you might need to support multiple schema versions or implement a migration strategy.

Best Practices

• Version Your Schemas: Clearly label schema versions (e.g., /schemas/user/v1.json, /schemas/user/v2.json).
• Automate Tests: Integrate compatibility tests into your Continuous Integration (CI/CD) pipeline. Every schema change should automatically trigger these checks.
• Maintain Comprehensive Test Data: Build a rich suite of valid JSON examples for each schema version, covering edge cases and all defined constraints.
• Prefer Backward Compatible Changes: Design schema evolutions to be backward compatible whenever possible (e.g., adding optional fields instead of removing required ones).
• Document Changes: Clearly document schema changes and their compatibility implications for consumers.
• Use Schema Comparison Tools: Augment data validation tests with structural schema comparison tools for a more complete picture.

Tools and Libraries

Popular libraries for JSON Schema validation in JavaScript/TypeScript include:

• AJV (Another JSON Schema Validator): Fast and comprehensive validator.
• json-schema-validator: A lightweight alternative.
• For schema comparison tools, search for "JSON schema diff" or "JSON schema compatibility checker".

Note that while libraries like Zod or Valibot are excellent for defining and validating data shapes in TypeScript, they often use their own DSLs rather than standard JSON Schema. Ensure the tool/library you choose supports the standard JSON Schema specification if that is your requirement.

Conclusion

Compatibility testing is not an optional step but a fundamental practice when managing evolving JSON schemas, especially in API development and distributed systems. By proactively testing backward and forward compatibility using a combination of data validation and schema comparison tools, you can significantly reduce the risk of introducing breaking changes, ensure smoother deployments, and build more robust applications.

Embrace schema versioning and automated compatibility checks in your development workflow to maintain reliable data contracts between different parts of your system and external consumers.