Need help with your JSON?

Try our JSON Formatter tool to automatically identify and fix syntax errors in your JSON. JSON Formatter tool

Duplicate Keys in JSON: Detection and Resolution Strategies

One of the more subtle issues that can arise in JSON documents is the presence of duplicate keys within the same object. While many JSON formatters will parse such documents without error, duplicate keys can lead to unpredictable results and data loss. This article explores how duplicate keys are handled in different environments, how to detect them, and strategies for resolving these issues.

The Problem with Duplicate Keys

According to the JSON specification (ECMA-404 and RFC 8259), names within a JSON object SHOULD be unique, but the standard does not explicitly forbid duplicate keys. This has led to inconsistent handling across different parsers and environments.

Simple Example of Duplicate Keys:

{
  "user": "alice",
  "score": 42,
  "user": "bob"    // Duplicate key: "user"
}

In this example, the key "user" appears twice. While syntactically valid, this creates ambiguity about which value should be used.

How Different Environments Handle Duplicates

Behavior Across Environments:

Environment	Behavior with Duplicate Keys
JavaScript	Uses the last occurrence of the key (later values overwrite earlier ones)
Python	Depends on the version and library; typically uses the last value
Java (Jackson)	By default uses the last value, but can be configured to throw an error
Ruby	Uses the last occurrence
PHP	Last value wins
JSON Schema Validators	Many validate against duplicates and report them as errors

Important Warning:

Even though many environments "accept" duplicate keys, relying on this behavior is dangerous. The same JSON may be interpreted differently across systems, leading to subtle bugs and security vulnerabilities. Always ensure your JSON has unique keys.

Real-World Issues Caused by Duplicate Keys

1. Data Loss

When a parser encounters duplicate keys, one value inevitably overrides the other, causing data loss.

Data Loss Example:

// Original JSON with duplicate keys
{
  "settings": {
    "notifications": true,
    "theme": "dark",
    "notifications": false
  }
}

// After parsing in JavaScript
{
  "settings": {
    "notifications": false,  // The first value (true) was lost
    "theme": "dark"
  }
}

2. Security Vulnerabilities

Duplicate keys can be exploited to create security vulnerabilities, especially in systems that use JSON for configuration.

Security Example:

// Configuration JSON
{
  "user": {
    "role": "user",
    "permissions": ["read"],
    "role": "admin"  // Malicious duplicate could escalate privileges
  }
}

If the system parses this JSON and uses the last value for "role", it could result in unauthorized privilege escalation.

3. Inconsistent Cross-Platform Behavior

When the same JSON is processed by different systems, duplicate keys can lead to inconsistent application behavior.

Detecting Duplicate Keys

1. Using Linters and Validators

Many JSON linters and validators can detect duplicate keys during the validation process.

JSON Lint Example Output:

Error: Duplicate key 'user' on line 5
Parse error on line 5:
...  "role": "user",  "permissions":
----------------------^
Expecting 'STRING', 'NUMBER', 'NULL', 'TRUE', 'FALSE', '{', '['

2. Programmatic Detection

You can write custom code to detect duplicate keys during parsing or validation.

JavaScript Detection Example:

function detectDuplicateKeys(jsonString) {
  const duplicates = [];
  
  // Use a reviver function with JSON.parse
  JSON.parse(jsonString, (key, value, path = '') => {
    if (key === '') return value; // Root object
    
    // For objects, check for duplicate keys
    if (typeof value === 'object' && value !== null && !Array.isArray(value)) {
      const keys = {};
      const originalJson = jsonString.substring(
        jsonString.indexOf('{', jsonString.indexOf(key)) + 1,
        jsonString.indexOf('}', jsonString.indexOf(key))
      );
      
      // Use regex to find all keys in the original JSON string
      const keyPattern = /"([^"]+)"s*:/g;
      let match;
      while ((match = keyPattern.exec(originalJson)) !== null) {
        const foundKey = match[1];
        if (keys[foundKey]) {
          duplicates.push({
            key: foundKey,
            path: path + '.' + key
          });
        } else {
          keys[foundKey] = true;
        }
      }
    }
    
    return value;
  });
  
  return duplicates;
}

3. Custom JSON Parsers

For more complex scenarios, you may need to implement or configure a custom JSON parser that explicitly checks for duplicates.

Jackson Configuration (Java):

ObjectMapper mapper = new ObjectMapper();
mapper.enable(JsonParser.Feature.STRICT_DUPLICATE_DETECTION);

try {
    MyObject obj = mapper.readValue(jsonString, MyObject.class);
    // Processing successful
} catch (JsonParseException e) {
    // Will catch duplicate key errors
    System.err.println("Duplicate key found: " + e.getMessage());
}

Strategies for Resolving Duplicate Keys

1. Manual Deduplication

For smaller JSON documents, manually identifying and removing duplicate keys is a straightforward approach.

Before Deduplication:

{
  "product": "Laptop",
  "price": 999.99,
  "inStock": true,
  "price": 899.99
}

After Deduplication:

{
  "product": "Laptop",
  "price": 899.99,  // Kept the last/newer price
  "inStock": true
}

In this example, we manually resolved the duplicate "price" key by keeping only the last occurrence.

2. Automated Merging

For more complex cases, you might want to merge the values of duplicate keys according to specific rules.

JavaScript Merging Example:

function mergeJsonWithDuplicateKeys(jsonString) {
  const result = {};
  const tempObj = {};
  
  // First, construct an object that preserves all values
  JSON.parse(jsonString, (key, value) => {
    if (key === '') return value;
    
    if (!tempObj[key]) {
      tempObj[key] = [value];
    } else {
      tempObj[key].push(value);
    }
    
    return value;
  });
  
  // Then apply merging strategies based on key names or value types
  for (const key in tempObj) {
    const values = tempObj[key];
    
    if (values.length === 1) {
      // No duplicates, just use the value
      result[key] = values[0];
    } else {
      // Custom merging strategies based on the key name or value type
      if (Array.isArray(values[0])) {
        // Merge arrays
        result[key] = [].concat(...values);
      } else if (typeof values[0] === 'object' && values[0] !== null) {
        // Recursively merge objects
        result[key] = values.reduce((merged, obj) => ({...merged, ...obj}), {});
      } else if (key === 'permissions' || key.includes('list')) {
        // For specific keys that should be combined (like permission lists)
        result[key] = Array.from(new Set([].concat(...values.map(v => 
          Array.isArray(v) ? v : [v]
        ))));
      } else {
        // Default: use the last value
        result[key] = values[values.length - 1];
      }
    }
  }
  
  return result;
}

3. Key Renaming

Sometimes the best strategy is to rename duplicate keys to make them unique while preserving all values.

Before Renaming:

{
  "address": "123 Main St",
  "address": "Apt 4B"
}

After Renaming:

{
  "addressLine1": "123 Main St",
  "addressLine2": "Apt 4B"
}

4. Array Transformation

For cases where all values should be retained, convert duplicate keys into an array.

Before Transformation:

{
  "tag": "important",
  "tag": "urgent",
  "tag": "review"
}

After Transformation:

{
  "tags": ["important", "urgent", "review"]
}

Here, we've renamed the key to a plural form and converted the multiple values into an array.

Prevention: Best Practices

Validate JSON before processing or storing it, using tools that detect duplicate keys
Generate JSON programmatically rather than manually to reduce the risk of duplicates
Use JSON Schema to define and enforce your data structure
Implement duplicate key detection in your application's data validation pipeline
Configure parsers to be strict about duplicate keys where possible

JSON Schema Example for Validation:

{
  "$schema": "http://json-schema.org/draft-07/schema#",
  "type": "object",
  "required": ["product", "price"],
  "additionalProperties": false,
  "properties": {
    "product": {
      "type": "string"
    },
    "price": {
      "type": "number"
    },
    "inStock": {
      "type": "boolean"
    }
  }
}

Using a JSON Schema with additionalProperties: false helps ensure only defined properties are allowed, effectively preventing duplicate keys with different names.

Conclusion

Duplicate keys in JSON documents are a subtle but significant issue that can lead to data integrity problems, security vulnerabilities, and inconsistent application behavior. While JSON parsers may handle duplicates differently, best practice is to ensure keys are unique within each object.

By implementing proper detection mechanisms and following the resolution strategies outlined in this article, you can maintain the integrity of your JSON data across different systems and avoid the pitfalls associated with duplicate keys. As with many aspects of software development, prevention through validation and good tooling is ultimately more efficient than fixing problems after they occur.

Need help with your JSON?

Try our JSON Formatter tool to automatically identify and fix syntax errors in your JSON. JSON Formatter tool