Merging JSON Configurations Across Environments

Managing application configuration for different environments (like development, staging, and production) is a common task for developers. You often have settings that are shared across all environments (e.g., API endpoints, feature flags) and settings that are specific to each environment (e.g., database credentials, logging levels). Storing these settings in JSON files is a popular approach. However, simply replacing the entire configuration file based on the environment can lead to duplication and make updates tedious. This is where merging JSON configurations becomes incredibly useful.

The Challenge: Environment-Specific Settings

Imagine you have a configuration file for your application.

// config.development.json
{
  "apiUrl": "https://dev.api.example.com",
  "database": {
    "host": "localhost",
    "port": 27017,
    "name": "mydb_dev"
  },
  "logging": {
    "level": "debug"
  },
  "featureFlags": {
    "newDashboard": true
  }
}

// config.production.json
{
  "apiUrl": "https://api.example.com",
  "database": {
    "host": "prod-db.example.com",
    "port": 27017, // Same port
    "name": "mydb_prod"
  },
  "logging": {
    "level": "info"
  },
  "performance": { // New section
    "cacheEnabled": true
  }
}

Notice the overlap and the differences. If you just load `config.development.json` in development and `config.production.json` in production, you have to maintain potentially large parts of the configuration in multiple places. Adding a new common setting means editing every environment file. This is inefficient and error-prone.

Introducing Configuration Merging

A better approach is to define a base configuration that contains all shared settings and then have environment-specific files that only contain the overrides and additions for that particular environment.

// config.base.json
{
  "apiUrl": "https://default.api.example.com", // A reasonable default
  "database": {
    "host": "localhost", // Default for local dev
    "port": 27017,
    "name": "default_db"
  },
  "logging": {
    "level": "warn", // Default logging
    "format": "json"
  },
  "featureFlags": {
    "newDashboard": false,
    "adminPanel": true
  }
}

// config.development.json
{
  "apiUrl": "https://dev.api.example.com", // Override base
  "database": {
    "name": "mydb_dev" // Override base name, keep host/port from base
  },
  "logging": {
    "level": "debug" // Override base level, keep format from base
  },
  "featureFlags": {
    "newDashboard": true // Override base
  }
}

// config.production.json
{
  "apiUrl": "https://api.example.com", // Override base
  "database": {
    "host": "prod-db.example.com", // Override base host
    "name": "mydb_prod" // Override base name
    // port is inherited from base
  },
  "logging": {
    "level": "info" // Override base level
    // format is inherited from base
  },
  "performance": { // New section, added to the merged config
    "cacheEnabled": true
  }
  // featureFlags are inherited from base
}

With this structure, you load the base configuration first, and then merge the environment-specific configuration on top of it. The values from the environment file override the values in the base file where they exist, and new sections/keys in the environment file are added to the final configuration.

How Merging Works: Shallow vs. Deep

There are two main strategies for merging JSON objects:

Shallow Merge

A shallow merge only copies top-level properties from the source object to the target object. If a property's value is an object, the object itself is copied by reference or replaced entirely; its nested properties are not merged recursively.

// Example of shallow merge (using Object.assign or spread syntax)
const baseConfig = {
  "a": 1,
  "b": { "c": 2, "d": 3 },
  "e": [1, 2]
};

const devConfig = {
  "a": 10, // Overrides 'a'
  "b": { "c": 20 }, // Replaces 'b' object entirely
  "f": "new" // Adds 'f'
};

// Shallow merge: devConfig onto baseConfig
const mergedConfigShallow = { ...baseConfig, ...devConfig };

console.log(mergedConfigShallow);
// Output:
// {
//   "a": 10,
//   "b": { "c": 20 }, // Notice 'd' is gone
//   "e": [1, 2],
//   "f": "new"
// }

As you can see, the nested `database` or `logging` objects from the base config would be entirely replaced by the objects in the environment config, losing any properties not explicitly listed in the environment config. This is often not the desired behavior for configurations.

Deep Merge

A deep merge recursively merges nested objects. If both the source and target have a property that is an object, the merge function calls itself on those nested objects. If a property's value is primitive (string, number, boolean, null) or an array, it's typically overwritten by the source value.

// Conceptual Deep Merge Example
const baseConfig = {
  "a": 1,
  "b": { "c": 2, "d": 3 },
  "e": [1, 2]
};

const devConfig = {
  "a": 10, // Overrides 'a'
  "b": { "c": 20 }, // Merges into 'b' object
  "f": "new" // Adds 'f'
};

// Deep merge: devConfig onto baseConfig
// (Implementation shown below)
const mergedConfigDeep = deepMerge(baseConfig, devConfig);

console.log(mergedConfigDeep);
// Output:
// {
//   "a": 10,
//   "b": { "c": 20, "d": 3 }, // Notice 'd' is preserved
//   "e": [1, 2], // Array is overwritten/kept as is (typical)
//   "f": "new"
// }

Deep merging is usually what you want for JSON configurations, as it allows you to override specific nested settings without having to repeat the entire nested structure in your environment files.

Implementing a Deep Merge Function

Here's a basic TypeScript implementation of a deep merge function that handles objects and overwrites primitives and arrays.

type JsonValue =
  | string
  | number
  | boolean
  | null
  | { [key: string]: JsonValue }
  | JsonValue[];

function isObject(item: any): item is { [key: string]: JsonValue } & null {
  return (item && typeof item === 'object' && !Array.isArray(item));
}

/**
 * Deeply merges two JSON-like objects.
 * Properties in source will override properties in target.
 * Nested objects are merged recursively.
 * Arrays are overwritten (not merged element by element).
 *
 * @param target The object to merge properties into.
 * @param source The object to merge properties from.
 * @returns A new object representing the merged configuration.
 */
function deepMerge<T extends JsonValue, S extends JsonValue>(
  target: T,
  source: S
): T & S { // Return type is simplified, assumes source overrides target
  // Create a deep copy of the target to avoid modifying the original
  const output = JSON.parse(JSON.stringify(target)); // Simple deep clone

  if (isObject(target) && isObject(source)) {
    Object.keys(source).forEach(key => {
      if (isObject(source[key])) {
        if (!(key in output)) {
          // If the key doesn't exist in target, just assign the source object
          Object.assign(output, { [key]: source[key] });
        } else {
          // If the key exists and both are objects, recurse
          output[key] = deepMerge(output[key] as {[k: string]: JsonValue}, source[key] as {[k: string]: JsonValue});
        }
      } else {
        // Otherwise, overwrite the value in the target
        Object.assign(output, { [key]: source[key] });
      }
    });
  } else if (Array.isArray(target) && Array.isArray(source)) {
     // If both are arrays, source array overwrites target array
     return source as any; // Coercion needed because of simplified return type
  }
   else {
    // If target is not an object/array or source is not,
    // or if types mismatch (e.g., merging object onto string), source overwrites
     return source as any; // Coercion needed
  }


  return output;
}

// Example usage (conceptual, as this is a static page)
/*
const base = { "a": 1, "b": { "c": 2, "d": 3 }, "e": [1, 2] };
const overrides = { "a": 10, "b": { "c": 20, "f": 4 }, "e": [3, 4, 5], "g": "hello" };

const finalConfig = deepMerge(base, overrides);
console.log(finalConfig);
// Expected Output:
// {
//   "a": 10,
//   "b": { "c": 20, "d": 3, "f": 4 },
//   "e": [3, 4, 5],
//   "g": "hello"
// }
*/

This function takes two objects (target and source) and recursively merges them. Keys present in the `source` object will override keys in the `target` object. If both keys contain objects, those objects are merged recursively. Primitive values and arrays from the `source` simply replace those in the `target`.

In a real application, you would typically load `config.base.json` and then load `config.${NODE_ENV}.json` and apply the deep merge.

// Conceptual loading and merging logic
import fs from 'fs';
import path from 'path';

// Assume deepMerge function is defined as above

const env = process.env.NODE_ENV || 'development'; // Get current environment
const baseConfigPath = path.resolve('./config.base.json');
const envConfigPath = path.resolve(`./config.${env}.json`);

let finalConfig: JsonValue;

try {
  const baseConfig = JSON.parse(fs.readFileSync(baseConfigPath, 'utf8'));
  finalConfig = baseConfig; // Start with base config

  if (fs.existsSync(envConfigPath)) {
    const envConfig = JSON.parse(fs.readFileSync(envConfigPath, 'utf8'));
    finalConfig = deepMerge(baseConfig, envConfig); // Merge environment overrides
    console.log(`Configuration loaded and merged for environment: ${env}`);
  } else {
    console.warn(`Environment config file not found: ${envConfigPath}. Using base config only.`);
  }

  // You can now use finalConfig throughout your application

} catch (error) {
  console.error("Failed to load or merge configuration:", error);
  // Handle error, perhaps exit or use default settings
  // process.exit(1);
}

Advanced Considerations

Array Merging

The provided `deepMerge` function overwrites arrays. For configurations, this is often acceptable (e.g., a list of allowed origins might be completely different per environment). However, sometimes you might want different array merging strategies:

• Concatenation: Combine elements from both arrays.
• Merging Objects in Arrays: If an array contains objects with a unique identifier (`id`, `name`), you might want to merge objects based on this ID (e.g., list of users where you only update properties for a specific user by ID).
• Custom Logic: More complex scenarios might require custom merge rules for specific array properties.

Implementing these requires a more sophisticated merge function, potentially with options passed to control array behavior or even a convention like `_mergeStrategy: "concat"`. Many libraries exist that offer configurable deep merging.

Handling Sensitive Data

Never store sensitive credentials (passwords, API keys) directly in JSON configuration files, especially not in files that might be checked into source control. Use environment variables or a dedicated secrets management system instead. Your application code should read secrets from these secure sources and merge them with the non-sensitive configuration loaded from files.

Configuration Validation

After merging, it's crucial to validate the final configuration structure and types to ensure your application receives the expected data. Libraries like Zod, Joi, or custom validation logic can be used here.

Benefits of Using Merging

• Reduced Duplication: Common settings live only in the base config.
• Improved Maintainability: Changes to common settings are made in one place. Environment-specific changes are isolated.
• Clear Overrides: It's explicit what settings are being overridden for each environment.
• Flexibility: Easily add new environments or feature flags by creating minimal override files.

Conclusion

Merging JSON configurations across environments is a powerful pattern that significantly simplifies application setup and maintenance. By defining a base configuration and using environment-specific overrides combined with a deep merge strategy, you can create a flexible, readable, and less error-prone configuration system. Remember to use secure practices for sensitive data and validate your final configuration to ensure reliability.