Handling Incomplete JSON Data Streams

Incomplete JSON is usually a protocol problem, not a parsing trick. A normal JSON document becomes valid only when the full serialized value arrives, so cutting a stream in the middle of an object or array leaves you with invalid JSON. The practical fix is to decide whether you are receiving one complete document, or a stream of many smaller JSON records that needs explicit framing.

1. Why Plain JSON Breaks In Streams

Plain JSON does not tell the reader where one logical record ends unless the entire value is already complete. That is why naive brace counting is fragile: braces can appear inside strings, top-level values can be arrays, numbers, booleans, or null, and a truncated escape sequence can make the remaining buffer ambiguous.

If you control the producer, the safest improvement is usually to stop streaming one massive JSON array and instead emit self-contained records.

2. Prefer Framed Records For Long-Lived Streams

Two formats are especially useful when a connection may pause, break, or reconnect:

NDJSON
{"id":1,"type":"login"}
{"id":2,"type":"view"}

RFC 7464 JSON text sequence
\u001E{"id":1,"type":"login"}
\u001E{"id":2,"type":"view"}

3. Safe Browser Pattern For NDJSON

In current browsers, fetch() exposes the body as a ReadableStream, and TextDecoderStream is widely available. A safe reader keeps a text buffer, splits on newlines, parses only complete lines, and treats any leftover tail at end-of-stream as truncated data.

async function consumeNdjson(url, { onItem, signal } = {}) {
  const response = await fetch(url, {
    headers: { Accept: "application/x-ndjson, application/json" },
    signal,
  });

  if (!response.ok || !response.body) {
    throw new Error(`Request failed: ${response.status}`);
  }

  const reader = response.body
    .pipeThrough(new TextDecoderStream())
    .getReader();

  let buffer = "";
  let lineNumber = 0;

  try {
    while (true) {
      const { value, done } = await reader.read();
      if (done) break;

      buffer += value;
      const lines = buffer.split(/\r?\n/);
      buffer = lines.pop() ?? "";

      for (const line of lines) {
        if (!line.trim()) continue;
        lineNumber += 1;

        try {
          onItem?.(JSON.parse(line));
        } catch (error) {
          throw new Error(
            `Invalid JSON record on line ${lineNumber}: ${error.message}`
          );
        }
      }
    }

    if (buffer.trim()) {
      throw new Error(
        "Stream ended with a partial JSON record. Retry from the last confirmed offset."
      );
    }
  } finally {
    reader.releaseLock();
  }
}

4. Recovery Strategy When The Stream Cuts Off

The recovery path should be explicit. Do not silently invent closing braces or brackets unless you are doing a one-off repair by hand and already know the missing content.

1. Keep the unparsed tail separate from confirmed records.
2. Checkpoint a cursor, byte offset, sequence number, or last event ID as you process each record.
3. Retry from the last confirmed checkpoint instead of replaying from the beginning when possible.
4. Make downstream processing idempotent so duplicates are safe after reconnects.
5. Log the raw fragment, response headers, and checkpoint metadata for debugging.

5. If You Cannot Change The Producer

Sometimes a third-party service sends one huge JSON document and you still need early results. In that case, use a real incremental parser for your runtime rather than writing a homegrown state machine. Node.js, Python, Java, Go, and Rust all have mature streaming parser options that can emit tokens or objects as the buffer grows.

Even then, the goal is usually to read one valid value safely, not to guess the missing tail of a broken one. If you need to inspect the fragment manually, validate the recovered piece in the Offline Tools JSON Formatter before replaying it or storing it permanently.

6. Practical Decision Guide

• If the API returns one document, buffer the full response and parse once.
• If the API emits many records, switch to NDJSON, JSON text sequences, or SSE.
• If the connection may resume, attach sequence IDs and build around checkpoints.
• If truncation is common, fix the producer contract before tuning the parser.