Send XML Data to Google BigQuery Using Node.js

XML still shows up everywhere data gets exchanged — SOAP APIs, vendor exports, RSS feeds, decades-old ERP integrations. BigQuery, meanwhile, does not ingest XML natively: its load jobs and streaming API speak JSON, CSV, Avro, and Parquet. So every “XML to BigQuery” pipeline is really a three-step move: parse the XML to JavaScript objects, map those objects to your table schema, then load the rows. Here’s the whole pattern in Node.js.

1. Set up the project

In the Google Cloud Console, create (or pick) a project and enable the BigQuery API. Create a service account with the BigQuery Data Editor role, download its JSON key, and point your environment at it:

export GOOGLE_APPLICATION_CREDENTIALS="/path/to/service-account-key.json"

Then install the two libraries doing the heavy lifting — the BigQuery client and an XML parser:

npm install @google-cloud/bigquery fast-xml-parser

We use fast-xml-parser here because it’s fast, dependency-free, and turns XML into plain objects with one call. xml2js works just as well if you’re already using it.

2. Create the dataset and table

Give the data a real schema instead of dumping raw XML strings into a single column — future-you writing SQL will be grateful:

const { BigQuery } = require('@google-cloud/bigquery');
const bigquery = new BigQuery();

async function createDatasetAndTable() {
  const [dataset] = await bigquery.createDataset('vendor_feed');
  await dataset.createTable('orders', {
    schema: [
      { name: 'order_id', type: 'STRING', mode: 'REQUIRED' },
      { name: 'customer', type: 'STRING' },
      { name: 'total', type: 'NUMERIC' },
      { name: 'ordered_at', type: 'TIMESTAMP' },
    ],
  });
}

3. Parse the XML

Say the feed looks like this:

<orders>
  <order>
    <id>A-1001</id>
    <customer>Acme Corp</customer>
    <total>149.99</total>
    <date>2022-12-28T14:03:00Z</date>
  </order>
</orders>

XMLParser turns it into objects, and a small mapping function shapes each one to match the table schema exactly:

const { XMLParser } = require('fast-xml-parser');

function xmlToRows(xml) {
  const parsed = new XMLParser().parse(xml);
  // one <order> parses to an object, many to an array — normalize to array
  const orders = [].concat(parsed.orders.order);
  return orders.map((o) => ({
    order_id: String(o.id),
    customer: o.customer,
    total: o.total,
    ordered_at: o.date,
  }));
}

That normalize-to-array line matters: XML parsers return a bare object when an element appears once and an array when it repeats, and single-record feeds are exactly the edge case that breaks pipelines on quiet days.

4. Insert the rows

For feeds that arrive continuously, streaming inserts get rows into the table within seconds:

async function insertRows(rows) {
  try {
    await bigquery.dataset('vendor_feed').table('orders').insert(rows);
    console.log(`Inserted ${rows.length} rows`);
  } catch (err) {
    // PartialFailureError: some rows landed, some didn't — log the per-row reasons
    if (err.name === 'PartialFailureError') {
      for (const failure of err.errors) {
        console.error(failure.errors, JSON.stringify(failure.row));
      }
    } else {
      throw err;
    }
  }
}

const fs = require('fs');
insertRows(xmlToRows(fs.readFileSync('orders.xml', 'utf8')));

The PartialFailureError branch is the part most examples skip: BigQuery’s streaming API can accept half your batch and reject the rest (usually a type mismatch), and unless you inspect the per-row errors those records vanish silently.

Streaming vs. batch loads

Streaming inserts cost money per MB and have per-table quotas. If your XML arrives as a nightly file rather than a live feed, skip streaming: write the mapped rows to a newline-delimited JSON file and use a load job (table.load(file)), which is free. The parse-and-map code above stays identical — only the last step changes. A good rule: files load, events stream.

Where this pattern goes next

This is the same skeleton we use for every API-to-warehouse sender: swap the parse step and the schema, keep the load-and-error-handling core. If your source is a REST API instead of an XML file, these walk through the same pattern: Auth0 to BigQuery, Sage API to BigQuery, and Twitter to BigQuery.

And when the one-off script becomes a fleet of feeds with schedules, retries, and monitoring, that’s the point where a real pipeline earns its keep — our data engineering consulting practice builds exactly these.