Working with BigCommerce data 

You will often want to store BigCommerce data in your Gadget database after subscribing to BigCommerce webhooks.

This guide will show you how to read and write data to BigCommerce using Gadget, and how you can store BigCommerce data in your Gadget database.

Calling the BigCommerce API 

Before you can store data, you need to fetch it from BigCommerce! BigCommerce webhooks often only provide a resource ID, so you will need to use the BigCommerce API to fetch the full resource data.

The best way to call the BigCommerce API is to use the included BigCommerce API client.

Here's a simple example of using the API client to read products from BigCommerce in a global action.

Read product data from BigCommerce in a global action
JavaScript
Copy
1import { ReadProductsGlobalActionContext } from "gadget-server";
2
3/**
4 * @param { ReadProductsGlobalActionContext } context
5 */
6export async function run({ params, logger, api, connections }) {
7  // get BigCommerce API client for the store
8  const bigcommerce = await connections.bigcommerce.forStoreHash(
9    "<add-your-store-hash>"
10  );
11  // use the API client to fetch 5 products, and return
12  const products = await bigcommerce.v3.get("/catalog/products?limit=5");
13  return products;
14}

You can also use the API client to write data back to a BigCommerce store:

Write product data to BigCommerce
JavaScript
Copy
1import { SendGlobalActionContext } from "gadget-server";
2
3/**
4 * @param { SendGlobalActionContext } context
5 */
6export async function run({ params, logger, api, connections }) {
7  // get BigCommerce API client for the store
8  const bigcommerce = await connections.bigcommerce.forStoreHash(
9    "<add-your-store-hash>"
10  );
11  // use the API client create a new product
12  await bigcommerce.v3.post("/catalog/products", {
13    body: {
14      name: "My new product",
15      type: "physical",
16      price: 10,
17      weight: 1,
18    },
19  });
20}

Store context for API client 

The connections.bigcommerce object provides access to a BigCommerce API client. You can use this client to read and write data to BigCommerce.

Depending on how an action was triggered, the current store context may be available on the connections object. For example, if you trigger a global action from a BigCommerce webhook, or call an action from a single-click frontend, the current store context will be available with connections.bigcommerce.current:

example of init a BigCommerce client using connections.bigcommerce.current
JavaScript
Copy
const bigcommerce = connections.bigcommerce.current;
logger.info(
  await bigcommerce.v3.get("/catalog/products?limit=1"),
  "read 1 bigcommerce product"
);

If the current store context is not available, such as calling an action from another action in Gadget, you can use the forStoreHash method to get the API client for a specific store hash:

init BigCommerce API client using forStoreHash
JavaScript
Copy
1export default async function ({ logger, connections }) {
2  const bigcommerce = await connections.bigcommerce.forStoreHash(
3    "<add-your-store-hash>"
4  );
5  logger.info(
6    await bigcommerce.v3.get("/catalog/products?limit=1"),
7    "read 1 product from BigCommerce"
8  );
9}

This is common when calling model actions from webhook-triggered global actions.

Read more about connections.bigcommerce in the gadget-server reference.

Storing data in Gadget 

You can store data read from BigCommerce in Gadget data models. This allows you to access all data required to power your application in a single, highly available and performant, non-rate limited database.

When you set up your BigCommerce connection, a store data model was created at api/models/bigcommerce/store. You can also add custom models to the bigcommerce namespace.

To add a custom BigCommerce data model:

1. Right-click on the api/models/bigcommerce directory in the Gadget editor and select Add model
2. Give your model a name, for example product or order
3. Add a field to store the BigCommerce id
4. Add a relationship field between the new model and the bigcommerce/store model so that bigcommerce/store has many bigcommerce/newModel
5. Add fields to store the rest of the BigCommerce data you need to power your app

You can use the auto-generated CRUD (create, read, update, delete) API to interact with your new model.

For example, you can call a bigcommerce/product model's create action to store a new record in Gadget:

Create a new product record
JavaScript
Copy
1import { HandleProductCreateGlobalActionContext } from "gadget-server";
2
3/**
4 * @param { HandleProductCreateGlobalActionContext } context
5 */
6export async function run({ params, logger, api, connections }) {
7  // get the BigCommerce API client for the current store
8  const bigcommerce = connections.bigcommerce.current;
9  // fetch the product data
10  const product = await bigcommerce.v3.get("/catalog/products/{product_id}", {
11    path: {
12      product_id: params.id,
13    },
14  });
15
16  // store product data in the product data model
17  await api.bigcommerce.product.create({
18    bigcommerceId: product.id,
19    store: {
20      // get the bigcommerce/store id for the record stored in Gadget
21      _link: connections.bigcommerce.currentStoreId,
22    },
23    name: product.name,
24    // ... any other product fields that need to be stored
25  });
26}
27
28export const options = {
29  triggers: {
30    bigcommerce: {
31      webhooks: ["store/product/created"],
32    },
33  },
34};

Storing BigCommerce IDs 

It is important to store BigCommerce resource IDs in Gadget! This is the unique identifier for the record in BigCommerce, and is required to handle update or delete webhooks, and associate data stored in Gadget with BigCommerce records.

It's also a good idea to add both the Required and Uniqueness validations to the bigcommerceId field on your models! For multi-tenant applications, you should also scope this uniqueness validation to the store relationship so that each store can only have one record with a given bigcommerceId.

Upserting data and handling /updated webhooks 

When you are storing data from BigCommerce in Gadget, you may want to update existing records if they already exist, or create a new record if it doesn't. This is known as upserting data. This is particularly useful when handling /updated webhooks, where a record may or may not already exist in your Gadget database.

Your Gadget API provides an upsert meta action that manages the upsert logic for you. For BigCommerce data, you often want to upsert based on the bigcommerceId field. For multi-tenant applications, you should also scope the upsert to the store relationship. This ensures that you only update or create records for the current store.

Here is an example of how you could upsert product data in Gadget when a store/product/updated webhook is received:

Upsert product data on a store/product/updated webhook
JavaScript
Copy
1import { HandleProductUpdateWebhookGlobalActionContext } from "gadget-server";
2
3/**
4 * @param { HandleProductUpdateWebhookGlobalActionContext } context
5 */
6export async function run({ params, api, connections }) {
7  // get the BigCommerce API client for the current store
8  const bigcommerce = connections.bigcommerce.current;
9  // fetch the product data
10  const product = await bigcommerce.v3.get("/catalog/products/{product_id}", {
11    path: {
12      product_id: params.id,
13    },
14  });
15
16  // upsert product data in the product data model
17  await api.bigcommerce.product.upsert({
18    bigcommerceId: product.id,
19    store: {
20      // get the bigcommerce/store id for the record stored in Gadget
21      _link: connections.bigcommerce.currentStoreId,
22    },
23    name: product.name,
24    // ... any other product fields that need to be stored
25    // use bigcommerceId and store to identify unique records for upsert
26    on: ["bigcommerceId", "store"],
27  });
28}

This will check to see if a product with the given bigcommerceId and store relationship already exists in Gadget. If it does, it will update the existing record. If it doesn't, it will create a new record.

Storing data from child models 

You can store data from child models in Gadget by adding relationships between models. When you create a new model, you can add a relationship field to associate the new model with the parent model.

An example of this relationship is products and product variants. You can create a bigcommerce/productVariant model and associate it with the bigcommerce/product model using a relationship field.

Whether you are syncing or creating data from a webhook, you can include the required child data when reading from BigCommerce so it is returned in the same payload.

Include variant information when retrieving product data
JavaScript
Copy
1const product = await bigcommerce.v3.get("/catalog/products/{product_id}", {
2  path: {
3    product_id: params.id,
4  },
5  query: {
6    include: "variants",
7  },
8});

You also have the option to make a separate API call to retrieve the child data and store it in Gadget. Refer to the BigCommerce API reference for more information on retrieving child data.

Once you have the child data, you can store it in the child model using the CRUD API.

For example, here is how you could store product variant data in Gadget when a store/product/created webhook is received:

Store product and variant data in Gadget
JavaScript
Copy
1import { HandleProductCreateGlobalActionContext } from "gadget-server";
2
3/**
4 * @param { HandleProductCreateGlobalActionContext } context
5 */
6export async function run({ params, logger, api, connections }) {
7  // get the BigCommerce API client for the current store
8  const bigcommerce = connections.bigcommerce.current;
9  // fetch the product data
10  const product = await bigcommerce.v3.get("/catalog/products/{product_id}", {
11    path: {
12      product_id: params.id,
13    },
14    query: {
15      include: "variants",
16    },
17  });
18  // log the product data and webhook payload
19  logger.info({ product, params }, "product data and webhook payload");
20
21  // iterate over the variants and store them in an array
22  const variants = [];
23  for (const variant of product.variants) {
24    variants.push({
25      bigcommerceId: variant.id,
26      store: {
27        _link: connections.bigcommerce.currentStoreId,
28      },
29      // ... any other product variant fields that need to be stored
30    });
31  }
32
33  // store product data in the product data model
34  const productRecord = await api.bigcommerce.product.upsert({
35    bigcommerceId: product.id,
36    name: product.name,
37    store: {
38      // get the bigcommerce/store id for the record stored in Gadget
39      _link: connections.bigcommerce.currentStoreId,
40    },
41    variants: {
42      // use a _converge to relate the variants to the product and add them to the db
43      _converge: {
44        values: variants,
45      },
46    },
47  });
48}
49
50export const options = {
51  triggers: {
52    bigcommerce: {
53      webhooks: ["store/product/created"],
54    },
55  },
56};

Syncing all existing data 

If you need to sync all existing data from a BigCommerce store, you can use the BigCommerce API client to fetch all data and store it in Gadget. There are a couple of things you need to be aware of and handle properly, including the BigCommerce API rate limits and max page size for fetching data.

Gadget's built-in background actions queue can help you manage this process. Background actions allow developers to set custom concurrency controls to adhere to a rate limit, and have built-in retry logic for failed actions, ensuring that all data is synced.

To maximize the speed of the sync, you can use bulk actions to enqueue multiple actions at once. For a data sync, you will often want to use Gadget's upsert meta API so that you can update existing records and create new records in a single action.

Here is an example of how you could sync all product data from a BigCommerce store:

Sync all product data from BigCommerce
JavaScript
Copy
1import { SyncProductsGlobalActionContext } from "gadget-server";
2
3/**
4 * @param { SyncProductsGlobalActionContext } context
5 */
6export async function run({ params, logger, api, connections }) {
7  // set the batch size to 50 for bulk upsert
8  const BATCH_SIZE = 50;
9
10  const bigcommerce = connections.bigcommerce.current;
11  // use the API client to fetch all products, and return
12  const products = await bigcommerce.v3.list(`/catalog/products`);
13
14  const productPayload = [];
15  // use a for await loop to iterate over the AsyncIterables, add to an array
16  for await (const product of products) {
17    productPayload.push({
18      // store the BigCommerce ID
19      bigcommerceId: product.id,
20      // associate the product with the current store
21      store: {
22        _link: connections.bigcommerce.currentStoreId,
23      },
24      name: product.name,
25      // ... add more fields as needed
26      // use bigcommerceId and store to identify unique records for upsert
27      on: ["bigcommerceId", "store"],
28    });
29
30    // enqueue 50 actions at a time
31    if (productPayload.length >= BATCH_SIZE) {
32      const section = productPayload.splice(0, BATCH_SIZE);
33      // bulk enqueue create action
34      await api.enqueue(api.bigcommerce.product.bulkUpsert, section, {
35        queue: { name: "product-sync" },
36      });
37    }
38  }
39
40  // enqueue any remaining products
41  await api.enqueue(api.bigcommerce.product.bulkUpsert, productPayload, {
42    queue: { name: "product-sync" },
43  });
44}
45
46export const options = {
47  timeoutMS: 900000, // 15 minute timeout for the sync
48};

This will sync all products from a BigCommerce store to Gadget. It may take some time before all records are synced!

If you need to sync on a regular schedule, you can use a scheduled action to run the sync at a set interval.

Handle BigCommerce rate limits 

Rate limits in BigCommerce are per store and shared across all apps installed on a store. This means that 429 response codes must be handled properly.

The included BigCommerce API client will automatically handle rate limits for you and will retry requests that return 429 or 5XX response codes using an exponential backoff.

Gadget's background actions can also be used to manage rate limits. You can set the concurrency of a background action to match the rate limit of the BigCommerce API and use the built-in retry logic to handle rate limit errors.

To do this, use the maxConcurrency option in the enqueue function to limit how many actions are run concurrently.

For example, to create new products in BigCommerce, you would run the following action in a background action:

Action to write data to BigCommerce
JavaScript
Copy
1import { BigCommerceCreateProductGlobalActionContext } from "gadget-server";
2
3/**
4 * @param { BigCommerceCreateProductGlobalActionContext } context
5 */
6export async function run({ params, logger, api, connections }) {
7  // get BigCommerce API client for the store
8  const bigcommerce = await connections.bigcommerce.forStoreHash(params.storeHash);
9
10  // use the API client to fetch 5 products, and return
11  await bigcommerce.v3.post("/catalog/products", {
12    body: {
13      name: "My new product",
14      type: "physical",
15      price: 10,
16      weight: 1,
17    },
18  });
19}
20
21// define product param
22export const params = {
23  product: {
24    type: "object",
25    properties: {
26      name: { type: "string" },
27      price: { type: "number" },
28      weight: { type: "number" },
29    },
30  },
31  storeHash: { type: "string" },
32};

And then enqueue the action with a maxConcurrency option:

Action to run write action using background actions
JavaScript
Copy
1import { BigCommerceCreateMultipleProductsGlobalActionContext } from "gadget-server";
2
3/**
4 * @param { BigCommerceCreateMultipleProductsGlobalActionContext } context
5 */
6export async function run({ params, logger, api, connections }) {
7  // get products from passed in params
8  const products = params.products;
9  products.map(async (product) => {
10    await api.enqueue(
11      api.bigcommerce.createProduct,
12      {
13        product,
14        storeHash: connections.bigcommerce.currentStoreHash,
15      },
16      {
17        // use maxConcurrency to limit how fast enqueued actions are run
18        queue: {
19          name: "create-products",
20          maxConcurrency: 2,
21        },
22      }
23    );
24  });
25}
26
27export const params = {
28  products: {
29    type: "array",
30    items: {
31      type: "object",
32      properties: {
33        name: { type: "string" },
34        price: { type: "number" },
35        weight: { type: "number" },
36      },
37    },
38  },
39};

Working with BigCommerce metafields 

Similar to any other resource in BigCommerce, you can read and write metafields using the BigCommerce API client. You might wish to store metafield data in line with the resource data in Gadget.

There are two ways to store metafield data next to your model data:

1. Subscribe to metafield webhooks in a global action and update the model data in Gadget when a metafield is created, updated, or deleted. Resources that can store metafield data have their own metafield webhook topics, so you can subscribe to these topics and update your model data accordingly.

2. Fetch metafield data when you fetch the resource data and store it in the model. This would be required when syncing data into your Gadget database and the code would look similar to the sync example.

Subscribe to metafield webhooks 

Metafield payloads always have a metafield_id and a resource_id field, which you can use to update the correct record in your model:

json
Copy
1{
2  "metafield_id": 2,
3  "namespace": "Sales Department",
4  "resource_id": "133",
5  "storeHash": "bbcolvaxts"
6}

Both the resource_id and metafield_id should be stored in your model to be able to update and delete metafields.

Here is an example of a store/product/metafield/created webhook subscription that updates the bigcommerce/product model when a metafield is created for a product. The product model stores resource_id as bigcommerceId and metafield_id as metafieldId.

Add metafield data to an existing product record
JavaScript
Copy
1import { BigCommerceCreateMetafieldGlobalActionContext } from "gadget-server";
2
3/**
4 * @param { BigCommerceCreateMetafieldGlobalActionContext } context
5 */
6export async function run({ params, logger, api, connections }) {
7  // get the BigCommerce API client
8  const bigcommerce = connections.bigcommerce.current;
9
10  // get ids from webhook payload
11  const productId = params.resource_id;
12  const metafieldId = params.metafield_id;
13
14  // fetch the metafield value from BigCommerce
15  const metafield = await bigcommerce.v3.get(
16    "/catalog/products/{product_id}/metafields/{metafield_id}",
17    {
18      path: {
19        product_id: productId,
20        metafield_id: metafieldId,
21      },
22    }
23  );
24
25  // see if the product exists in the Gadget db
26  let product = await api.bigcommerce.product.maybeFindByBigCommerceId(
27    parseInt(productId)
28  );
29
30  // if the product doesn't exist in Gagdet, fetch it
31  if (!product) {
32    product = await bigcommerce.v3.get("/catalog/products/{product_id}", {
33      path: {
34        product_id: productId,
35      },
36    });
37
38    // and add the product to the Gadget db
39    return await api.bigcommerce.product.create({
40      bigcommerceId: product.id,
41      store: {
42        // get the bigcommerce/store id for the record stored in Gadget
43        _link: connections.bigcommerce.currentStoreId,
44      },
45      name: product.name,
46      metafieldId,
47      metafieldValue: metafield.value,
48    });
49  }
50
51  // update the existing product
52  return await api.bigcommerce.product.update(product.id, {
53    metafieldId,
54    metafieldValue: metafield.value,
55  });
56}
57
58export const options = {
59  triggers: {
60    api: false,
61    bigcommerce: {
62      webhooks: ["store/product/metafield/created"],
63    },
64  },
65};

You can also use the namespace and key to identify the metafield, which is useful when you have multiple metafields for a resource.

Data security and multi-tenancy 

If you are building a public app, you will need to ensure that data is stored securely and that one store's data is not accessible by another store. Gadget provides tools that help manage multi-tenancy and row-level security (RLS) for models and actions.

Securing models and model actions 

Gadget provides a set of data access tools that can be used to secure models and their actions. These tools include:

• relationship fields on models so that you can associate data with a store record
• a Uniqueness validation to make sure that each store has only one record with a given ID
• built-in functions for preventing cross-shop data access in model actions
• filtered model permissions that restrict access based on the related store

Follow these steps to set up these tools and enforce multi-tenancy on a model:

1. If you haven't already done so, add a relationship field to any custom model that associates the model with a store record

For example, add a belongs to relationship field on the bigcommerce/product model. The field should be named store and should reference the bigcommerce/store model, so that product belongsTo store. Then define the inverse of the relationship so that store hasMany products.

2. Add a Uniqueness validation to the BigCommerce resource ID field on your model and scope it to the store relationship. This is important to ensure that stored IDs are unique per store. See storing BigCommerce IDs for more information.

3. In your custom model, import preventCrossStoreDataAccess from gadget-server/bigcommerce and use it in your model actions

For example, in the bigcommerce/product model, you can call preventCrossStoreDataAccess in the run function of the create action to ensure that a product can only be created for the current store:

api/models/bigcommerce/product/actions/create.js
JavaScript
Copy
1import {
2  applyParams,
3  save,
4  ActionOptions,
5  CreateBigCommerceProductActionContext,
6} from "gadget-server";
7// import the preventCrossStoreDataAccess function
8import { preventCrossStoreDataAccess } from "gadget-server/bigcommerce";
9
10/**
11 * @param { CreateBigCommerceProductActionContext } context
12 */
13export async function run({ params, record, logger, api, connections }) {
14  applyParams(params, record);
15  // use the preventCrossStoreDataAccess function to make sure
16  // the product is created for the current store
17  await preventCrossStoreDataAccess(params, record);
18  await save(record);
19}
20
21/**
22 * @param { CreateBigCommerceProductActionContext } context
23 */
24export async function onSuccess({ params, record, logger, api, connections }) {
25  // Your logic goes here
26}
27
28/** @type { ActionOptions } */
29export const options = {
30  actionType: "create",
31};

Each time preventCrossStoreDataAccess runs, it will make sure the given record has the correct storeId for the store processing this action. For existing records, that means it will verify that the storeId of the saved record the matches the storeId on the incoming record.

4. Add a tenancy filter to the read action for the custom model in accessControl/permissions.

Tenancy filters are expressed with Gelly, Gadget's data access language. These filters check the store ID of the current session against the ID of the store that the data being read belongs to.

Most tenancy filters for BigCommerce look similar to this:

accessControl/filters/bigcommerce/product.gelly
gelly
Copy
filter ($session: Session) on BigCommerceStore [
  where storeId == $session.bigcommerceStoreId
]

Securing global actions and HTTP routes 

When using global actions or HTTP routes to read or write data, you cannot use Gelly tenancy filters or functions like preventCrossStoreDataAccess because there is no backing model or relationship to a store record.

For global actions or HTTP routes, you often need to filter data based on the bigcommerceStoreId, which is included as a properly on the connections parameter. Filtering is built into your app's API and available on most field types. Docs for filtering are available in the API reference.

For example, if I am fetching bigcommerce/order data in a global action before doing some processing and returning the results, I would do the following:

api/actions/processOrders.js
JavaScript
Copy
1import { BigCommerceProcessOrdersGlobalActionContext } from "gadget-server";
2
3/**
4 * @param { BigCommerceProcessOrdersGlobalActionContext } context
5 */
6export async function run({ params, logger, api, connections }) {
7  const bigcommerceStoreId = connections.bigcommerce.currentStoreId;
8
9  // filter results by the store ID
10  const orders = await api.bigcommerce.orders.findMany({
11    filter: {
12      storeId: {
13        equals: bigcommerceStoreId,
14      },
15    },
16  });
17
18  return orders;
19}

This will only work when there is an authenticated session!

Otherwise, you will need to manually fetch the store ID from bigcommerce/store using the store hash:

JavaScript
Copy
1import { BigCommerceProcessOrdersGlobalActionContext } from "gadget-server";
2
3/**
4 * @param { BigCommerceProcessOrdersGlobalActionContext } context
5 */
6export async function run({ params, logger, api, connections }) {
7  // get the store ID from the record in Gadget
8  const bigcommerceStore = await api.bigcommerce.store.findByStoreHash(
9    params.storeHash,
10    {
11      // only select the id field
12      select: {
13        id: true,
14      },
15    }
16  );
17
18  // filter results by the store ID
19  const orders = await api.bigcommerce.orders.findMany({
20    filter: {
21      storeId: {
22        equals: bigcommerceStore.id,
23      },
24    },
25  });
26
27  return orders;
28}
29
30// allow the storeHash to be passed as a parameter
31export const params = {
32  storeHash: { type: "string" },
33};

The same pattern can be used for HTTP routes, where you would fetch the store ID from the bigcommerce/store model and then filter the results based on the store ID.