# Working with BigCommerce data 

You will often want to store BigCommerce data in your Gadget database after subscribing to BigCommerce webhooks.

This guide will show you how to read and write data to BigCommerce using Gadget, and how you can store BigCommerce data in your Gadget database.

## Calling the BigCommerce API 

Before you can store data, you need to fetch it from BigCommerce! BigCommerce webhooks often only provide a resource ID, so you will need to use the BigCommerce API to fetch the full resource data.

The best way to call the BigCommerce API is to use the included [BigCommerce API client](https://github.com/Space48/bigcommerce-api-js).

Here's a simple example of using the API client to read products from BigCommerce in a global action.

```typescript
export const run: ActionRun = async ({ params, logger, api, connections }) => {
  // get BigCommerce API client for the store
  const bigcommerce = await connections.bigcommerce.forStoreHash(
    "<add-your-store-hash>"
  );
  // use the API client to fetch 5 products, and return
  const products = await bigcommerce?.v3.get("/catalog/products", {
    query: { limit: 5 },
  });
  return products;
};
```

You can also use the API client to write data back to a BigCommerce store:

```typescript
export const run: ActionRun = async ({ params, logger, api, connections }) => {
  // get BigCommerce API client for the store
  const bigcommerce = await connections.bigcommerce.forStoreHash(
    "<add-your-store-hash>"
  );
  // use the API client create a new product
  await bigcommerce?.v3.post("/catalog/products", {
    body: {
      name: "My new product",
      type: "physical",
      price: 10,
      weight: 1,
    },
  });
};
```

Data stored in Gadget models is not automatically synced with BigCommerce. The best way to keep data in BigCommerce and Gadget in sync is to write data back to BigCommerce using the API client, and then allow webhooks to update data in Gadget.

### Store context for API client 

The `connections.bigcommerce` object provides access to a BigCommerce API client. You can use this client to read and write data to BigCommerce.

Depending on how an action was triggered, the current store context may be available on the `connections` object. For example, if you trigger a global action from a BigCommerce webhook, or call an action from a single-click frontend, the current store context will be available with `connections.bigcommerce.current`:

```typescript
const bigcommerce = connections.bigcommerce.current;
logger.info(
  await bigcommerce?.v3.get("/catalog/products", { query: { limit: 1 } }),
  "read 1 bigcommerce product"
);
```

If the current store context is not available, such as calling an action from another action in Gadget, you can use the `forStoreHash` method to get the API client for a specific store hash:

```typescript
const bigcommerce = await connections.bigcommerce.forStoreHash(
  "<add-your-store-hash>"
);
logger.info(
  await bigcommerce?.v3.get("/catalog/products", { query: { limit: 1 } }),
  "read 1 product from BigCommerce"
);
```

This is common when [calling model actions from webhook-triggered global actions](https://docs.gadget.dev/guides/plugins/bigcommerce/webhooks#persisting-session-data-in-webhook-triggered-actions).

Read more about [`connections.bigcommerce` in the `gadget-server` reference](https://docs.gadget.dev/reference/gadget-server#connections-bigcommerce).

## Storing data in Gadget 

You can store data read from BigCommerce in Gadget data models. This allows you to access all data required to power your application in a single, highly available and performant, non-rate limited database.

When you set up your BigCommerce connection, a `store` data model was created at `api/models/bigcommerce/store`. You can also add custom models to the `bigcommerce` namespace.

It's also important to relate any additional BigCommerce resource models to the `bigcommerce/store` model so that you can associate data with a specific store!

To add a custom BigCommerce data model:

1.  Right-click on the `api/models/bigcommerce` directory in the Gadget editor and select **Add model**
2.  Give your model a name, for example `product` or `order`
3.  Add a field to 
4.  Add a relationship field between the new model and the `bigcommerce/store` model so that `bigcommerce/store` has many `bigcommerce/newModel`
5.  Add fields to store the rest of the BigCommerce data you need to power your app

You can use the auto-generated CRUD (create, read, update, delete) API to interact with your new model.

For example, you can call a `bigcommerce/product` model's `create` action to store a new record in Gadget:

```typescript
import { ActionOptions } from "gadget-server";

export const run: ActionRun = async ({ params, logger, api, connections }) => {
  // get the BigCommerce API client for the current store
  const bigcommerce = connections.bigcommerce.current;

  if (!bigcommerce) {
    throw new Error("Missing bigcommerce connection");
  }

  // fetch the product data
  const product = await bigcommerce.v3.get("/catalog/products/{product_id}", {
    path: {
      product_id: params.id,
    },
  });

  if (!product) {
    throw new Error("Missing product");
  }

  // store product data in the product data model
  await api.bigcommerce.product.create({
    bigcommerceId: product.id,
    store: {
      // get the bigcommerce/store id for the record stored in Gadget
      _link: connections.bigcommerce.currentStoreId,
    },
    name: product.name,
    // ... any other product fields that need to be stored
  });
};

export const options: ActionOptions = {
  triggers: {
    bigcommerce: {
      webhooks: ["store/product/created"],
    },
  },
};
```

### Storing BigCommerce IDs 

It is important to store BigCommerce resource IDs in Gadget! This is the unique identifier for the record in BigCommerce, and is required to handle update or delete webhooks, and associate data stored in Gadget with BigCommerce records.

It's also a good idea to add both the [Required](https://docs.gadget.dev/guides/models/fields#required-validation) and [Uniqueness](https://docs.gadget.dev/guides/models/fields#uniqueness-validation) validations to the `bigcommerceId` field on your models! For multi-tenant applications, you should also scope this uniqueness validation to the `store` relationship so that each store can only have one record with a given `bigcommerceId`.

### Upserting data and handling `/updated` webhooks 

When you are storing data from BigCommerce in Gadget, you may want to update existing records if they already exist, or create a new record if it doesn't. This is known as upserting data. This is particularly useful when handling `/updated` webhooks, where a record may or may not already exist in your Gadget database.

Your Gadget API provides an [`upsert` meta action](https://docs.gadget.dev/guides/actions/code#upsert-meta-action) that manages the upsert logic for you. For BigCommerce data, you often want to upsert based on the `bigcommerceId` field. For multi-tenant applications, you should also scope the upsert to the `store` relationship. This ensures that you only update or create records for the current store.

Here is an example of how you could upsert product data in Gadget when a `store/product/updated` webhook is received:

```typescript
export const run: ActionRun = async ({ params, api, connections }) => {
  // get the BigCommerce API client for the current store
  const bigcommerce = connections.bigcommerce.current;

  if (!bigcommerce) {
    throw new Error("Missing bigcommerce connection");
  }
  // fetch the product data
  const product = await bigcommerce.v3.get("/catalog/products/{product_id}", {
    path: {
      product_id: params.id,
    },
  });

  if (!product) {
    throw new Error("Missing product");
  }

  // upsert product data in the product data model
  await api.bigcommerce.product.upsert({
    bigcommerceId: product.id,
    store: {
      // get the bigcommerce/store id for the record stored in Gadget
      _link: connections.bigcommerce.currentStoreId,
    },
    name: product.name,
    // ... any other product fields that need to be stored
    // use bigcommerceId and store to identify unique records for upsert
    on: ["bigcommerceId", "store"],
  });
};

export const options: ActionOptions = {
  triggers: {
    bigcommerce: {
      webhooks: ["store/product/updated"],
    },
  },
};
```

This will check to see if a product with the given `bigcommerceId` and `store` relationship already exists in Gadget. If it does, it will update the existing record. If it doesn't, it will create a new record.

### Storing data from child models 

You can store data from child models in Gadget by adding [relationships](https://docs.gadget.dev/guides/models/relationships) between models. When you create a new model, you can add a relationship field to associate the new model with the parent model.

An example of this relationship is products and product variants. You can create a `bigcommerce/productVariant` model and associate it with the `bigcommerce/product` model using a relationship field.

Whether you are syncing or creating data from a webhook, you can include the required child data when reading from BigCommerce so it is returned in the same payload.

```typescript
const product = await bigcommerce.v3.get("/catalog/products/{product_id}", {
  path: {
    product_id: parseInt(params.id!),
  },
  query: {
    include: ["variants"],
  },
});
```

You also have the option to make a separate API call to retrieve the child data and store it in Gadget. Refer to the [BigCommerce API reference](https://developer.bigcommerce.com/docs/api) for more information on retrieving child data.

Once you have the child data, you can store it in the child model using the CRUD API.

For example, here is how you could store product variant data in Gadget when a `store/product/created` webhook is received:

```typescript
import { ActionOptions } from "gadget-server";

export const run: ActionRun = async ({ params, logger, api, connections }) => {
  // get the BigCommerce API client for the current store
  const bigcommerce = connections.bigcommerce.current;

  if (!bigcommerce) {
    throw new Error("Missing bigcommerce connection");
  }

  // fetch the product data
  const product = await bigcommerce.v3.get("/catalog/products/{product_id}", {
    path: {
      product_id: params.id,
    },
    query: {
      include: ["variants"],
    },
  });

  if (!product) {
    throw new Error("Missing product");
  }

  // log the product data and webhook payload
  logger.info({ product, params }, "product data and webhook payload");

  // iterate over the variants and store them in an array
  const variants = [];
  for (const variant of product.variants) {
    variants.push({
      bigcommerceId: variant.id,
      store: {
        _link: connections.bigcommerce.currentStoreId,
      },
      // ... any other product variant fields that need to be stored
    });
  }

  // store product data in the product data model
  const productRecord = await api.bigcommerce.product.upsert({
    bigcommerceId: product.id,
    name: product.name,
    store: {
      // get the bigcommerce/store id for the record stored in Gadget
      _link: connections.bigcommerce.currentStoreId,
    },
    variants: {
      // use a _converge to relate the variants to the product and add them to the db
      _converge: {
        values: variants,
      },
    },
  });
};

export const options: ActionOptions = {
  triggers: {
    bigcommerce: {
      webhooks: ["store/product/created"],
    },
  },
};
```

### Syncing all existing data 

If you need to sync all existing data from a BigCommerce store, you can use the BigCommerce API client to fetch all data and store it in Gadget. There are a couple of things you need to be aware of and handle properly, including the [BigCommerce API rate limits](https://developer.bigcommerce.com/docs/start/best-practices#api-rate-limits) and max page size for fetching data.

Gadget's built-in [background actions](https://docs.gadget.dev/guides/actions/background) queue can help you manage this process. Background actions allow developers to set custom concurrency controls to adhere to a rate limit, and have built-in retry logic for failed actions, ensuring that all data is synced.

To maximize the speed of the sync, you can use bulk actions to enqueue multiple actions at once. For a data sync, you will often want to use Gadget's [`upsert` meta API](https://docs.gadget.dev/guides/actions/code#upsert-meta-action) so that you can update existing records and create new records in a single action.

Here is an example of how you could sync all product data from a BigCommerce store:

```typescript
import { ActionOptions } from "gadget-server";

export const run: ActionRun = async ({ params, logger, api, connections }) => {
  // set the batch size to 50 for bulk upsert
  const BATCH_SIZE = 50;

  const bigcommerce = connections.bigcommerce.current;

  if (!bigcommerce) {
    throw new Error("Missing bigcommerce connection");
  }

  // use the API client to fetch all products, and return
  const products = await bigcommerce.v3.list(`/catalog/products`);

  const productPayload = [];
  // use a for await loop to iterate over the AsyncIterables, add to an array
  for await (const product of products) {
    productPayload.push({
      // store the BigCommerce ID
      bigcommerceId: product.id,
      // associate the product with the current store
      store: {
        _link: connections.bigcommerce.currentStoreId,
      },
      name: product.name,
      // ... add more fields as needed
      // use bigcommerceId and store to identify unique records for upsert
      on: ["bigcommerceId", "store"],
    });

    // enqueue 50 actions at a time
    if (productPayload.length >= BATCH_SIZE) {
      const section = productPayload.splice(0, BATCH_SIZE);
      // bulk enqueue create action
      await api.enqueue(api.bigcommerce.product.bulkUpsert, section, {
        queue: { name: "product-sync" },
      });
    }
  }

  // enqueue any remaining products
  await api.enqueue(api.bigcommerce.product.bulkUpsert, productPayload, {
    queue: { name: "product-sync" },
  });
};

export const options: ActionOptions = {
  // 15 minute timeout for the sync
  timeoutMS: 900000,
};
```

This will sync all products from a BigCommerce store to Gadget. It may take some time before all records are synced!

This action has a timeout of 15 minutes, which is the maximum time an action can run in Gadget. Contact the Gadget team on [Discord](https://ggt.link/discord) if your sync takes longer than 15 minutes!

If you need to sync on a regular schedule, you can use a [scheduled action](https://docs.gadget.dev/guides/actions/triggers#scheduler-trigger) to run the sync at a set interval.

### Handle BigCommerce rate limits 

Rate limits in BigCommerce are per store and shared across all apps installed on a store. This means that 429 response codes must be handled properly.

The included [BigCommerce API client](https://github.com/Space48/bigcommerce-api-js/blob/master/README.md) will automatically handle rate limits for you and will retry requests that return 429 or 5XX response codes using an exponential backoff.

Gadget's [background actions](https://docs.gadget.dev/guides/actions/background) can also be used to manage rate limits. You can set the concurrency of a background action to match the rate limit of the BigCommerce API and use the built-in retry logic to handle rate limit errors.

To do this, use the `maxConcurrency` option in the `enqueue` function to limit how many actions are run concurrently.

For example, to create new products in BigCommerce, you would run the following action in a background action:

```typescript
export const run: ActionRun = async ({ params, logger, api, connections }) => {
  // get BigCommerce API client for the store
  const bigcommerce = await connections.bigcommerce.forStoreHash(params.storeHash!);

  // use the API client to fetch 5 products, and return
  await bigcommerce?.v3.post("/catalog/products", {
    body: {
      name: "My new product",
      type: "physical",
      price: 10,
      weight: 1,
    },
  });
};

// define product param
export const params = {
  product: {
    type: "object",
    properties: {
      name: { type: "string" },
      price: { type: "number" },
      weight: { type: "number" },
    },
  },
  storeHash: { type: "string" },
};
```

And then enqueue the action with a `maxConcurrency` option:

```typescript
export const run: ActionRun = async ({ params, logger, api, connections }) => {
  if (!params.products) {
    throw new Error("The `products` param is required to run this action");
  }
  // loop over products from passed in params
  for (const product of params.products) {
    await api.enqueue(
      api.bigcommerce.createProduct,
      {
        product,
        storeHash: connections.bigcommerce.currentStoreHash,
      },
      {
        // use maxConcurrency to limit how fast enqueued actions are run
        queue: {
          name: "create-products",
          maxConcurrency: 2,
        },
      }
    );
  }
};

export const params = {
  products: {
    type: "array",
    items: {
      type: "object",
      properties: {
        name: { type: "string" },
        price: { type: "number" },
        weight: { type: "number" },
      },
    },
  },
};
```

When using background actions, you will need to pass the `storeHash` to the enqueued action as a parameter. This is because the current session and data tenancy is not passed when you use the `api` to make requests.

## Working with BigCommerce metafields 

Similar to any other resource in BigCommerce, you can read and write metafields using the BigCommerce API client. You might wish to store metafield data in line with the resource data in Gadget.

There are two ways to store metafield data next to your model data:

1.  Subscribe to [metafield webhooks](https://developer.bigcommerce.com/docs/integrations/webhooks/events#metafields) in a global action and update the model data in Gadget when a metafield is created, updated, or deleted. [Resources that can store metafield data](https://developer.bigcommerce.com/docs/integrations/metafields) have their own metafield webhook topics, so you can subscribe to these topics and update your model data accordingly.
    
2.  Fetch metafield data when you fetch the resource data and store it in the model. This would be required when syncing data into your Gadget database and the code would look similar to the .
    

Subscribe to [metafield webhooks](https://developer.bigcommerce.com/docs/integrations/webhooks/events#metafields) in a global action and update the model data in Gadget when a metafield is created, updated, or deleted. [Resources that can store metafield data](https://developer.bigcommerce.com/docs/integrations/metafields) have their own metafield webhook topics, so you can subscribe to these topics and update your model data accordingly.

Fetch metafield data when you fetch the resource data and store it in the model. This would be required when syncing data into your Gadget database and the code would look similar to the .

### Subscribe to metafield webhooks 

Metafield payloads always have a `metafield_id` and a `resource_id` field, which you can use to update the correct record in your model:

```json
{
  "metafield_id": 2,
  "namespace": "Sales Department",
  "resource_id": "133",
  "storeHash": "bbcolvaxts"
}
```

Both the `resource_id` and `metafield_id` should be stored in your model to be able to update and delete metafields.

Here is an example of a `store/product/metafield/created` webhook subscription that updates the `bigcommerce/product` model when a metafield is created for a product. The `product` model stores `resource_id` as `bigcommerceId` and `metafield_id` as `metafieldId`.

```typescript
import { ActionOptions } from "gadget-server";

export const run: ActionRun = async ({ params, logger, api, connections }) => {
  // get the BigCommerce API client
  const bigcommerce = connections.bigcommerce.current;

  if (!bigcommerce) {
    throw new Error("Missing bigcommerce connection");
  }

  // get ids from webhook payload
  const productId = params.resource_id;
  const metafieldId = params.metafield_id;

  // fetch the metafield value from BigCommerce
  const metafield = await bigcommerce.v3.get(
    "/catalog/products/{product_id}/metafields/{metafield_id}",
    {
      path: {
        product_id: productId,
        metafield_id: metafieldId,
      },
    }
  );

  if (!metafield) {
    throw new Error("Missing metafield");
  }

  // see if the product exists in the Gadget db
  let product = await api.bigcommerce.product.maybeFindByBigCommerceId(
    parseInt(productId)
  );

  // if the product doesn't exist in Gagdet, fetch it
  if (!product) {
    product = await bigcommerce.v3.get("/catalog/products/{product_id}", {
      path: {
        product_id: productId,
      },
    });

    // and add the product to the Gadget db
    return await api.bigcommerce.product.create({
      bigcommerceId: product.id,
      store: {
        // get the bigcommerce/store id for the record stored in Gadget
        _link: connections.bigcommerce.currentStoreId,
      },
      name: product.name,
      metafieldId,
      metafieldValue: metafield.value,
    });
  }

  // update the existing product
  return await api.bigcommerce.product.update(product.id, {
    metafieldId,
    metafieldValue: metafield.value,
  });
};

export const options: ActionOptions = {
  triggers: {
    api: false,
    bigcommerce: {
      webhooks: ["store/product/metafield/created"],
    },
  },
};
```

You can also use the namespace and key to identify the metafield, which is useful when you have multiple metafields for a resource.

## Data security and multi-tenancy 

If you are building a public app, you will need to ensure that data is stored securely and that one store's data is not accessible by another store. Gadget provides tools that help manage multi-tenancy and row-level security (RLS) for models and actions.

### Securing models and model actions 

Gadget provides a set of data access tools that can be used to secure models and their actions. These tools include:

*   [relationship fields](https://docs.gadget.dev/guides/models/relationships) on models so that you can associate data with a `store` record
*   [a Uniqueness validation](https://docs.gadget.dev/guides/models/fields#uniqueness-validation) to make sure that each store has only one record with a given ID
*   built-in functions for [preventing cross-shop data access](https://docs.gadget.dev/reference/gadget-server#preventcrossshopdataaccess-params-record-options) in model actions
*   [filtered model permissions](https://docs.gadget.dev/guides/access-control#filtered-model-permissions) that restrict access based on the related `store`

Follow these steps to set up these tools and enforce multi-tenancy on a model:

1.  If you haven't already done so, add a relationship field to any custom model that associates the model with a `store` record

For example, add a belongs to relationship field on the `bigcommerce/product` model. The field should be named `store` and should reference the `bigcommerce/store` model, so that `product belongsTo store`. Then define the inverse of the relationship so that `store hasMany products`.

2.  Add a Uniqueness validation to the BigCommerce resource ID field on your model and scope it to the `store` relationship. This is important to ensure that stored IDs are unique per store. See  for more information.
    
3.  In your custom model, import `preventCrossStoreDataAccess` from `gadget-server/bigcommerce` and use it in your model actions
    

Add a Uniqueness validation to the BigCommerce resource ID field on your model and scope it to the `store` relationship. This is important to ensure that stored IDs are unique per store. See  for more information.

In your custom model, import `preventCrossStoreDataAccess` from `gadget-server/bigcommerce` and use it in your model actions

For example, in the `bigcommerce/product` model, you can call `preventCrossStoreDataAccess` in the `run` function of the `create` action to ensure that a product can only be created for the current store:

```typescript
import { applyParams, save, ActionOptions } from "gadget-server";
// import the preventCrossStoreDataAccess function
import { preventCrossStoreDataAccess } from "gadget-server/bigcommerce";

export const run: ActionRun = async ({
  params,
  record,
  logger,
  api,
  connections,
}) => {
  applyParams(params, record);
  // use the preventCrossStoreDataAccess function to make sure
  // the product is created for the current store
  await preventCrossStoreDataAccess(params, record);
  await save(record);
};

export const options: ActionOptions = {
  actionType: "create",
};
```

Each time `preventCrossStoreDataAccess` runs, it will make sure the given record has the correct `storeId` for the store processing this action. For existing records, that means it will verify that the `storeId` of the saved record the matches the `storeId` on the incoming `record`.

4.  Add a tenancy filter to the `read` action for the custom model in `accessControl/permissions`.

Tenancy filters are expressed with Gelly, Gadget's data access language. These filters check the store ID of the current session against the ID of the store that the data being read belongs to.

Most tenancy filters for BigCommerce look similar to this:

```gelly
// in accessControl/filters/bigcommerce/product.gelly
filter ($session: Session) on BigCommerceStore [
  where storeId == $session.bigcommerceStoreId
]
```

When you set up a relationship field in Gadget, an `id` field for the related model is automatically generated on the belongs to side of the relationship. For example, when a relationship field named `store` is added a `storeId` field will also be available on the child model. This field is used to store the ID of the associated `store` record and is useful in tenancy filters or when manually filtering API requests per store.

### Securing global actions and HTTP routes 

When using global actions or HTTP routes to read or write data, you cannot use Gelly tenancy filters or functions like `preventCrossStoreDataAccess` because there is no backing model or relationship to a `store` record.

For global actions or HTTP routes, you often need to filter data based on the `bigcommerceStoreId`, which is included as a properly on the [`connections` parameter](https://docs.gadget.dev/guides/actions/code#connections). Filtering is built into your app's API and available on most field types. Docs for filtering are available in the [API reference](https://docs.gadget.dev/api/example-app/development/sorting-and-filtering#filtering).

For example, if I am fetching `bigcommerce/order` data in a global action before doing some processing and returning the results, I would do the following:

```typescript
export const run: ActionRun = async ({ params, logger, api, connections }) => {
  const bigcommerceStoreId = connections.bigcommerce.currentStoreId;

  // filter results by the store ID
  const orders = await api.bigcommerce.order.findMany({
    filter: {
      storeId: {
        equals: bigcommerceStoreId,
      },
    },
  });

  return orders;
};
```

This will only work when there is an authenticated `session`!

Otherwise, you will need to manually fetch the store ID from `bigcommerce/store` using the store hash:

```typescript
export const run: ActionRun = async ({ params, logger, api, connections }) => {
  // get the store ID from the record in Gadget
  const bigcommerceStore = await api.bigcommerce.store.findByStoreHash(
    params.storeHash!,
    {
      // only select the id field
      select: {
        id: true,
      },
    }
  );

  // filter results by the store ID
  const orders = await api.bigcommerce.order.findMany({
    filter: {
      storeId: {
        equals: bigcommerceStore.id,
      },
    },
  });

  return orders;
};

// allow the storeHash to be passed as a parameter
export const params = {
  storeHash: { type: "string" },
};
```

The same pattern can be used for HTTP routes, where you would fetch the store ID from the `bigcommerce/store` model and then filter the results based on the store ID.