You will often want to store BigCommerce data in your Gadget database after subscribing to BigCommerce webhooks.
This guide will show you how to read and write data to BigCommerce using Gadget, and how you can store BigCommerce data in your Gadget database.
Calling the BigCommerce API
Before you can store data, you need to fetch it from BigCommerce! BigCommerce webhooks often only provide a resource ID, so you will need to use the BigCommerce API to fetch the full resource data.
Data stored in Gadget models is not automatically synced with BigCommerce. The best way to keep data in BigCommerce and Gadget in sync is
to write data back to BigCommerce using the API client, and then allow webhooks to update data in Gadget.
Store context for API client
The connections.bigcommerce object provides access to a BigCommerce API client. You can use this client to read and write data to BigCommerce.
Depending on how an action was triggered, the current store context may be available on the connections object. For example, if you trigger a global action from a BigCommerce webhook, or call an action from a single-click frontend, the current store context will be available with connections.bigcommerce.current:
init a BigCommerce client using connections.bigcommerce.current
If the current store context is not available, such as calling an action from another action in Gadget, you can use the forStoreHash method to get the API client for a specific store hash:
You can store data read from BigCommerce in Gadget data models. This allows you to access all data required to power your application in a single, highly available and performant, non-rate limited database.
When you set up your BigCommerce connection, a store data model was created at api/models/bigcommerce/store. You can also add custom models to the bigcommerce namespace.
It's also important to relate any additional BigCommerce resource models to the bigcommerce/store model so that you can associate data
with a specific store!
To add a custom BigCommerce data model:
Right-click on the api/models/bigcommerce directory in the Gadget editor and select Add model
Give your model a name, for example product or order
26// get the bigcommerce/store id for the record stored in Gadget
27_link: connections.bigcommerce.currentStoreId,
28},
29name: product.name,
30// ... any other product fields that need to be stored
31});
32};
33
34exportconstoptions:ActionOptions={
35triggers:{
36bigcommerce:{
37webhooks:["store/product/created"],
38},
39},
40};
Storing BigCommerce IDs
It is important to store BigCommerce resource IDs in Gadget! This is the unique identifier for the record in BigCommerce, and is required to handle update or delete webhooks, and associate data stored in Gadget with BigCommerce records.
It's also a good idea to add both the Required and Uniqueness validations to the bigcommerceId field on your models! For multi-tenant applications, you should also scope this uniqueness validation to the store relationship so that each store can only have one record with a given bigcommerceId.
Upserting data and handling /updated webhooks
When you are storing data from BigCommerce in Gadget, you may want to update existing records if they already exist, or create a new record if it doesn't. This is known as upserting data. This is particularly useful when handling /updated webhooks, where a record may or may not already exist in your Gadget database.
Your Gadget API provides an upsert meta action that manages the upsert logic for you. For BigCommerce data, you often want to upsert based on the bigcommerceId field. For multi-tenant applications, you should also scope the upsert to the store relationship. This ensures that you only update or create records for the current store.
Here is an example of how you could upsert product data in Gadget when a store/product/updated webhook is received:
19// upsert product data in the product data model
20await api.bigcommerce.product.upsert({
21bigcommerceId: product.id,
22store:{
23// get the bigcommerce/store id for the record stored in Gadget
24_link: connections.bigcommerce.currentStoreId,
25},
26name: product.name,
27// ... any other product fields that need to be stored
28// use bigcommerceId and store to identify unique records for upsert
29on:["bigcommerceId","store"],
30});
31};
32
33exportconstoptions:ActionOptions={
34triggers:{
35bigcommerce:{
36webhooks:["store/product/updated"],
37},
38},
39};
This will check to see if a product with the given bigcommerceId and store relationship already exists in Gadget. If it does, it will update the existing record. If it doesn't, it will create a new record.
Storing data from child models
You can store data from child models in Gadget by adding relationships between models. When you create a new model, you can add a relationship field to associate the new model with the parent model.
An example of this relationship is products and product variants. You can create a bigcommerce/productVariant model and associate it with the bigcommerce/product model using a relationship field.
Whether you are syncing or creating data from a webhook, you can include the required child data when reading from BigCommerce so it is returned in the same payload.
Include variant information when retrieving product data
You also have the option to make a separate API call to retrieve the child data and store it in Gadget. Refer to the BigCommerce API reference for more information on retrieving child data.
Once you have the child data, you can store it in the child model using the CRUD API.
For example, here is how you could store product variant data in Gadget when a store/product/created webhook is received:
45// get the bigcommerce/store id for the record stored in Gadget
46_link: connections.bigcommerce.currentStoreId,
47},
48variants:{
49// use a _converge to relate the variants to the product and add them to the db
50_converge:{
51values: variants,
52},
53},
54});
55};
56
57exportconstoptions:ActionOptions={
58triggers:{
59bigcommerce:{
60webhooks:["store/product/created"],
61},
62},
63};
Syncing all existing data
If you need to sync all existing data from a BigCommerce store, you can use the BigCommerce API client to fetch all data and store it in Gadget. There are a couple of things you need to be aware of and handle properly, including the BigCommerce API rate limits and max page size for fetching data.
Gadget's built-in background actions queue can help you manage this process. Background actions allow developers to set custom concurrency controls to adhere to a rate limit, and have built-in retry logic for failed actions, ensuring that all data is synced.
To maximize the speed of the sync, you can use bulk actions to enqueue multiple actions at once. For a data sync, you will often want to use Gadget's upsert meta API so that you can update existing records and create new records in a single action.
Here is an example of how you could sync all product data from a BigCommerce store:
This will sync all products from a BigCommerce store to Gadget. It may take some time before all records are synced!
Action timeout
This action has a timeout of 15 minutes, which is the maximum time an action can run in Gadget. Contact the Gadget team on Discord
if your sync takes longer than 15 minutes!
If you need to sync on a regular schedule, you can use a scheduled action to run the sync at a set interval.
Handle BigCommerce rate limits
Rate limits in BigCommerce are per store and shared across all apps installed on a store. This means that 429 response codes must be handled properly.
The included BigCommerce API client will automatically handle rate limits for you and will retry requests that return 429 or 5XX response codes using an exponential backoff.
Gadget's background actions can also be used to manage rate limits. You can set the concurrency of a background action to match the rate limit of the BigCommerce API and use the built-in retry logic to handle rate limit errors.
To do this, use the maxConcurrency option in the enqueue function to limit how many actions are run concurrently.
For example, to create new products in BigCommerce, you would run the following action in a background action:
14// use maxConcurrency to limit how fast enqueued actions are run
15queue:{
16name:"create-products",
17maxConcurrency:2,
18},
19}
20);
21}
22};
23
24exportconst params ={
25products:{
26type:"array",
27items:{
28type:"object",
29properties:{
30name:{type:"string"},
31price:{type:"number"},
32weight:{type:"number"},
33},
34},
35},
36};
Passing the storeHash
When using background actions, you will need to pass the storeHash to the enqueued action as a parameter. This is because the current
session and data tenancy is not passed when you use the api to make requests.
Working with BigCommerce metafields
Similar to any other resource in BigCommerce, you can read and write metafields using the BigCommerce API client. You might wish to store metafield data in line with the resource data in Gadget.
There are two ways to store metafield data next to your model data:
Subscribe to metafield webhooks in a global action and update the model data in Gadget when a metafield is created, updated, or deleted. Resources that can store metafield data have their own metafield webhook topics, so you can subscribe to these topics and update your model data accordingly.
Fetch metafield data when you fetch the resource data and store it in the model. This would be required when syncing data into your Gadget database and the code would look similar to the sync example.
Subscribe to metafield webhooks
Metafield payloads always have a metafield_id and a resource_id field, which you can use to update the correct record in your model:
json
1{
2"metafield_id":2,
3"namespace":"Sales Department",
4"resource_id":"133",
5"storeHash":"bbcolvaxts"
6}
Both the resource_id and metafield_id should be stored in your model to be able to update and delete metafields.
Here is an example of a store/product/metafield/created webhook subscription that updates the bigcommerce/product model when a metafield is created for a product. The product model stores resource_id as bigcommerceId and metafield_id as metafieldId.
You can also use the namespace and key to identify the metafield, which is useful when you have multiple metafields for a resource.
Data security and multi-tenancy
If you are building a public app, you will need to ensure that data is stored securely and that one store's data is not accessible by another store. Gadget provides tools that help manage multi-tenancy and row-level security (RLS) for models and actions.
Securing models and model actions
Gadget provides a set of data access tools that can be used to secure models and their actions. These tools include:
relationship fields on models so that you can associate data with a store record
Follow these steps to set up these tools and enforce multi-tenancy on a model:
If you haven't already done so, add a relationship field to any custom model that associates the model with a store record
For example, add a belongs to relationship field on the bigcommerce/product model. The field should be named store and should reference the bigcommerce/store model, so that product belongsTo store. Then define the inverse of the relationship so that store hasMany products.
Add a Uniqueness validation to the BigCommerce resource ID field on your model and scope it to the store relationship. This is important to ensure that stored IDs are unique per store. See storing BigCommerce IDs for more information.
In your custom model, import preventCrossStoreDataAccess from gadget-server/bigcommerce and use it in your model actions
For example, in the bigcommerce/product model, you can call preventCrossStoreDataAccess in the run function of the create action to ensure that a product can only be created for the current store:
Each time preventCrossStoreDataAccess runs, it will make sure the given record has the correct storeId for the store processing this action. For existing records, that means it will verify that the storeId of the saved record the matches the storeId on the incoming record.
Add a tenancy filter to the read action for the custom model in accessControl/permissions.
Tenancy filters are expressed with Gelly, Gadget's data access language. These filters check the store ID of the current session against the ID of the store that the data being read belongs to.
Most tenancy filters for BigCommerce look similar to this:
accessControl/filters/bigcommerce/product.gelly
gelly
filter ($session: Session)onBigCommerceStore[
where storeId ==$session.bigcommerceStoreId
]
Auto-generated ID fields
When you set up a relationship field in Gadget, an id field for the related model is automatically generated on the
belongs to side of the relationship. For example, when a relationship field named store is added a storeId field will
also be available on the child model. This field is used to store the ID of the associated store record and is useful in tenancy filters
or when manually filtering API requests per store.
Securing global actions and HTTP routes
When using global actions or HTTP routes to read or write data, you cannot use Gelly tenancy filters or functions like preventCrossStoreDataAccess because there is no backing model or relationship to a store record.
For global actions or HTTP routes, you often need to filter data based on the bigcommerceStoreId, which is included as a properly on the connections parameter. Filtering is built into your app's API and available on most field types. Docs for filtering are available in the API reference.
For example, if I am fetching bigcommerce/order data in a global action before doing some processing and returning the results, I would do the following:
25// allow the storeHash to be passed as a parameter
26exportconst params ={
27storeHash:{type:"string"},
28};
The same pattern can be used for HTTP routes, where you would fetch the store ID from the bigcommerce/store model and then filter the results based on the store ID.