Working with Shopify webhooks 

Shopify webhooks within model actions 

Shopify webhooks within model actions in Gadget ensure your application is kept up to date with the latest Shopify store data.

When an event occurs in your Shopify store, such as the creation, update, or deletion of a product, Shopify sends a webhook to your Gadget application. This webhook contains information about the event that just occurred.

In Gadget, each Shopify model has corresponding actions (create, update, delete) that are triggered by these webhooks. For instance, if Shopify sends a products/create webhook, Gadget will run the create action on the Product model in your app. This action uses the incoming parameters from the webhook to create a new record in your Gadget database.

To handle these webhooks:

1. Gadget verifies the authenticity of the webhook to ensure it's coming from Shopify.
2. It then triggers the corresponding action associated with the webhook event.
3. The complete payload of the webhook is accessible within the trigger object passed to your action code, allowing you to use the data within your action's logic.
4. If an action fails, Gadget will retry it up to 10 times with increasing delays between attempts. If the action fails on the last attempt, the webhook is considered "lost", though Gadget will later attempt to automatically reconcile the missing webhook with Shopify.

Reconciliation of webhooks 

As described in the above process, in the case when a webhook fails or is missed, Gadget automatically catches the missed webhook during a daily reconciliation and re-runs the corresponding action.

Effectively, this means that your data will always go back in sync with Shopify, even after an outage, because of the automated reconciliation that is built into our webhook processing.

Gadget's reconciliation for missed webhooks includes support for webhooks that have filters or includeFields options set, and will run any missed actions with the same params as a webhook-triggered action would receive.

Shopify webhooks within global actions 

Shopify webhooks can trigger global actions if a Shopify connection has already been set up on a Gadget app. To run a global action when a webhook is received, add the Shopify data trigger to your global action, and configure which webhook topics to listen to. Gadget will register the webhook topics, and trigger the global action when Shopify sends webhook payloads.

Shopify data triggers on global actions are similar to Shopify data triggers on model actions, however, there is no nightly reconciliation for global action webhook triggers. If a webhook is missed, its payload will not be retrieved at a later time and the action will not be run.

Only webhook topics granted by the access scopes selected in a Shopify connection will be available as global action triggers. You can select multiple webhook topics to dispatch to the same global action as well.

If one or more webhooks have been disabled for a particular shop, those webhooks will also no longer trigger any global actions.

Adding a Shopify webhook trigger to a global action 

To add a Shopify data trigger to a global action:

1. Select your global action in api/actions
2. Click + in the TRIGGERS panel and select Shopify
3. Select one or more webhook topics

The webhook payload can be found in the trigger object provided by the action context parameter.

1export const run: ActionRun = async ({ trigger, logger }) => {
2  // access product payload attributes at trigger.payload.id
3  // access the topic at trigger.topic
4  const isShopifyTrigger = trigger.type === "shopify_webhook";
5  if (isShopifyTrigger) {
6    logger.info(
7      {
8        productID: trigger.payload.id,
9        title: trigger.payload.title,
10        topic: trigger.topic,
11      },
12      "product update webhook happened"
13    );
14  }
15};
16
17export const options = {
18  triggers: {
19    shopify: {
20      webhooks: ["products/update"],
21    },
22    api: false,
23  },
24};

1export const run: ActionRun = async ({ trigger, logger }) => {
2  // access product payload attributes at trigger.payload.id
3  // access the topic at trigger.topic
4  const isShopifyTrigger = trigger.type === "shopify_webhook";
5  if (isShopifyTrigger) {
6    logger.info(
7      {
8        productID: trigger.payload.id,
9        title: trigger.payload.title,
10        topic: trigger.topic,
11      },
12      "product update webhook happened"
13    );
14  }
15};
16
17export const options = {
18  triggers: {
19    shopify: {
20      webhooks: ["products/update"],
21    },
22    api: false,
23  },
24};

When to use webhook triggers within global actions 

It's best to use webhook triggers on global actions when you want to run code that doesn't affect the state of your application or database. This is because there is no reconciliation for these triggers, so data integrity and consistency can't be guaranteed.

Below are some examples where you might use webhook triggers within global actions.

Use cases 

• Real-time notifications and alerts: You can use webhooks to send notifications to your team or customers. For example, you might send a Slack message to your fulfillment team when a new order is created, or notify a customer via SMS or email when their order status changes.
• Forward data to an external service: You can forward a webhook payload to an existing service, for example pushing Shopify order data to an ERP. Depending on the type of data being forwarded, you may want the resiliency built into model action webhooks for this use case.

Disable webhook processing per shop 

Gadget supports limiting which webhooks are processed by certain shops to reduce webhook volume. Certain Shopify shops generate a ton of webhooks. Processing and storing these webhooks can add up to cost too much, especially if you aren't making use you don't want to pay to process and consume webhooks for all models for all shops that have installed your app. For example, within your Shopify app, you may want to disable certain webhooks from processing on Shopify models for free trial users so you do not end up adding up your costs for users on your free trial.

Within Gadget, you can disable certain webhooks from processing so you have greater control over your app. Every shopifyShop model has a disabledWebhooks field which is a boolean value set to false by default. When the value is changed to true this will disable webhooks within the scheduled sync from syncing data concerning the model associated with the disabledWebhooks value changing.

For example, let's take a look below at our update action with our shopifyShop model where within the run function we'll set our disabledWebhooks field to true when it's a shopifyProduct model. What we've defined here is any reconciliation sync going forward on webhook re-registrations will disable webhook processing and won't sync data for the shopifyProduct model. If we also head to our shopifyShop data we'll notice within the registeredWebhooks column that the shopifyProduct is no longer associated with its value.

1import { applyParams, save, ActionOptions } from "gadget-server";
2import { preventCrossShopDataAccess } from "gadget-server/shopify";
3
4export const run: ActionRun = async ({
5  params,
6  record,
7  logger,
8  api,
9  connections,
10}) => {
11  applyParams(params, record);
12
13  record.disabledWebhooks = { shopifyProduct: true };
14  // Disabling all webhooks that sync data for the shopifyProduct model
15
16  await preventCrossShopDataAccess(params, record);
17  await save(record);
18};
19
20export const onSuccess: ActionOnSuccess = async ({
21  params,
22  record,
23  logger,
24  api,
25  connections,
26}) => {
27  // Your logic goes here
28};
29
30export const options: ActionOptions = {
31  actionType: "update",
32};

1import { applyParams, save, ActionOptions } from "gadget-server";
2import { preventCrossShopDataAccess } from "gadget-server/shopify";
3
4export const run: ActionRun = async ({
5  params,
6  record,
7  logger,
8  api,
9  connections,
10}) => {
11  applyParams(params, record);
12
13  record.disabledWebhooks = { shopifyProduct: true };
14  // Disabling all webhooks that sync data for the shopifyProduct model
15
16  await preventCrossShopDataAccess(params, record);
17  await save(record);
18};
19
20export const onSuccess: ActionOnSuccess = async ({
21  params,
22  record,
23  logger,
24  api,
25  connections,
26}) => {
27  // Your logic goes here
28};
29
30export const options: ActionOptions = {
31  actionType: "update",
32};

Handling deletion webhooks 

Any deletion webhooks will call the delete Action on the model, along with running the delete Action for all Shopify records associated with that model. For example, when a product is deleted, we delete all of its options, images, and variants. All other webhooks will call either the create or update Action, based on whether or not the record currently exists in Gadget.

Registering Shopify webhooks 

In some cases, you may need to manually register webhooks to ensure your Shopify app receives the necessary events from a store.

You should only have to manually register webhooks on development environments. Gadget automatically updates webhook registrations during deployment, so you should not need to manually register webhooks for a production environment. If the selected access scopes have changed and are deployed to production, stores that have installed your app will need to re-authenticate to grant your app access to the new scopes, and webhook registration will be done at that time.

Here are the scenarios when you should manually register Shopify webhooks in development environments:

1. When editing connections: If you edit a connection to include new models, you may need to register new webhooks for those models. For example, if you add the Collection model, you should register webhooks to receive collections/create, collections/update, and collections/delete. If you edit the connection to include additional access scopes, you will have to re-authenticate your app on your development store to grant the app access to the new scopes, and new webhooks will be automatically registered.

2. When missing webhooks: If you notice that certain webhooks are not registered, as indicated on the Installs page of your Shopify connection, you should manually register the missing webhooks. This can happen if you are missing access scopes or topics/namespaces for your registered webhooks.

3. API version upgrade: Shopify releases a new API version every 3 months, and when you upgrade your Shopify Connection's API version, you should ensure that your webhooks are still correctly registered for the new version.

4. When accessing protected customer data: If you are using an Event subscriptions version later than 2022-07, you will need to request access permissions in the Protected customer data access section on the API access page of your Shopify Partners app. After completing the required sections, return to your Gadget app and register the webhooks.

How to register Shopify webhooks 

To register Shopify webhooks, you can:

1. Navigate to the Installs page for your Connection in Gadget (Settings -> Plugins -> Shopify)
2. Click the ⋮ button for any stores you have re-authenticated, and select the Register Webhooks option:

Filtering Shopify webhooks beta 

Shopify allows you to filter the events that trigger a webhook for your application. There's two ways to filter webhooks in Shopify:

• limit which records are delivered to your app, which you can do with a webhook filter
• limit which fields of each record are delivered to your app, which you can do with an includeFields list

Both of these strategies can be used individually or together to limit the number of webhook Shopify sends to your app, and lower your total Gadget compute time.

Webhook filters 

In Gadget, you can use the same filter syntax provided by Shopify to filter your action's Shopify data triggers. When a filter is added, Gadget will register the webhook with the filter using Shopify's API, and Shopify will start only sending data to your app that passes the filter. Your filter must be written in Shopify's filter syntax, and can be applied to any Shopify model actions or global actions with the Shopify Data trigger added.

Note that there are some notable differences between Shopify's filtering and search syntax that are important to be aware of when building webhook filters.

Applying a webhook filter 

You can apply webhook filters or include fields lists to root model actions triggered by Shopify webhooks, or to webhook-triggered global actions.

To apply a filter to a Shopify data trigger:

1. Navigate to an action triggered by a Shopify webhook.
2. Open the menu on the filter card and select Filter.
3. Enter your filter in the trigger card on the action. This will add the trigger to your action options in your source code.
4. Gadget will automatically re-register your webhooks in your development environment, and re-register them in your production environment when you deploy.

For example, you could use the following filter on product webhook triggers if you only wanted webhooks to be fired when the product vendor is Nike:

1// a sample filter for a product webhook
2export const options: ActionOptions = {
3  triggers: {
4    shopify: {
5      shopifyFilter: "vendor:Nike",
6    },
7  },
8};

1// a sample filter for a product webhook
2export const options: ActionOptions = {
3  triggers: {
4    shopify: {
5      shopifyFilter: "vendor:Nike",
6    },
7  },
8};

You can also use filters on fields of child models, such as the price of product variants. Webhooks will be fired if ANY of the children pass the filter condition.

For example, you can filter to only receive webhooks for products if any of that product's variants have a price greater than $10.00:

1// a sample filter for a product webhook that filters on variant fields
2export const options: ActionOptions = {
3  triggers: {
4    shopify: {
5      shopifyFilter: "variants.price:>=10.00",
6    },
7  },
8};

1// a sample filter for a product webhook that filters on variant fields
2export const options: ActionOptions = {
3  triggers: {
4    shopify: {
5      shopifyFilter: "variants.price:>=10.00",
6    },
7  },
8};

Filters are also applied to the nightly reconciliation job that Gadget runs for your Shopify apps. Only missed events that would have triggered a webhook will be re-run.

Filters are not applied to any Shopify data syncs.

Syntax errors in webhook filters will be reported by Shopify and will produce an error in your Gadget logs. Gadget will also alert you to a failed webhook registration. See Shopify's guide on debugging filters for more details.

Modify webhook payloads with includeFields 

Gadget also supports Shopify's includeFields option that allows you to modify your webhook payload. This allows you to limit the data Shopify adds to a webhook payload so only a subset of the typical payload data is returned. If a record changes, but no fields in your includeFields list change, Shopify will not send a webhook, and you won't incur any Gadget compute time. If fields in your includeFields list do change, the webhook will be sent with only those fields, so you can still subscribe to the data you care about, and ignore the data you don't.

To modify webhook payloads in Gadget:

1. Navigate to an action triggered by a Shopify webhook.
2. Open the menu on the filter card and select Include fields.
3. Enter a comma-separated list of fields that will be included in the webhook payload. Ensure you include the id of any records for model actions, as it is necessary for Gadget to identify the record.
4. Gadget will automatically re-register your webhooks in your development environment, and re-register them in your production environment when you deploy.

For example, here is how you could limit a products/create webhook payloads to only include the product title:

1// a sample includeFields for a products/update webhook
2// note: we must include the id field
3export const options: ActionOptions = {
4  triggers: {
5    shopify: {
6      includeFields: ["id", "title"],
7    },
8  },
9};

1// a sample includeFields for a products/update webhook
2// note: we must include the id field
3export const options: ActionOptions = {
4  triggers: {
5    shopify: {
6      includeFields: ["id", "title"],
7    },
8  },
9};

You can also include fields across child models, such as the price on all a product's variants:

1// a sample includeFields for a products/update webhook
2// note: we must include the id field
3export const options: ActionOptions = {
4  triggers: {
5    shopify: {
6      includeFields: ["id", "variants.id", "variants.price"],
7    },
8  },
9};

1// a sample includeFields for a products/update webhook
2// note: we must include the id field
3export const options: ActionOptions = {
4  triggers: {
5    shopify: {
6      includeFields: ["id", "variants.id", "variants.price"],
7    },
8  },
9};

Handling duplicate webhook payloads from Shopify 

Using includeFields with Shopify webhooks helps reduce unnecessary processing in two ways:

1. It reduces payload size by only sending specified fields
2. Shopify doesn't send duplicate payloads

When includeFields is used, the webhook payloads contain less data, and often exclude fields that may have changed. Shopify compares these new, smaller payloads against recently sent ones. If they're identical, Shopify skips sending the duplicate. Without includeFields, every payload contains all fields and is considered unique, so all are delivered.

Shopify's deduplication system is best-effort and only works within a short time window. Our observations suggest this window is approximately 10 minutes, though this duration appears to vary. This means you may still receive duplicate webhooks if they're sent outside this window.

Gadget's additional deduplication layer 

To provide more reliable deduplication, Gadget deduplicates webhooks sent to model actions to further reduce unnecessary processing:

• When Gadget receives a webhook payload, it compares the incoming data against existing model data. If the model has an updated_at field, and the field is included in the webhook payload, Gadget will use that value to determine if the record has changed. Otherwise, Gadget will compare each incoming field in the payload to what is stored in the database for each record in the payload.
• For any records with detected differences, Gadget will run actions to update the stored record with the new data. For records that haven't changed, Gadget skips running any actions.
• If nothing has changed, Gadget skips running any actions entirely.

This deduplication process happens automatically and adds no additional compute time.

Excluding the updated_at field 

The updated_at timestamp field from Shopify changes almost every time a new webhook is delivered, as Shopify bumps it when any relevant changes are made to a resource. For this reason, if you include it in your includeFields list, you will still receive a webhook for every change to a resource, which can trigger a lot of unnecessary actions. If updated_at is in your included field list, Gadget will still deduplicate webhooks, but since the value is different for most payloads, your actions will still run most of the time.

If you want to minimize action processing time, and you don't need the updated_at field's value, Gadget recommends excluding the updated_at field from your includeFields list. With this in place, you will receive fewer webhooks from Shopify, and Gadget's deduplication will be much more effective at preventing unnecessary actions.

Combining webhook filters and payload modification 

You can combine webhook filters and includeFields to limit both which records and which fields are delivered to your app. If you are using webhook filtering at the same time as includeFields for the same webhook, you must ensure that all fields used in the filter are part of the webhook payload. Otherwise Shopify may not be able to register the webhook, or the filter condition could not work as expected.

Using different filters on different model actions 

You can use different webhook filters or includeFields trigger settings for topics on the same Shopify resource. To do this, set the filter for the Gadget model action corresponding to the topic you want to adjust.

For the shopifyProduct model for example, here's how the model actions correspond to the Shopify webhook topics Gadget will register:

To filter on nested models, like shopifyProductVariant, you generally need to set filters or includeFields on both the parent model's create and update actions. This is because the update topic for a resource can create, update, or delete children records, so you need to set the filter or includeFields on both actions to ensure you get all the data you need.

For example, if you want to know everything about a product when it is created, but then only find out about changes to the title in the future as products are updated, you can set no filter on the shopifyProduct.create action, and set includeFields on the shopifyProduct.update action to only include id and title.

Using multiple different filters on the same topics 

Gadget supports only one filter and includeFields per Shopify Data trigger, so Shopify model actions can only have one filter or includeFields set. You can combine different filters with Shopify's filtering syntax using the and and or operators.

If you need two independent filters, you can also use Gadget's global actions with the Shopify Data trigger. To register two different, independent filters, create two different global actions, and add the Shopify Data trigger to each. You can then pick which topics each global action should register webhooks for, and set independent filters or includeFields on each. Under the hood, Gadget will register one webhook for each global action if their filters are different, and Shopify will send webhooks to the global actions accordingly.

Filling in ignored records or fields later 

If you are using a webhook filter to ignore certain records, or includeFields to ignore changes to certain fields, Gadget won't automatically populate that data later. If you do need the data eventually, you can fill in those records or fields later in your app by running a sync. It's up to you to decide how often you want to run a sync to fill in the data.

A common pattern is to schedule a daily sync to run against models you want to fill in all the data for. You can run syncs for all your installed shops on a schedule with a global action like this:

1import { ActionOptions } from "gadget-server";
2import { globalShopifySync } from "gadget-server/shopify";
3
4const HourInMs = 60 * 60 * 1000;
5
6export const run: ActionRun = async ({ params, logger, api, connections }) => {
7  const syncSince = new Date(Date.now() - 25 * HourInMs);
8
9  await globalShopifySync({
10    apiKeys: connections.shopify.apiKeys,
11    syncSince,
12    models: ["shopifyProduct", "shopifyProductVariant"],
13  });
14};
15
16export const options: ActionOptions = {
17  triggers: {
18    scheduler: [
19      {
20        every: "day",
21        at: "08:00 UTC",
22      },
23    ],
24  },
25};

1import { ActionOptions } from "gadget-server";
2import { globalShopifySync } from "gadget-server/shopify";
3
4const HourInMs = 60 * 60 * 1000;
5
6export const run: ActionRun = async ({ params, logger, api, connections }) => {
7  const syncSince = new Date(Date.now() - 25 * HourInMs);
8
9  await globalShopifySync({
10    apiKeys: connections.shopify.apiKeys,
11    syncSince,
12    models: ["shopifyProduct", "shopifyProductVariant"],
13  });
14};
15
16export const options: ActionOptions = {
17  triggers: {
18    scheduler: [
19      {
20        every: "day",
21        at: "08:00 UTC",
22      },
23    ],
24  },
25};

This strategy achieves a good balance of cost vs data freshness. Webhooks will deliver up to date data in real-time for your app, so for the things you really care about freshness for you can use them, but filter out data you don't care about. You can then ensure it doesn't drift too far out of sync by running a sync to refresh the other data at whatever schedule meets your requirements.

Shopify webhook status 

The status and timeline of a Shopify webhook can be observed within the Queues dashboard in Gadget. Here you'll be able to view all Shopify webhooks running and observe the progress in how they're running.