# Syncing  The syncing system for the Shopify connection works by leveraging actions. Each time Gadget goes to sync the connected shop, Gadget creates a new `shopifySync` record and dispatches the `run` and either the `complete` or `error` actions based on the outcome of the sync. There are two types of syncs for the Shopify connection: API-triggered or manual. ## API-triggered Syncs  An API-triggered sync can be initiated via the API Playground, JS clients, or the `api` object in any action or route code. There are many ways that you can use API-triggered syncs. The three main ways are after a shop installation, on a model's action, and syncing data within a specified date range. Calls to `api.shopifySync.run()` require two parameters: * `domain`, the `.myshopify.com` domain of the shop to sync. * `shop`, a link to the `shopifyShop` record passed as `{ _link: shopId }`. Avoid adding an API-triggered sync to the `shopifySync` model's actions! This will cause an infinite loop of syncs. Instead, we recommend adding API-triggered syncs to the `shopifyShop` model's `install` action to sync store data on app installation, or using a custom action to trigger data syncs. ### Sync on shop install  It may be necessary to run an initial sync upon installation. For example, you may want to display all products on a user interface immediately after a merchant installs your app. To do this, use the `onSuccess` function on the Shopify Shop model's `install` action with the following code: ```typescript // in api/models/shopifyShop/actions/install.ts export const onSuccess: ActionOnSuccess = async ({ record, api }) => { await api.shopifySync.run({ domain: record.domain, shop: { _link: record.id, }, }); }; ``` ```graphql mutation ($shopifySync: RunShopifySyncInput) { runShopifySync(shopifySync: $shopifySync) { success errors { message ... on InvalidRecordError { validationErrors { apiIdentifier message } record model { apiIdentifier } } } shopifySync { __typename id state createdAt domain errorDetails errorMessage models syncSince updatedAt } } } ``` ```json { "shopifySync": { "shop": { "_link": "SHOPID" }, "domain": "SHOPDOMAIN" } } ``` ### Sync by model  You can optionally pass a `models` parameter to the `runSync` call to specify which particular Shopify models to sync. For example, if you wish to sync all `products` in your backend, but not `orders`, the `models` should be passed as an array of `model` API identifiers. If the `models` parameter is omitted or is an empty array, all Shopify models will be synced. ```typescript // in an action file export const run: ActionRun = async ({ params, logger, api, connections }) => { const shop = await api.shopifyShop.findOne("123"); await api.shopifySync.run({ domain: shop.domain, shop: { _link: params.shopId, }, // optional parameter models: ["shopifyProduct"], }); }; export const params = { shopId: { type: "string" }, }; ``` ```graphql mutation ($shopifySync: RunShopifySyncInput) { runShopifySync(shopifySync: $shopifySync) { success errors { message ... on InvalidRecordError { validationErrors { apiIdentifier message } record model { apiIdentifier } } } shopifySync { __typename id state createdAt domain errorDetails errorMessage models syncSince updatedAt } } } ``` ```json { "shopifySync": { "shop": { "_link": "SHOPID" }, "domain": "SHOPDOMAIN", "models": ["shopifyProduct"] } } ``` ### Sync by date-time  You may want to limit syncing to recent records. To do this, you can pass the `syncSince` parameter to indicate which records Gadget should sync. Without this parameter, the API-triggered sync will copy every record to Gadget, regardless of when it was created or updated. By default, Gadget will check the `shopifyUpdatedAt` field for each Shopify record, and only sync records created or updated within the specified window. The timestamp used to sync records can be configured using `syncSinceBy`: * `syncSinceBy: "updated_at"`: Use `shopifyUpdatedAt` to sync records updated in the `syncSince` time window (default). * `syncSinceBy: "created_at"`: Use `shopifyCreatedAt` to sync records created in the `syncSince` time window. ```typescript // in an action file const millisecondsPerDay = 1000 * 60 * 60 * 24; export const run: ActionRun = async ({ api, params }) => { const shop = await api.shopifyShop.findOne("123"); const syncSince = new Date(); // sync from 5 days ago syncSince.setTime(syncSince.getTime() - 5 * millisecondsPerDay); await api.shopifySync.run({ domain: shop.domain, // optional parameter syncSince, shop: { _link: params.shopId }, }); }; export const params = { shopId: { type: "string" }, }; ``` ```graphql mutation ($shopifySync: RunShopifySyncInput) { runShopifySync(shopifySync: $shopifySync) { success errors { message ... on InvalidRecordError { validationErrors { apiIdentifier message } record model { apiIdentifier } } } shopifySync { __typename id state createdAt domain errorDetails errorMessage models syncSince updatedAt } } } ``` ```json { "shopifySync": { "shop": { "_link": "SHOPID" }, "domain": "SHOPDOMAIN", "syncSince": "ISOSTRING" } } ``` You can also sync the most recently created records with `syncSinceBy: "created_at"`: ```typescript // in an action file const millisecondsPerDay = 1000 * 60 * 60 * 24; export const run: ActionRun = async ({ api, params }) => { const shop = await api.shopifyShop.findOne("123"); const syncSince = new Date(); // sync from 5 days ago syncSince.setTime(syncSince.getTime() - 5 * millisecondsPerDay); await api.shopifySync.run({ domain: shop.domain, // optional parameter syncSince, syncSinceBy: "created_at", shop: { _link: params.shopId }, }); }; export const params = { shopId: { type: "string" }, }; ``` ```graphql mutation ($shopifySync: RunShopifySyncInput) { runShopifySync(shopifySync: $shopifySync) { success errors { message ... on InvalidRecordError { validationErrors { apiIdentifier message } record model { apiIdentifier } } } shopifySync { __typename id state createdAt domain errorDetails errorMessage models syncSince syncSinceBy updatedAt } } } ``` ```json { "shopifySync": { "shop": { "_link": "SHOPID" }, "domain": "SHOPDOMAIN", "syncSince": "ISOSTRING", "syncSinceBy": "created_at" } } ``` ### Sync by count  You can limit the number of records synced with the `syncLast` parameter. This will sync only the most recent `N` records, which is useful when you only need a subset of the latest data rather than all records. By default, records are ordered by the `updated_at` timestamp coming from Shopify (the `shopifyUpdatedAt` field in Gadget), but you can control this with the `syncLastBy` parameter: * `syncLastBy: "updated_at"`: Use `shopifyUpdatedAt` to sort records by last updated (default). * `syncLastBy: "created_at"`: Use `shopifyCreatedAt` to sort records by created date. ```typescript // in an action file export const run: ActionRun = async ({ api, params }) => { const shop = await api.shopifyShop.findOne("123"); await api.shopifySync.run({ domain: shop.domain, shop: { _link: params.shopId }, // sync only the last 100 products by updated date syncLast: 100, syncLastBy: "updated_at", // optional - this is the default }); }; export const params = { shopId: { type: "string" }, }; ``` ```graphql mutation ($shopifySync: RunShopifySyncInput) { runShopifySync(shopifySync: $shopifySync) { success errors { message ... on InvalidRecordError { validationErrors { apiIdentifier message } record model { apiIdentifier } } } shopifySync { __typename id state createdAt domain errorDetails errorMessage models syncSince syncLast syncLastBy updatedAt } } } ``` ```json { "shopifySync": { "shop": { "_link": "SHOPID" }, "domain": "SHOPDOMAIN", "syncLast": 100, "syncLastBy": "updated_at" } } ``` You can also sync the most recently created records with `syncLastBy: "created_at"`: ```typescript // in an action file export const run: ActionRun = async ({ api, params }) => { const shop = await api.shopifyShop.findOne("123"); await api.shopifySync.run({ domain: shop.domain, shop: { _link: params.shopId }, // sync the 50 most recently created orders syncLast: 50, syncLastBy: "created_at", }); }; export const params = { shopId: { type: "string" }, }; ``` ```graphql mutation ($shopifySync: RunShopifySyncInput) { runShopifySync(shopifySync: $shopifySync) { success errors { message ... on InvalidRecordError { validationErrors { apiIdentifier message } record model { apiIdentifier } } } shopifySync { __typename id state createdAt domain errorDetails errorMessage models syncSince syncLast syncLastBy updatedAt } } } ``` ```json { "shopifySync": { "shop": { "_link": "SHOPID" }, "domain": "SHOPDOMAIN", "syncLast": 50, "syncLastBy": "created_at" } } ``` #### Sync by count and date-time  You can combine `syncLast` with `syncSince` to get the most recent `N` records within a specific date range: ```typescript // in an action file const millisecondsPerDay = 1000 * 60 * 60 * 24; export const run: ActionRun = async ({ api, params }) => { const shop = await api.shopifyShop.findOne("123"); const syncSince = new Date(); // sync from 30 days ago syncSince.setTime(syncSince.getTime() - 30 * millisecondsPerDay); await api.shopifySync.run({ domain: shop.domain, shop: { _link: params.shopId }, syncSince, // sync only the last 50 records updated in the past 30 days syncLast: 50, syncLastBy: "updated_at", }); }; export const params = { shopId: { type: "string" }, }; ``` ```graphql mutation ($shopifySync: RunShopifySyncInput) { runShopifySync(shopifySync: $shopifySync) { success errors { message ... on InvalidRecordError { validationErrors { apiIdentifier message } record model { apiIdentifier } } } shopifySync { __typename id state createdAt domain errorDetails errorMessage models syncSince syncLast syncLastBy updatedAt } } } ``` ```json { "shopifySync": { "shop": { "_link": "SHOPID" }, "domain": "SHOPDOMAIN", "syncSince": "ISOSTRING", "syncLast": 50, "syncLastBy": "updated_at" } } ``` ### Sync priority  You can control the execution priority of syncs by passing a `priority` parameter. This is useful when you need certain syncs to be processed before others, such as prioritizing important stores or urgent data updates. The `priority` parameter accepts three values: * `"low"`: Lower priority syncs that can wait * `"default"`: Normal priority (used when priority is not specified) * `"high"`: Higher priority syncs that should be processed first, both `api.shopifySync.run()` and all requests to Shopify triggered by the sync can use [surge compute](https://docs.gadget.dev/guides/account-and-billing#surge-compute) if it is enabled in production ```typescript // in an action file export const run: ActionRun = async ({ api, params }) => { const shop = await api.shopifyShop.findOne("123"); await api.shopifySync.run({ domain: shop.domain, shop: { _link: params.shopId }, // optional parameter priority: "high", }); }; export const params = { shopId: { type: "string" }, }; ``` ```graphql mutation ($shopifySync: RunShopifySyncInput) { runShopifySync(shopifySync: $shopifySync) { success errors { message ... on InvalidRecordError { validationErrors { apiIdentifier message } record model { apiIdentifier } } } shopifySync { __typename id state priority createdAt domain errorDetails errorMessage models syncSince updatedAt } } } ``` ```json { "shopifySync": { "shop": { "_link": "SHOPID" }, "domain": "SHOPDOMAIN", "priority": "high" } } ``` Common use cases for high-priority syncs are: * Running syncs for premium or VIP stores that require faster data updates * Processing urgent data changes that impact other business logic, like retrieving up-to-date inventory information in one high priority sync, but allowing order data to be delayed by running in a default priority sync * Running the first sync for a freshly installed Shopify store to get data in the app's UI quickly ### Forced syncs  A sync will not update any fields or run any actions if the `shopifyUpdatedAt` value in the payload is less than or equal to the `shopifyUpdatedAt` value currently stored on the record. There are certain situations, such as , where you may want to backfill new values or re-run your actions on existing records. In these situations, you can run a sync with the `force` field set to `true`. A forced sync will update all values and re-run all actions on applicable records. You can run a forced sync on an individual store by calling the `runSync` function on the `shopifySync` model: ```typescript // in an action file const millisecondsPerDay = 1000 * 60 * 60 * 24; export const run: ActionRun = async ({ api, params }) => { const shop = await api.shopifyShop.findOne("123"); const syncSince = new Date(); // sync from 5 days ago syncSince.setTime(syncSince.getTime() - 5 * millisecondsPerDay); await api.shopifySync.run({ domain: shop.domain, // optional parameter syncSince, shop: { _link: shop.id, }, force: true, }); }; export const params = { shopId: { type: "string" }, }; ``` ```graphql mutation ($shopifySync: RunShopifySyncInput) { runShopifySync(shopifySync: $shopifySync) { success errors { message ... on InvalidRecordError { validationErrors { apiIdentifier message } record model { apiIdentifier } } } shopifySync { __typename id state createdAt domain errorDetails errorMessage models syncSince updatedAt } } } ``` ```json { "shopifySync": { "shop": { "_link": "SHOPID" }, "domain": "SHOPDOMAIN", "syncSince": "ISOSTRING", "force": true } } ``` The [`globalShopifySync` API](https://docs.gadget.dev/reference/gadget-server#globalshopifysync-params) can be used to force a sync across multiple stores. ## Manual syncs  A full data sync, without a `syncSince` set, can be queued using the **Sync all data** option under the vertical ellipses on the **Installs** page. This calls the `shopifySync` model's `run` action. The install page can be found on the left nav bar or by navigating to **Settings** > **Plugins** > **Shopify** in the web editor. This sync may take some time to complete, depending on how much data the shop has. ## Non-webhook models  When connecting to Shopify, you may have opted to connect to Shopify models that do not offer webhooks. Gadget refers to these as **non-webhook** models, meaning that they can only be kept up to date by running a sync on a schedule. When you add one or more non-webhook models to your Gadget database, Gadget will also add a global action, `api/actions/scheduledShopifySync.js`, that runs once per day to fetch and save data for those models. You can customize this action's schedule by altering its trigger and change how it behaves by editing the code. For more information on these models, refer to the [list of non-webhook Shopify models](https://docs.gadget.dev/guides/plugins/shopify/non-webhook-fields-and-models#non-webhook-models). ### Non-webhook fields  A small number of fields on Shopify models are not included in webhook payloads and are only set during a sync. See [non-webhook fields and models](https://docs.gadget.dev/guides/plugins/shopify/non-webhook-fields-and-models#available-non-webhook-fields) for the full list and configuration details. ### `scheduledShopifySync` global action  If you add any non-webhook models to your Shopify connection, Gadget will automatically place a global action called `scheduledShopifySync` in your project's global actions folder. This action allows you to control how the scheduled sync is run. It calls on the [`globalShopifySync`](https://docs.gadget.dev/reference/gadget-server#globalshopifysync-params) function to run the sync process. This function takes in the following parameters for the sync: * **apiKeys:** the list of Shopify APP API keys to trigger a sync for * **syncSince:** the start date to start syncing data from, by default: In the action code we create a new `Date` object representing a specific point in time, `const syncSince = new Date(Date.now() - 25 * HourInMs)` sets `syncSince` to a `Date` object that represents the time exactly 25 hours before the current time. The reason we've set the start of the sync to 25 hours before the current time is as a best practice to leave additional leeway time to avoid any possible errors there might be in the exact timing of the sync. * **syncSinceBy:** whether to use `"created_at"` or `"updated_at"` as a time window when using `syncSince` (optional, defaults to `"updated_at"`) * **syncLast:** the maximum number of records to sync per model (optional) * **syncLastBy:** whether to order by `"created_at"` or `"updated_at"` when using `syncLast` (optional, defaults to `"updated_at"`) * **priority:** the execution priority for the sync (optional, values: `"low"`, `"default"`, `"high"`, defaults to normal priority) * **models:** the list of model API identifiers to trigger syncs for, by default: We create a `syncOnlyModels` object that maps through an array of the existing available models with the Shopify connection and filter through identifying for non-webhook models ```typescript import { globalShopifySync } from "gadget-server"; const HourInMs = 60 * 60 * 1000; export const run: ActionRun = async ({ params, logger, api, connections }) => { const syncOnlyModels = connections.shopify.enabledModels .filter((model) => model.syncOnly) .map((model) => model.apiIdentifier); const syncSince = new Date(Date.now() - 24 * HourInMs); await globalShopifySync({ apiKeys: connections.shopify.apiKeys, syncSince, models: syncOnlyModels, }); }; ``` You can also edit the rest of the action to configure it to your preference, by default the action is scheduled to run daily but you can configure the time it's run, head to the [scheduler trigger](https://docs.gadget.dev/guides/actions/triggers#scheduler-trigger) associated with the action and set a time interval of your preference, like below: If you also want to configure and add additional models to be synced with the non-webhook models, you can do so by adding your custom preference in the models param as an array like below: ```typescript import { globalShopifySync } from "gadget-server"; const HourInMs = 60 * 60 * 1000; export const run: ActionRun = async ({ params, logger, api, connections }) => { const syncOnlyModels = connections.shopify.enabledModels .filter((model) => model.syncOnly) .map((model) => model.apiIdentifier); const syncSince = new Date(Date.now() - 24 * HourInMs); await globalShopifySync({ apiKeys: connections.shopify.apiKeys, syncSince, // Pass the list of models to sync as an array of strings models: [...syncOnlyModels, "shopifyRefund"], }); }; ``` To remove a non-webhook model from being included in the scheduled sync, head to **Settings** -> **Plugins** -> **Shopify** and click the **Edit** button to edit your API scopes and simply unselect the model. Removing the `globalShopifySync` call from this action will stop syncing any data from Shopify, stopping your app from getting any updates. Removing your Shopify connection will not remove the `scheduledShopifySync` action file. You must remove that yourself if you no longer need it. ### Filtering syncs  Gadget supports filtering which records are synced when you run a sync by only these parameters: * `syncSince` to sync data changed since a specific date * `syncSinceBy` to control whether `syncSince` uses `"created_at"` or `"updated_at"` * `syncLast` to sync only the most recent N records per model * `syncLastBy` to control whether `syncLast` orders by `"created_at"` or `"updated_at"` * `models` to sync only specific set of models * `apiKeys` to sync data for only specific Shopify apps in your Gadget app Filtering by other parameters is not supported, but you can always modify your model actions to return without saving, or modify the parameters before saving to adjust what data is synced. #### Syncs and webhook filters  When a sync is run, it syncs all the enabled fields for the models you have selected, and all rows in the models you have selected. Webhook filters or the `includeFields` option that apply to webhooks will have no effect on the sync. ## Sync error handling  If an error occurs during a sync, the Shopify Sync model's Error action will be called. For example, here's how you teach Gadget to alert you via SMS when a sync fails. ```typescript import { ActionOptions } from "gadget-server"; import Twilio from "twilio"; // Environment variables set in Gadget const twilio = Twilio( process.env["TWILIO_ACCOUNT_SID"], process.env["TWILIO_ACCOUNT_AUTH_TOKEN"] ); export const run: ActionRun = async ({ params, logger }) => { const errors = params.shopifySync?.errorDetails?.split("\n") ?? []; // send the errors individually to the logger for (const error of errors) { logger.error({ error }, "an error occurred syncing"); } // send an SMS notification await twilio.messages.create({ to: "+11112223333", from: "+11112223333", body: `Shopify sync failed with message: ${params.shopifySync?.errorMessage}`, }); }; export const options: ActionOptions = { actionType: "update", }; ``` * `id` is the Shopify Sync id that was created * `errorMessage` contains a generate description of the error or errors * `errorDetails` contains a all error messages (if more than 1 occurred) joined by a `\n` The best way to debug why a sync may have failed is to search for errors through your logs. Use the following log query to filter by sync ID (replace `{syncId}` with ID of the sync that has failed): ## Abort a running sync  If you've accidentally synced the wrong model or perhaps have a long ongoing sync, you do have the ability to abort a running sync. 1. During an ongoing sync, head over to your `shopifySync` model. 2. Select the `abort` action and then click **Run Action** During an ongoing sync, head over to your `shopifySync` model. Select the `abort` action and then click **Run Action** 3. Then invoke your action within the GraphQL playground, and once ran, your sync is then successfully aborted #### Adding the `abort` action to older `shopifySync` models  You only need to add the `abort` action to your `shopifySync` model if it was created before November 17, 2023 and you wish to enable the ability to abort a running sync 1. To add the abort action, head to your `shopifySync` model and create a new action called **abort** 2. Then paste the following code within the `abort` action code file To add the abort action, head to your `shopifySync` model and create a new action called **abort** Then paste the following code within the `abort` action code file ```typescript import { transitionState, applyParams, save, ActionOptions, ShopifySyncState, } from "gadget-server"; import { preventCrossShopDataAccess, abortSync } from "gadget-server/shopify"; export const run: ActionRun = async ({ params, record, logger, api, connections, }) => { transitionState(record, { from: ShopifySyncState.Running, to: ShopifySyncState.Errored, }); applyParams(params, record); await preventCrossShopDataAccess(params, record); await abortSync(params, record); await save(record); }; export const onSuccess: ActionOnSuccess = async ({ params, record, logger, api, connections, }) => { // Your logic goes here }; export const options: ActionOptions = { actionType: "update", }; ``` 3. Now that you've successfully added the action you can abort a running sync and can follow the steps above to do so.