How to: use Gadget as middleware between Shopify and an existing service 

Gadget simplifies the process of sending Shopify data to an ERP or other existing systems. Integrating with Shopify can be challenging, which is why many builders use Gadget as an in-between step. Whether you need to synchronize data with Shopify, respond to webhooks, or handle authorization, Gadget can serve as middleware to facilitate the seamless movement of data between services.

In this guide, we will use Gadget to build a middleware application that forwards new orders from Shopify to an existing service. Keep in mind that existing services may have certain restrictions, such as rate limits, that need to be carefully managed. As an example, we will utilize Google Cloud Tasks to demonstrate how to queue requests and payloads for forwarding to an existing service.

You can fork this Gadget project and try it out yourself.

After forking, you will still need to:

• add a Shopify Connection in Gadget
• set up a Google Cloud Tasks project, queue, and service account
• add environment variables to your Gadget app

Create a Gadget app and connect to Shopify 

Our first step will be to set up a Gadget project and connect our backend to a Shopify store via the Shopify connection. Create a new Gadget application at gadget.new and select the Shopify app template.

To connect your app to a Shopify store, you have two options. The recommended approach that we walk through here is to create a custom app via the Shopify Partners dashboard. Alternatively, you can create a custom application on your Shopify store Admin page.

Connect to Shopify through the Partners dashboard 

Our first step is going to be setting up a custom Shopify application in the Partners dashboard.

• Go to the Shopify Partners dashboard
• Click on the link to the Apps page

• Click the Create App button

• Click the Create app manually button and enter a name for your Shopify app

• Click on Settings in the side nav bar
• Click on Plugins in the modal that opens
• Select Shopify from the list of plugins and connections

• Copy the Client ID and Client secret from your newly created Shopify app and paste the values into the Gadget Connections page

• Click Connect to move to scope and model selection

Now we get to select what Shopify scopes we give our application access to, while also picking what Shopify data models we want to import into our Gadget app.

• Select the Orders Read scope and the underlying Order model that you want to import into Gadget

• Click Confirm at the bottom of the page

Now we want to connect our Gadget app to our custom app in the Partners dashboard.

• In your Shopify app in the Partners dashboard, click on Configuration in the side nav bar so you can edit the App URL and Allowed redirection URL(s) fields
• Copy the App URL and Allowed redirection URL from the Gadget Connections page and paste them into your custom Shopify App

Now we need to install our Shopify app on a store.

• Go back to the Shopify Partners dashboard
• Click on Apps to go to the Apps page again
• Click on your custom app
• Click on Select store

• Click on the store we want to use to develop our app
• You may be prompted about Store transfer being disabled. This is okay, click Install anyway
• Click Install app to install your Gadget app on your Shopify store

You will be redirected to an embedded admin app that has been generated for you. The code for this app template can be found in web/routes/index.jsx.

Gadget will copy the selected Shopify Order models, their types, validations, and associations into your Gadget backend. These models are ready to process webhooks as soon as you install the app on a Shopify store.

Set up Google Cloud Tasks 

Google Cloud Tasks is a fully managed service that facilitates the management, execution, dispatch, and delivery of a large number of distributed tasks. Cloud Tasks will be utilized to queue requests and payloads for forwarding to an existing service.

Before proceeding, ensure that you already have a Google Cloud account. If not, you can create one here. Note that using Cloud Tasks will require you to provide billing information.

You can set up a Cloud Tasks queue using Google's Cloud Console or the gcloud CLI. This guide uses the Cloud Console to set up a Cloud Tasks queue.

onSuccess in shopifyOrder/actions/create.js
JavaScript
Copy
1/**
2 * @param { CreateShopifyOrderActionContext } context
3 */
4export async function onSuccess({ trigger, logger }) {
5  const existingServiceUrl = "https://eo8l1h133a8wqhs.m.pipedream.net";
6  try {
7    const response = await fetch(existingServiceUrl, {
8      method: "POST",
9      body: JSON.stringify(trigger),
10    });
11    logger.info({ response });
12  } catch (error) {
13    logger.error({ error });
14  }
15}

Create a new Google Cloud project and Cloud Tasks queue 

If you don't already have a Google Cloud project, you will need to create one, enable the Cloud Tasks API, and create a queue:

• Create a new project using the Google Cloud Console

• Enable the Cloud Tasks API for your project by navigating to the Cloud Tasks API page and clicking ENABLE
  • If you haven't already done so, you will need to enter billing information. Google has a generous free tier and trial period, so you shouldn't be charged for test usage
• Navigate to the Cloud Tasks management page in the Cloud Console and click CREATE PUSH QUEUE to create a new queue
• Give your queue a name and select a region
• Under SHOW ADVANCED SETTINGS you can configure your queue and how it handles retries and rate limits

• Click CREATE to create the new queue

Create a service account and generate a key 

Now that you have a queue, you can set up a Google service account to handle HTTP Target authentication. This service account will be used to add credentials to the Cloud Tasks API. You can follow Google's instructions, or do the following:

• Navigate to the Service Accounts page in the Cloud Console
• Select your project
• Click + CREATE SERVICE ACCOUNT
• Give your service account a name and description, and click CREATE AND CONTINUE

• Select Cloud Tasks > Cloud Tasks Enqueuer as the role
• Click + ADD ANOTHER ROLE and select Service Accounts > Service Account User

• Click Continue to save the selected roles
• Click Done to finish creating the service account

Once you have created your service account, you can generate a key that will be used to authenticate the client used to enqueue tasks from Gadget.

• On the service account page, open the menu for your service account and click Manage keys
• Click ADD KEY > Create new key and select JSON

A .json file containing the private key should be downloaded on your local machine. You will need to add the private key to your Gadget app as an environment variable.

Add environment variables 

Now that a Google Cloud Tasks queue has been set up, it is time to set up the client in Gadget so you can push tasks to the queue. You will need to add the following environment variables to your Gadget app:

• Go to Settings -> Environment Variables in Gadget
• Add new environment variables by clicking + Add Variable
• Create the following variables:
  • GCLOUD_SERVICE_ACCOUNT_PRIVATE_KEY: add the private key generated from the service account - you can encrypt this variable by clicking on the lock icon
  • GCLOUD_SERVICE_ACCOUNT_EMAIL: add the email of your service account - this is available on the Details page for the service account user in Google Cloud Console
  • GCLOUD_PROJECT_ID: add the project ID of your Google Cloud project
  • GCLOUD_QUEUE: add the queue name of your Cloud Tasks queue
  • GCLOUD_LOCATION: add the location of your Cloud Tasks queue
  • TARGET_SERVICE_URL: add the URL of the service you want to forward data to

These variables will all be used to create tasks that will be pushed to the queue. In this example, a RequestBin is used to receive the tasks and respond with a 200 status code.

Install Google Cloud Tasks client 

Before adding any code in Gadget, you can add the Google Cloud Tasks Node client to your Gadget project.

• Open the Gadget command palette using P or Ctrl P
• Enter > in the palette to allow you to run yarn commands
• Run the following yarn add command to install the client

yarn
Copy
yarn add @google-cloud/tasks

The @google-cloud/tasks dependency will be added to your Gadget project's package.json file.

Now you can make use of the client in some custom code.

Write code to add tasks to the queue 

With the connection to a Shopify store and the Google Cloud Tasks queue in place, you can add some custom code to forward any new orders to our existing service. You can do this by creating new tasks that are pushed to the Cloud Tasks queue. Once a task is pushed to the queue, Google Cloud Tasks will handle the rest, including retries and rate limits. The steps to add a success effect and code file are the same for other imported Shopify models if you are not using orders.

• Click on the shopifyOrder model in the left nav
• Click on the create action

The following snippet will forward new order information to an existing backend service that was also built using Gadget. You will get a simple response with the id of the order that was forwarded. The trigger param that is used here contains the entire webhook payload from Shopify.

• Paste the following snippet in shopifyOrder/actions/create.js. The Google Cloud Tasks client is called from the onSuccess function:

shopifyOrder/actions/create.js
JavaScript
Copy
1import {
2  applyParams,
3  save,
4  ActionOptions,
5  CreateShopifyOrderActionContext,
6} from "gadget-server";
7import { preventCrossShopDataAccess } from "gadget-server/shopify";
8const { CloudTasksClient } = require("@google-cloud/tasks");
9
10// set up Google Cloud Task client with service account credentials
11const client = new CloudTasksClient({
12  credentials: {
13    client_email: process.env["GCLOUD_SERVICE_ACCOUNT_EMAIL"],
14    private_key: process.env["GCLOUD_SERVICE_ACCOUNT_PRIVATE_KEY"]
15      .split(String.raw`\n`)
16      .join("\n"),
17  },
18});
19
20/**
21 * @param { CreateShopifyOrderActionContext } context
22 */
23export async function run({ params, record, logger, api }) {
24  applyParams(params, record);
25  await preventCrossShopDataAccess(params, record);
26  await save(record);
27}
28
29/**
30 * @param { CreateShopifyOrderActionContext } context
31 */
32export async function onSuccess({ trigger, logger }) {
33  // only forward data if trigger is a webhook or sync
34  if (trigger.type === "shopify_webhook" || trigger.type === "shopify_sync") {
35    await createHttpTask(JSON.stringify(trigger), logger);
36  }
37}
38
39/** @type { ActionOptions } */
40export const options = {
41  actionType: "create",
42};
43
44async function createHttpTask(webhookPayload, logger) {
45  // Google Cloud project id
46  const project = process.env["GCLOUD_PROJECT_ID"];
47  // Google Cloud Tasks queue variables
48  const queue = process.env["GCLOUD_QUEUE"];
49  const location = process.env["GCLOUD_LOCATION"];
50  // url of existing service
51  const url = process.env["TARGET_SERVICE_URL"];
52
53  // number of seconds to delay the request
54  const inSeconds = 60;
55
56  // construct the fully qualified queue name
57  const parent = client.queuePath(project, location, queue);
58
59  // create a new task
60  const task = {
61    httpRequest: {
62      headers: {
63        "Content-Type": "application/json",
64      },
65      httpMethod: "POST",
66      url,
67    },
68  };
69
70  // add webhook payload to task body
71  if (webhookPayload) {
72    task.httpRequest.body = Buffer.from(webhookPayload).toString("base64");
73  }
74
75  // add delay to task
76  if (inSeconds) {
77    // the time when the task is scheduled to be attempted
78    task.scheduleTime = {
79      seconds: inSeconds + Date.now() / 1000,
80    };
81  }
82
83  // send create task request
84  const request = { parent: parent, task: task };
85  logger.debug({ request }, "sending task");
86  const [response] = await client.createTask(request);
87  logger.info({ response }, `created task ${response.name}`);
88}

If you are using shopifyOrder data, you can make a test order on your development store to test your app. If not, complete an action that triggers a webhook to fire for your Shopify data model of interest. You should see a new task in the Google Cloud Tasks queue. The default example above has a 60-second delay between the task being added to the queue and the task being executed. You can change this by modifying the inSeconds variable in the code snippet. After the delay, you should see a new request made against your existing service or RequestBin.

If you also want to run this code on an update webhook from Shopify:

• Create a new code file in shopifyOrder, for example shopifyOrder/util.js
• Move the CloudTasksClient dependency import and initialization to the new file
• Move the createHttpTask function to the new file
• export to the createHttpTask function:

shopifyOrder/util.js
js
Copy
export async function createHttpTask(webhookPayload, logger) {
  /** function code */
}

• Import createHttpTask into both shopifyOrder/actions/create.js and shopifyOrder/actions/update.js and call the function from the onSuccess functions

Now order webhooks will be queued and forwarded to your existing service when an order is created or updated. You can also add code snippets to other actions on the Shopify models, such as delete.

Deploy to Production 

Some things to remember before deploying to Production:

• You may or may not want to set up another project or Tasks queue in the Google Cloud Console
• Make sure you have set up your Production environment variables
• Create another app in Shopify and create the Production connection

Deploying your application to Production can be done in two clicks:

• Click Deploy in the bottom right corner of your Gadget app
• Click the Deploy Changes button to deploy your current Development app to Production

Questions? 

That is all that is required to use Gadget as middleware. In this example, Gadget gives you hands-off data sync and webhook handling with the Shopify connection, while allowing you to continue to use your existing service to manage your store's data.

If you have any questions about this guide, feel free to reach out on Gadget's developer Discord.