Storing files 

Within Gadget, you have the ability to store files in 2 different ways depending on your application's use case.

1. Static Assets: Files that are stored and served as-is without any processing or modification by the application. These assets are typically fixed and do not change frequently. Examples of static assets include images, CSS files, JavaScript files, and other files that make up the frontend of your application.

2. Dynamic Assets: Files that are generated or processed by the application at runtime. These assets are typically generated dynamically based on user input, database queries, or other dynamic factors, and can change frequently. For example, in a social media application, user-uploaded profile pictures or images in posts would be considered dynamic assets, as they are generated or processed by the application based on user input.

Static assets within the frontend 

If you require external assets to enhance your app such as images, videos, fonts, stylesheets, etc, frontend hosting within Gadgets supports storing and serving these static assets.

It's easy to import static assets into your Gadget projects. Drag and drop your file assets to the frontend folder in your Gadget editor and they are immediately ready to be imported wherever needed.

For more information on handling static assets check out our building frontends guide.

Dynamic assets as file field types 

Dynamic files in Gadget are stored as file fields on a Gadget model. Files are uploaded when creating or updating a record and then deleted when the file is no longer referenced by a record. File URLs can then be fetched alongside other data for those records (or related records) using the GraphQL API. Gadget hosts and stores files securely in the cloud. Only users with permission to access the file's owner records can access the stored files.

Adding a new file field 

To store files, you must add a new file field to one of your Gadget models. Each file field can store one file, so if you need to store multiple files associated with the same record, you can add several file fields to the model, or create a new child model with the file field and a belongs to relationship back to the parent.

File field validations 

If desired, you can use Gadget's built-in validations for file fields to limit the size of uploaded files or to only accept images. You can also use custom code validations to implement custom business logic to work with the uploaded file and decide if it is valid or not.

1import got from "got";
2
3export default async ({ record, field, errors }) => {
4  // the value of a field inside a validation is a metadata blob describing the uploaded file
5  const metadata = record[field.apiIdentifier];
6  // you can access the file's mimeType to get a MIME type string
7  metadata.mimeType; // #=> "image/png"
8  // you can access the file's size in bytes
9  metadata.byteSize; // #=> 70
10  // you can access the file's URL in cloud storage
11  metadata.url; // #=> "https://storage.gadget.dev/files/123/456/aaaaaa/test-image.png"
12  // you must explicitly retrieve the file if you want to work with the contents of it
13  const contents = await got(metadata.url);
14  // do stuff with the file contents as a Buffer to validate it
15  if (hasBadBytes(contents)) {
16    errors.add(field.apiIdentifier, "has bad bytes");
17  }
18};

You can use modules from npm to work with file contents as well to validate them. For example, you can validate if a file is a valid PDF file by retrieving it from cloud storage using got, and then using the is-pdf-valid module to check the file's contents:

1import got from "got";
2import validPDF from "is-pdf-valid";
3
4export default async ({ record, field, errors }) => {
5  const metadata = record[field.apiIdentifier];
6  const contents = await got(metadata.url).buffer();
7  // do stuff with the file contents as a Buffer to validate it
8  if (!validPDF(contents)) {
9    errors.add(field.apiIdentifier, "is not a valid PDF file");
10  }
11};

File fields in actions 

Records with file fields can be read and written inside actions. The record object passed to your function will have a stored file's metadata present for inspection by default, and the file contents can be accessed by downloading the file from cloud storage.

1import got from "got";
2
3export default async ({ record, field, errors }) => {
4  // the value of a field inside a validation is a metadata blob describing the uploaded file
5  const metadata = record.someFileField;
6  // you can access the file's name. either the original one provided or a generate one if none was specified
7  metadata.fileName; // #=> "test-image.png"
8  // you can access the file's mimeType to get a MIME type string
9  metadata.mimeType; // #=> "image/png"
10  // you can access the file's size in bytes
11  metadata.byteSize; // #=> 70
12  // you can access the file's URL in cloud storage
13  metadata.url; // #=> "https://storage.gadget.dev/files/123/456/aaaaaa/test-image.png"
14  // you must explicitly retrieve the file if you want to work with the contents of it
15  const got = require("got");
16  const fileContents = await got(metadata.url).buffer();
17  // do stuff with the file contents
18};

For example, we could download a CSV file and work with it:

1import got from "got";
2import { parse } from "csv-parse/sync";
3
4export default async ({ api, record }) => {
5  // retrieve the file contents from cloud storage
6  const fileContents = await got(record.someFileField.url).buffer();
7
8  // parse the file contents as a CSV and log the first row
9  const parsed = parse(fileContents, { columns: true, skip_empty_lines: true });
10  console.log(parsed[0]);
11};

Or, we could send any changed file contents off to a third-party service in an action:

1import got from "got";
2
3export default async ({ api, record }) => {
4  if (record.changed("someFileField")) {
5    // retrieve the file contents from cloud storage
6    const fileContents = await got(record.someFileField.url).buffer();
7
8    // send the file contents to a third party service
9    await got.post("https://some-service.com/upload", { body: fileContents });
10  }
11};

Saving files to a record 

file fields are set the same way as other fields in Gadget using GraphQL mutations. The input file data must be provided in one of the supported formats, and then an optional fileName and mimeType can also be set on the file. Gadget's API will store the file in cloud storage, and then return a URL to access the file when the record is queried later.

File data can be submitted in a number of ways: as a string, as a multipart request, as a URL, or directly uploaded to cloud storage.

Each of these methods is used by passing an object with specific keys as the input value for the field in an action on the model.

The StoredFileInput object 

All file field inputs must be given as objects conforming to the GraphQL StoredFileInput object schema, both in GraphQL calls and calls made using the generated JavaScript client.

GraphQL
Copy
1type StoredFileInput {
2  # Sets the file contents using this string, interpreting the string as base64 encoded bytes. This is useful for creating files quickly and easily if you have the file contents available already, but, it doesn't support files larger than 10MB, and is slower to process for the backend. Using multipart file uploads or direct-to-storage file uploads is preferable.
3  base64: GraphQLString
4  # Sets the file contents using binary bytes sent alongside a GraphQL mutation as a multipart POST request. Gadget expects this multipart POST request to be formatted according to the GraphQL multipart request spec defined at https://github.com/jaydenseric/graphql-multipart-request-spec. Sending files as a multipart POST request is supported natively by the generated Gadget JS client using File objects as variables in API calls. This method supports files up to 100MB.
5  file: GraphQLUpload
6  # Sets the file contents by fetching a remote URL and saving a copy to cloud storage. File downloads happen as the request is processed so they can be validated, which means large files can take some time to download from the existing URL. If the file can't be fetched from this URL, the action will fail.
7  copyURL: GraphQLURL
8  # Sets the file contents using a token from a separate upload request made with the Gadget storage service. Uploading files while a user is completing the rest of a form gives a great user experience and supports much larger files, but requires client side code to complete the upload, and then pass the returned token for this field.
9  directUploadToken: GraphQLString
10  # Sets this file's mime type, which will then be used when serving the file during read requests as the `Content-Type` HTTP header. If not set, Gadget will infer a content type based on the file's contents.
11  mimeType: GraphQLString
12  # Sets this file's stored name, which will then be used as the file's name when serving the file during read requests. If not set, Gadget will infer a filename if possible.
13  fileName: GraphQLString
14}

base64 encoded string uploads 

Files can be uploaded when creating records by using the contents of the file inline as as base64 encoded string using the base64 field of the input.

For example, if we have an Image model that has a string name field and a file image field, we can create a new Image record like this:

1api.image.create({
2  name: "A Test Image",
3  image: {
4    base64:
5      "iVBORw0KGgoAAAANSUhEUgAAAAEAAAABCAYAAAAfFcSJAAAADUlEQVR42mP8z8BQDwAEhQGAhKmMIQAAAABJRU5ErkJggg==",
6    fileName: "test-image.png",
7  },
8});

1mutation {
2  createImage(
3    image: {
4      name: "A Test Image"
5      image: {
6        base64: "iVBORw0KGgoAAAANSUhEUgAAAAEAAAABCAYAAAAfFcSJAAAADUlEQVR42mP8z8BQDwAEhQGAhKmMIQAAAABJRU5ErkJggg=="
7        fileName: "test-image.png"
8      }
9    }
10  ) {
11    success
12    image {
13      id
14      name
15      image {
16        url
17      }
18    }
19  }
20}

When queried, the created Image record will have an id, createdAt, and updatedAt field like any other model, the name field set to A Test Image, and the image file field set to an object with a url field.

base64 performance 

base64 encoded string file uploads are convenient if you have the file contents in memory already and want an easy way to submit small files to the Gadget server. But, base64 uploads don't perform well compared to the other methods. Embedding the file contents in the request like this causes delayed processing in the Gadget API, as the whole request string needs to be sent to the API before it can successfully parse the whole JSON blob. Large files have to be base64 encoded which adds extra processing time.

If you are working in a browser or using the generated JavaScript client for your Gadget app, prefer using the file key or the directUploadToken key, especially if your users are ever uploading large files.

Uploads using the File object 

Files can be uploaded when creating records by using the File object that browsers expose. File objects can be accessed from <input type="file" />s or from drag and drop operations. For more information on using Files in a web application, see MDN.

To upload a File, pass it at the file key in the input object for a file field.

For example, if we have an Image model that has a string name field and a file image field, we can create a new Image record like this:

1// get the file from an <input type="file" />
2// see https://developer.mozilla.org/en-US/docs/Web/API/File/Using_files_from_web_applications#getting_information_about_selected_files
3const file = document.getElementById("#some-file-input").files[0];
4// use the File object at the file key of the input for the image field
5api.image.create({
6  name: "A Test Image",
7  image: {
8    file: file,
9  },
10});

File object performance 

Uploading File objects with the file works well for small files, as the browser is able to stream the initial GraphQL request, and then the files to the backend API, allowing processing of the request while the files are being uploaded to cloud storage. Large files can take a long time to stream, so if your users intend to upload large files, the Direct Upload method is best.

Gadget supports a maximum file size of 100MB per file uploaded with the file key and a maximum of 10 files per request.

Uploads using the Multipart Request spec 

Gadget GraphQL APIs implement the GraphQL Multipart Request specification which allows submitting files alongside other variables in a GraphQL mutation. The Gadget-generated JavaScript client uses this specification to upload File objects automatically, but if you're working in a different language or with a different client, you can pass files to Gadget using any client which implements this spec.

For example, to upload a test.png file with cURL according to the multipart request spec, you can run the following command:

terminal
Copy
curl https://example.gadget.app/api/graphql \
  -F operations='{ "query": "mutation ($file: Upload!) { createImage(image: { name: "A test image", image: { file: $file } } )) { success image { id image { url }} } }", "variables": { "file": null } }' \
  -F map='{ "0": ["variables.file"] }' \
  -F 0=@test.png

Uploads from a URL 

Gadget supports storing files directly from an existing URL on the web. Instead of a client uploading a local file, Gadget will copy a file from a URL into cloud storage.

To set the value of a file field, use the copyURL key of the input object in a GraphQL mutation or API client call.

1await api.photo.create({
2  name: "A photo!",
3  image: {
4    // create a record copying an image from an existing URL
5    copyURL: "https://example.com/image.png",
6  },
7});

1query CreatePhoto {
2  createPhoto(
3    photo: {
4      name: "A photo!"
5      image: {
6        # create a record copying an image from an existing URL
7        copyURL: "https://example.com/image.png"
8      }
9    }
10  ) {
11    photo {
12      id
13      name
14      image {
15        url
16      }
17    }
18  }
19}

Direct URL upload performance 

When possible, copying from remote URLs to upload files tends to perform well. Server-to-server communication over the internet is generally much faster than a user's connection, so if files are already hosted somewhere, copying them into Gadget can be faster and more convenient for the user than downloading and re-uploading them.

Direct uploads using tokens 

Uploading files to a remote server can be slow sometimes, especially if the files are large or the user's internet connection is constrained. Often, users upload files to an application at the same time as filling in other data, and they select which file they want to upload before filling out the rest of the form. That means there's a good opportunity to give users a good experience by uploading their files in the background while they complete the rest of the form. If the file is large, starting the upload as soon as possible helps as well to get the user to complete the submission faster by starting the upload sooner.

To support this workflow, Gadget supports direct uploads, which allows the user's browser to start an upload as soon as the file is ready, and upload it directly to cloud storage. The form can then be submitted to Gadget later with a token for the file.

The process works like this:

1. The user's browser requests the /api/files/upload-target Gadget API to retrieve an upload URL and a file token
2. The user's browser uploads a file to the upload URL
3. The user finishes filling out the form with any other data
4. Once the upload is complete and any other client-side validations are satisfied, the user's browser submits the form to Gadget and runs a model action. The action input references the file using the directUploadToken key of the StoredFileInput for the file field on the model.

Gadget will retrieve metadata from the uploaded file in cloud storage and associate it with the file field on the record.

Getting a file upload URL and token 

Direct upload URLs and tokens need to be fetched before a user uploads a file or submits a form. To get a token, you can use the await api.getDirectUploadToken() API client function, the gadgetMeta { directUploadToken { url token } } GraphQL query, or the /api/files/upload-target REST endpoint.

1import got from "got";
2// get an upload URL and token
3const { url, token } = await api.getDirectUploadToken();
4// upload the image to cloud storage
5await got(url, {
6  method: "PUT",
7  headers: {
8    "Content-Type": "text/plain",
9  },
10  body: "an example text file",
11});
12// create a record using the returned token that is now associated with the file
13await api.attachment.create({
14  name: "A file!",
15  file: {
16    directUploadToken: token,
17    fileName: "example.txt",
18  },
19});

1query GetDirectUploadToken {
2  gadgetMeta {
3    directUploadToken {
4      url
5      token
6    }
7  }
8}

curl https://my-gadget-app.gadget.app/api/files/upload-target

Once you have a direct upload url, you must PUT the file's contents to the url. The url is signed for security and expires after 3 hours to avoid abuse. This means you can't just generate the url once and hardcode it in your code -- you must generate a new url for each upload by your form.

Once the upload has been completed, use the token as the value of the directUploadToken key in the input to the file field.

1const { url, token } = await api.getDirectUploadToken();
2// upload image to cloud storage
3api.image.create({
4  name: "A Test Image",
5  image: {
6    // pass the token instead of a file key or base64 key
7    directUploadToken: token,
8    // set the filename or the mime type if needed
9    fileName: "test-image.png",
10  },
11});

1mutation {
2  createImage(
3    image: {
4      name: "A Test Image"
5      image: {
6        # pass the token instead of a file key or base64 key
7        directUploadToken: "v4.local.someopaquetoken"
8        # set the filename or the mime type if needed
9        fileName: "test-image.png"
10      }
11    }
12  ) {
13    success
14    image {
15      id
16      name
17      image {
18        url
19      }
20    }
21  }
22}

Direct upload tokens can be re-used to associate the same uploaded file with many records. If you complete the upload to the url once, you can call many Gadget actions with the same token. Direct upload urls can't be re-used to upload more than one file. You must create a new url and token for each unique file uploaded by the user.

Direct upload tokens expire after 3 hours. Users can't take longer than this between uploading a file to the url and using the token in an action.

Writing frontend upload forms 

Some frontend code is required to coordinate file upload when using the direct token upload method. The token should only be sent to Gadget once the upload is complete and the user has completed the form. The general flow is:

1. Wait for the user to submit a file
2. Get a url and token before you start the file upload with api.getDirectUploadToken()
3. Upload the file to cloud storage in the background with fetch or similar
4. Store the token in the state while the user completes the rest of the form
5. Submit the action to Gadget once the user has finished the form and the upload has completed

Using React as an example, we can listen to a drop event on an element and upload the dropped file right to cloud storage:

JavaScript
Copy
1import React, { useState } from "react";
2import { api } from "../api"; // or wherever your Gadget client is constructed
3
4function Uploader(props) {
5  const [isUploading, setIsUploading] = useState(false);
6  const [fileName, setFileName] = useState(null);
7
8  return (
9    <div
10      onDragOver={(event) => event.preventDefault()}
11      onDrop={async (event) => {
12        event.preventDefault();
13        const file = event.dataTransfer.files[0];
14        // send the file to cloud storage
15        setIsUploading(true);
16
17        const { url, token } = await api.getDirectUploadToken();
18
19        try {
20          await fetch(url, {
21            method: "PUT",
22            headers: {
23              "Content-Type": file.type,
24            },
25            body: file,
26          });
27          // once it has been uploaded, give the parent component the token and file name to submit with the other data in the form
28          props.onFileTokenChange(token, file.name);
29          setFileName(file.name);
30        } finally {
31          setIsUploading(false);
32        }
33      }}
34    >
35      {isUploading ? "Uploading ..." : fileName ? fileName : "Drop a file here"}
36    </div>
37  );
38}

Direct file upload performance 

Direct file uploads perform the best for both the user and the systems involved out of any of the upload methods. The user benefits because the file upload is started (and completed) earlier, which allows their form submission to complete as soon as they have finished filling it out. If the form is complicated or long and the user is likely to get some validation errors, it's also great to be able to re-use the token for the uploaded file for a second or third submission instead of re-uploading the file again.

Direct file uploads also support the biggest files as there is no file size limit. Because the user's browser is uploading directly to cloud storage, the cloud storage system's optimized frontends can handle the upload and give it the best chance of completing with very high request timeouts and great internet connectivity.

Storing new files within actions 

To store a new file programmatically within an action, make an API call with the api object to run an action with the same inputs as detailed above. The api object available in the action context works the same as an api object in an external script, so all the upload methods are available and ready for use inside actions as well.

For example, we can create file contents by serializing an object to JSON, and then store the JSON contents in a file in cloud storage:

1import got from "got";
2
3export default async ({ api, record }) => {
4  // build some example data
5  const data = {
6    name: record.name,
7    description: record.description,
8  };
9
10  const fileContents = JSON.stringify(data);
11  // create a new record of a model with a File field named `file`
12  await api.someModel.create({
13    file: {
14      // set the filename
15      fileName: "example.json",
16      // base64 encode the file contents to pass to Gadget
17      base64: new Buffer(fileContents).toString("base64"),
18    },
19  });
20};

The copyURL upload method can also be used inside actions to save a record with a file downloaded from a URL:

1import got from "got";
2
3export default async ({ api, record }) => {
4  // create a new record of a model with a File field named `file`
5  await api.someModel.create({
6    file: {
7      // base64 encode the file contents to pass to Gadget
8      copyURL: "https://assets.gadget.dev/assets/favicon-32.png",
9    },
10  });
11};

Currently, the Internal API does not support uploading new files to Gadget, so you must use your application's Public API to manage files inside effects.

Retrieving files 

file fields are accessed like other fields in Gadget using the generated GraphQL API. Gadget returns several details about a stored file like it's url, fileName, mimeType, and byteSize. File field details can be fetched when querying individual records or lists of records.

1api.image.findMany({
2  select: {
3    // get details of the owning record
4    id: true,
5    name: true,
6    image: {
7      // get the file's URL, suitable for embedding in an <img src="..."/> tag
8      url: true,
9      // get the file's filename, including extension
10      fileName: true,
11      // get the file's mime type
12      mimeType: true,
13      // get the file's size in bytes
14      byteSize: true,
15    },
16  },
17});

1query {
2  images {
3    edges {
4      node {
5        # get details of the owning record
6        id
7        name
8        image {
9          # get the file's URL, suitable for embedding in an <img src="..."/> tag
10          url
11          # get the file's filename, including extension
12          fileName
13          # get the file's mime type
14          mimeType
15          # get the file's size in bytes
16          byteSize
17        }
18      }
19    }
20  }
21}

Marking files as private 

file fields in Gadget can be marked as private or not. When the field is private, unauthenticated users aren't able to access the URL for a file at all. For authenticated users, Gadget generates securely signed URLs to files that expire after 3 hours. This is ideal because it means if a user logs out or their access is revoked, they can no longer access the file after the 3 hour window has elapsed. Private files are not CDN cached which is often desirable if they can contain any kind of sensitive data.

If you want anyone to be able to access files, like the images on a blog, or profile photos on a social network, you can mark the field as not private. In this instance, Gadget will generate URLs that never expire so users are always able to access the file, and the files will be cached on Gadget's CDN for maximum performance.

If you change a file field's private configuration, existing uploaded files are not re-permissioned. If you want to make all the files stored by a field private, you would need to re-upload each one to recreate them with the new access permissions.

Deleting stored files 

Files are deleted from storage when they are no longer referenced by a record. There are 2 ways to delete stored files:

1. Delete the record that owns the file. When a record is deleted, all files associated with it are deleted as well.
2. Set a file field to null.

Updating a file field on a record also deletes the original file from storage.

Where are files stored? 

Under the hood, Gadget uses Google Cloud Storage to store files. The Gadget API uploads submitted files to a private bucket and uses Google Cloud Storage's Signed URL functionality to grant access to files when a user can prove they have access to the owning record.

It's not currently possible to use your own Cloud Storage bucket for files.