Storing files

Gadget supports uploading and downloading files like images, audio, PDFs, binary blobs or anything else to your Gadget application. Files in Gadget are stored as File fields on a Gadget model. Files are uploaded when creating or updating a record, and then deleted when that record is deleted. File URLs can then be fetched alongside other data for those records (or related records) using the GraphQL API. Gadget hosts and stores files securely in the cloud, allowing access to files to users with permission to access to the owner records.

Adding a new file field

To store files, you must add a new File field to one of your Gadget models. Each File field can store one file, so if you need to store multiple files associated with the same record, you can add several File fields to the model, or create a new child model with the File field and a BelongsTo relationship back to the parent.

File field validations

If desired, you can use Gadget's built in validations for file fields to limit the size of uploaded files, or to only accept images. You can also use custom code validations to implement custom business logic to work with the uploaded file and decide if it is valid or not.

Validating a file field
JavaScript
1const got = require("got");
2
3module.exports = async ({ record, field, errors }) => {
4 // the value of a field inside a validation is a metadata blob describing the uploaded file
5 const metadata = record[field.apiIdentifier];
6 // you can access the file's mimeType to get a MIME type string
7 metadata.mimeType; // #=> "image/png"
8 // you can access the file's size in bytes
9 metadata.byteSize; // #=> 70
10 // you can access the file's URL in cloud storage
11 metadata.url; // #=> "https://storage.gadget.dev/files/123/456/aaaaaa/test-image.png"
12 // you must explicitly retrieve the file if you want to work with the contents of it
13 const contents = await got(metadata.url);
14 // do stuff with the file contents as a Buffer to validate it
15 if (hasBadBytes(contents)) {
16 errors.add(field.apiIdentifier, "has bad bytes");
17 }
18};

You can use modules from npm to work with file contents as well in order to validate them. For example, you can validate if a file is a valid PDF file by retrieving it from cloud storage using got, and then using the is-pdf-valid module to check the file's contents:

Validating a PDF file field
JavaScript
1const got = require("got");
2const validPDF = require("is-pdf-valid");
3
4module.exports = async ({ record, field, errors }) => {
5 const metadata = record[field.apiIdentifier];
6 const contents = await got(metadata.url).buffer();
7 // do stuff with the file contents as a Buffer to validate it
8 if (!validPDF(contents)) {
9 errors.add(field.apiIdentifier, "is not a valid PDF file");
10 }
11};

File fields in Code Effects

Records with File fields can be read and written inside Code Effects. The record object passed to your effect function will have a stored file's metadata present for inspection by default, and the file contents can be accessed by downloading the file from cloud storage.

Accessing file content in an effect
JavaScript
1const got = require("got");
2
3module.exports = async ({ record, field, errors }) => {
4 // the value of a field inside a validation is a metadata blob describing the uploaded file
5 const metadata = record.someFileField;
6 // you can access the file's name. either the original one provided or a generate one if none was specified
7 metadata.fileName; // #=> "test-image.png"
8 // you can access the file's mimeType to get a MIME type string
9 metadata.mimeType; // #=> "image/png"
10 // you can access the file's size in bytes
11 metadata.byteSize; // #=> 70
12 // you can access the file's URL in cloud storage
13 metadata.url; // #=> "https://storage.gadget.dev/files/123/456/aaaaaa/test-image.png"
14 // you must explicitly retrieve the file if you want to work with the contents of it
15 const got = require("got");
16 const fileContents = await got(metadata.url).buffer();
17 // do stuff with the file contents
18};

For example, we could download a CSV file and work with it:

Parsing CSV files
JavaScript
1const got = require("got");
2const { parse } = require("csv-parse/sync");
3
4module.exports = async ({ api, record }) => {
5 // retrieve the file contents from cloud storage
6 const fileContents = await got(record.someFileField.url).buffer();
7
8 // parse the file contents as a CSV and log the first row
9 const parsed = parse(fileContents, { columns: true, skip_empty_lines: true });
10 console.log(parsed[0]);
11};

Or, we could send any changed file contents off to a third party service in an effect:

Sending file content elsewhere if changed
JavaScript
1const got = require("got");
2
3module.exports = async ({ api, record }) => {
4 if (record.changed("someFileField")) {
5 // retrieve the file contents from cloud storage
6 const fileContents = await got(record.someFileField.url).buffer();
7
8 // send the file contents to a third party service
9 await got.post("https://some-service.com/upload", { body: fileContents });
10 }
11};

Saving files to a record

File fields are set the same way as other fields in Gadget using GraphQL mutations. The input file data must be provided in one of the supported formats, and then an optional fileName and mimeType can also be set on the file. Gadget's API will store the file in cloud storage, and then return a URL to access the file when the record is queries later.

File data can submitted in a number of ways: as a string, as a multipart request, as a URL, or directly uploaded to cloud storage.

ApproachDescriptionMax File SizePerformanceInput Key
base64 variableFile contents POSTed as a base64 encoded string10MBShould only be used for small filesbase64more info
File object variableFile contents POSTed as a multipary request by the generated API client100MBShould only be used for smallish filesfilemore info
Existing public URLFile contents downloaded by Gadget from a URL100MBGood performance, depends on existing hostcopyURLmore info
Direct uploadsFiles uploaded directly to cloud storage, then associated with a record using a token10GBBest user experience, best performancedirectUploadTokenmore info

Each of these methods is used by passing an object with specific keys as the input value for the field in an action on the model.

The StoredFileInput object

All File field inputs must be given as objects conforming to the GraphQL StoredFileInput object schema, both in GraphQL calls and calls made using the generated JavaScript client.


GraphQL
1type StoredFileInput {
2 # Sets the file contents using this string, interpreting the string as base64 encoded bytes. This is useful for creating files quickly and easily if you have the file contents available already, but, it doesn't support files larger than 10MB, and is slower to process for the backend. Using multipart file uploads or direct-to-storage file uploads is preferable.
3 base64: GraphQLString
4 # Sets the file contents using binary bytes sent along side a GraphQL mutation as a multipart POST request. Gadget expects this multipart POST request to be formatted according to the GraphQL multipart request spec defined at https://github.com/jaydenseric/graphql-multipart-request-spec. Sending files as a multipart POST requests is supported natively by the generated Gadget JS client using File objects as variables in API calls. This method supports files up to 100MB.
5 file: GraphQLUpload
6 # Sets the file contents by fetching a remote URL and saving a copy to cloud storage. File downloads happen as the request is processed so they can be validated, which means large files can take some time to download from the existing URL. If the file can't be fetched from this URL, the action will fail.
7 copyURL: GraphQLURL
8 # Sets the file contents using a token from a separate upload request made with the Gadget storage service. Uploading files while a user is completing the rest of a form gives a great user experience and supports much larger files, but requires client side code to complete the upload, and then pass the returned token for this field.
9 directUploadToken: GraphQLString
10 # Sets this file's mime type, which will then be used when serving the file during read requests as the `Content-Type` HTTP header. If not set, Gadget will infer a content type based on the file's contents.
11 mimeType: GraphQLString
12 # Sets this file's stored name, which will then be used as the file's name when serving the file during read requests. If not set, Gadget will infer a filename if possible.
13 fileName: GraphQLString
14}

base64 encoded string uploads

Files can be uploaded when creating records by using the contents of the file inline as as base64 encoded string using the base64 field of the input.

For example, if we have an Image model that has a String name field and a File image field, we can create a new Image record like this:

Setting an image using base64
1api.image.create({
2 image: {
3 name: "A Test Image",
4 image: {
5 base64:
6 "iVBORw0KGgoAAAANSUhEUgAAAAEAAAABCAYAAAAfFcSJAAAADUlEQVR42mP8z8BQDwAEhQGAhKmMIQAAAABJRU5ErkJggg==",
7 fileName: "test-image.png",
8 },
9 },
10});
1mutation {
2 createImage(
3 image: {
4 name: "A Test Image"
5 image: {
6 base64: "iVBORw0KGgoAAAANSUhEUgAAAAEAAAABCAYAAAAfFcSJAAAADUlEQVR42mP8z8BQDwAEhQGAhKmMIQAAAABJRU5ErkJggg=="
7 fileName: "test-image.png"
8 }
9 }
10 ) {
11 success
12 image {
13 id
14 name
15 image {
16 url
17 }
18 }
19 }
20}

When queries, the created Image record will have an id, createdAt and updatedAt field like any other model, the name field set to A Test Image, and the image File field set to an object with a url field.

Performance

base64 encoded string file uploads are convenient if you have the file contents in memory already and want an easy way to submit small files to the Gadget server. But, base64 uploads don't perform well compared to the other methods. Embedding the file contents in the request like this causes delayed processin in the Gadget API, as the whole request string needs to be sent to the API before it can successfully parse the whole JSON blob. Large files have to be base64 encoded which adds extra processing time.

If you are working in a browser or using the generated JavaScript client for your Gadget app, prefer using the file key or the directUploadToken key, especially if your users are ever uploading large files.

Uploads using the File object

Files can be uploaded when creating records by using the File object that browsers expose. File objects can be accessed from <input type="file" />s or from drag and drop operations. For more information on using Files in a web application, see MDN.

To upload a File, pass it at the file key in the input object for a File field.

For example, if we have an Image model that has a String name field and a File image field, we can create a new Image record like this:

Setting an image using a File object
JavaScript
1// get the file from an <input type="file" />
2// see https://developer.mozilla.org/en-US/docs/Web/API/File/Using_files_from_web_applications#getting_information_about_selected_files
3const file = document.getElementById("#some-file-input").files[0];
4// use the File object at the file key of the input for the image field
5api.image.create({
6 image: {
7 name: "A Test Image",
8 image: {
9 file: file,
10 },
11 },
12});
Performance

Uploading File objects with the file works well for small files, as the browser is able to stream the initial GraphQL request, and then the files to the backend API, allwing processing of the request while the files are being uploaded to cloud storage. Large files can take a long time to stream however, so if your users intend to upload large files, the Direct Upload method is best.

Gadget supports a maximum file size of 100MB per file uploaded with the file key and a maximum of 10 files per request.

Uploads using the Multipart Request spec

Gadget GraphQL APIs implement the GraphQL Multipart Request specification which allows submitting files alongside other variables in a GraphQL mutation. The Gadget-generated JavaScript client uses this specification to upload File objects automatically, but if you're working in a different language or with a different client, you can pass files to Gadget using any client which implements this spec.

For example, to upload a test.png file with cURL according to the multipart request spec, you can run the following command:

Shell
curl https://example.gadget.app/api/graphql \
-F operations='{ "query": "mutation ($file: Upload!) { createImage(image: { name: "A test image", image: { file: $file } } )) { success image { id image { url }} } }", "variables": { "file": null } }' \
-F map='{ "0": ["variables.file"] }' \
-F 0=@test.png

Uploads from a URL

Gadget supports storing files directly from an existing URL on the web. Instead of a client uploading a local file, Gadget will copy a file from a URL into cloud storage.

Gadget copies files from remote URLs once when the URL is first submitted, and then uses its own cloud storage system to host the file. Once a File field has been set on a record, take care to use the URL that Gadget generates for the file, instead of the original URL. The Gadget-generated URL will be governed by the Gadget permissions system, and use Gadget's performant CDN for hosting.

To set the value of a File field, use the copyURL key of the input object in a GraphQL mutation or api client call.

Using copyURL to download an image
1await api.photo.create({
2 photo: {
3 name: "A photo!",
4 image: {
5 // create a record copying an image from an existing URL
6 copyURL: "https://example.com/image.png",
7 },
8 },
9});
1query CreatePhoto {
2 createPhoto(
3 photo: {
4 name: "A photo!"
5 image: {
6 # create a record copying an image from an existing URL
7 copyURL: "https://example.com/image.png"
8 }
9 }
10 ) {
11 photo {
12 id
13 name
14 image {
15 url
16 }
17 }
18 }
19}
Performance

When possible, copying from remote URLs to upload files tends to perform well. Server to server communication over the internet is generally much faster than a user's connection, so if files are already hosted somewhere, copying them into Gadget can be faster and more convenient for the user than downloading and re-uploading them.

Direct uploads using tokens

Uploading files to a remote server can be slow sometimes, especially if the files are large or the user's internet connection is constrainted. Often, users upload files to an application at the same time as filling in other data, and they select which file they want to upload before filling out the rest of the form. That means there's a good opportunity to give users a good experience by uploading their files in the background while they complete the rest of the form. If the file is large, starting the upload as soon as possible helps as well to get the user to complete the submission faster by starting the upload sooner.

To support this workflow, Gadget supports direct uploads, which allows the user's browser to start an upload as soon as file is ready, and upload it directly to cloud storage. The form is then be submitted to Gadget later with a token for the file.

The process works like this:

  • the user's browser requests the /api/files/upload-target Gadget API to retrieve an upload URL and a file token
  • the user's browser uploads a file to the upload URL
  • the user finishes filling out the form with any other data
  • once the upload is complete and any other client side validations are satisfied, the user's browser submits the form to Gadget and runs a model action. The action input references the file using the directUploadToken key of the StoredFileInput for the File field on the model.

Gadget will retrieve metadata from the uploaded file in cloud storage and associate it with the File field on the record.

Getting a file upload URL and token

Direct upload URLs and tokens need to be fetched before a user uploads a file or submits a form. To do get a token, you can use the await api.getDirectUploadToken() API client function, the gadgetMeta { directUploadToken { url token } } GraphQL query, or the /api/files/upload-target REST endpoint.

Getting a direct upload token
1import got from "got";
2// get an upload URL and token
3const { url, token } = await api.getDirectUploadToken();
4// upload the image to cloud storage
5await got(url, {
6 method: "PUT",
7 headers: {
8 "Content-Type": "text/plain",
9 },
10 body: "an example text file",
11});
12// create a record using the returned token that is now associated with the file
13await api.attachment.create({
14 attachment: {
15 name: "A file!",
16 file: {
17 directUploadToken: token,
18 fileName: "example.txt",
19 },
20 },
21});
1query GetDirectUploadToken {
2 gadgetMeta {
3 directUploadToken {
4 url
5 token
6 }
7 }
8}
curl https://my-gadget-app.gadget.app/api/files/upload-target

Once you have a direct upload url, you must PUT the file's contents to the url. The url is signed for security and expires after 3 hours to avoid abuse. This means you can't just generate the url once and hardcode it in your code -- you must generate a new url for each upload by your form.

Once the upload has been completed, use the token as the value of the directUploadToken key in the input to the File field.

Setting an image using a direct upload token
1const { url, token } = await api.getDirectUploadToken();
2// upload image to cloud storage
3api.image.create({
4 image: {
5 name: "A Test Image",
6 image: {
7 // pass the token instead of a file key or base64 key
8 directUploadToken: token,
9 // set the filename or the mime type if needed
10 fileName: "test-image.png",
11 },
12 },
13});
1mutation {
2 createImage(
3 image: {
4 name: "A Test Image"
5 image: {
6 # pass the token instead of a file key or base64 key
7 directUploadToken: "v4.local.someopaquetoken"
8 # set the filename or the mime type if needed
9 fileName: "test-image.png"
10 }
11 }
12 ) {
13 success
14 image {
15 id
16 name
17 image {
18 url
19 }
20 }
21 }
22}

Direct upload tokens can be re-used to associate the same uploaded file with many records. If you complete the upload to the url once, you can call many Gadget actions with the same token. Direct upload urls can't be re-used to upload more than one file. You must create a new url and token for each unique file uploaded by the user.

Direct upload tokens expire after 3 hours. Users can't take longer than this between uploading a file to the url and using the token in an action.

Writing frontend upload forms

Some frontend code is required to coordinate file upload when using the direct token upload method. The token should only be sent to Gadget once the upload is complete and the user has completed the form. The general flow is:

  • wait for the user to submit a file
  • get a url and token before you start the file upload with api.getDirectUploadToken()
  • upload the file to cloud storage in the background with fetch or similar
  • store the token in the state while the user completes the rest of the form
  • submit the action to Gadget once the user has finished the form and the upload has completed

Using React as an example, we can listen to a drop event on an element and upload the dropped file right to cloud storage:

JavaScript
1import React, { useState } from "react";
2import { api } from "../api"; // or wherever your Gadget client is constructed
3
4function Uploader(props) {
5 const [isUploading, setIsUploading] = useState(false);
6 const [fileName, setFileName] = useState(null);
7
8 return (
9 <div
10 onDragOver={(event) => event.preventDefault()}
11 onDrop={async (event) => {
12 event.preventDefault();
13 const file = event.dataTransfer.files[0];
14 // send the file to cloud storage
15 setIsUploading(true);
16
17 const { url, token } = await api.getDirectUploadToken();
18
19 try {
20 await fetch(url, {
21 method: "PUT",
22 headers: {
23 "Content-Type": file.type,
24 },
25 body: file,
26 });
27 // once it has been uploaded, give the parent component the token and file name to submit with the other data in the form
28 props.onFileTokenChange(token, file.name);
29 setFileName(file.name);
30 } finally {
31 setIsUploading(false);
32 }
33 }}
34 >
35 {isUploading ? "Uploading ..." : fileName ? fileName : "Drop a file here"}
36 </div>
37 );
38}

More examples of uploading files to Gadget can be found in the Gadget Examples on GitHub.

Performance

Direct file uploads perform the best for both the user and the systems involved out of any of the upload methods. The user benefits because the file upload is started (and completed) earlier, which allows their form submission to complete as soon as they have finished filling it out. If the form is complicated or long and the user is likely to get some validation errors, it's also great to be able to re-use the token for the uploaded file for a second or third submission instead of re-uploading the file again.

Direct file uploads also support the biggest files as there is no file size limit. Because the user's browser is uploading directly to cloud storage, the cloud storage system's optimized frontends can handle the upload and give it the best chance of completing with very high request timeouts and great internet connectivity.

Storing new files within Effects

To store a new file programmatically within an Effect, make an API call with the api object to run an action with the same inputs as detailed above. The api object available in the effect context works the same as an api object in an external script, so all the upload methods are available and ready for use inside effects as well.

For example, we can create file contents by serializing an object to JSON, and then store the JSON contents in a file in cloud storage:

Creating a File in an effect
JavaScript
1const got = require("got");
2
3module.exports = async ({ api, record }) => {
4 // build some example data
5 const data = {
6 name: record.name,
7 description: record.description,
8 };
9
10 const fileContents = JSON.stringify(data);
11 // create a new record of a model with a File field named `file`
12 await api.someModel.create({
13 someModel: {
14 file: {
15 // set the filename
16 fileName: "example.json",
17 // base64 encode the file contents to pass to Gadget
18 base64: new Buffer(fileContents).toString("base64"),
19 },
20 },
21 });
22};

The copyURL upload method can also be used inside effects to save a record with a file downloaded from a URL:

Creating a record with a File from a URL
JavaScript
1const got = require("got");
2
3module.exports = async ({ api, record }) => {
4 // create a new record of a model with a File field named `file`
5 await api.someModel.create({
6 someModel: {
7 file: {
8 // base64 encode the file contents to pass to Gadget
9 copyURL: "https://assets.gadget.dev/assets/favicon-32.png",
10 },
11 },
12 });
13};

Currently, the Internal API does not support uploading new files to Gadget, so you must use your application's Public API to manage files inside effects.

Retrieving files

File fields are accessed like other fields in Gadget using the generated GraphQL API. Gadget returns several details about a stored file like it's url, fileName, mimeType, and byteSize. File field details can be fetched when querying individual records or lists of records.

Getting a file
1api.image.findMany({
2 select: {
3 // get details of the owning record
4 id: true,
5 name: true,
6 image: {
7 // get the file's URL, suitable for embedding in an <img src="..."/> tag
8 url: true,
9 // get the file's filename, including extension
10 fileName: true,
11 // get the file's mime type
12 mimeType: true,
13 // get the file's size in bytes
14 byteSize: true,
15 },
16 },
17});
1query {
2 images {
3 edges {
4 node {
5 # get details of the owning record
6 id
7 name
8 image {
9 # get the file's URL, suitable for embedding in an <img src="..."/> tag
10 url
11 # get the file's filename, including extension
12 fileName
13 # get the file's mime type
14 mimeType
15 # get the file's size in bytes
16 byteSize
17 }
18 }
19 }
20 }
21}

Marking files as private

File fields in Gadget can be marked as private or not. When the field is private, unauthenticated users aren't able to access the URL for a file at all. For authenticated users, Gadget generates securely signed URLs to files that expire after 3 hours. This is ideal because it means if a user logs out or their access is revoked, they are no longer able to access the file after the 3 hour window has elapsed. Private files are not CDN cached which is often desirable if they can contain any kind of sensitive data.

If you want anyone to be able to access files, like say the images on a blog or profile photos on a social network, you can mark the field as not private. In this instance, Gadget will generate URLs that never expire so users are always able to access the file, and the files will be cached on Gadget's CDN for maximum performance.

If you change a File field's private configuration, existing uploaded files are not re-permissioned. If you want to make all the files stored by a field private, you would need to re-upload each one to recreate them with the new access permissions.

Where are files stored?

Under the hood Gadget uses Google Cloud Storage to store files. The Gadget API uploads submitted files to a private bucket and uses Google Cloud Storage's Signed URL functionality to grant access to files when a user can prove they have access to the owning record.

It's not currently possible to use your own Cloud Storage bucket for files.