Storing files

Gadget supports uploading and downloading files like images, audio, PDFs, binary blobs or anything else to your Gadget application. Files in Gadget are stored as File fields on a Gadget model. Files are uploaded when creating or updating a record, and then deleted when that record is deleted. File URLs can then be fetched alongside other data for those records (or related records) using the GraphQL API. Gadget hosts and stores files securely in the cloud, allowing access to files to users with permission to access to the owner records.

Adding a new file field

To store files, you must add a new File field to one of your Gadget models. Each File field can store one file, so if you need to store multiple files associated with the same record, you can add several File fields to the model, or create a new child model with the File field and a BelongsTo relationship back to the parent.

File field validations

If desired, you can use Gadget's built in validations for file fields to limit the size of uploaded files, or to only accept images. You can also use custom code validations to implement custom business logic to work with the uploaded file and decide if it is valid or not.

Validating a file field
JavaScript
1module.exports = ({ record, field, errors }) => {
2 // the value of a field inside a validation is a metadata blob describing the uploaded file
3 const blob = record[field.apiIdentifier];
4 // you can access the file's mimeType to get a MIME type string
5 blob.mimeType; // #=> "image/png"
6 // you can access the file's size in bytes
7 blob.byteSize; // #=> 70
8 // you can access the file's URL in cloud storage
9 blob.url; // #=> "https://storage.gadget.dev/files/123/456/aaaaaa/test-image.png"
10 // you must explicitly retrieve the file if you want to work with the contents of it
11 const got = require("got");
12 const file = await got(blob.url);
13 // do stuff with the file contents as a Buffer to validate it
14 if (hasBadBytes(file)) {
15 errors.add(field.apiIdentifier, "has bad bytes");
16 }
17};

You can use modules from npm to work with file contents as well in order to validate them. For example, you can validate if a file is a valid PDF file by retrieving it from cloud storage using got, and then using the is-pdf-valid module to check the file's contents:

Validating a PDF file field
JavaScript
1const got = require("got");
2const validPDF = require("is-pdf-valid");
3module.exports = ({ record, field, errors }) => {
4 const blob = record[field.apiIdentifier];
5 const contents = await got(blob.url).buffer();
6 // do stuff with the file contents as a Buffer to validate it
7 if (!validPDF(contents)) {
8 errors.add(field.apiIdentifier, "is not a valid PDF file");
9 }
10};

Saving files to a record

File fields are set the same way as other fields in Gadget using GraphQL mutations. The input file data must be provided in one of the supported formats, and then an optional fileName and mimeType can also be set on the file. Gadget's API will store the file in cloud storage, and then return a URL to access the file when the record is queries later.

File data can submitted in a number of ways: as a string, as a multipart request, as a URL, or directly uploaded to cloud storage.

ApproachDescriptionMax File SizePerformanceInput Key
base64 variableFile contents POSTed as a base64 encoded string10MBShould only be used for small filesbase64more info
File object variableFile contents POSTed as a multipary request by the generated API client100MBShould only be used for smallish filesfilemore info
Existing public URLFile contents downloaded by Gadget from a URL100MBGood performance, depends on existing hostcopyURLmore info
Direct uploadsFiles uploaded directly to cloud storage, then associated with a record using a token10GBBest user experience, best performancedirectUploadTokenmore info

Each of these methods is used by passing an object with specific keys as the input value for the field in an action on the model.

The StoredFileInput object

All File field inputs must be given as objects conforming to the GraphQL StoredFileInput object schema, both in GraphQL calls and calls made using the generated JavaScript client.


GraphQL
1type StoredFileInput {
2 # Sets the file contents using this string, interpreting the string as base64 encoded bytes. This is useful for creating files quickly and easily if you have the file contents available already, but, it doesn't support files larger than 10MB, and is slower to process for the backend. Using multipart file uploads or direct-to-storage file uploads is preferable.
3 base64: GraphQLString
4 # Sets the file contents using binary bytes sent along side a GraphQL mutation as a multipart POST request. Gadget expects this multipart POST request to be formatted according to the GraphQL multipart request spec defined at https://github.com/jaydenseric/graphql-multipart-request-spec. Sending files as a multipart POST requests is supported natively by the generated Gadget JS client using File objects as variables in API calls. This method supports files up to 100MB.
5 file: GraphQLUpload
6 # Sets the file contents by fetching a remote URL and saving a copy to cloud storage. File downloads happen as the request is processed so they can be validated, which means large files can take some time to download from the existing URL. If the file can't be fetched from this URL, the action will fail.
7 copyURL: GraphQLURL
8 # Sets the file contents using a token from a separate upload request made with the Gadget storage service. Uploading files while a user is completing the rest of a form gives a great user experience and supports much larger files, but requires client side code to complete the upload, and then pass the returned token for this field.
9 directUploadToken: GraphQLString
10 # Sets this file's mime type, which will then be used when serving the file during read requests as the `Content-Type` HTTP header. If not set, Gadget will infer a content type based on the file's contents.
11 mimeType: GraphQLString
12 # Sets this file's stored name, which will then be used as the file's name when serving the file during read requests. If not set, Gadget will infer a filename if possible.
13 fileName: GraphQLString
14}

base64 encoded string uploads

Files can be uploaded when creating records by using the contents of the file inline as as base64 encoded string using the base64 field of the input.

For example, if we have an Image model that has a String name field and a File image field, we can create a new Image record like this:

Setting an image using base64
1api.image.create({
2 image: {
3 name: "A Test Image",
4 image: {
5 base64:
6 "iVBORw0KGgoAAAANSUhEUgAAAAEAAAABCAYAAAAfFcSJAAAADUlEQVR42mP8z8BQDwAEhQGAhKmMIQAAAABJRU5ErkJggg==",
7 fileName: "test-image.png",
8 },
9 },
10});
1mutation {
2 createImage(
3 image: {
4 name: "A Test Image"
5 image: {
6 base64: "iVBORw0KGgoAAAANSUhEUgAAAAEAAAABCAYAAAAfFcSJAAAADUlEQVR42mP8z8BQDwAEhQGAhKmMIQAAAABJRU5ErkJggg=="
7 fileName: "test-image.png"
8 }
9 }
10 ) {
11 success
12 image {
13 id
14 name
15 image {
16 url
17 }
18 }
19 }
20}

When queries, the created Image record will have an id, createdAt and updatedAt field like any other model, the name field set to A Test Image, and the image File field set to an object with a url field.

Performance

base64 encoded string file uploads are convenient if you have the file contents in memory already and want an easy way to submit small files to the Gadget server. But, base64 uploads don't perform well compared to the other methods. Embedding the file contents in the request like this causes delayed processin in the Gadget API, as the whole request string needs to be sent to the API before it can successfully parse the whole JSON blob. Large files have to be base64 encoded which adds extra processing time.

If you are working in a browser or using the generated JavaScript client for your Gadget app, prefer using the file key or the directUploadToken key, especially if your users are ever uploading large files.

Uploads using the File object

Files can be uploaded when creating records by using the File object that browsers expose. File objects can be accessed from <input type="file" />s or from drag and drop operations. For more information on using Files in a web application, see MDN.

To upload a File, pass it at the file key in the input object for a File field.

For example, if we have an Image model that has a String name field and a File image field, we can create a new Image record like this:

Setting an image using a File object
JavaScript
1// get the file from an <input type="file" />
2// see https://developer.mozilla.org/en-US/docs/Web/API/File/Using_files_from_web_applications#getting_information_about_selected_files
3const file = document.getElementById("#some-file-input").files[0];
4// use the File object at the file key of the input for the image field
5api.image.create({
6 image: {
7 name: "A Test Image",
8 image: {
9 file: file,
10 },
11 },
12});
Performance

Uploading File objects with the file works well for small files, as the browser is able to stream the initial GraphQL request, and then the files to the backend API, allwing processing of the request while the files are being uploaded to cloud storage. Large files can take a long time to stream however, so if your users intend to upload large files, the Direct Upload method is best.

Gadget supports a maximum file size of 100MB per file uploaded with the file key and a maximum of 10 files per request.

Uploads using the Multipart Request spec

Gadget GraphQL APIs implement the GraphQL Multipart Request specification which allows submitting files alongside other variables in a GraphQL mutation. The Gadget-generated JavaScript client uses this specification to upload File objects automatically, but if you're working in a different language or with a different client, you can pass files to Gadget using any client which implements this spec.

For example, to upload a test.png file with cURL according to the multipart request spec, you can run the following command:

Shell
curl https://example.gadget.app/api/graphql \
-F operations='{ "query": "mutation ($file: Upload!) { createImage(image: { name: "A test image", image: { file: $file } } )) { success image { id image { url }} } }", "variables": { "file": null } }' \
-F map='{ "0": ["variables.file"] }' \
-F 0=@test.png

Uploads from a URL

Gadget supports storing files directly from an existing URL on the web. Instead of a client uploading a local file, Gadget will copy a file from a URL into cloud storage.

Gadget copies files from remote URLs once when the URL is first submitted, and then uses its own cloud storage system to host the file. Once a File field has been set on a record, take care to use the URL that Gadget generates for the file, instead of the original URL. The Gadget-generated URL will be governed by the Gadget permissions system, and use Gadget's performant CDN for hosting.

To set the value of a File field, use the copyURL key of the input object in a GraphQL mutation or api client call.

Using copyURL to download an image
1await api.photo.create({
2 photo: {
3 name: "A photo!",
4 image: {
5 // create a record copying an image from an existing URL
6 copyURL: "https://example.com/image.png",
7 },
8 },
9});
1query CreatePhoto {
2 createPhoto(
3 photo: {
4 name: "A photo!"
5 image: {
6 # create a record copying an image from an existing URL
7 copyURL: "https://example.com/image.png"
8 }
9 }
10 ) {
11 photo {
12 id
13 name
14 image {
15 url
16 }
17 }
18 }
19}
Performance

When possible, copying from remote URLs to upload files tends to perform well. Server to server communication over the internet is generally much faster than a user's connection, so if files are already hosted somewhere, copying them into Gadget can be faster and more convenient for the user than downloading and re-uploading them.

Direct uploads using tokens

Uploading files to a remote server can be slow sometimes, especially if the files are large or the user's internet connection is constrainted. Often, users upload files to an application at the same time as filling in other data, and they select which file they want to upload before filling out the rest of the form. That means there's a good opportunity to give users a good experience by uploading their files in the background while they complete the rest of the form. If the file is large, starting the upload as soon as possible helps as well to get the user to complete the submission faster by starting the upload sooner.

To support this workflow, Gadget supports direct uploads, which allows the user's browser to start an upload as soon as file is ready, and upload it directly to cloud storage. The form is then be submitted to Gadget later with a token for the file.

The process works like this:

  • the user's browser requests the /api/files/upload-target Gadget API to retrieve an upload URL and a file token
  • the user's browser uploads a file to the upload URL
  • the user finishes filling out the form with any other data
  • once the upload is complete and any other client side validations are satisfied, the user's browser submits the form to Gadget and runs a model action. The action input references the file using the directUploadToken key of the StoredFileInput for the File field on the model.

Gadget will retrieve metadata from the uploaded file in cloud storage and associate it with the File field on the record.

Getting a file upload URL and token

Direct upload URLs and tokens need to be fetched before a user uploads a file or submits a form. To do get a token, you can use the await api.getDirectUploadToken() API client function, the gadgetMeta { directUploadToken { url token } } GraphQL query, or the /api/files/upload-target REST endpoint.

Getting a direct upload token
1import got from "got";
2// get an upload URL and token
3const { url, token } = await api.getDirectUploadToken();
4// upload the image to cloud storage
5await got(url, {
6 method: "PUT",
7 headers: {
8 "Content-Type": "text/plain",
9 },
10 body: "an example text file",
11});
12// create a record using the returned token that is now associated with the file
13await api.attachment.create({
14 attachment: {
15 name: "A file!",
16 file: {
17 directUploadToken: token,
18 fileName: "example.txt",
19 },
20 },
21});
1query GetDirectUploadToken {
2 gadgetMeta {
3 directUploadToken {
4 url
5 token
6 }
7 }
8}
curl https://my-gadget-app.gadget.app/api/files/upload-target

Once you have a direct upload url, you must PUT the file's contents to the url. The url is signed for security and expires after 30 minutes. Direct upload URLs can't be re-used, so you must fetch a new url and token combination for each new upload you want to make.

Once the upload has been completed, use the token as the value of the directUploadToken key in the input to the File field.

Setting an image using a direct upload token
1const { url, token } = await api.getDirectUploadToken();
2// upload image to cloud storage
3api.image.create({
4 image: {
5 name: "A Test Image",
6 image: {
7 // pass the token instead of a file key or base64 key
8 directUploadToken: token,
9 // set the filename or the mime type if needed
10 fileName: "test-image.png",
11 },
12 },
13});
1mutation {
2 createImage(
3 image: {
4 name: "A Test Image"
5 image: {
6 # pass the token instead of a file key or base64 key
7 directUploadToken: "v4.local.someopaquetoken"
8 # set the filename or the mime type if needed
9 fileName: "test-image.png"
10 }
11 }
12 ) {
13 success
14 image {
15 id
16 name
17 image {
18 url
19 }
20 }
21 }
22}

Direct upload tokens can be re-used, so if you want to associate the same uploaded file with many records, you can re-use a token in the parameters of different actions.

Performance

Direct file uploads perform the best for both the user and the systems involved out of any of the upload methods. The user benefits because the file upload is started (and completed) earlier, which allows their form submission to complete as soon as they have finished filling it out. If the form is complicated or long and the user is likely to get some validation errors, it's also great to be able to re-use the token for the uploaded file for a second or third submission instead of re-uploading the file again.

Direct file uploads also support the biggest files as there is no file size limit. Because the user's browser is uploading directly to cloud storage, the cloud storage system's optimized frontends can handle the upload and give it the best chance of completing with very high request timeouts and great internet connectivity.

Retrieving files

File fields are accessed like other fields in Gadget using the generated GraphQL API. Gadget returns several details about a stored file like it's url, fileName, mimeType, and byteSize. File field details can be fetched when querying individual records or lists of records.

Getting a file
1api.image.findMany({
2 select: {
3 // get details of the owning record
4 id: true,
5 name: true,
6 image: {
7 // get the file's URL, suitable for embedding in an <img src="..."/> tag
8 url: true,
9 // get the file's filename, including extension
10 fileName: true,
11 // get the file's mime type
12 mimeType: true,
13 // get the file's size in bytes
14 byteSize: true,
15 },
16 },
17});
1query {
2 images {
3 edges {
4 node {
5 # get details of the owning record
6 id
7 name
8 image {
9 # get the file's URL, suitable for embedding in an <img src="..."/> tag
10 url
11 # get the file's filename, including extension
12 fileName
13 # get the file's mime type
14 mimeType
15 # get the file's size in bytes
16 byteSize
17 }
18 }
19 }
20 }
21}

Marking files as private

File fields in Gadget can be marked as private or not. When the field is private, unauthenticated users aren't able to access the URL for a file at all. For authenticated users, Gadget generates securely signed URLs to files that expire after 3 hours. This is ideal because it means if a user logs out or their access is revoked, they are no longer able to access the file after the 3 hour window has elapsed. Private files are not CDN cached which is often desirable if they can contain any kind of sensitive data.

If you want anyone to be able to access files, like say the images on a blog or profile photos on a social network, you can mark the field as not private. In this instance, Gadget will generate URLs that never expire so users are always able to access the file, and the files will be cached on Gadget's CDN for maximum performance.

If you change a File field's private configuration, existing uploaded files are not re-permissioned. If you want to make all the files stored by a field private, you would need to re-upload each one to recreate them with the new access permissions.

Where are files stored?

Under the hood Gadget uses Google Cloud Storage to store files. The Gadget API uploads submitted files to a private bucket and uses Google Cloud Storage's Signed URL functionality to grant access to files when a user can prove they have access to the owning record.

It's not currently possible to use your own Cloud Storage bucket for files.