# Storing files 

Within Gadget, you have the ability to store files in 2 different ways depending on your application's use case.

1.  **Static Assets:** Files that are stored and served as-is without any processing or modification by the application. These assets are typically fixed and do not change frequently. Examples of static assets include images, CSS files, JavaScript files, and other files that make up the frontend of your application.
    
2.  **Dynamic Assets:** Files that are generated or processed by the application at runtime. These assets are typically generated dynamically based on user input, database queries, or other dynamic factors, and can change frequently. For example, in a social media application, user-uploaded profile pictures or images in posts would be considered dynamic assets, as they are generated or processed by the application based on user input.
    

**Static Assets:** Files that are stored and served as-is without any processing or modification by the application. These assets are typically fixed and do not change frequently. Examples of static assets include images, CSS files, JavaScript files, and other files that make up the frontend of your application.

**Dynamic Assets:** Files that are generated or processed by the application at runtime. These assets are typically generated dynamically based on user input, database queries, or other dynamic factors, and can change frequently. For example, in a social media application, user-uploaded profile pictures or images in posts would be considered dynamic assets, as they are generated or processed by the application based on user input.

## Static assets within the frontend 

If you require external assets to enhance your app such as images, videos, fonts, stylesheets, etc, Gadget's frontend hosting will store and serve these assets for you. Add your static assets to your project by dragging and dropping into the web editor or use `ggt`, and then import them in your code!

See the [building frontends guide](https://docs.gadget.dev/guides/frontend/building-frontends#static-asset-handling) for more information on referencing static assets in your code.

## Dynamic assets as file field types 

Dynamic files in Gadget are stored as file fields on a Gadget model. Files are uploaded when creating or updating a record and then deleted when the file is no longer referenced by a record. File URLs can then be fetched alongside other data for those records (or related records) using the GraphQL API. Gadget hosts and stores files securely in the cloud. Only users with permission to access the file's owner records can access the stored files.

## Adding a new file field 

To store files, you must add a new file field to one of your Gadget models. Each file field can store one file, so if you need to store multiple files associated with the same record, you can add several file fields to the model, or create a new child model with the file field and a belongs to relationship back to the parent.

### File field validations 

If desired, you can use Gadget's built-in validations for file fields to limit the size of uploaded files or to only accept images. You can also use custom code validations to implement custom business logic to work with the uploaded file and decide if it is valid or not.

```typescript
import got from "got";
import { ValidationContext } from "gadget-server";
import { StoredFile } from "@gadget-client/<your-app-name>";

export default async ({ record, field, errors }: ValidationContext) => {
  // the value of a field inside a validation is a metadata blob describing the uploaded file
  const metadata = record[field.apiIdentifier as keyof typeof record] as StoredFile;

  const {
    // you can access the file's mimeType to get a MIME type string
    // Example => "image/png"
    mimeType,

    // you can access the file's size in bytes
    // Example => 70
    byteSize,

    // you can access the file's URL in cloud storage
    // Example => "https://storage.gadget.dev/files/123/456/aaaaaa/test-image.png"
    url,
  } = metadata;

  // you must explicitly retrieve the file if you want to work with the contents of it
  const contents = await got(metadata.url);
  // do stuff with the file contents as a Buffer to validate it
  if (hasBadBytes(contents)) {
    errors.add(field.apiIdentifier, "has bad bytes");
  }
};
```

You can use modules from npm to work with file contents as well to validate them. For example, you can validate if a file is a valid PDF file by retrieving it from cloud storage using `got`, and then using the `is-pdf-valid` module to check the file's contents:

```typescript
import got from "got";
import validPDF from "is-pdf-valid";
import { ValidationContext } from "gadget-server";
import { StoredFile } from "@gadget-client/<your-app-name>";

export default async ({ record, field, errors }: ValidationContext) => {
  const metadata = record[field.apiIdentifier as keyof typeof record] as StoredFile;

  const contents = await got(metadata.url).buffer();
  // do stuff with the file contents as a Buffer to validate it
  if (!validPDF(contents.toString())) {
    errors.add(field.apiIdentifier, "is not a valid PDF file");
  }
};
```

### File fields in actions 

Records with file fields can be read and written inside actions. The `record` object passed to your function will have a stored file's metadata present for inspection by default, and the file contents can be accessed by downloading the file from cloud storage.

```typescript
import got from "got";

export const run: ActionRun = async ({ record }) => {
  // the value of a field inside a validation is a metadata blob describing the uploaded file
  const metadata = record.someFileField;
  if (!metadata) {
    return;
  }

  const {
    // you can access the file's name. either the original one provided or a generate one if none was specified
    // Example => "test-image.png"
    fileName,

    // you can access the file's mimeType to get a MIME type string
    // Example => "image/png"
    mimeType,

    // you can access the file's size in bytes
    // Example => 70
    byteSize,

    // you can access the file's URL in cloud storage
    // Example => "https://storage.gadget.dev/files/123/456/aaaaaa/test-image.png"
    url,
  } = metadata;

  // you must explicitly retrieve the file if you want to work with the contents of it
  const response = await got(metadata.url);
  // do stuff with the file contents in response.body
  // ...
};
```

For example, we could download a CSV file and work with it:

```typescript
import got from "got";
import { parse } from "csv-parse/sync";

export const run: ActionRun = async ({ api, record, logger }) => {
  // retrieve the file contents from cloud storage
  const fileContents = await got(record.someFileField.url).buffer();

  // parse the file contents as a CSV and log the first row
  const parsed = parse(fileContents, { columns: true, skip_empty_lines: true });
  logger.info(parsed[0]);
};
```

Or, we could send any changed file contents off to a third-party service in an action:

```typescript
import got from "got";

export const run: ActionRun = async ({ api, record }) => {
  if (record.changed("someFileField")) {
    // retrieve the file contents from cloud storage
    const fileContents = await got(record.someFileField.url).buffer();

    // send the file contents to a third party service
    await got.post("https://some-service.com/upload", { body: fileContents });
  }
};
```

## Saving files to a record 

file fields are set the same way as other fields in Gadget using GraphQL mutations. The input file data must be provided in one of the supported formats, and then an optional `fileName` and `mimeType` can also be set on the file. Gadget's API will store the file in cloud storage, and then return a URL to access the file when the record is queried later.

File data can be submitted in a number of ways: as a string, as a multipart request, as a URL, or directly uploaded to cloud storage.

| Approach | Description | Max file size | Performance | Input key |     |
| --- | --- | --- | --- | --- | --- |
| base64 variable | File contents POSTed as a base64 encoded string | 10MB | Should only be used for small files | `base64` |     |
| `File` object variable | File contents POSTed as a multipart request by the generated API client | 100MB | Should only be used for smallish files | `file` |     |
| Existing public URL | File contents downloaded by Gadget from a URL | 100MB | Good performance, depends on existing host | `copyURL` |     |
| Direct uploads | Files uploaded directly to cloud storage, then associated with a record using a token | 10GB | Best user experience, best performance | `directUploadToken` |     |

Each of these methods is used by passing an object with specific keys as the input value for the field in an action on the model.

### The `StoredFileInput` object 

All file field inputs must be given as objects conforming to the GraphQL `StoredFileInput` object schema, both in GraphQL calls and calls made using the generated JavaScript client.

```graphql
type StoredFileInput {
  # Sets the file contents using this string, interpreting the string as base64 encoded bytes. This is useful for creating files quickly and easily if you have the file contents available already, but, it doesn't support files larger than 10MB, and is slower to process for the backend. Using multipart file uploads or direct-to-storage file uploads is preferable.
  base64: GraphQLString
  # Sets the file contents using binary bytes sent alongside a GraphQL mutation as a multipart POST request. Gadget expects this multipart POST request to be formatted according to the GraphQL multipart request spec defined at https://github.com/jaydenseric/graphql-multipart-request-spec. Sending files as a multipart POST request is supported natively by the generated Gadget JS client using File objects as variables in API calls. This method supports files up to 100MB.
  file: GraphQLUpload
  # Sets the file contents by fetching a remote URL and saving a copy to cloud storage. File downloads happen as the request is processed so they can be validated, which means large files can take some time to download from the existing URL. If the file can't be fetched from this URL, the action will fail.
  copyURL: GraphQLURL
  # Sets the file contents using a token from a separate upload request made with the Gadget storage service. Uploading files while a user is completing the rest of a form gives a great user experience and supports much larger files, but requires client side code to complete the upload, and then pass the returned token for this field.
  directUploadToken: GraphQLString
  # Sets this file's mime type, which will then be used when serving the file during read requests as the `Content-Type` HTTP header. If not set, Gadget will infer a content type based on the file's contents.
  mimeType: GraphQLString
  # Sets this file's stored name, which will then be used as the file's name when serving the file during read requests. If not set, Gadget will infer a filename if possible.
  fileName: GraphQLString
}
```

### base64 encoded string uploads 

Files can be uploaded when creating records by using the contents of the file inline as as base64 encoded string using the `base64` field of the input.

For example, if we have an **Image** model that has a string `name` field and a file `image` field, we can create a new **Image** record like this:

```typescript
await api.image.create({
  name: "A Test Image",
  image: {
    base64:
      "iVBORw0KGgoAAAANSUhEUgAAAAEAAAABCAYAAAAfFcSJAAAADUlEQVR42mP8z8BQDwAEhQGAhKmMIQAAAABJRU5ErkJggg==",
    fileName: "test-image.png",
  },
});
```

```graphql
mutation {
  createImage(
    image: {
      name: "A Test Image"
      image: {
        base64: "iVBORw0KGgoAAAANSUhEUgAAAAEAAAABCAYAAAAfFcSJAAAADUlEQVR42mP8z8BQDwAEhQGAhKmMIQAAAABJRU5ErkJggg=="
        fileName: "test-image.png"
      }
    }
  ) {
    success
    image {
      id
      name
      image {
        url
      }
    }
  }
}
```

When queried, the created **Image** record will have an `id`, `createdAt`, and `updatedAt` field like any other model, the `name` field set to `A Test Image`, and the `image` file field set to an object with a `url` field.

##### base64 performance 

base64 encoded string file uploads are convenient if you have the file contents in memory already and want an easy way to submit small files to the Gadget server. But, base64 uploads don't perform well compared to the other methods. Embedding the file contents in the request like this causes delayed processing in the Gadget API, as the whole request string needs to be sent to the API before it can successfully parse the whole JSON blob. Large files have to be base64 encoded which adds extra processing time.

If you are working in a browser or using the generated JavaScript client for your Gadget app, prefer using the `file` key or the `directUploadToken` key, especially if your users are ever uploading large files.

### Uploads using the File object 

Files can be uploaded when creating records by using the [`File`](https://developer.mozilla.org/en-US/docs/Web/API/File) object that browsers expose. `File` objects can be accessed from `<input type="file" />`s or from drag and drop operations. For more information on using `File`s in a web application, see [MDN](https://developer.mozilla.org/en-US/docs/Web/API/File/Using_files_from_web_applications).

To upload a `File`, pass it at the `file` key in the input object for a file field.

For example, if we have an **Image** model that has a string `name` field and a file `image` field, we can create a new **Image** record like this:

```javascript

  // get the file from an <input type="file" />
  // see https://developer.mozilla.org/en-US/docs/Web/API/File/Using_files_from_web_applications#getting_information_about_selected_files
  const file = document.getElementById("#some-file-input").files[0];
  // use the File object at the file key of the input for the image field
  api.image.create({
    name: "A Test Image",
    image: {
      file: file,
    }
  })
```

##### File object performance 

Uploading `File` objects with the `file` works well for small files, as the browser is able to stream the initial GraphQL request, and then the files to the backend API, allowing processing of the request while the files are being uploaded to cloud storage. Large files can take a long time to stream, so if your users intend to upload large files, the  method is best.

Gadget supports a maximum file size of 100MB per file uploaded with the `file` key and a maximum of 10 files per request.

### Uploads using the Multipart Request spec 

Gadget GraphQL APIs implement the [GraphQL Multipart Request](https://github.com/jaydenseric/graphql-multipart-request-spec) specification which allows submitting files alongside other variables in a GraphQL mutation. The Gadget-generated JavaScript client uses this specification to upload `File` objects automatically, but if you're working in a different language or with a different client, you can pass files to Gadget using any client which implements this spec.

For example, to upload a `test.png` file with cURL according to the multipart request spec, you can run the following command:

```shell
curl https://example.gadget.app/api/graphql \
  -F operations='{ "query": "mutation ($file: Upload!) { createImage(image: { name: "A test image", image: { file: $file } } )) { success image { id image { url }} } }", "variables": { "file": null } }' \
  -F map='{ "0": ["variables.file"] }' \
  -F 0=@test.png
```

### Uploads from a URL 

Gadget supports storing files directly from an existing URL on the web. Instead of a client uploading a local file, Gadget will copy a file from a URL into cloud storage.

Gadget copies files from remote URLs once when the URL is first submitted, and then uses its cloud storage system to host the file. Once a file field has been set on a record, take care to use the URL that Gadget generates for the file, instead of the original URL. The Gadget-generated URL will be governed by the Gadget permissions system, and use Gadget's performant CDN for hosting.

To set the value of a file field, use the `copyURL` key of the input object in a GraphQL mutation or API client call.

```javascript
await api.photo.create({
  name: "A photo!",
  image: {
    // create a record copying an image from an existing URL
    copyURL: "https://example.com/image.png",
  },
});
```

```graphql
query CreatePhoto {
  createPhoto(
    photo: {
      name: "A photo!"
      image: {
        # create a record copying an image from an existing URL
        copyURL: "https://example.com/image.png"
      }
    }
  ) {
    photo {
      id
      name
      image {
        url
      }
    }
  }
}
```

##### Direct URL upload performance 

When possible, copying from remote URLs to upload files tends to perform well. Server-to-server communication over the internet is generally much faster than a user's connection, so if files are already hosted somewhere, copying them into Gadget can be faster and more convenient for the user than downloading and re-uploading them.

### Direct uploads using tokens 

Uploading files to a remote server can be slow sometimes, especially if the files are large or the user's internet connection is constrained. Often, users upload files to an application at the same time as filling in other data, and they select which file they want to upload before filling out the rest of the form. That means there's a good opportunity to give users a good experience by uploading their files in the background while they complete the rest of the form. If the file is large, starting the upload as soon as possible helps as well to get the user to complete the submission faster by starting the upload sooner.

To support this workflow, Gadget supports _direct uploads_, which allows the user's browser to start an upload as soon as the file is ready, and upload it directly to cloud storage. The form can then be submitted to Gadget later with a token for the file.

The process works like this:

1.  The user's browser requests the `/api/files/upload-target` Gadget API to retrieve an upload URL and a file token
2.  The user's browser uploads a file to the upload URL
3.  The user finishes filling out the form with any other data
4.  Once the upload is complete and any other client-side validations are satisfied, the user's browser submits the form to Gadget and runs a model action. The action input references the file using the `directUploadToken` key of the `StoredFileInput` for the file field on the model.

Gadget will retrieve metadata from the uploaded file in cloud storage and associate it with the file field on the record.

#### Getting a file upload URL and token 

Direct upload URLs and tokens need to be fetched _before_ a user uploads a file or submits a form. To get a token, you can use the `await api.getDirectUploadToken()` API client function, the `gadgetMeta { directUploadToken { url token } }` GraphQL query, or the `/api/files/upload-target` REST endpoint.

```javascript
import got from "got";
// get an upload URL and token
const { url, token } = await api.getDirectUploadToken();
// upload the image to cloud storage
await got(url, {
  method: "PUT",
  headers: {
    "Content-Type": "text/plain",
  },
  body: "an example text file",
});
// create a record using the returned token that is now associated with the file
await api.attachment.create({
  name: "A file!",
  file: {
    directUploadToken: token,
    fileName: "example.txt",
  },
});
```

```graphql
query GetDirectUploadToken {
  gadgetMeta {
    directUploadToken {
      url
      token
    }
  }
}
```

```curl
curl https://my-gadget-app.gadget.app/api/files/upload-target
```

Once you have a direct upload `url`, you must `PUT` the file's contents to the `url`. The `url` is signed for security and expires after 3 hours to avoid abuse. This means you can't just generate the `url` once and hardcode it in your code -- you must generate a new `url` for each upload by your form.

Once the upload has been completed, use the `token` as the value of the `directUploadToken` key in the input to the file field.

```javascript
const { url, token } = await api.getDirectUploadToken();
// upload image to cloud storage
api.image.create({
  name: "A Test Image",
  image: {
    // pass the token instead of a file key or base64 key
    directUploadToken: token,
    // set the filename or the mime type if needed
    fileName: "test-image.png",
  },
});
```

```graphql
mutation {
  createImage(
    image: {
      name: "A Test Image"
      image: {
        # pass the token instead of a file key or base64 key
        directUploadToken: "v4.local.someopaquetoken"
        # set the filename or the mime type if needed
        fileName: "test-image.png"
      }
    }
  ) {
    success
    image {
      id
      name
      image {
        url
      }
    }
  }
}
```

Direct upload `token`s can be re-used to associate the same uploaded file with many records. If you complete the upload to the `url` once, you can call many Gadget actions with the same `token`. Direct upload `url`s can't be re-used to upload more than one file. You must create a new `url` and `token` for each unique file uploaded by the user.

Direct upload tokens expire after 3 hours. Users can't take longer than this between uploading a file to the `url` and using the `token` in an action.

#### Writing frontend upload forms 

Some frontend code is required to coordinate file upload when using the direct token upload method. The `token` should only be sent to Gadget once the upload is complete and the user has completed the form. The general flow is:

1.  Wait for the user to submit a file
2.  Get a `url` and `token` before you start the file upload with `api.getDirectUploadToken()`
3.  Upload the file to cloud storage in the background with `fetch` or similar
4.  Store the `token` in the state while the user completes the rest of the form
5.  Submit the action to Gadget once the user has finished the form and the upload has completed

Using React as an example, we can listen to a `drop` event on an element and upload the dropped file right to cloud storage:

```tsx
import React, { useState } from "react";

// Import `api` from wherever your Gadget client is constructed
import { api } from "../api";

function Uploader(props: { onFileTokenChange: (token: string, fileName: string) => void }) {
  const [isUploading, setIsUploading] = useState(false);
  const [fileName, setFileName] = useState<string | null>(null);

  return (
    <div
      onDragOver={(event) => event.preventDefault()}
      onDrop={async (event) => {
        event.preventDefault();
        const file = event.dataTransfer.files[0];
        // send the file to cloud storage
        setIsUploading(true);

        const { url, token } = await api.getDirectUploadToken();

        try {
          await fetch(url, {
            method: "PUT",
            headers: {
              "Content-Type": file.type,
            },
            body: file,
          });
          // once it has been uploaded, give the parent component the token and file name to submit with the other data in the form
          props.onFileTokenChange(token, file.name);
          setFileName(file.name);
        } finally {
          setIsUploading(false);
        }
      }}
    >
      {isUploading ? "Uploading ..." : fileName ? fileName : "Drop a file here"}
    </div>
  );
}
```

##### Direct file upload performance 

Direct file uploads perform the best for both the user and the systems involved out of any of the upload methods. The user benefits because the file upload is started (and completed) earlier, which allows their form submission to complete as soon as they have finished filling it out. If the form is complicated or long and the user is likely to get some validation errors, it's also great to be able to re-use the token for the uploaded file for a second or third submission instead of re-uploading the file again.

Direct file uploads also support the biggest files as there is no file size limit. Because the user's browser is uploading directly to cloud storage, the cloud storage system's optimized frontends can handle the upload and give it the best chance of completing with very high request timeouts and great internet connectivity.

### Storing new files within actions 

To store a new file programmatically within an action, make an API call with the `api` object to run an action with the same inputs as detailed above. The `api` object available in the action context works the same as an `api` object in an external script, so all the upload methods are available and ready for use inside actions as well.

For example, we can create file contents by serializing an object to JSON, and then store the JSON contents in a file in cloud storage:

```typescript
import got from "got";

export const run: ActionRun = async ({ api, record }) => {
  // build some example data
  const data = {
    name: record.name,
    description: record.description,
  };

  const fileContents = JSON.stringify(data);
  // create a new record of a model with a File field named `file`
  await api.someModel.create({
    file: {
      // set the filename
      fileName: "example.json",
      // base64 encode the file contents to pass to Gadget
      base64: new Buffer(fileContents).toString("base64"),
    },
  });
};
```

The `copyURL` upload method can also be used inside actions to save a record with a file downloaded from a URL:

```typescript
import got from "got";

export const run: ActionRun = async ({ api, record }) => {
  // create a new record of a model with a File field named `file`
  await api.someModel.create({
    file: {
      // base64 encode the file contents to pass to Gadget
      copyURL: "https://assets.gadget.dev/assets/favicon-32.png",
    },
  });
};
```

Currently, the Internal API does not support uploading new files to Gadget, so you must use your application's Public API to manage files inside effects.

## Retrieving files 

file fields are accessed like other fields in Gadget using the generated GraphQL API. Gadget returns several details about a stored file like it's `url`, `fileName`, `mimeType`, and `byteSize`. File field details can be fetched when querying individual records or lists of records.

```javascript
api.image.findMany({
  select: {
    // get details of the owning record
    id: true,
    name: true,
    image: {
      // get the file's URL, suitable for embedding in an <img src="..."/> tag
      url: true,
      // get the file's filename, including extension
      fileName: true,
      // get the file's mime type
      mimeType: true,
      // get the file's size in bytes
      byteSize: true,
    },
  },
});
```

```graphql
query {
  images {
    edges {
      node {
        # get details of the owning record
        id
        name
        image {
          # get the file's URL, suitable for embedding in an <img src="..."/> tag
          url
          # get the file's filename, including extension
          fileName
          # get the file's mime type
          mimeType
          # get the file's size in bytes
          byteSize
        }
      }
    }
  }
}
```

## Marking files as private 

file fields in Gadget can be marked as private or not. When the field is private, unauthenticated users aren't able to access the URL for a file at all. For authenticated users, Gadget generates securely signed URLs to files that expire after 3 hours. This is ideal because it means if a user logs out or their access is revoked, they can no longer access the file after the 3 hour window has elapsed. Private files are not CDN cached which is often desirable if they can contain any kind of sensitive data.

If you want anyone to be able to access files, like the images on a blog, or profile photos on a social network, you can mark the field as not private. In this instance, Gadget will generate URLs that never expire so users are always able to access the file, and the files will be cached on Gadget's CDN for maximum performance.

If you change a file field's private configuration, existing uploaded files are not re-permissioned. If you want to make all the files stored by a field private, you would need to re-upload each one to recreate them with the new access permissions.

## Deleting stored files 

Files are deleted from storage when they are no longer referenced by a record. There are 2 ways to delete stored files:

1.  Delete the record that owns the file. When a record is deleted, all files associated with it are deleted as well.
2.  Set a file field to `null`.

Updating a file field on a record also deletes the original file from storage.

## Where are files stored? 

Under the hood, Gadget uses Google Cloud Storage to store files. The Gadget API uploads submitted files to a private bucket and uses Google Cloud Storage's [Signed URL](https://cloud.google.com/storage/docs/access-control/signed-urls) functionality to grant access to files when a user can prove they have access to the owning record.

It's not currently possible to use your own Cloud Storage bucket for files.