Building with OpenAI
Gadget has a variety of purpose-built features for building apps that use the exciting innovations in artificial intelligence.
- vector allows storing and retrieving vectors for implementing semantic search for LLM memory
- file allows storing generated artifacts like images, audio, or video synthesized by AI models
- HTTP Routes streaming support support for sending slow-to-generate responses from upstream APIs
- Support for the
openai
andlangchain
node modules (without any annoying edge-environment drawbacks)
Working with LLMs
Large Language Model (LLM) APIs allow developers to build incredible apps that can understand and generate text. To work with LLMs like OpenAI's GPT-4 or Anthropic's Claude, you can make API calls from your Gadget app to these services.
Using vectors for LLM state
Developers often need to pass specific context to a call to an LLM. Content like:
- the current user's details
- documentation relevant to the user's questions
- up-to-date information about current events
are all commonly inserted into an LLM prompt to refine the results. This is by no means a complete list -- just about any text can be sent in. However, since the LLMs have a limited input context size, you can't just pass everything under the sun. Instead, you must choose a relevant subset of your available context to the LLM.
Identifying and retrieving the most relevant context to pass is a common problem with a lot of depth. Most folks turn to a vector similarity search to solve this problem. Vector similarity searches allow you to store a big library of potential context entries and retrieve only the most relevant entries for each prompt. Then, only the relevant entries can be inserted into the prompt, keeping it short enough to be processed by the model. Relevance can be assessed without having to build a big rules system or a deep understanding of the plain-language that users might use to describe their problems.
Gadget supports vector similarity searching, filtering and sorting using the vector field on your app's data models.
Vector embeddings of strings
Vector search requires computing a vector for the content you want to search on. Vector similarity search allows you to ask the database to return records with vectors that are similar to an input vector, so you need a tool to convert any incoming strings (or images, etc) into a vector that you can then store and search with. This conversion-to-a-vector process is called embedding. The quality of the embedding system is what allows strings that are semantically similar but different character-wise to have similar vectors.
For example, let's say a user is asking a chatbot the question "What will keep me dry in the rain?". If we have a bunch of stored content that we might want to feed to the LLM, we don't want to just search for the term "dry" or "rain" in the text, we'd ideally search for related terms, like "umbrella", "shelter", or "waterproof". Vector embeddings are the fancy algorithms that group all these different English words together by producing similar vectors for similar text.
Gadget recommends using OpenAI's embeddings API for computing vectors from strings. You can use the openai
package to make calls to the embeddings API:
JavaScriptconst response = await openai.createEmbedding({model: "text-embedding-ada-002",input: "what will keep me dry in the rain",});const vector = response.data.data[0].embedding;
Read more about installing the openai
npm package below.
Storing vectors on models
To create a database of context for your LLM apps, you can store context as a Model in Gadget. Models allow you to store structured, unstructured, and vector data together in a single record within your app's serverless database.
For example, if we were building a chatbot for a product to answer questions powered by existing documentation, we could store each paragraph of the documentation as a record in a Documentation model, and then use this stored documentation to power the prompt generation for an LLM. We can create a model with the following fields:
- a string
body
field to store the plain text - a url
url
field for where the documentation actually came from - a vector
embedding
field for storing a vector embeddings
Then, when creating a new record, we can compute the embedding for the body
string automatically, and store it in the embedding
field. In the create
action of the model, we can add code to the onSuccess
function to compute the embedding and save it.
documentation/create.jsJavaScript1// import the openai client from a file where we construct it2import { openai } from "../../openai";34export async function run({ api, record }) {5 // get an embedding for the record.body field6 const response = await openai.createEmbedding({7 model: "text-embedding-ada-002",8 input: record.body,9 });10 const vector = response.data.data[0].embedding;1112 // store the vector in the db on this record13 await api.internal.description.update(record.id, {14 description: { embedding: vector },15 });16}
Retrieving similar records
Once you have records stored in your models with a vector field, you can query these records to retrieve records that are similar to a given vector.
Using the example Documentation model mentioned earlier, we can query the stored records to retrieve the body of the documentation snippets most relevant to a given prompt:
JavaScript1const relevantRecords = await api.documentation.findMany({2 sort: {3 embedding: {4 cosineSimilarityTo: [5 0.1, 0.15, 0.1, 0.5,6 // ... the remainder of a vector embedding7 ],8 },9 },10 first: 10,11 select: {12 id: true,13 body: true,14 },15});
This query will retrieve the top 10 most similar records to the given vector, and return only the id
and body
fields of each record. The embedding
field does not need to be selected in this case as it's only being used to power the sorting of the results, and we don't actually need the stored vectors client side.
Read more about the sorts and filters offered by your models with vector fields in the API Reference.
When sorting or filtering by vectors, you need to use the same embedding API to compute the vectors stored on models and the vectors
from any user input. For OpenAI, this means you must pass the same model
parameter to all your text embedding calls so that the vectors
returned are all in the same vector space.
Passing context to an LLM
With your data embedded into vectors within your app's database, you can query it for context to add to a prompt for an LLM. The flow is generally this:
- Take the user's input and embed it into a vector
- Query the database to find the most similar context using vector similarity against the input vector
- Build a bigger, internal string prompt for the LLM by adding the retrieved context to the user's prompt
- Send the prompt to the LLM and forward along its response to the user.
For example, if we have documentation for a product stored in a Documentation model as mentioned above, we can use the most relevant documentation to build a prompt for an LLM. We can accept the user's input as a param to an HTTP Route and then embed it into a vector, retrieve the most relevant records, and then send a combined prompt to OpenAI's chat completion API:
routes/GET-ask.jsJavaScript1import { openai } from "../openai";23export default async function ({ api, request, reply }) {4 // get the user's question from a ?question= query param5 const question = request.query.question;67 // get a vector embedding for the question8 const response = await openai.createEmbedding({9 model: "text-embedding-ada-002",10 input: question,11 });12 const questionVector = response.data.data[0].embedding;1314 // find the most relevant records from the documentation model15 const relevantRecords = await api.documentation.findMany({16 sort: {17 embedding: {18 cosineSimilarityTo: questionVector,19 },20 },21 first: 5,22 select: {23 id: true,24 body: true,25 },26 });2728 // build a big prompt out of a base prompt, the user's question, and the retrieved context29 const completion = await openai.createChatCompletion({30 model: "gpt-3.5-turbo",31 messages: [32 {33 role: "system",34 content:35 "You are a helpful assistant who answers users questions with some related documentation.",36 },37 {38 role: "user",39 content: `40 Related documentation: ${relevantRecords41 .map((record) => record.body)42 .join("\n")}4344 Question: ${question}45 `,46 },47 ],48 });4950 await reply.send(completion.data.choices[0].message);51}
Then, we can call this route to get a chat completion:
terminal> curl https://example-app--development.gadget.app/ask?question=what%20will%2keep%20me%20dry%20in%20the%20rainA raincoat or shelter will keep you dry in the rain.
At this point, you've successfully implemented semantic search for LLM prompt augmentation in your app! Well done!
For vector similarity to work, we need to also embed the user's input into a vector, and compare that vector to the vectors in the database. We compare vectors to vectors, so anything that got embedded on the way into the database has to be compared to other embeddings only.
Working with Image Generators
Text-to-image APIs like Stable Diffusion's or DALL-E can be used with Gadget to build incredible image generation apps.
Generating images
Images can be generated within Gadget apps by making API calls to one of the outstanding image generation models out there, like Stable Diffusion, or OpenAI's DALL-E.
Using Stable Diffusion
For generating images with Stable Diffusion, we recommend using got
for making HTTP requests. Add got
version 11 to your package.json:
package.jsonjson{"dependencies": {"got": "^11"}}
and hit the Run yarn install
button to install the package. Then, add a STABLE_DIFFUSION_API_KEY
environment variable to your Environment Variables in your Settings.
Now, you can use got
to make requests to Stable Diffusion in your action code or HTTP routes:
JavaScript1const response = await got("https://stablediffusionapi.com/api/v3/text2img", {2 method: "POST",3 json: {4 key: process.env["STABLE_DIFFUSION_API_KEY"],5 prompt: "a cat",6 samples: 1,7 width: "512",8 height: "512",9 },10}).json();1112// will be a URL to an image of a cat! this url will expire after 1 hour13const url = generateResponse.output[0];
Using DALL-E
To use DALL-E, we recommend using the openai
client from npm. Install the openai
package using the instructions below.
Then, use the openai
package in your action code or routes:
JavaScript1import { Configuration, OpenAIApi } from "openai";2const configuration = new Configuration({ apiKey: process.env["OPENAI_API_KEY"] });3const openai = new OpenAIApi(configuration);45const response = await openai.createImage({6 prompt: "a cat",7 n: 1,8 size: "1024x1024",9});1011// will be a URL to an image of a cat! this url will expire after 1 hour12const url = response.data.data[0].url;
Storing images
Text-to-image APIs typically have a short expiration time for the images they generate. As a result, the URLs returned from these APIs will stop working after a short time. To preserve any images for later use, you need to copy them into your own storage.
Gadget's file field type works well for storing any blob content, including generated images. You can copy files from external URLs in your Gadget app's storage with the copyURL: "https://some-url"
input format for file.
To generate and store an image, you can call your preferred image generation API, then pass a URL to a generated image to the copyURL
input of a file field on a model of your choosing.
For example, we can create a new generate
Action on an example image
Model, and add code to the run
function for generating the image with the Stable Diffusion API:
image/onGenerate.jsJavaScript1import got from "got";23/**4 * Action code for the generate action on the Image model5 */6export async function run({ api, record, logger }) {7 const generateResponse = await got(8 "https://stablediffusionapi.com/api/v3/text2img",9 {10 method: "POST",11 json: {12 key: process.env["STABLE_DIFFUSION_API_KEY"],13 // pass a prompt from a field named `prompt` on the image model14 prompt: record.prompt,15 samples: 1,16 width: "512",17 height: "512",18 },19 }20 ).json();2122 if (generateResponse.status != "success") {23 logger.error({ response: generateResponse }, "error generating image");24 throw new Error("generating image failed");25 }2627 const url = generateResponse.output[0];28 logger.info({ url }, "generated image");2930 // update the image field to store this generated image by passing the url to the `copyURL` input param31 await api.image.update(record.id, { image: { image: { copyURL: url } } });32 logger.info("stored generated image in Gadget");33}
Now, you can generate new images with your api
client by calling the generate
action, and access the generated images from Gadget's high-performance cloud storage:
JavaScriptconst record = await api.image.generate({ image: { prompt: "a cool looking cat" } });// record.image.url will be a URL to an image of a cat! this url will not expire!console.log(record.image.url);
Installing dependencies
Gadget apps are built with Node.js, and you can install any dependencies that communicate with third-party AI APIs like OpenAI's by adding packages to your package.json
.
Installing OpenAI
To install the openai
package from npm, add it to your package.json
:
package.jsonjson{"dependencies": {"openai": "^4.55.7"}}
and hit the Run yarn install
button to install the package.
Then, add an OPENAI_API_KEY
environment variable in your Environment Variables in your app's Settings section. You can get an OpenAI API Key from the OpenAI developer portal.
Next, construct an instance of the client object in a helper file:
/openai.jsJavaScriptimport { Configuration, OpenAIApi } from "openai";const configuration = new Configuration({ apiKey: process.env["OPENAI_API_KEY"] });export const openai = new OpenAIApi(configuration);
Now, you can use the openai
package in your action code or routes:
routes/GET-example.jsJavaScript1import { openai } from "../openai";23// ... somewhere in your route4const response = await openai.createImage({5 prompt: "a cat",6 n: 1,7 size: "1024x1024",8});910// will be a URL to an image of a cat!11const url = response.data.data[0].url;