Choosing a runtime

Modal runs all your code in containers that start from an image, on CPUs or GPUs you choose. Its runtimes differ in what starts a container and what stops it: Modal itself, scaling with calls or requests, or your code, creating a sandbox and terminating it.

Modal manages less as you decide more

Functionper call

Serverper request

programSandbox + main

sandboxper task

your code

you

you

you

you

image

Modal

Modal

Modal

you

scaling

Modal

Modal

you

you

container lifetime

Modal

Modal

you

you

hardware

Modal

Modal

Modal

Modal

From left to right, Modal manages less and you decide more. Functions and Servers scale on their own; your code creates each sandbox and decides when it stops.

The same program on every runtime

A Function, a Server and a program Sandbox are each a class that holds its code. The program inside is the same shape, an Effect that returns fetch and any methods you want to call:

Effect.gen(function* () {
  return {
    fetch: Effect.succeed(HttpServerResponse.text("ok")),
    embed: Effect.fn(function* (text: string) {
      return { tokens: text.split(/\s+/).length };
    }),
  };
});

Alchemy bundles main into an image, and Modal builds it at deploy.

Functions: scale with calls

A Modal.Function runs when it’s called. Modal starts containers as calls arrive, runs up to maxConcurrentInputs in each, and stops them when they’re idle. Callers use its methods by name, a schedule calls it on a cron, or web: true serves its fetch at a *.modal.run URL.

src/embed.ts

export default class Embed extends Modal.Function<Embed>()(
  "Embed",
  { main: import.meta.url, app: Inference, gpu: "L4" },
  program,
) {}

Choose it for inference, batch jobs and anything you fan out. See Functions.

callcold start

Each container takes one call at a time here. A second container starts when calls pile up, and both stop once idle.

Servers: scale with concurrent requests

A Modal.Server is an HTTP server behind one routing region. Each container serves many requests at once, and Modal adds a container when they pass targetConcurrency.

src/api.ts

export default class Api extends Modal.Server<Api>()(
  "Api",
  { main: import.meta.url, app: Backend, targetConcurrency: 20 },
  program,
) {}

Choose it for APIs, streaming responses and WebSockets. See Servers.

One container serves requests side by side. Modal scales on how many are in flight, not on a queue of calls.

Program sandboxes: your program, one container per create

A Modal.Sandbox with a main bundles your program into an image. Your code starts one with create, reaches the program’s port through a tunnel, and terminates it, or lets its timeout stop it.

const sandboxes = yield* Modal.CreateSandbox(MySandbox);
const sandbox = yield* sandboxes.create({ timeout: "10 minutes" });
const port = yield* sandbox.getTcpPort();

Choose it when each user or session needs a copy of your own server. See Programs.

Each create starts one container running your program. It runs until your code terminates it or its timeout runs out.

Sandboxes created at run time: one per task

Modal.CreateSandbox bound to an image gives a Worker or Lambda a client that creates sandboxes for code you don’t trust. Your code runs commands in each one, reads and writes its files, and terminates it when the task is done. idleTimeout stops one nobody is using.

const sandboxes = yield* Modal.CreateSandbox(Warm);
const sandbox = yield* sandboxes.create({ idleTimeout: "10 minutes" });
yield* sandbox.exec(["pnpm", "test"], { cwd: "/work" });

Choose it for agent loops, code interpreters and test runs. See Sandboxes.

commandidle, still billed

Each task gets its own sandbox. Its files and processes last between commands, until your code terminates it or the idle timeout does.

A persistent sandbox keeps its files between sessions instead, and resumes where the last one stopped.

Mix runtimes in one app

A Worker creates a sandbox per user and calls a GPU Function, and both read model weights from one Volume:

ApiWorker

MyEnvenvironment

session-samSandbox

EmbedFunction · L4

WeightsVolume

The Worker, Api, runs outside Modal and reaches both runtimes through Modal's API. The sandbox and the Function mount the same Volume in their environment.