Modal.Endpoint reference

Endpoint

Source: packages/modal/src/Endpoint.ts

A Modal Endpoint: a managed model server that speaks the OpenAI Chat Completions, OpenAI Responses or Anthropic Messages API, from a catalog model, a Hugging Face repo, or weights on a Modal Volume.

Reconcile waits until the endpoint is live, which takes minutes while the model loads. Modal can’t change an endpoint in place, so any change replaces it.

Caution

A dedicated endpoint reserves GPUs and is billed while it runs. Destroy it (or use servingMode: "shared") when you’re done.

Endpoint: Serving a model

A catalog model behind the OpenAI API

import * as Modal from "@alchemy.run/modal";

export const Qwen = Modal.Endpoint("Qwen", {
  model: { repoId: "Qwen/Qwen3-8B" },
  apis: ["openai-chat-completions", "anthropic-messages"],
});

Fine-tuned weights from a Volume

import * as Modal from "@alchemy.run/modal";

export const Weights = Modal.Volume("Weights", {});
export const Tuned = Modal.Endpoint("Tuned", {
  model: { volume: Weights, path: "/checkpoints/final", baseModel: "Qwen/Qwen3-8B" },
});

Endpoint: Calling it

import * as Cloudflare from "@alchemy.run/cloudflare";
import * as Modal from "@alchemy.run/modal";
import * as Effect from "effect/Effect";
import * as HttpClientRequest from "effect/http/HttpClientRequest";
import * as HttpServerResponse from "effect/http/HttpServerResponse";
import { Qwen } from "./Qwen.ts";

export default class Chat extends Cloudflare.Worker<Chat>()(
  "Chat",
  { main: import.meta.url },
  Effect.gen(function* () {
    // Creates a webhook token for the Worker and sends it as Modal's proxy auth.
    const qwen = yield* Modal.FetchServer(Qwen);
    return {
      fetch: Effect.gen(function* () {
        const response = yield* qwen.fetch(
          HttpClientRequest.post("/v1/chat/completions").pipe(
            HttpClientRequest.bodyJsonUnsafe({
              model: "Qwen/Qwen3-8B",
              messages: [{ role: "user", content: "hi" }],
            }),
          ),
        );
        return HttpServerResponse.json(yield* response.json);
      }).pipe(Effect.orDie),
    };
  }).pipe(Effect.provide(Modal.FetchServerHttp)),
) {}