GCP.DataLineage reference

Process

Source: src/GCP/DataLineage/Process.ts

A Data Lineage process — the definition of a data transformation.

Processes have no labels field, so Alchemy stamps ownership into attributes for list / nuke. processId and location are identity — changing either replaces the process. Display name, origin, and attributes update in place.

Process: Creating a Process

Generated name

const process = yield* GCP.DataLineage.Process("Etl", {
  displayName: "nightly etl",
  attributes: { env: "dev" },
});

Named process with a custom origin

const process = yield* GCP.DataLineage.Process("Etl", {
  processId: "orders-etl",
  location: "us-central1",
  displayName: "orders etl",
  origin: { sourceType: "CUSTOM", name: "dbt" },
  attributes: { env: "prod" },
});

ProcessesRun

Source: src/GCP/DataLineage/ProcessesRun.ts

A Data Lineage run — one execution of a process that produces lineage events.

Runs have no labels field, so Alchemy stamps ownership into attributes for list / nuke. Parent process, runId, and location are identity. State, timestamps, display name, and attributes update in place.

ProcessesRun: Creating a Run

Started run under a process

const run = yield* GCP.DataLineage.ProcessesRun("Nightly", {
  process: process.name,
  startTime: "2024-01-01T00:00:00Z",
  state: "STARTED",
  attributes: { env: "dev" },
});

Completed run

const run = yield* GCP.DataLineage.ProcessesRun("Nightly", {
  process: process.name,
  startTime: "2024-01-01T00:00:00Z",
  endTime: "2024-01-01T01:00:00Z",
  state: "COMPLETED",
  displayName: "nightly 2024-01-01",
});

ProcessesRunsLineageEvent

Source: src/GCP/DataLineage/ProcessesRunsLineageEvent.ts

A Data Lineage event — one operation that moved data from source assets to target assets.

Events are immutable after create (no patch API). Identity is the parent run plus lineageEventId. Changing links or timestamps replaces the event. Events have no labels, so list / nuke walks alchemy-owned parent processes.

ProcessesRunsLineageEvent: Creating a Lineage Event

const event = yield* GCP.DataLineage.ProcessesRunsLineageEvent("Load", {
  run: run.name,
  startTime: "2024-01-01T00:00:00Z",
  links: [
    {
      source: { fullyQualifiedName: "custom:proj.raw.orders" },
      target: { fullyQualifiedName: "custom:proj.dw.orders" },
    },
  ],
});