GCP.DataPipelines reference

Pipeline

Source: src/GCP/DataPipelines/Pipeline.ts

A Data Pipelines pipeline that launches recurring Dataflow jobs.

Pipelines have no labels field, so Alchemy stamps ownership into pipelineSources for list / nuke. Pipeline id, location, and type are identity — changing any of them replaces the pipeline. Display name, workload, schedule, and scheduler service account update in place. Archiving uses pipelines.stop and is permanent. Creating a pipeline requires the Data Pipelines API (datapipelines.googleapis.com).

Pipeline: Creating a Pipeline

Generated name

const pipeline = yield* GCP.DataPipelines.Pipeline("Batch", {
  type: "PIPELINE_TYPE_BATCH",
});

Classic Word Count template on a yearly cron

const pipeline = yield* GCP.DataPipelines.Pipeline("WordCount", {
  pipelineId: "word-count",
  type: "PIPELINE_TYPE_BATCH",
  displayName: "word-count",
  scheduleInfo: { schedule: "0 0 1 1 *", timeZone: "UTC" },
  workload: {
    dataflowLaunchTemplateRequest: {
      projectId: "my-project",
      location: "us-central1",
      gcsPath: "gs://dataflow-templates/latest/Word_Count",
      launchParameters: {
        jobName: "word-count",
        parameters: {
          inputFile: "gs://dataflow-samples/shakespeare/kinglear.txt",
          output: "gs://my-bucket/out",
        },
        environment: { tempLocation: "gs://my-bucket/tmp" },
      },
    },
  },
});

Pipeline: Updating a Pipeline

Change props on the same logical id; the engine keeps the physical id.

const pipeline = yield* GCP.DataPipelines.Pipeline("WordCount", {
  type: "PIPELINE_TYPE_BATCH",
  displayName: "word-count-v2",
  scheduleInfo: { schedule: "0 0 1 2 *", timeZone: "UTC" },
});

RunPipeline

Source: src/GCP/DataPipelines/RunPipeline.ts

Runtime binding for Data Pipelines pipelines.run.

Creates a job for the bound pipeline immediately. Use this when the internal scheduler is not configured. Bind in a Function/Action init phase and provide RunPipelineHttp.

RunPipeline: Running Pipelines

const runPipeline = yield* GCP.DataPipelines.RunPipeline(batch);
const response = yield* runPipeline();

RunPipelineHttp

Source: src/GCP/DataPipelines/RunPipelineHttp.ts Kind: Layer · Provides: GCP.DataPipelines.RunPipeline

HTTP implementation of RunPipeline.

StopPipeline

Source: src/GCP/DataPipelines/StopPipeline.ts

Runtime binding for Data Pipelines pipelines.stop.

Grants roles/datapipelines.admin on the project because datapipelines.pipelines.stop is only in that role and Data Pipelines has no per-resource IAM.

Freezes pipeline execution permanently (state becomes STATE_ARCHIVED) and deletes any attached scheduler job. Bind in a Function/Action init phase and provide StopPipelineHttp.

StopPipeline: Stopping Pipelines

const stopPipeline = yield* GCP.DataPipelines.StopPipeline(batch);
const archived = yield* stopPipeline();

StopPipelineHttp

Source: src/GCP/DataPipelines/StopPipelineHttp.ts Kind: Layer · Provides: GCP.DataPipelines.StopPipeline

HTTP implementation of StopPipeline.