Job

Source: src/AWS/Glue/Job.ts

An AWS Glue job — a Spark (glueetl), Python shell (pythonshell), or streaming ETL job definition (script in S3 + IAM role + arguments). The definition lifecycle is instant and free; job runs are billed and are started via startJobRun.

Creating Jobs

Python Shell Job

import * as AWS from "alchemy/AWS";

const job = yield* AWS.Glue.Job("Etl", {
  role: jobRole.roleArn,
  command: {
    name: "pythonshell",
    pythonVersion: "3.9",
    scriptLocation: "s3://my-bucket/scripts/etl.py",
  },
  maxCapacity: 0.0625,
  glueVersion: "3.0",
  defaultArguments: { "--job-language": "python" },
});

Spark ETL Job

const job = yield* AWS.Glue.Job("SparkEtl", {
  role: jobRole.roleArn,
  command: {
    name: "glueetl",
    scriptLocation: "s3://my-bucket/scripts/spark.py",
  },
  glueVersion: "4.0",
  workerType: "G.1X",
  numberOfWorkers: 2,
  timeout: "1 hour",
});

Running Jobs

// init
const startJobRun = yield* AWS.Glue.StartJobRun(job);

// runtime
const { JobRunId } = yield* startJobRun({});