Crawler

Source: src/AWS/Glue/Crawler.ts

An AWS Glue crawler — connects to an S3 (or JDBC/DynamoDB/catalog) data store, infers schemas, and populates the Glue Data Catalog with tables. Runs are asynchronous: create the crawler, then invoke startCrawler (or attach a schedule).

Creating Crawlers

S3 Crawler

import * as AWS from "alchemy/AWS";

const database = yield* AWS.Glue.Database("Analytics", {
  databaseName: "analytics",
});

const crawler = yield* AWS.Glue.Crawler("EventsCrawler", {
  role: crawlerRole.roleArn,
  databaseName: database.databaseName,
  targets: {
    s3Targets: [{ path: "s3://my-data-lake/events/" }],
  },
});

Scheduled Crawler with Schema Policy

const crawler = yield* AWS.Glue.Crawler("EventsCrawler", {
  role: crawlerRole.roleArn,
  databaseName: database.databaseName,
  targets: { s3Targets: [{ path: "s3://my-data-lake/events/" }] },
  schedule: "cron(0 12 * * ? *)",
  tablePrefix: "raw_",
  schemaChangePolicy: {
    updateBehavior: "UPDATE_IN_DATABASE",
    deleteBehavior: "DEPRECATE_IN_DATABASE",
  },
});