Dataset

Source: src/AWS/DataBrew/Dataset.ts

An AWS Glue DataBrew dataset — a pointer to source data (S3 file/prefix, Glue Data Catalog table, or JDBC query) plus parsing options. The dataset definition itself stores no data and is free; it is consumed by DataBrew projects and jobs.

Creating Datasets

CSV Dataset from S3

import * as AWS from "alchemy/AWS";

const dataset = yield* AWS.DataBrew.Dataset("Sales", {
  format: "CSV",
  formatOptions: { csv: { delimiter: ",", headerRow: true } },
  input: {
    s3InputDefinition: {
      bucket: bucket.bucketName,
      key: "raw/sales.csv",
    },
  },
});

JSON Dataset

const dataset = yield* AWS.DataBrew.Dataset("Events", {
  format: "JSON",
  formatOptions: { json: { multiLine: false } },
  input: {
    s3InputDefinition: { bucket: bucket.bucketName, key: "events/" },
  },
});

Glue Data Catalog

const dataset = yield* AWS.DataBrew.Dataset("Curated", {
  input: {
    dataCatalogInputDefinition: {
      databaseName: glueDatabase.databaseName,
      tableName: "curated_sales",
    },
  },
});