Cluster

Source: src/AWS/EMR/Cluster.ts

A provisioned Amazon EMR cluster (job flow) running open-source big-data frameworks such as Apache Spark and Hadoop on EC2 instances.

Clusters take roughly 10-15 minutes to reach WAITING and bill per instance-hour while they exist. Each cluster needs an EMR service role and an EC2 instance profile (job-flow role); destroy clusters you are not using, or set autoTerminationPolicy as a safety net.

Creating a Cluster

Spark Cluster in a Default-VPC Subnet

const cluster = yield* Cluster("Analytics", {
  releaseLabel: "emr-7.5.0",
  applications: ["Spark", "Hadoop"],
  serviceRole: serviceRole.roleName,
  jobFlowRole: instanceProfile.instanceProfileName,
  logUri: Output.interpolate`s3://${logsBucket.bucketName}/logs/`,
  instances: {
    masterInstanceType: "m5.xlarge",
    coreInstanceType: "m5.xlarge",
    coreInstanceCount: 1,
    ec2SubnetId: subnetId,
  },
});

Cluster with an Auto-Termination Safety Net

const cluster = yield* Cluster("Batch", {
  releaseLabel: "emr-7.5.0",
  applications: ["Spark"],
  serviceRole: serviceRole.roleName,
  jobFlowRole: instanceProfile.instanceProfileName,
  autoTerminationPolicy: { idleTimeout: "1 hour" },
  stepConcurrencyLevel: 4,
});

Applying a Security Configuration

const config = yield* SecurityConfiguration("Encryption", {
  securityConfiguration: {
    EncryptionConfiguration: {
      EnableInTransitEncryption: false,
      EnableAtRestEncryption: false,
    },
  },
});
const cluster = yield* Cluster("Secure", {
  releaseLabel: "emr-7.5.0",
  serviceRole: serviceRole.roleName,
  jobFlowRole: instanceProfile.instanceProfileName,
  securityConfiguration: config.securityConfigurationName,
});