Pipeline

Source: src/AWS/OSIS/Pipeline.ts

An Amazon OpenSearch Ingestion (OSIS) pipeline — a managed Data Prepper pipeline that ingests, transforms, and delivers data to OpenSearch domains, serverless collections, or S3.

Pipelines take roughly 5-10 minutes to provision and are billed per Ingestion-OCU-hour while they exist (minimum 1 OCU). Destroy pipelines you are not using.

Creating a Pipeline

HTTP Source to S3 Sink

const pipeline = yield* Pipeline("Logs", {
  minUnits: 1,
  maxUnits: 1,
  pipelineConfigurationBody: Output.interpolate`version: "2"
log-pipeline:
  source:
    http:
      path: "/logs"
  sink:
    - s3:
        aws:
          sts_role_arn: "${role.roleArn}"
          region: "us-west-2"
        bucket: "${bucket.bucketName}"
        threshold:
          event_collect_timeout: "60s"
        codec:
          ndjson:
`,
});

Pipeline with CloudWatch Logging

const pipeline = yield* Pipeline("Logs", {
  minUnits: 1,
  maxUnits: 2,
  pipelineConfigurationBody: configYaml,
  logPublishingOptions: {
    isLoggingEnabled: true,
    cloudWatchLogDestination: {
      logGroup: "/aws/vendedlogs/OpenSearchIngestion/logs",
    },
  },
});