GCP.Datastream reference

ConnectionProfile

Source: src/GCP/Datastream/ConnectionProfile.ts

A Datastream connection profile: credentials and connectivity for a source database or a BigQuery / Cloud Storage destination.

connectionProfileId, location, and the engine kind (mysqlProfile / postgresqlProfile / oracleProfile / sqlServerProfile / mongodbProfile / spannerProfile / gcsProfile / bigqueryProfile / salesforceProfile / salesforceMarketingCloudProfile / serviceNowProfile / dataverseProfile) plus Spanner database are replacement triggers. Display name, labels, credentials, and connectivity update in place. Passwords, SSL keys, and OAuth secrets are input-only and never returned.

ConnectionProfile: Creating a Connection Profile

BigQuery destination

const dest = yield* GCP.Datastream.ConnectionProfile("BqDest", {
  bigqueryProfile: {},
  labels: { env: "test" },
});

MySQL source

const source = yield* GCP.Datastream.ConnectionProfile("MysqlSrc", {
  mysqlProfile: {
    hostname: "10.0.0.8",
    port: 3306,
    username: "datastream",
    password: process.env.MYSQL_PASSWORD,
  },
  staticServiceIpConnectivity: {},
  force: true,
});

Cloud Storage destination

const gcs = yield* GCP.Datastream.ConnectionProfile("GcsDest", {
  gcsProfile: { bucket: bucket.bucketName, rootPath: "/data" },
});

PrivateConnection

Source: src/GCP/Datastream/PrivateConnection.ts

A Datastream private connection: VPC peering or a PSC interface between Datastream and a consumer network.

The API has no patch method. privateConnectionId, location, vpcPeeringConfig, pscInterfaceConfig, display name, and labels are replacement triggers. Delete always force-deletes child routes.

PrivateConnection: Creating a Private Connection

const network = yield* GCP.Compute.Network("DsVpc", {
  autoCreateSubnetworks: false,
});
const peering = yield* GCP.Datastream.PrivateConnection("DsPeer", {
  displayName: "ds-peer",
  vpcPeeringConfig: {
    vpc: network.networkName,
    subnet: "10.9.0.0/29",
  },
  labels: { env: "test" },
});

PrivateConnectionsRoute

Source: src/GCP/Datastream/PrivateConnectionsRoute.ts

A Datastream route attached to a private connection. Routes advertise a destination address (and optional port) through the private connection’s VPC peering or PSC interface.

The API has no patch method. routeId, parent private connection, destination address/port, display name, and labels are replacement triggers.

PrivateConnectionsRoute: Creating a Route

const peering = yield* GCP.Datastream.PrivateConnection("DsPeer", {
  vpcPeeringConfig: {
    vpc: network.networkName,
    subnet: "10.9.0.0/29",
  },
});
const route = yield* GCP.Datastream.PrivateConnectionsRoute("DbHost", {
  privateConnection: peering.name,
  destinationAddress: "10.0.0.8",
  destinationPort: 3306,
  labels: { env: "test" },
});

Stream

Source: src/GCP/Datastream/Stream.ts

A Datastream stream: continuously replicates data from a source connection profile to a BigQuery or Cloud Storage destination.

streamId, location, customerManagedEncryptionKey, and the source / destination connection profile identities are replacement triggers. Display name, labels, include/exclude objects, backfill strategy, rule sets, and desired state update in place.

Stream: Creating a Stream

const source = yield* GCP.Datastream.ConnectionProfile("MysqlSrc", {
  mysqlProfile: {
    hostname: "10.0.0.8",
    username: "datastream",
    password: process.env.MYSQL_PASSWORD,
  },
  force: true,
});
const dest = yield* GCP.Datastream.ConnectionProfile("BqDest", {
  bigqueryProfile: {},
});
const stream = yield* GCP.Datastream.Stream("MysqlToBq", {
  sourceConfig: {
    sourceConnectionProfile: source.name,
    mysqlSourceConfig: {
      includeObjects: { mysqlDatabases: [{ database: "app" }] },
    },
  },
  destinationConfig: {
    destinationConnectionProfile: dest.name,
    bigqueryDestinationConfig: {
      singleTargetDataset: { datasetId: "app_replica" },
    },
  },
  backfillAll: {},
  force: true,
});