Skip to content

GCP.Datastream reference

Source: packages/gcp/src/Datastream/ConnectionProfile.ts

A Datastream connection profile: credentials and connectivity for a source database or a BigQuery / Cloud Storage destination.

connectionProfileId, location, and the engine kind (mysqlProfile / postgresqlProfile / oracleProfile / sqlServerProfile / mongodbProfile / spannerProfile / gcsProfile / bigqueryProfile / salesforceProfile / salesforceMarketingCloudProfile / serviceNowProfile / dataverseProfile) plus Spanner database are replacement triggers. Display name, labels, credentials, and connectivity update in place. Passwords, SSL keys, and OAuth secrets are input-only and never returned.

ConnectionProfile: Creating a Connection Profile

Section titled “ConnectionProfile: Creating a Connection Profile”

BigQuery destination

const dest = yield* GCP.Datastream.ConnectionProfile("BqDest", {
bigqueryProfile: {},
labels: { env: "test" },
});

MySQL source

const source = yield* GCP.Datastream.ConnectionProfile("MysqlSrc", {
mysqlProfile: {
hostname: "10.0.0.8",
port: 3306,
username: "datastream",
password: process.env.MYSQL_PASSWORD,
},
staticServiceIpConnectivity: {},
force: true,
});

Cloud Storage destination

const gcs = yield* GCP.Datastream.ConnectionProfile("GcsDest", {
gcsProfile: { bucket: bucket.bucketName, rootPath: "/data" },
});

ConnectionProfile: Looking up an existing connection profile

Section titled “ConnectionProfile: Looking up an existing connection profile”

ConnectionProfile.lookup references a connection profile Alchemy doesn’t manage (created in the console or by another stack). It is read during plan/deploy, never created, updated or deleted, and binds like a declared connection profile.

const connectionProfile = GCP.Datastream.ConnectionProfile.lookup({
connectionProfileId: "my-connection-profile",
});

Source: packages/gcp/src/Datastream/PrivateConnection.ts

A Datastream private connection: VPC peering or a PSC interface between Datastream and a consumer network.

The API has no patch method. privateConnectionId, location, vpcPeeringConfig, pscInterfaceConfig, display name, and labels are replacement triggers. Delete always force-deletes child routes.

PrivateConnection: Creating a Private Connection

Section titled “PrivateConnection: Creating a Private Connection”
const network = yield* GCP.Compute.Network("DsVpc", {
autoCreateSubnetworks: false,
});
const peering = yield* GCP.Datastream.PrivateConnection("DsPeer", {
displayName: "ds-peer",
vpcPeeringConfig: {
vpc: network.networkName,
subnet: "10.9.0.0/29",
},
labels: { env: "test" },
});

PrivateConnection: Looking up an existing private connection

Section titled “PrivateConnection: Looking up an existing private connection”

PrivateConnection.lookup references a private connection Alchemy doesn’t manage (created in the console or by another stack). It is read during plan/deploy, never created, updated or deleted, and binds like a declared private connection.

const privateConnection = GCP.Datastream.PrivateConnection.lookup({
privateConnectionId: "my-private-connection",
});

Source: packages/gcp/src/Datastream/PrivateConnectionsRoute.ts

A Datastream route attached to a private connection. Routes advertise a destination address (and optional port) through the private connection’s VPC peering or PSC interface.

The API has no patch method. routeId, parent private connection, destination address/port, display name, and labels are replacement triggers.

const peering = yield* GCP.Datastream.PrivateConnection("DsPeer", {
vpcPeeringConfig: {
vpc: network.networkName,
subnet: "10.9.0.0/29",
},
});
const route = yield* GCP.Datastream.PrivateConnectionsRoute("DbHost", {
privateConnection: peering.name,
destinationAddress: "10.0.0.8",
destinationPort: 3306,
labels: { env: "test" },
});

PrivateConnectionsRoute: Looking up an existing private connections route

Section titled “PrivateConnectionsRoute: Looking up an existing private connections route”

PrivateConnectionsRoute.lookup references a private connections route Alchemy doesn’t manage (created in the console or by another stack). It is read during plan/deploy, never created, updated or deleted, and binds like a declared private connections route.

const privateConnectionsRoute = GCP.Datastream.PrivateConnectionsRoute.lookup({
routeId: "my-route",
privateConnection: "my-private-connection",
});

Source: packages/gcp/src/Datastream/Stream.ts

A Datastream stream: continuously replicates data from a source connection profile to a BigQuery or Cloud Storage destination.

streamId, location, customerManagedEncryptionKey, and the source / destination connection profile identities are replacement triggers. Display name, labels, include/exclude objects, backfill strategy, rule sets, and desired state update in place.

const source = yield* GCP.Datastream.ConnectionProfile("MysqlSrc", {
mysqlProfile: {
hostname: "10.0.0.8",
username: "datastream",
password: process.env.MYSQL_PASSWORD,
},
force: true,
});
const dest = yield* GCP.Datastream.ConnectionProfile("BqDest", {
bigqueryProfile: {},
});
const stream = yield* GCP.Datastream.Stream("MysqlToBq", {
sourceConfig: {
sourceConnectionProfile: source.name,
mysqlSourceConfig: {
includeObjects: { mysqlDatabases: [{ database: "app" }] },
},
},
destinationConfig: {
destinationConnectionProfile: dest.name,
bigqueryDestinationConfig: {
singleTargetDataset: { datasetId: "app_replica" },
},
},
backfillAll: {},
force: true,
});

Stream.lookup references a stream Alchemy doesn’t manage (created in the console or by another stack). It is read during plan/deploy, never created, updated or deleted, and binds like a declared stream.

const stream = GCP.Datastream.Stream.lookup({ streamId: "my-stream" });