Note
Access to this page requires authorization. You can try signing in or changing directories.
Access to this page requires authorization. You can try changing directories.
This article shows you how to add a MongoDB Change Data Capture(CDC) source to an eventstream.
The MongoDB CDC Connector for Eventstream allows you to stream Change Data Capture (CDC) events from MongoDB into Fabric Eventstream. It supports multiple MongoDB deployment types, including on-premises, cloud-hosted, and MongoDB Atlas, enabling a wide range of CDC scenarios. With this connector, you can capture real-time database changes and stream them directly into Eventstream for immediate processing and analytics.
Note
This source is not supported in the following regions of your workspace capacity: West US3, Switzerland West.
Prerequisites
- A workspace in Fabric capacity or Trial license mode, with Contributor or higher permissions.
- A MongoDB cluster that is accessible from your client IP address.
- Change Data Capture (CDC) enabled for the collections you want to capture.
- An eventstream in Fabric. If you don’t have one, create an eventstream.
Set up a MongoDB instance
This example uses MongoDB Atlas, the managed MongoDB service on MongoDB Cloud.
To capture changes, you must enable Change Data Capture (CDC) for the target collections.
Run the following command in the MongoDB shell to enable CDC for a collection:
db.runCommand({
collMod: "<collectionName>",
changeStreamPreAndPostImages: { enabled: true }
});
Note
You need a user with the atlasAdmin
role, which includes the collMod action, to run this command. If you don’t have these permissions, ask a colleague with the atlasAdmin
role to enable Change Data Capture (CDC) for the target collections before capturing changes in your eventstream.
You must also create or use an existing MongoDB user with the read
role (or higher) on the target database. In MongoDB Atlas on MongoDB Cloud, go to Database Access to verify the user’s role.
In MongoDB Atlas on MongoDB Cloud, go to Network Access and add your client IP address to the IP Access List.
Add MongoDB (CDC) as a source
If you haven't added any source to your eventstream yet, select Use external source tile.
If you're adding the source to an already published eventstream, switch to Edit mode, select Add source on the ribbon, and then select External sources.
On the Select a data source page, search for and select Connect on the MongoDB (CDC) tile.
Configure and connect to MongoDB (CDC)
Select MongoDB instance type: either MongoDB Atlas or MongoDB (Self-managed).
Connection:
If you choose MongoDB Atlas:
On the Connect screen, under Connection, select New connection to create a cloud connection.
Enter the following Connection settings and Connection credentials:
Server: The connection string for your Atlas cluster, for example
cluster0.example.mongodb.net
.Cluster: This field is optional. The cluster name is already part of the Server, so you can leave this blank.
Connection name: Automatically generated, or you can enter a new name for this connection.
Username and Password: Credentials for a user with at least the
read
role. Make sure the user has access to the target database and collections.
If you choose MongoDB (Self-managed):
On the Connect screen, under Connection, select New connection to create a cloud connection.
Enter the following Connection settings and Connection credentials:
Server: The connection string for your self-managed MongoDB server, for example
mongodb0.example.com:27017
.Connection name: Automatically generated, or you can enter a new name for this connection.
Username and Password: Use a user with read or higher permissions to access the target database and collections.
Enter the following information to configure the MongoDB CDC data source, and then select Next.
- Databases: Choose All (Default) or Enter database name(s). If you choose the latter, provide an optional comma-separated list of regular expressions that match the database names to monitor.
- Collections: Select All (Default) or Enter collection name(s). If you choose the latter, provide a comma-separated list of regular expressions that match fully qualified namespaces (for example
dbName.collectionName
) of the MongoDB collections to monitor.
You may expand Advanced settings to access more configuration options for the MongoDB CDC source:
- Snapshot mode: Options are:
- initial (default): Specifies that the connector reads a snapshot when either no offset is found or if the oplog/change stream no longer contains the previous offset.
- initial_only: The connector performs a database snapshot. After the snapshot completes, the connector stops, and doesn't stream event records for subsequent database changes.
- no_data: The connector captures the structure of all relevant tables, but it doesn't create READ events to represent the data set at the point of the connector’s start-up.
You can also edit source name by selecting the Pencil button for Source name in the Stream details section to the right.
- Snapshot mode: Options are:
On the Review + connect page, after reviewing the summary for MongoDB CDC source, select Add to complete the configuration.
View updated eventstream
You see the MongoDB (CDC) source added to your eventstream in Edit mode.
You see the eventstream in Live mode. Select Edit on the ribbon to get back to the Edit mode to update the eventstream.
Limitation
- The MongoDB CDC source currently does not support CI/CD features, including Git Integration and Deployment Pipeline. Attempting to export or import an Eventstream item with this source to a Git repository may result in errors.
Related content
Other connectors: