Edit

Share via


Add MongoDB CDC source to an eventstream (preview)

This article shows you how to add a MongoDB Change Data Capture(CDC) source to an eventstream.

The MongoDB CDC Connector for Eventstream allows you to stream Change Data Capture (CDC) events from MongoDB into Fabric Eventstream. It supports multiple MongoDB deployment types, including on-premises, cloud-hosted, and MongoDB Atlas, enabling a wide range of CDC scenarios. With this connector, you can capture real-time database changes and stream them directly into Eventstream for immediate processing and analytics.

Note

This source is not supported in the following regions of your workspace capacity: West US3, Switzerland West.

Prerequisites

  • A workspace in Fabric capacity or Trial license mode, with Contributor or higher permissions.
  • A MongoDB cluster that is accessible from your client IP address.
  • Change Data Capture (CDC) enabled for the collections you want to capture.
  • An eventstream in Fabric. If you don’t have one, create an eventstream.

Set up a MongoDB instance

This example uses MongoDB Atlas, the managed MongoDB service on MongoDB Cloud.

To capture changes, you must enable Change Data Capture (CDC) for the target collections.

Run the following command in the MongoDB shell to enable CDC for a collection:

db.runCommand({
  collMod: "<collectionName>",
  changeStreamPreAndPostImages: { enabled: true }
});

Note

You need a user with the atlasAdmin role, which includes the collMod action, to run this command. If you don’t have these permissions, ask a colleague with the atlasAdmin role to enable Change Data Capture (CDC) for the target collections before capturing changes in your eventstream.

You must also create or use an existing MongoDB user with the read role (or higher) on the target database. In MongoDB Atlas on MongoDB Cloud, go to Database Access to verify the user’s role.

Screenshot that shows how to edit role for MongoDB database.

In MongoDB Atlas on MongoDB Cloud, go to Network Access and add your client IP address to the IP Access List.

Screenshot that shows how to add ID list.

Add MongoDB (CDC) as a source

If you haven't added any source to your eventstream yet, select Use external source tile.

Screenshot that shows the selection of Use external source tile.

If you're adding the source to an already published eventstream, switch to Edit mode, select Add source on the ribbon, and then select External sources.

Screenshot that shows the selection of Add source to External sources menu.

On the Select a data source page, search for and select Connect on the MongoDB (CDC) tile.

Screenshot that shows the selection of MongoDB (CDC) as the source type in the Get events wizard.

Configure and connect to MongoDB (CDC)

  1. Select MongoDB instance type: either MongoDB Atlas or MongoDB (Self-managed).

  2. Connection:

    1. If you choose MongoDB Atlas:

      1. On the Connect screen, under Connection, select New connection to create a cloud connection.

        Screenshot that shows the Connect page.

      2. Enter the following Connection settings and Connection credentials:

      • Server: The connection string for your Atlas cluster, for example cluster0.example.mongodb.net.

      • Cluster: This field is optional. The cluster name is already part of the Server, so you can leave this blank.

      • Connection name: Automatically generated, or you can enter a new name for this connection.

      • Username and Password: Credentials for a user with at least the read role. Make sure the user has access to the target database and collections.

        A screenshot of the connection settings for MongoDB Atlas.

    2. If you choose MongoDB (Self-managed):

      1. On the Connect screen, under Connection, select New connection to create a cloud connection.

        Screenshot that shows the Connect page.

      2. Enter the following Connection settings and Connection credentials:

      • Server: The connection string for your self-managed MongoDB server, for example mongodb0.example.com:27017.

      • Connection name: Automatically generated, or you can enter a new name for this connection.

      • Username and Password: Use a user with read or higher permissions to access the target database and collections.

        A screenshot of the connection settings for MongoDB (Self-managed).

  3. Enter the following information to configure the MongoDB CDC data source, and then select Next.

    • Databases: Choose All (Default) or Enter database name(s). If you choose the latter, provide an optional comma-separated list of regular expressions that match the database names to monitor.
    • Collections: Select All (Default) or Enter collection name(s). If you choose the latter, provide a comma-separated list of regular expressions that match fully qualified namespaces (for example dbName.collectionName) of the MongoDB collections to monitor.
  4. You may expand Advanced settings to access more configuration options for the MongoDB CDC source:

    • Snapshot mode: Options are:
      • initial (default): Specifies that the connector reads a snapshot when either no offset is found or if the oplog/change stream no longer contains the previous offset.
      • initial_only: The connector performs a database snapshot. After the snapshot completes, the connector stops, and doesn't stream event records for subsequent database changes.
      • no_data: The connector captures the structure of all relevant tables, but it doesn't create READ events to represent the data set at the point of the connector’s start-up.

    You can also edit source name by selecting the Pencil button for Source name in the Stream details section to the right.

    A screenshot of selecting Tables, Server ID, and Port for the MongoDB (CDC) connection.

  5. On the Review + connect page, after reviewing the summary for MongoDB CDC source, select Add to complete the configuration.

    Screenshot that shows the Review + connect page with the Add button selected.

View updated eventstream

You see the MongoDB (CDC) source added to your eventstream in Edit mode.

A screenshot of the added MongoDB CDC source in Edit mode with the Publish button highlighted.

You see the eventstream in Live mode. Select Edit on the ribbon to get back to the Edit mode to update the eventstream.

A screenshot of the added MongoDB CDC source in Live mode.

Limitation

  • The MongoDB CDC source currently does not support CI/CD features, including Git Integration and Deployment Pipeline. Attempting to export or import an Eventstream item with this source to a Git repository may result in errors.

Other connectors: