Add Azure Event Hubs source to an eventstream

1. On the Connect page, confirm that Basic is selected for the feature level, and then select New connection.

   

   If there's an existing connection to your Azure event hub, you select that existing connection as shown in the following image, and then move on to the step to configure Data format in the following steps.

   

2. In the Connection settings section, do these steps:

   1. Enter the name of the Event Hubs namespace.

   2. Enter the name of the event hub.

      

3. In the Connection credentials section, do these steps:

   1. For Connection name, enter a name for the connection to the event hub.

   2. For Authentication kind, confirm that Shared Access Key is selected.

   3. For Shared Access Key Name, enter the name of the shared access key. For instructions on getting an access key, see Get connection string.

   4. For Shared Access Key, enter the value of the shared access key.

   5. Select Connect at the bottom of the page.

      

4. Now, on the Connect page of wizard, for Consumer group, enter the name of the consumer group. By default, $Default is selected, which is the default consumer group for the event hub.

5. For Data format, select a data format of the incoming real-time events that you want to get from your Azure event hub. You can select from JSON, Avro, and CSV (with header) data formats.

   

6. In the Stream details pane to the right, select Pencil icon next to the source name, and enter a name for the source. This step is optional.

   

7. Select Next at the bottom of the page.

   

8. On the Review + connect page, review settings, and select Add.

   

1. On the Connect page, for Feature level, select Extended features (Preview).

   

   If there's an existing connection to your Azure event hub, you select that existing connection as shown in the following image, and then move on to the step to configure Data format in the following steps.

2. In the Connection settings section, do these steps:

   1. Enter the name of the Event Hubs namespace.

   2. Enter the name of the event hub.

      

3. In the Connection credentials section, do these steps:

   1. For Connection name, enter a name for the connection to the event hub.

   2. For Authentication kind, confirm that Shared Access Key is selected.

   3. For Shared Access Key Name, enter the name of the shared access key. For instructions on getting an access key, see Get connection string.

   4. For Shared Access Key, enter the value of the shared access key.

   5. Select Connect at the bottom of the page.

      

4. Now, on the Connect page of wizard, for Consumer group, enter the name of the consumer group. By default, $Default is selected, which is the default consumer group for the event hub.

   

5. (Optional) To customize the source's name, select Pencil icon next to the source name in the Stream details pane to the right, and enter a name for the source. Then, select Next at the bottom of the page.

   

6. In the Schema handling page, you must provide rules to handle events received from the selected Eventhub, so that Eventstream can apply them correctly. The mapping rules depend on how you model the events.

   If you have one schema that governs all of the events, you choose fixed schema.

   Alternatively, if you have multiple schemas that represent the different incoming events, you can define matching rules to apply your schemas. To choose this mode, select Dynamic schema via header option, and then specify the rules, using header and value pairs, to select each schema. The header is a custom Kafka header property that is part of the event metadata. The value is the expected value for that property.

   

7. For both modes, you can choose the schemas by clicking on Add more schemas drop-down, and choosing one or more existing schemas from the event schema registry. If you don't have schemas to choose from, you can create new schemas from this view. To learn how to define a new event schema, see Create and manage event schemas in schema sets.

   

   If you select Choose from event schema registry option, you see the Associate an event schema page. Select one or more schemas from the registry, depending on your schema matching mode, and then select Choose at the bottom of the page.

   

8. If you selected the Fixed schema option, you don't need to provide any more rules to match the schema. You can continue to the next step. If you selected the Dynamic schema via headers option, specify the Kafka header property and expected value that maps to the schema. Add more schemas and specify different header properties and/or different values to map to those schemas.

   Note

   When you define the mapping rules, each value of the header MUST be unique. If you try to reuse a schema, you see a warning message indicating that you might break existing streams. As long as the mapping rules are the same, you can reuse a schema. If this limitation affects your use, do reach out to your Microsoft representative to share your feedback. We're actively working on removing this limitation.

   

9. Once schemas for all expected events are mapped, select Next at the bottom of the Schema handling page.

   

10. On the Review + connect page, review settings, and select Connect.

    

11. On the Review + connect page, select Add now.

    

View updated eventstream

1. You see that the Event Hubs source is added to your eventstream on the canvas in the Edit mode. Select Refresh in the bottom pane, which shows you preview of the data in the event hub. To implement this newly added Azure event hub, select Publish on the ribbon.

   

2. After you complete these steps, the Azure event hub is available for visualization in the Live view. Select the Event hub tile in the diagram to see the page similar to the following one.

   

View updated eventstream

1. You see that the Event Hubs source is added to your eventstream on the canvas in the Edit mode. Select Refresh in the bottom pane, which shows you preview of the data in the event hub. To implement this newly added Azure event hub, select Publish on the ribbon.

   

2. After you complete these steps, the Azure event hub is available for visualization in the Live view. Select the Event hub tile in the diagram to see the page similar to the following one.

   

Configure eventstream destinations to use schemas

Currently, only the Eventhouse, custom endpoint, and derived stream destinations are supported for eventstreams with extended features (like schema support) enabled. This section shows you how to add and configure an Eventhouse destination when extended features (like schema support) are enabled for the eventstream.

Configure schema for a custom endpoint destination

1. Select Transform events or add destination, and then select Custom endpoint.

2. In the Custom endpoint pane, specify a name for the destination as usual.

3. For Input schema, select the schema for events. This field is the extra field you fill when you enable the schema support for an eventstream.

   

   For detailed steps on configuring a custom endpoint destination, see Add a custom endpoint or custom app destination to an eventstream.

Configure schemas for an eventhouse destination

1. Select Transform events or add destination, and then select Eventhouse.
2. On the Eventhouse page, configure the following schema-related settings:

   1. For Input schema, select one or more schemas from the drop-down list.

      

      Note

      If you selected Dynamic schema via headers option when configuring an Event Hubs source, you might have configured multiple schemas for the source and map them to different properties and their values.

   2. For Table creation method, select one of the options depending on your requirements: A single table with all schemas combined or Separate tables for each schema.

      

   3. For Write data with, select one of the following options:

      • Payload only: To write extracted payload data to the table. If there are multiple input schemas, data is sent to multiple tables.

      • Metadata and payload: Write metadata and payload data to a single table. Example columns: source , subject, type, data, etc.

        

      For detailed steps on configuring an Eventhouse destination, see Add an Eventhouse destination to an eventstream.