az datafactory pipeline

Note

This reference is part of the datafactory extension for the Azure CLI (version 2.15.0 or higher). The extension will automatically install the first time you run an az datafactory pipeline command. Learn more about extensions.

Manage pipeline with datafactory.

Commands

Name	Description	Type	Status
az datafactory pipeline create	Create a pipeline.	Extension	GA
az datafactory pipeline create-run	Creates a run of a pipeline.	Extension	GA
az datafactory pipeline delete	Deletes a pipeline.	Extension	GA
az datafactory pipeline list	Lists pipelines.	Extension	GA
az datafactory pipeline show	Gets a pipeline.	Extension	GA
az datafactory pipeline update	Update a pipeline.	Extension	GA

az datafactory pipeline create

Create a pipeline.

az datafactory pipeline create --factory-name
                               --name --pipeline-name
                               --pipeline
                               --resource-group
                               [--if-match]

Examples

Pipelines_Create

az datafactory pipeline create --factory-name "exampleFactoryName" --pipeline "{\"activities\":[{\"name\":\"ExampleForeachActivity\",\"type\":\"ForEach\",\"typeProperties\":{\"activities\":[{\"name\":\"ExampleCopyActivity\",\"type\":\"Copy\",\"inputs\":[{\"type\":\"DatasetReference\",\"parameters\":{\"MyFileName\":\"examplecontainer.csv\",\"MyFolderPath\":\"examplecontainer\"},\"referenceName\":\"exampleDataset\"}],\"outputs\":[{\"type\":\"DatasetReference\",\"parameters\":{\"MyFileName\":{\"type\":\"Expression\",\"value\":\"@item()\"},\"MyFolderPath\":\"examplecontainer\"},\"referenceName\":\"exampleDataset\"}],\"typeProperties\":{\"dataIntegrationUnits\":32,\"sink\":{\"type\":\"BlobSink\"},\"source\":{\"type\":\"BlobSource\"}}}],\"isSequential\":true,\"items\":{\"type\":\"Expression\",\"value\":\"@pipeline().parameters.OutputBlobNameList\"}}}],\"parameters\":{\"JobId\":{\"type\":\"String\"},\"OutputBlobNameList\":{\"type\":\"Array\"}},\"variables\":{\"TestVariableArray\":{\"type\":\"Array\"}},\"runDimensions\":{\"JobId\":{\"type\":\"Expression\",\"value\":\"@pipeline().parameters.JobId\"}},\"duration\":\"0.00:10:00\"}" --name "examplePipeline" --resource-group "exampleResourceGroup"

Required Parameters

--factory-name

The factory name.

--name --pipeline-name -n

The pipeline name.

--pipeline

Pipeline resource definition. Expected value: json-string/json-file/@json-file.

--resource-group -g

Name of resource group. You can configure the default group using az configure --defaults group=<name>.

Optional Parameters

The following parameters are optional, but depending on the context, one or more might become required for the command to execute successfully.

--if-match

ETag of the pipeline entity. Should only be specified for update, for which it should match existing entity or can be * for unconditional update.

Global Parameters

--debug

Increase logging verbosity to show all debug logs.

Property	Value
Default value:	False

--help -h

Show this help message and exit.

--only-show-errors

Only show errors, suppressing warnings.

Property	Value
Default value:	False

--output -o

Output format.

Property	Value
Default value:	json
Accepted values:	json, jsonc, none, table, tsv, yaml, yamlc

--query

JMESPath query string. See http://jmespath.org/ for more information and examples.

--subscription

Name or ID of subscription. You can configure the default subscription using az account set -s NAME_OR_ID.

--verbose

Increase logging verbosity. Use --debug for full debug logs.

Property	Value
Default value:	False

az datafactory pipeline create-run

Creates a run of a pipeline.

az datafactory pipeline create-run --factory-name
                                   --name --pipeline-name
                                   --resource-group
                                   [--is-recovery {false, true}]
                                   [--parameters]
                                   [--reference-pipeline-run-id]
                                   [--start-activity-name]
                                   [--start-from-failure {false, true}]

Examples

Pipelines_CreateRun

az datafactory pipeline create-run --factory-name "exampleFactoryName" --parameters "{\"OutputBlobNameList\":[\"exampleoutput.csv\"]}" --name "examplePipeline" --resource-group "exampleResourceGroup"

Required Parameters

--factory-name

The factory name.

--name --pipeline-name -n

The pipeline name.

--resource-group -g

Name of resource group. You can configure the default group using az configure --defaults group=<name>.

Optional Parameters

The following parameters are optional, but depending on the context, one or more might become required for the command to execute successfully.

--is-recovery

Recovery mode flag. If recovery mode is set to true, the specified referenced pipeline run and the new run will be grouped under the same groupId.

Property	Value
Accepted values:	false, true

--parameters

Parameters of the pipeline run. These parameters will be used only if the runId is not specified. Expected value: json-string/json-file/@json-file.

--reference-pipeline-run-id

The pipeline run identifier. If run ID is specified the parameters of the specified run will be used to create a new run.

--start-activity-name

In recovery mode, the rerun will start from this activity. If not specified, all activities will run.

--start-from-failure

In recovery mode, if set to true, the rerun will start from failed activities. The property will be used only if startActivityName is not specified.

Property	Value
Accepted values:	false, true

Global Parameters

--debug

Increase logging verbosity to show all debug logs.

Property	Value
Default value:	False

--help -h

Show this help message and exit.

--only-show-errors

Only show errors, suppressing warnings.

Property	Value
Default value:	False

--output -o

Output format.

Property	Value
Default value:	json
Accepted values:	json, jsonc, none, table, tsv, yaml, yamlc

--query

JMESPath query string. See http://jmespath.org/ for more information and examples.

--subscription

Name or ID of subscription. You can configure the default subscription using az account set -s NAME_OR_ID.

--verbose

Increase logging verbosity. Use --debug for full debug logs.

Property	Value
Default value:	False

az datafactory pipeline delete

Deletes a pipeline.

az datafactory pipeline delete [--factory-name]
                               [--ids]
                               [--name --pipeline-name]
                               [--resource-group]
                               [--subscription]
                               [--yes]

Examples

Pipelines_Delete

az datafactory pipeline delete --factory-name "exampleFactoryName" --name "examplePipeline" --resource-group "exampleResourceGroup"

Optional Parameters

The following parameters are optional, but depending on the context, one or more might become required for the command to execute successfully.

--factory-name

The factory name.

Property	Value
Parameter group:	Resource Id Arguments

--ids

One or more resource IDs (space-delimited). It should be a complete resource ID containing all information of 'Resource Id' arguments. You should provide either --ids or other 'Resource Id' arguments.

Property	Value
Parameter group:	Resource Id Arguments

--name --pipeline-name -n

The pipeline name.

Property	Value
Parameter group:	Resource Id Arguments

--resource-group -g

Name of resource group. You can configure the default group using az configure --defaults group=<name>.

Property	Value
Parameter group:	Resource Id Arguments

--subscription

Name or ID of subscription. You can configure the default subscription using az account set -s NAME_OR_ID.

Property	Value
Parameter group:	Resource Id Arguments

--yes -y

Do not prompt for confirmation.

Property	Value
Default value:	False

Global Parameters

--debug

Increase logging verbosity to show all debug logs.

Property	Value
Default value:	False

--help -h

Show this help message and exit.

--only-show-errors

Only show errors, suppressing warnings.

Property	Value
Default value:	False

--output -o

Output format.

Property	Value
Default value:	json
Accepted values:	json, jsonc, none, table, tsv, yaml, yamlc

--query

JMESPath query string. See http://jmespath.org/ for more information and examples.

--verbose

Increase logging verbosity. Use --debug for full debug logs.

Property	Value
Default value:	False

az datafactory pipeline list

Lists pipelines.

az datafactory pipeline list --factory-name
                             --resource-group

Examples

Pipelines_ListByFactory

az datafactory pipeline list --factory-name "exampleFactoryName" --resource-group "exampleResourceGroup"

Required Parameters

--factory-name

The factory name.

--resource-group -g

Name of resource group. You can configure the default group using az configure --defaults group=<name>.

Global Parameters

--debug

Increase logging verbosity to show all debug logs.

Property	Value
Default value:	False

--help -h

Show this help message and exit.

--only-show-errors

Only show errors, suppressing warnings.

Property	Value
Default value:	False

--output -o

Output format.

Property	Value
Default value:	json
Accepted values:	json, jsonc, none, table, tsv, yaml, yamlc

--query

JMESPath query string. See http://jmespath.org/ for more information and examples.

--subscription

Name or ID of subscription. You can configure the default subscription using az account set -s NAME_OR_ID.

--verbose

Increase logging verbosity. Use --debug for full debug logs.

Property	Value
Default value:	False

az datafactory pipeline show

Gets a pipeline.

az datafactory pipeline show [--factory-name]
                             [--ids]
                             [--if-none-match]
                             [--name --pipeline-name]
                             [--resource-group]
                             [--subscription]

Examples

Pipelines_Get

az datafactory pipeline show --factory-name "exampleFactoryName" --name "examplePipeline" --resource-group "exampleResourceGroup"

Optional Parameters

The following parameters are optional, but depending on the context, one or more might become required for the command to execute successfully.

--factory-name

The factory name.

Property	Value
Parameter group:	Resource Id Arguments

--ids

Property	Value
Parameter group:	Resource Id Arguments

--if-none-match

ETag of the pipeline entity. Should only be specified for get. If the ETag matches the existing entity tag, or if * was provided, then no content will be returned.

--name --pipeline-name -n

The pipeline name.

Property	Value
Parameter group:	Resource Id Arguments

--resource-group -g

Name of resource group. You can configure the default group using az configure --defaults group=<name>.

Property	Value
Parameter group:	Resource Id Arguments

--subscription

Name or ID of subscription. You can configure the default subscription using az account set -s NAME_OR_ID.

Property	Value
Parameter group:	Resource Id Arguments

Global Parameters

--debug

Increase logging verbosity to show all debug logs.

Property	Value
Default value:	False

--help -h

Show this help message and exit.

--only-show-errors

Only show errors, suppressing warnings.

Property	Value
Default value:	False

--output -o

Output format.

Property	Value
Default value:	json
Accepted values:	json, jsonc, none, table, tsv, yaml, yamlc

--query

JMESPath query string. See http://jmespath.org/ for more information and examples.

--verbose

Increase logging verbosity. Use --debug for full debug logs.

Property	Value
Default value:	False

az datafactory pipeline update

Update a pipeline.

az datafactory pipeline update [--activities]
                               [--add]
                               [--annotations]
                               [--concurrency]
                               [--description]
                               [--duration]
                               [--factory-name]
                               [--folder-name]
                               [--force-string]
                               [--ids]
                               [--if-match]
                               [--if-none-match]
                               [--name --pipeline-name]
                               [--parameters]
                               [--remove]
                               [--resource-group]
                               [--run-dimensions]
                               [--set]
                               [--subscription]
                               [--variables]

Examples

Pipelines_Update

az datafactory pipeline update --factory-name "exampleFactoryName" --description "Example description" --activities "[{\"name\":\"ExampleForeachActivity\",\"type\":\"ForEach\",\"typeProperties\":{\"activities\":[{\"name\":\"ExampleCopyActivity\",\"type\":\"Copy\",\"inputs\":[{\"type\":\"DatasetReference\",\"parameters\":{\"MyFileName\":\"examplecontainer.csv\",\"MyFolderPath\":\"examplecontainer\"},\"referenceName\":\"exampleDataset\"}],\"outputs\":[{\"type\":\"DatasetReference\",\"parameters\":{\"MyFileName\":{\"type\":\"Expression\",\"value\":\"@item()\"},\"MyFolderPath\":\"examplecontainer\"},\"referenceName\":\"exampleDataset\"}],\"typeProperties\":{\"dataIntegrationUnits\":32,\"sink\":{\"type\":\"BlobSink\"},\"source\":{\"type\":\"BlobSource\"}}}],\"isSequential\":true,\"items\":{\"type\":\"Expression\",\"value\":\"@pipeline().parameters.OutputBlobNameList\"}}}]" --parameters "{\"OutputBlobNameList\":{\"type\":\"Array\"}}" --duration "0.00:10:00" --name "examplePipeline" --resource-group "exampleResourceGroup"

Optional Parameters

The following parameters are optional, but depending on the context, one or more might become required for the command to execute successfully.

--activities

List of activities in pipeline. Expected value: json-string/json-file/@json-file.

--add

Add an object to a list of objects by specifying a path and key value pairs. Example: --add property.listProperty <key=value, string or JSON string>.

Property	Value
Parameter group:	Generic Update Arguments
Default value:	[]

--annotations

List of tags that can be used for describing the Pipeline. Expected value: json-string/json-file/@json-file.

--concurrency

The max number of concurrent runs for the pipeline.

--description

The description of the pipeline.

--duration

TimeSpan value, after which an Azure Monitoring Metric is fired. Expected value: json-string/json-file/@json-file.

Property	Value
Parameter group:	Policy Elapsed Time Metric Arguments

--factory-name

The factory name.

Property	Value
Parameter group:	Resource Id Arguments

--folder-name

The name of the folder that this Pipeline is in.

Property	Value
Parameter group:	Folder Arguments

--force-string

When using 'set' or 'add', preserve string literals instead of attempting to convert to JSON.

Property	Value
Parameter group:	Generic Update Arguments
Default value:	False

--ids

Property	Value
Parameter group:	Resource Id Arguments

--if-match

ETag of the pipeline entity. Should only be specified for update, for which it should match existing entity or can be * for unconditional update.

--if-none-match

ETag of the pipeline entity. Should only be specified for get. If the ETag matches the existing entity tag, or if * was provided, then no content will be returned. Default value is None.

--name --pipeline-name -n

The pipeline name.

Property	Value
Parameter group:	Resource Id Arguments

--parameters

List of parameters for pipeline. Expected value: json-string/json-file/@json-file.

--remove

Remove a property or an element from a list. Example: --remove property.list <indexToRemove> OR --remove propertyToRemove.

Property	Value
Parameter group:	Generic Update Arguments
Default value:	[]

--resource-group -g

Name of resource group. You can configure the default group using az configure --defaults group=<name>.

Property	Value
Parameter group:	Resource Id Arguments

--run-dimensions

Dimensions emitted by Pipeline. Expected value: json-string/json-file/@json-file.

--set

Update an object by specifying a property path and value to set. Example: --set property1.property2=<value>.

Property	Value
Parameter group:	Generic Update Arguments
Default value:	[]

--subscription

Name or ID of subscription. You can configure the default subscription using az account set -s NAME_OR_ID.

Property	Value
Parameter group:	Resource Id Arguments

--variables

List of variables for pipeline. Expected value: json-string/json-file/@json-file.

Global Parameters

--debug

Increase logging verbosity to show all debug logs.

Property	Value
Default value:	False

--help -h

Show this help message and exit.

--only-show-errors

Only show errors, suppressing warnings.

Property	Value
Default value:	False

--output -o

Output format.

Property	Value
Default value:	json
Accepted values:	json, jsonc, none, table, tsv, yaml, yamlc

--query

JMESPath query string. See http://jmespath.org/ for more information and examples.

--verbose

Increase logging verbosity. Use --debug for full debug logs.

Property	Value
Default value:	False

Share via

az datafactory pipeline

Commands

az datafactory pipeline create

Examples

Required Parameters

Optional Parameters

az datafactory pipeline create-run

Examples

Required Parameters

Optional Parameters

az datafactory pipeline delete

Examples

Optional Parameters

az datafactory pipeline list

Examples

Required Parameters

az datafactory pipeline show

Examples

Optional Parameters

az datafactory pipeline update

Examples

Optional Parameters

Feedback