Share via

data_transfer Package

  • Reference

Classes

DataTransferCopy

Note

This is an experimental class, and may change at any time. Please see https://aka.ms/azuremlexperimental for more information.

Base class for data transfer copy node.

You should not instantiate this class directly. Instead, you should create from builder function: copy_data.
DataTransferCopyComponent

Note

This is an experimental class, and may change at any time. Please see https://aka.ms/azuremlexperimental for more information.

DataTransfer copy component version, used to define a data transfer copy component.
DataTransferExport

Note

This is an experimental class, and may change at any time. Please see https://aka.ms/azuremlexperimental for more information.

Base class for data transfer export node.

You should not instantiate this class directly. Instead, you should create from builder function: export_data.
DataTransferExportComponent

Note

This is an experimental class, and may change at any time. Please see https://aka.ms/azuremlexperimental for more information.

DataTransfer export component version, used to define a data transfer export component.
DataTransferImport

Note

This is an experimental class, and may change at any time. Please see https://aka.ms/azuremlexperimental for more information.

Base class for data transfer import node.

You should not instantiate this class directly. Instead, you should create from builder function: import_data.
DataTransferImportComponent

Note

This is an experimental class, and may change at any time. Please see https://aka.ms/azuremlexperimental for more information.

DataTransfer import component version, used to define a data transfer import component.
Database

Define a database class for a DataTransfer Component or Job.
FileSystem

Define a file system class of a DataTransfer Component or Job.

e.g. source_s3 = FileSystem(path='s3://my_bucket/my_folder', connection='azureml:my_s3_connection')

Functions

copy_data

Note

This is an experimental method, and may change at any time. Please see https://aka.ms/azuremlexperimental for more information.

Create a DataTransferCopy object which can be used inside dsl.pipeline as a function.

copy_data(*, name: str | None = None, description: str | None = None, tags: Dict | None = None, display_name: str | None = None, experiment_name: str | None = None, compute: str | None = None, inputs: Dict | None = None, outputs: Dict | None = None, is_deterministic: bool = True, data_copy_mode: str | None = None, **kwargs: Any) -> DataTransferCopy

Keyword-Only Parameters

Name Description
name
str

The name of the job.

Default value: None
description
str

Description of the job.

Default value: None
tags
dict[str, str]

Tag dictionary. Tags can be added, removed, and updated.

Default value: None
display_name
str

Display name of the job.

Default value: None
experiment_name
str

Name of the experiment the job will be created under.

Default value: None
compute
str

The compute resource the job runs on.

Default value: None
inputs
dict

Mapping of inputs data bindings used in the job.

Default value: None
outputs
dict

Mapping of outputs data bindings used in the job.

Default value: None
is_deterministic
bool

Specify whether the command will return same output given same input. If a command (component) is deterministic, when use it as a node/step in a pipeline, it will reuse results from a previous submitted job in current workspace which has same inputs and settings. In this case, this step will not use any compute resource. Default to be True, specify is_deterministic=False if you would like to avoid such reuse behavior.

Default value: True
data_copy_mode
str

data copy mode in copy task, possible value is "merge_with_overwrite", "fail_if_conflict".

Default value: None

Returns

Type Description
DataTransferCopyComponent

A DataTransferCopy object.

export_data

Note

This is an experimental method, and may change at any time. Please see https://aka.ms/azuremlexperimental for more information.

Create a DataTransferExport object which can be used inside dsl.pipeline.

export_data(*, name: str | None = None, description: str | None = None, tags: Dict | None = None, display_name: str | None = None, experiment_name: str | None = None, compute: str | None = None, sink: Dict | Database | FileSystem | None = None, inputs: Dict | None = None, **kwargs: Any) -> DataTransferExport

Keyword-Only Parameters

Name Description
name
str

The name of the job.

Default value: None
description
str

Description of the job.

Default value: None
tags
dict[str, str]

Tag dictionary. Tags can be added, removed, and updated.

Default value: None
display_name
str

Display name of the job.

Default value: None
experiment_name
str

Name of the experiment the job will be created under.

Default value: None
compute
str

The compute resource the job runs on.

Default value: None
sink
Union[ Dict, Database, FileSystem]

The sink of external data and databases.

Default value: None
inputs
dict

Mapping of inputs data bindings used in the job.

Default value: None

Returns

Type Description
<xref:azure.ai.ml.entities._job.pipeline._component_translatable.DataTransferExport>

A DataTransferExport object.

Exceptions

Type Description
ValidationException

If sink is not provided or exporting file system is not supported.

import_data

Note

This is an experimental method, and may change at any time. Please see https://aka.ms/azuremlexperimental for more information.

Create a DataTransferImport object which can be used inside dsl.pipeline.

import_data(*, name: str | None = None, description: str | None = None, tags: Dict | None = None, display_name: str | None = None, experiment_name: str | None = None, compute: str | None = None, source: Dict | Database | FileSystem | None = None, outputs: Dict | None = None, **kwargs: Any) -> DataTransferImport

Keyword-Only Parameters

Name Description
name
str

The name of the job.

Default value: None
description
str

Description of the job.

Default value: None
tags
dict[str, str]

Tag dictionary. Tags can be added, removed, and updated.

Default value: None
display_name
str

Display name of the job.

Default value: None
experiment_name
str

Name of the experiment the job will be created under.

Default value: None
compute
str

The compute resource the job runs on.

Default value: None
source
Union[Dict, Database, FileSystem]

The data source of file system or database.

Default value: None
outputs
dict

Mapping of outputs data bindings used in the job. The default will be an output port with the key "sink" and type "mltable".

Default value: None

Returns

Type Description
<xref:azure.ai.ml.entities._job.pipeline._component_translatable.DataTransferImport>

A DataTransferImport object.