Automate web and desktop apps with computer use (preview)

[This article is prerelease documentation and is subject to change.]

Computer use is a tool in Copilot Studio that lets your agent interact with any system that has a graphical user interface. It works with websites and desktop apps by selecting buttons, choosing menus, and entering text into fields on the screen. Describe in natural language what you want computer use to do, and it performs the task on a computer you set up using a virtual mouse and keyboard. Agents complete tasks even when there isn't an API to connect to the system directly. If a person can use an app or website, computer use can too. Use it for tasks like automated data entry, invoice processing, and data extraction.

Computer use is powered by Computer-Using Agents (CUA), an AI model that combines vision capabilities with advanced reasoning to interact with graphical user interfaces (GUIs). Because it's AI-powered, it adapts to interface changes. For example, when buttons or screens change, the tool continues working without breaking your flow. It's simple to use. Just describe what you want in natural language. You don't need to write code.

Watch this video to learn how the computer use tool lets an agent interact with a web application.

Requirements

• Computer use is a preview feature available to a limited number of customers to explore early capabilities with known limitations.

• Your Microsoft Entra tenant must be located in the United States.

• Purchase at least 500,000 Copilot Studio messages.

• The feature rolls out in May and June, starting with US-based early release cycle environments and then expanding to all US environments.

• Make sure your agent has the generative orchestrator enabled to use computer use.

• Register a machine in Power Automate and enable it for computer use. See the machine registration section for details.

Machine registration

To interact with websites and desktop apps for task automation, computer use runs on a Windows machine. You register and manage machines in Power Automate.

Here are the key requirements to use a machine for computer use:

• Install Power Automate for desktop version 2.56 or later.

• After registering the machine, turn on computer use in the machine settings.

Learn more in prerequisites and limitations.

Register a new machine

Because computer use runs on the machine, dedicate a machine to computer use to avoid interruptions.

To register a new machine:

1. Install the latest version of Power Automate on your machine. During installation, make sure you check the Install the machine-runtime app to connect to the Power Automate cloud portal option.

2. Open the Power Automate machine runtime app and sign in.

3. Register the machine to the environment you want to use with computer use.

See register a new machine for detailed instructions.

Enable computer use

After you register your machine, enable it for computer use.

1. Sign in to Power Automate.
2. Go to Machines.
3. Select the machine you registered.
4. On the machine detail page, select Settings.
5. Turn on Enable for computer use.
6. Select Save.

Add computer use to your agent

After you register a machine in Power Automate and enable it for computer use, add the computer use tool to your agent.

1. Go to the Tools section in your agent and select Add tool.

2. In the Add tool dialog, select New tool.

3. Select Computer use.

   If you don’t have a machine registered for computer use, a dialog appears that tells you to register a new machine.

4. Select an existing connection or create a new connection. If you create a new connection, select:

   • Machine: The machine you want to use for computer use. If you don’t see the machine, see the Register a new machine section.

   • Domain and username: Enter the username to connect to the machine.

   • Password: Enter the password to connect to the machine.

   Note

   Ensure the account you use has the necessary permissions to access and automate tasks on the selected machine.

5. After you set up the connection, select Add and configure.

6. On the configuration page, configure these three fields:

   • Name: Enter the display name for the computer use tool. This name helps you differentiate it from other tools you add to your agent.

   • Description: Enter a short description of what this tool does and when to use it. This text lets your agent know when to use this tool.

   • Instructions: List the steps the tool should perform, including URLs and application names. For tips, see Best practices for instructions for computer use.

     Tip

     Looking for ideas of what you can do with computer use? Check these examples.

   Review these other fields and settings on the configuration page that might be relevant:

   • Inputs: Use Inputs to define dynamic values that change each time computer use runs. For example, if you want to fill out a form with a different value on every run, create an input for that field. At execution time, computer use combines your instructions with the input values to complete the task.

   • Machine: This value is the name of the machine where computer use runs. Select Manage machine to view more machine settings.

   • Connection: This value is the connection for this tool. Update the connection to change the credentials and machine for computer use.

   • Authentication: Specify how computer use authenticates during execution:

     • Copilot author authentication (default): This option uses the maker’s credentials and is suitable for autonomous agents.

       Warning

       If you share an agent with this setting, anyone using it can act with the original author’s access on the configured machine.

     • User authentication: This option uses the credentials of the person interacting with the agent. Each user must have access credentials to the machine.

7. Select Save.

Test computer use

Test computer use as a key step in the authoring journey. After you enter a name, description, and instructions, and save the tool, select Test to start the test experience.

After a short loading period, the test experience appears:

• The left panel shows your instructions and a step-by-step log of the tool’s reasoning and actions.
• The right panel shows a preview of the actions on the machine you set up for computer use.

When the task finishes, you see a "Test completed" message. While the test is in progress, you can select Stop testing to immediately stop all actions on the machine.

If the result isn't what you expect, go to the configuration page and refine your instructions. Add more detail to improve accuracy. For guidance, see best practices for writing effective instructions.

Publish an agent with computer use

Set up computer use, then publish your agent. How your agent runs depends on the scenario. It can be autonomous or conversational:

• Autonomous agents run automatically and perform tasks in the background.

• Conversational agents let users interact through channels like Microsoft Teams.

Computer use works best for autonomous agents, performing tasks in the background without user interaction.

Computer use can also be applied in conversational experiences, but keep these considerations in mind:

1. If you select User authentication as the authentication setting, each user interacting with the agent in a conversation needs valid credentials for the machine used by computer use.

2. When the tool runs, it shares reasoning messages and screenshots of the machine's activity in the chat.

Monitor computer use runs

To check what computer use did during a run:

1. Go to the Activity section of your agent and select a run.

2. On the run details page, switch between the Activity map and Transcript views.

The Transcript view shows how computer use follows your instructions, providing a step-by-step log with reasoning messages and screenshots for each action.

Best practices

To stay productive and safe in today’s digital environments, follow best practices for computer use, especially when writing clear instructions and securing machines.

Best practices for securing machines

When setting up machines for computer use that lets AI perform tasks using natural language, consider these security recommendations:

Best practices for instructions for computer use

The instructions you write determine how well computer use works. Specific, detailed instructions help computer use complete tasks accurately. Think of it as explaining a task to a colleague. Clear, step-by-step guidance helps ensure success.

Tips for writing effective instructions:

• Be specific about websites and applications. Always include the full URL of any website and the exact name of any application the tool should use. Example: "Open https://www.microsoft.com and go to 'Company news'."

• Clearly state relevant actions. If you want something done, say it explicitly - especially actions like submitting a form or sending an email. Example: "Once you fill in the form, select the 'Submit' button. No need to ask for permission."

• Break down complex interactions. For areas where the UI might be more complex to navigate, explain each step in detail. Example: "Select the three-dot menu in the top right corner. A dropdown appears. Once it opens, select the last item in the list."

• Use step-by-step formatting for longer tasks. Long instructions are easier to follow when you format them as a list.

Best practices for credentials

Computer use doesn't currently support interactive authentication flows. To access websites or applications that require authentication, ensure these services are signed in on the machine where computer use runs. For example, set up the browser or application to remember sign-in credentials, so the required sessions are active when computer use begins its tasks.

Sample instructions

Explore these sample instructions to try out computer use or use them as a reference for writing your own.

Disable computer use

To disable computer use in an environment, follow these steps based on whether you're using the new admin center or the classic admin center.

Licensing

While computer use is in preview, it's billed using the Agent action feature with a billing rate of five messages. Learn more about Microsoft Copilot Studio billing rate and management.

Each computer use run can perform several actions, with each action costing five messages.

For example, if you set up computer use to fill out a web-based timesheet form, it performs these actions each time it's triggered:

1. Launch browser.

2. Navigate to the timesheet web portal.

3. Select Create new timesheet.

4. Fill the Start Time form field.

5. Fill the End Time form field.

6. Fill the Project Code form field.

7. Select the Submit button.

In this example, computer use executes seven actions.

Limitations

Troubleshooting

Share your feedback

Do you have feedback about computer use? Let us know at computeruse-feedback@microsoft.com.

Related information

FAQ for the computer use tool