• Stars
    star
    607
  • Rank 73,845 (Top 2 %)
  • Language
    Go
  • License
    MIT License
  • Created almost 7 years ago
  • Updated about 2 months ago

Reviews

There are no reviews yet. Be the first to send feedback to the community and the maintainers!

Repository Details

The new Azure Storage data transfer utility - AzCopy v10

AzCopy v10

AzCopy v10 is a command-line utility that you can use to copy data to and from containers and file shares in Azure Storage accounts. AzCopy V10 presents easy-to-use commands that are optimized for high performance and throughput.

Features and capabilities

Use with storage accounts that have a hierarchical namespace (Azure Data Lake Storage Gen2).

Create containers and file shares.

Upload files and directories.

Download files and directories.

Copy containers, directories and blobs between storage accounts (Service to Service).

Synchronize data between Local <=> Blob Storage, Blob Storage <=> File Storage, and Local <=> File Storage.

Delete blobs or files from an Azure storage account

Copy objects, directories, and buckets from Amazon Web Services (AWS) to Azure Blob Storage (Blobs only).

Copy objects, directories, and buckets from Google Cloud Platform (GCP) to Azure Blob Storage (Blobs only).

List files in a container.

Recover from failures by restarting previous jobs.

Download AzCopy

The latest binary for AzCopy along with installation instructions may be found here.

Find help

For complete guidance, visit any of these articles on the docs.microsoft.com website.

✳️ Get started with AzCopy (download links here)

✳️ Upload files to Azure Blob storage by using AzCopy

✳️ Download blobs from Azure Blob storage by using AzCopy

✳️ Copy blobs between Azure storage accounts by using AzCopy

✳️ Synchronize between Local File System/Azure Blob Storage (Gen1)/Azure File Storage by using AzCopy

✳️ Transfer data with AzCopy and file storage

✳️ Transfer data with AzCopy and Amazon S3 buckets

✳️ Transfer data with AzCopy and Google GCP buckets

✳️ Use data transfer tools in Azure Stack Hub Storage

✳️ Configure, optimize, and troubleshoot AzCopy

✳️ AzCopy WiKi

Supported Operations

The general format of the AzCopy commands is: azcopy [command] [arguments] --[flag-name]=[flag-value]

  • bench - Runs a performance benchmark by uploading or downloading test data to or from a specified destination

  • copy - Copies source data to a destination location. The supported directions are:

    • Local File System <-> Azure Blob (SAS or OAuth authentication)
    • Local File System <-> Azure Files (Share/directory SAS authentication)
    • Local File System <-> Azure Data Lake Storage (ADLS Gen2) (SAS, OAuth, or SharedKey authentication)
    • Azure Blob (SAS or public) -> Azure Blob (SAS or OAuth authentication)
    • Azure Blob (SAS or public) -> Azure Files (SAS)
    • Azure Files (SAS) -> Azure Files (SAS)
    • Azure Files (SAS) -> Azure Blob (SAS or OAuth authentication)
    • AWS S3 (Access Key) -> Azure Block Blob (SAS or OAuth authentication)
    • Google Cloud Storage (Service Account Key) -> Azure Block Blob (SAS or OAuth authentication) [Preview]
  • sync - Replicate source to the destination location. The supported directions are:

    • Local File System <-> Azure Blob (SAS or OAuth authentication)
    • Local File System <-> Azure Files (Share/directory SAS authentication)
    • Azure Blob (SAS or public) -> Azure Files (SAS)
  • login - Log in to Azure Active Directory (AD) to access Azure Storage resources.

  • logout - Log out to terminate access to Azure Storage resources.

  • list - List the entities in a given resource

  • doc - Generates documentation for the tool in Markdown format

  • env - Shows the environment variables that you can use to configure the behavior of AzCopy.

  • help - Help about any command

  • jobs - Sub-commands related to managing jobs

  • load - Sub-commands related to transferring data in specific formats

  • make - Create a container or file share.

  • remove - Delete blobs or files from an Azure storage account

Find help from your command prompt

For convenience, consider adding the AzCopy directory location to your system path for ease of use. That way you can type azcopy from any directory on your system.

To see a list of commands, type azcopy -h and then press the ENTER key.

To learn about a specific command, just include the name of the command (For example: azcopy list -h).

AzCopy command help example

If you choose not to add AzCopy to your path, you'll have to change directories to the location of your AzCopy executable and type azcopy or .\azcopy in Windows PowerShell command prompts.

Frequently asked questions

What is the difference between sync and copy?

  • The copy command is a simple transferring operation. It scans/enumerates the source and attempts to transfer every single file/blob present on the source to the destination. The supported source/destination pairs are listed in the help message of the tool.

  • On the other hand, sync scans/enumerates both the source, and the destination to find the incremental change. It makes sure that whatever is present in the source will be replicated to the destination. For sync,

  • If your goal is to simply move some files, then copy is definitely the right command, since it offers much better performance. If the use case is to incrementally transfer data (files present only on source) then sync is the better choice, since only the modified/missing files will be transferred. Since sync enumerates both source and destination to find the incremental change, it is relatively slower as compared to copy

Will copy overwrite my files?

By default, AzCopy will overwrite the files at the destination if they already exist. To avoid this behavior, please use the flag --overwrite=false.

Will sync overwrite my files?

By default, AzCopy sync use last-modified-time to determine whether to transfer the same file present at both the source, and the destination. i.e, If the source file is newer compared to the destination file, we overwrite the destination You can change this default behaviour and overwrite files at the destination by using the flag --mirror-mode=true

Will 'sync' delete files in the destination if they no longer exist in the source location?

By default, the 'sync' command doesn't delete files in the destination unless you use an optional flag with the command. To learn more, see Synchronize files.

How to contribute to AzCopy v10

This project welcomes contributions and suggestions. Most contributions require you to agree to a Contributor License Agreement (CLA) declaring that you have the right to, and actually do, grant us the rights to use your contribution. For details, visit https://cla.microsoft.com.

When you submit a pull request, a CLA-bot will automatically determine whether you need to provide a CLA and decorate the PR appropriately (e.g., label, comment). Simply follow the instructions provided by the bot. You will only need to do this once across all repos using our CLA.

This project has adopted the Microsoft Open Source Code of Conduct. For more information see the Code of Conduct FAQ or contact [email protected] with any additional questions or comments.

More Repositories

1

azure-quickstart-templates

Azure Quickstart Templates
Bicep
13,949
star
2

azure-sdk-for-net

This repository is for active development of the Azure SDK for .NET. For consumers of the SDK we recommend visiting our public developer docs at https://learn.microsoft.com/dotnet/azure/ or our versioned developer docs at https://azure.github.io/azure-sdk-for-net.
C#
5,256
star
3

autorest

OpenAPI (f.k.a Swagger) Specification code generator. Supports C#, PowerShell, Go, Java, Node.js, TypeScript, Python
TypeScript
4,594
star
4

Azure-Sentinel

Cloud-native SIEM for intelligent security analytics for your entire enterprise.
Jupyter Notebook
4,559
star
5

azure-sdk-for-python

This repository is for active development of the Azure SDK for Python. For consumers of the SDK we recommend visiting our public developer docs at https://learn.microsoft.com/python/azure/ or our versioned developer docs at https://azure.github.io/azure-sdk-for-python.
Python
4,540
star
6

azure-powershell

Microsoft Azure PowerShell
C#
4,178
star
7

MachineLearningNotebooks

Python notebooks with ML and deep learning examples with Azure Machine Learning Python SDK | Microsoft
Jupyter Notebook
4,090
star
8

DotNetty

DotNetty project – a port of netty, event-driven asynchronous network application framework
C#
4,087
star
9

azure-cli

Azure Command-Line Interface
Python
3,930
star
10

draft-classic

A tool for developers to create cloud-native applications on Kubernetes.
Go
3,925
star
11

bicep

Bicep is a declarative language for describing and deploying Azure resources
Bicep
3,240
star
12

azure-rest-api-specs

The source for REST API specifications for Microsoft Azure.
HCL
2,634
star
13

azure-sdk-for-java

This repository is for active development of the Azure SDK for Java. For consumers of the SDK we recommend visiting our public developer docs at https://docs.microsoft.com/java/azure/ or our versioned developer docs at https://azure.github.io/azure-sdk-for-java.
Java
2,305
star
14

azure-sdk-for-js

This repository is for active development of the Azure SDK for JavaScript (NodeJS & Browser). For consumers of the SDK we recommend visiting our public developer docs at https://docs.microsoft.com/javascript/azure/ or our versioned developer docs at https://azure.github.io/azure-sdk-for-js.
TypeScript
2,051
star
15

AKS

Azure Kubernetes Service
HTML
1,965
star
16

azure-functions-host

The host/runtime that powers Azure Functions
C#
1,871
star
17

golua

A Lua 5.3 engine implemented in Go
Go
1,850
star
18

Azurite

A lightweight server clone of Azure Storage that simulates most of the commands supported by it with minimal dependencies
TypeScript
1,823
star
19

azureml-examples

Official community-driven Azure Machine Learning examples, tested with GitHub Actions.
Jupyter Notebook
1,733
star
20

Microsoft-Defender-for-Cloud

Welcome to the Microsoft Defender for Cloud community repository
PowerShell
1,677
star
21

Enterprise-Scale

The Azure Landing Zones (Enterprise-Scale) architecture provides prescriptive guidance coupled with Azure best practices, and it follows design principles across the critical design areas for organizations to define their Azure architecture
PowerShell
1,603
star
22

azure-sdk-for-go

This repository is for active development of the Azure SDK for Go. For consumers of the SDK we recommend visiting our public developer docs at:
Go
1,594
star
23

Stormspotter

Azure Red Team tool for graphing Azure and Azure Active Directory objects
Python
1,525
star
24

durabletask

Durable Task Framework allows users to write long running persistent workflows in C# using the async/await capabilities.
C#
1,516
star
25

azure-policy

Repository for Azure Resource Policy built-in definitions and samples
Open Policy Agent
1,485
star
26

iotedge

The IoT Edge OSS project
C#
1,421
star
27

aztfexport

A tool to bring existing Azure resources under Terraform's management
Go
1,335
star
28

review-checklists

This repo contains code and examples to operationalize Azure review checklists.
Python
1,187
star
29

azure-sdk-for-node

Azure SDK for Node.js - Documentation
JavaScript
1,187
star
30

azure-functions-core-tools

Command line tools for Azure Functions
C#
1,128
star
31

Azure-Functions

PowerShell
1,108
star
32

acs-engine

WE HAVE MOVED: Please join us at Azure/aks-engine!
Go
1,031
star
33

aks-engine

AKS Engine: legacy tool for Kubernetes on Azure (see status)
Go
1,027
star
34

data-api-builder

Data API builder provides modern REST and GraphQL endpoints to your Azure Databases and on-prem stores.
C#
912
star
35

terraform-azurerm-caf-enterprise-scale

Azure landing zones Terraform module
HCL
856
star
36

coco-framework

The Confidential Consortium Blockchain Framework is an open-source system that enables high-scale, confidential blockchain networks that meet all key enterprise requirements—providing a means to accelerate production enterprise adoption of blockchain technology.
834
star
37

azure-iot-sdks

SDKs for a variety of languages and platforms that help connect devices to Microsoft Azure IoT services
829
star
38

GPT-RAG

Sharing the learning along the way we been gathering to enable Azure OpenAI at enterprise scale in a secure manner. GPT-RAG core is a Retrieval-Augmented Generation pattern running in Azure, using Azure Cognitive Search for retrieval and Azure OpenAI large language models to power ChatGPT-style and Q&A experiences.
Bicep
822
star
39

AzurePublicDataset

Microsoft Azure Traces
Jupyter Notebook
790
star
40

azure-sql-database-samples

Azure SQL Database Samples and Reference Implementation Repository
Python
781
star
41

Azure-Network-Security

Resources for improving Customer Experience with Azure Network Security
Python
768
star
42

caf-terraform-landingzones

This solution, offered by the Open-Source community, will no longer receive contributions from Microsoft. Customers are encouraged to transition to Microsoft Azure Verified Modules for continued support and updates from Microsoft. Please note, this repository is scheduled for decommissioning and will be removed on July 1, 2025.
HCL
763
star
43

azure-search-vector-samples

A repository of code samples for Vector search capabilities in Azure AI Search.
Jupyter Notebook
750
star
44

azure-service-operator

Azure Service Operator allows you to create Azure resources using kubectl
Go
741
star
45

azure-functions-durable-extension

Durable Task Framework extension for Azure Functions
C#
714
star
46

azure-webjobs-sdk

Azure WebJobs SDK
C#
712
star
47

ResourceModules

This repository includes a CI platform for and collection of mature and curated Bicep modules. The platform supports both ARM and Bicep and can be leveraged using GitHub actions as well as Azure DevOps pipelines.
Bicep
710
star
48

ALZ-Bicep

This repository contains the Azure Landing Zones (ALZ) Bicep modules that help deliver and deploy the Azure Landing Zone conceptual architecture in a modular approach. https://aka.ms/alz/docs
Bicep
694
star
49

azure-sdk-for-rust

This repository is for active development of the *unofficial* Azure SDK for Rust. This repository is *not* supported by the Azure SDK team.
Rust
692
star
50

terraform

Source code for the Azure Marketplace Terraform development VM package.
HCL
690
star
51

azure-api-management-devops-resource-kit

Azure API Management DevOps Resource Kit
C#
684
star
52

application-gateway-kubernetes-ingress

This is an ingress controller that can be run on Azure Kubernetes Service (AKS) to allow an Azure Application Gateway to act as the ingress for an AKS cluster.
Go
668
star
53

CCOInsights

Welcome to the Continuous Cloud Optimization Power BI Dashboard GitHub Project. In this repository you will find all the guidance and files needed to deploy the Dashboard in your environment to take benefit of a single pane of glass to get insights about your Azure resources and services.
Mathematica
668
star
54

SimuLand

Understand adversary tradecraft and improve detection strategies
PowerShell
664
star
55

DeepLearningForTimeSeriesForecasting

A tutorial demonstrating how to implement deep learning models for time series forecasting
Jupyter Notebook
655
star
56

azure-xplat-cli

For ARM-based service please go to CLI 2.0.
JavaScript
651
star
57

azure-cosmos-dotnet-v3

.NET SDK for Azure Cosmos DB for the core SQL API
C#
637
star
58

node-sqlserver

C++
625
star
59

azure-devops-cli-extension

Azure DevOps Extension for Azure CLI
Python
621
star
60

azure-storage-fuse

A virtual file system adapter for Azure Blob storage
Go
613
star
61

Community-Policy

This repo is for Microsoft Azure customers and Microsoft teams to collaborate in making custom policies.
Open Policy Agent
612
star
62

azure-resource-manager-schemas

Schemas used to author and validate Resource Manager Templates. These schemas power the intellisense and syntax completion in our ARM Tools VSCode extension, as well as the Export Template API
TypeScript
610
star
63

counterfit

a CLI that provides a generic automation layer for assessing the security of ML models
Python
599
star
64

azure-iot-sdk-c

A C99 SDK for connecting devices to Microsoft Azure IoT services
C
587
star
65

azure-cosmos-dotnet-v2

Contains samples and utilities relating to the Azure Cosmos DB .NET SDK
577
star
66

azure-service-bus

☁️ Azure Service Bus service issue tracking and samples
571
star
67

aad-pod-identity

[DEPRECATED] Assign Azure Active Directory Identities to Kubernetes applications.
Go
569
star
68

Azure-Sentinel-Notebooks

Interactive Azure Sentinel Notebooks provides security insights and actions to investigate anomalies and hunt for malicious behaviors.
Jupyter Notebook
541
star
69

draft

A day 0 tool for getting your app on k8s fast
Go
541
star
70

AzureStack-QuickStart-Templates

Quick start ARM templates that deploy on Microsoft Azure Stack
PowerShell
540
star
71

azure-openai-samples

Azure OpenAI Samples is a collection of code samples illustrating how to use Azure Open AI in creating AI solution for various use cases across industries. This repository is mained by a community of volunters. We welcomed your contributions.
Jupyter Notebook
540
star
72

WALinuxAgent

Microsoft Azure Linux Guest Agent
Python
538
star
73

AI-in-a-Box

AI-in-a-Box leverages the expertise of Microsoft across the globe to develop and provide AI and ML solutions to the technical community. Our intent is to present a curated collection of solution accelerators that can help engineers establish their AI/ML environments and solutions rapidly and with minimal friction.
Jupyter Notebook
529
star
74

iot-edge-v1

Azure IoT Edge
C
524
star
75

Industrial-IoT

Azure Industrial IoT Platform
C#
521
star
76

MS-AMP

Microsoft Automatic Mixed Precision Library
Python
516
star
77

Mission-Critical

This repository provides a design methodology and approach to building highly-reliable applications on Microsoft Azure for mission-critical workloads.
510
star
78

static-web-apps-cli

Azure Static Web Apps CLI ✨
TypeScript
509
star
79

mlops-v2

Azure MLOps (v2) solution accelerators. Enterprise ready templates to deploy your machine learning models on the Azure Platform.
Shell
505
star
80

azure-docs-powershell-samples

Azure Powershell code samples, often used in docs.microsoft.com/Azure developer documentation
PowerShell
503
star
81

azqr

Azure Quick Review
Go
503
star
82

bicep-registry-modules

Bicep registry modules
Bicep
501
star
83

azure-storage-node

Microsoft Azure Storage SDK for Node.js
JavaScript
495
star
84

api-management-developer-portal

Developer portal provided by the Azure API Management service.
TypeScript
487
star
85

kubelogin

A Kubernetes credential (exec) plugin implementing azure authentication
Go
486
star
86

Azure-TDSP-ProjectTemplate

TDSP: Data science project template repository with standardized directory structure and document templates to support efficient project execution and collaboration.
R
483
star
87

azure-sdk

This is the Azure SDK parent repository and mostly contains documentation around guidelines and policies as well as the releases for the various languages supported by the Azure SDK.
PowerShell
478
star
88

azure-mobile-services

Mobile Services is deprecated - Use Mobile Apps instead
HTML
472
star
89

azure-devtestlab

Azure DevTestLab artifacts, scripts and samples
PowerShell
458
star
90

azure-iot-sdk-csharp

A C# SDK for connecting devices to Microsoft Azure IoT services
C#
455
star
91

actions-workflow-samples

Help developers to easily get started with GitHub Action workflows to deploy to Azure
Pug
450
star
92

azure-devops-utils

Azure DevOps Utilities
Shell
450
star
93

AzureDatabricksBestPractices

Version 1 of Technical Best Practices of Azure Databricks based on real world Customer and Technical SME inputs
446
star
94

app-service-announcements

Subscribe to this repo to be notified about major changes in App Service
446
star
95

azure-storage-net

Microsoft Azure Storage Libraries for .NET
C#
445
star
96

Azure-DataFactory

C#
444
star
97

arm-ttk

Azure Resource Manager Template Toolkit
PowerShell
442
star
98

RDS-Templates

ARM Templates for Remote Desktop Services deployments
PowerShell
427
star
99

Copilot-For-Security

Microsoft Copilot for Security is a generative AI-powered security solution that helps increase the efficiency and capabilities of defenders to improve security outcomes at machine speed and scale, while remaining compliant to responsible AI principles
PowerShell
426
star
100

enterprise-azure-policy-as-code

Enterprise-ready Azure Policy-as-Code (PaC) solution (includes Az DevOps pipeline)
PowerShell
423
star