Theoretical Foundation: Manage Data by Using Azure Storage Explorer and AzCopy

1. Initial Intuition

Imagine you need to manage files on a remote server. You have two options: use a program with a graphical interface, where you click, drag, and visualize files as if you were in Windows Explorer, or use a highly optimized command-line client, capable of transferring terabytes of data in parallel at maximum speed.

In Azure, these two tools exist and have specific names:

Azure Storage Explorer is the visual manager. You install it on your computer, connect to your Storage Accounts, and navigate through data as if they were local folders.

AzCopy is the command-line tool. It's a single executable, without complex installation, designed for high-performance transfers between local systems and Azure, or between different locations in Azure.

Both tools complement each other: Storage Explorer for one-off operations and visual exploration; AzCopy for automation, scripts, and large-scale transfers.

2. Context

2.1 Why these tools exist

The Azure portal has limited capabilities for data management in Storage Accounts. You can create containers, upload individual files, but you can't:

Copy thousands of files at once
Move data between two Storage Accounts in different regions
Synchronize local directories with containers
Download large volumes of data with optimized performance

Storage Explorer and AzCopy fill these gaps. They are the primary tools that an Azure administrator uses for data movement and management on a daily basis.

2.2 Position in the Azure ecosystem

100%

Scroll para zoom · Arraste para mover · 📱 Pinch para zoom no celular

3. Concept Construction

3.1 Azure Storage Explorer

What it is

Storage Explorer is a cross-platform desktop application (Windows, macOS, Linux) developed by Microsoft. It provides a graphical interface for managing Storage Accounts, including Blob Storage, Azure Files, Queue Storage, Table Storage, Azure Cosmos DB, and Azure Data Lake Storage.

Main capabilities

Functionality	Support
Browse containers, blobs, and directories	Yes
Upload and download files and folders	Yes
Copy and move data between accounts	Yes
Manage container permissions	Yes
Generate SAS tokens	Yes
Manage access policies	Yes
View and edit blob metadata	Yes
Manage Azure Files (SMB/NFS)	Yes
Edit data in Queue and Table Storage	Yes
Connect to local accounts (Azurite emulator)	Yes
Connect via Private Endpoint	Yes

Authentication forms in Storage Explorer

Storage Explorer supports multiple connection methods, and understanding each one is fundamental:

Method	When to use
Azure AD (Microsoft/organizational account)	RBAC-based access; recommended for production
Access key (Storage Account Key)	Full access; useful for troubleshooting; avoid in production
SAS token	Limited and temporary access; secure sharing
Connection string	Combination of key and endpoint; useful for local emulators
Public access (anonymous)	Containers with public access enabled
Managed Identity	When running on VM or Azure service with identity

3.2 AzCopy

What it is

AzCopy is a command-line utility optimized for high-performance data transfers involving Azure Storage. It's a single binary, with no dependencies, available for Windows, macOS, and Linux.

The main advantage of AzCopy over other CLI tools (like az storage blob upload) is its parallel transfer architecture: it splits large files into parts, sends multiple parts simultaneously, and uses all available machine cores.

Main capabilities

Functionality	Support
Upload files and directories	Yes
Download blobs	Yes
Copy between Storage Accounts (server-side)	Yes
Directory synchronization (`sync`)	Yes
Resumable transfer (journaling)	Yes
Integrity verification (MD5)	Yes
Filters by name and date pattern	Yes
Access tier definition	Yes
Cross-region and cross-subscription copy	Yes
Copy from Amazon S3 to Azure	Yes
Copy from Google Cloud Storage to Azure	Yes

Server-side copy: AzCopy's differentiator

When you copy data between two Storage Accounts, AzCopy uses server-side copy: data moves directly between Azure servers, without passing through your local machine. This means:

Speed is not limited by your internet connection
Your machine practically doesn't use bandwidth during the operation
Terabyte transfers complete in minutes instead of hours

100%

Scroll para zoom · Arraste para mover · 📱 Pinch para zoom no celular

3.3 Authentication in AzCopy

AzCopy supports three authentication methods:

1. Azure AD (recommended)

azcopy login

Opens browser for interactive authentication. After login, AzCopy uses OAuth2 tokens with 24-hour validity.

For automation (non-interactive):

# Service Principal
azcopy login --service-principal \
  --application-id <app-id> \
  --tenant-id <tenant-id>

Requires the Service Principal to have the Storage Blob Data Contributor role (or higher) on the Storage Account.

2. SAS Token

azcopy copy \
  'https://source.blob.core.windows.net/container/blob?<SAS-TOKEN>' \
  'https://dest.blob.core.windows.net/container/blob?<SAS-TOKEN>'

SAS tokens are directly appended to the URL. Useful when Azure AD is not available or for temporary access sharing.

3. Managed Identity

When AzCopy is executed within an Azure VM with Managed Identity, it can authenticate automatically:

azcopy login --identity

4. Structural View

100%

Scroll para zoom · Arraste para mover · 📱 Pinch para zoom no celular

5. Practical Operation

5.1 Azure Storage Explorer: essential operations

Connecting a Storage Account

In Storage Explorer, you can connect in three main ways:

Via Azure AD: Click "Add an Azure Account", authenticate, and all accounts in the subscription appear automatically.
Via Connection String or Key: Click "Add a Resource" > "Storage account or service" > "Account name and key" or "Connection string".
Via SAS URI: Click "Add a Resource" > "Blob container" > "Shared access signature URL". This connects only to the specific resource referenced by the SAS.

File upload

In Storage Explorer, you can drag files from your operating system directly to a container. Or use the "Upload" menu that offers options to:

Define blob type (Block, Append, Page)
Define access tier (Hot, Cool, Archive)
Define custom metadata
Choose Encryption Scope

Generating SAS tokens in Storage Explorer

Right-click on a container or blob and select "Get Shared Access Signature". Storage Explorer presents an interface to:

Define start and expiration date
Choose permissions (Read, Write, Delete, List, Add, Create)
Choose protocol (HTTPS only or HTTPS and HTTP)
Specify allowed IP

5.2 AzCopy: fundamental commands

Upload (local to Azure)

# Upload single file
azcopy copy \
  '/local/path/file.pdf' \
  'https://myaccount.blob.core.windows.net/mycontainer/file.pdf'

# Upload entire directory (recursive)
azcopy copy \
  '/local/path/directory/' \
  'https://myaccount.blob.core.windows.net/mycontainer/' \
  --recursive

# Upload with access tier definition
azcopy copy \
  '/local/path/archive-data/' \
  'https://myaccount.blob.core.windows.net/mycontainer/' \
  --recursive \
  --block-blob-tier Cool

Download (Azure to local)

# Download single blob
azcopy copy \
  'https://myaccount.blob.core.windows.net/mycontainer/file.pdf' \
  '/local/path/file.pdf'

# Download entire container
azcopy copy \
  'https://myaccount.blob.core.windows.net/mycontainer/' \
  '/local/path/directory/' \
  --recursive

Copy between Storage Accounts (server-side)

# Copy entire container between accounts
azcopy copy \
  'https://sourceaccount.blob.core.windows.net/source-container/' \
  'https://destaccount.blob.core.windows.net/dest-container/' \
  --recursive

# With SAS tokens in both accounts
azcopy copy \
  'https://sourceaccount.blob.core.windows.net/source-container/?<SAS>' \
  'https://destaccount.blob.core.windows.net/dest-container/?<SAS>' \
  --recursive

Synchronization (`sync`)

The sync command copies only files that are different between source and destination, and can delete files in the destination that no longer exist in the source:

# Synchronize local with Azure (unidirectional)
azcopy sync \
  '/local/directory/' \
  'https://myaccount.blob.core.windows.net/mycontainer/' \
  --recursive

# Synchronize with deletion at destination
azcopy sync \
  '/local/directory/' \
  'https://myaccount.blob.core.windows.net/mycontainer/' \
  --recursive \
  --delete-destination true

Non-obvious sync behavior: AzCopy compares files by last modified date and size, not by content (hash). Files with same content but different dates will be copied again.

Advanced filters

# Include only files with certain pattern
azcopy copy \
  '/local/path/' \
  'https://myaccount.blob.core.windows.net/container/' \
  --recursive \
  --include-pattern "*.log;*.csv"

# Exclude patterns
azcopy copy \
  '/local/path/' \
  'https://myaccount.blob.core.windows.net/container/' \
  --recursive \
  --exclude-pattern "*.tmp;*.bak"

# Include only files modified after certain date
azcopy copy \
  '/local/path/' \
  'https://myaccount.blob.core.windows.net/container/' \
  --recursive \
  --include-after "2025-01-01T00:00:00Z"

5.3 Job management in AzCopy

AzCopy maintains a journal of each operation. If a transfer is interrupted, it can be resumed from where it left off:

# List all jobs
azcopy jobs list

# View status of specific job
azcopy jobs show <job-id>

# Resume interrupted job
azcopy jobs resume <job-id>

# Clean completed jobs
azcopy jobs clean

Journals are stored in:

Windows: %USERPROFILE%\.azcopy
Linux/macOS: ~/.azcopy

6. Implementation Methods

6.1 Storage Explorer: when to use

Use Storage Explorer when:

You need to visually explore Storage Account content
You'll perform one-off operations (upload 10-20 files, rename, move)
You need to generate SAS tokens visually and in a controlled manner
You want to inspect metadata, properties, and access tiers of individual blobs
You're diagnosing access or data structure problems
You need to manage Queue Storage or Table Storage visually

Storage Explorer limitations:

Not suitable for automation or scripts
Performance inferior to AzCopy for large volumes
Doesn't support advanced inclusion/exclusion filters
Doesn't resume interrupted transfers granularly

6.2 AzCopy: when to use

Use AzCopy when:

Transferring large volumes of data (tens of GB or more)
You need a reproducible operation in scripts
Migrating data between Storage Accounts
Implementing automated backup via cron/Task Scheduler
You need inclusion/exclusion filters by name pattern
You need incremental directory synchronization

AzCopy limitations:

No graphical interface (can be intimidating for non-technical users)
Requires authentication configuration before use
Doesn't manage Queue Storage or Table Storage

6.3 Direct comparison

Criteria	Storage Explorer	AzCopy
Interface	Graphical (desktop)	Command line
Performance on large volumes	Moderate	High (parallel transfer)
Automation and scripts	No	Yes
Resume transfers	Limited	Yes (journal)
Advanced filters	No	Yes
Server-side copy	Yes	Yes
Incremental synchronization	No	Yes (sync command)
Queue and Table Storage	Yes	No
SAS generation	Yes (visual)	Via separate Azure CLI
Learning curve	Low	Medium

7. Control and Security

7.1 Recommended authentication by scenario

Scenario	Storage Explorer	AzCopy
Administrator in corporate environment	Azure AD	Azure AD (`azcopy login`)
Automation script in CI/CD pipeline	N/A	Service Principal with Azure AD
Script on Azure VM	N/A	Managed Identity (`azcopy login --identity`)
Share access with third parties	SAS token	SAS token in URL
Local development environment	Connection string (emulator)	Connection string

7.2 Principle of least privilege with SAS

When generating SAS tokens in Storage Explorer or for use with AzCopy, apply only necessary permissions:

Operation	Required SAS permissions
Upload only	Write, Add, Create
Download only	Read
List and download	Read, List
Upload and download	Read, Write, Add, Create
Delete	Delete (grant with extreme caution)

7.3 Environment variable for SAS in AzCopy

Instead of including SAS tokens directly in the command line (which remains visible in shell history), use environment variables:

export AZCOPY_SAS_TOKEN="?sv=2023-01-03&ss=b&..."
azcopy copy \
  "https://myaccount.blob.core.windows.net/container/$AZCOPY_SAS_TOKEN" \
  '/local/path/'

7.4 Logging and auditing in AzCopy

AzCopy generates detailed logs of each operation. Configure log level:

azcopy copy \
  '/source/' \
  'https://myaccount.blob.core.windows.net/container/' \
  --log-level INFO \
  --recursive

Available levels: NONE, DEBUG, INFO, WARNING, ERROR.

Logs are stored in the same directory as journals (~/.azcopy or %USERPROFILE%\.azcopy).

8. Decision Making

8.1 Tool selection by scenario

Situation	Best choice	Reason
Explore structure of unknown account	Storage Explorer	Visual interface facilitates navigation
Migrate 5 TB between two Storage Accounts	AzCopy	Server-side copy, parallel, no local bandwidth usage
Generate SAS token to share with partner	Storage Explorer	Clear visual interface to define permissions
Automated nightly backup via cron	AzCopy	Scriptable, resumable, no human interaction
Synchronize local directory with Azure	AzCopy sync	Native in AzCopy, incremental
Manage queues and tables	Storage Explorer	AzCopy doesn't support these services
Punctual upload of 3 files	Either one	Both suitable for small volumes
DevOps pipeline for asset deployment	AzCopy	Integration with CI/CD scripts
Copy from Amazon S3 to Azure	AzCopy	Native support for S3 as source
Blob metadata diagnostics	Storage Explorer	Rich property visualization

8.2 `copy` vs `sync`

Aspect	`azcopy copy`	`azcopy sync`
Behavior	Copies all specified files	Copies only differences
Delete at destination	No	Optional (`--delete-destination`)
Overwrite existing	Yes (default)	Only if different
Typical use	Migration, full backup	Continuous synchronization
File comparison	N/A	Modification date and size

9. Best Practices

For AzCopy:

Prefer Azure AD over SAS tokens for long-term automation. SAS tokens expire and need to be rotated.
Use Managed Identity on any Azure VM or service running AzCopy, eliminating credential management.
Always use --recursive explicitly when copying directories. The behavior without the flag can be confusing.
Test with --dry-run (where available) before executing destructive operations with sync --delete-destination.
Configure --cap-mbps in environments where bandwidth should be preserved for other applications:

azcopy copy '/source/' 'https://...' --recursive --cap-mbps 100

Monitor long-running jobs with azcopy jobs show <job-id> instead of leaving the terminal open.
Clean old journals periodically with azcopy jobs clean to free disk space.

For Storage Explorer:

Use Azure AD as the default connection method instead of access keys.
Never save connection strings with keys in shared configurations.
Keep Storage Explorer updated, as new versions add support for new features and fix vulnerabilities.
Use SAS tokens with short validity when sharing access with third parties via Storage Explorer.

10. Common Errors

Error	Why it happens	How to avoid
`AuthorizationPermissionMismatch` in AzCopy	Authenticated user doesn't have sufficient RBAC	Verify they have at least `Storage Blob Data Contributor`
Expired SAS token in AzCopy	Token generated with short validity	Generate tokens with adequate validity or use Azure AD
`sync` deleted files at destination unexpectedly	Using `--delete-destination true` unintentionally	Test with `--dry-run` first; omit flag when not needed
Slow upload despite good connection	`--cap-mbps` configured; or few threads	Check settings; AzCopy adjusts parallelism automatically
Transfer fails midway and doesn't resume	Job not identified correctly	Use `azcopy jobs resume <job-id>` explicitly
Blobs copied with wrong tier	Didn't specify `--block-blob-tier`	Always specify desired tier in cross-account copies
Storage Explorer doesn't show accounts	Subscription's Azure AD not selected	Check subscriptions in "Select Subscriptions"
Server-side copy fails with 403 error	Destination account SAS doesn't have Write permission	Include adequate permissions in destination SAS
AzCopy without access to Private Endpoint environment	Traffic blocked by Storage Account firewall	Run AzCopy from within VNet or via Jump Server

11. Operation and Maintenance

11.1 Checking AzCopy version

azcopy --version

AzCopy is updated frequently. Different versions may have different behaviors. Document the version used in production scripts.

11.2 AzCopy performance settings

AzCopy automatically adjusts parallelism based on machine resources, but you can control:

# Maximum number of concurrent operations
export AZCOPY_CONCURRENCY_VALUE=32

# Number of goroutines per operation (for large files)
export AZCOPY_CONCURRENT_FILES=10

11.3 Monitoring ongoing transfers

# View status of all active jobs
azcopy jobs list

# Details of a specific job (includes percentage progress)
azcopy jobs show <job-id>

11.4 Relevant limits

Limit	Value
Maximum blob size (Block Blob via AzCopy)	190.7 TiB
Multipart upload part size	100 MiB (default, configurable)
Default parallel connections	Based on CPU cores
Requests per second (Storage Account)	20,000 (service limit, not AzCopy)

12. Integration and Automation

12.1 AzCopy in Azure DevOps pipelines

- task: AzureCLI@2
  inputs:
    azureSubscription: 'MyServiceConnection'
    scriptType: 'bash'
    scriptLocation: 'inlineScript'
    inlineScript: |
      # Authenticate with Service Principal via Azure CLI
      # and use the token for AzCopy
      azcopy copy \
        './dist/' \
        'https://myaccount.blob.core.windows.net/$web/' \
        --recursive \
        --overwrite true

Authentication in pipelines: In Azure DevOps, use the Service Connection configured in the AzureCLI@2 task. AzCopy, when invoked within this task, inherits Azure CLI credentials automatically.

12.2 AzCopy with GitHub Actions

- name: Upload assets to Azure Storage
  run: |
    azcopy copy \
      './build/' \
      '${{ secrets.AZURE_STORAGE_URL }}/${{ secrets.AZURE_SAS_TOKEN }}' \
      --recursive

12.3 Automated backup with cron (Linux)

#!/bin/bash
# /opt/scripts/backup-to-azure.sh

# Authenticate via Managed Identity (on Azure VM)
azcopy login --identity

# Execute backup with timestamp
DATE=$(date +%Y-%m-%d)
azcopy copy \
  '/var/app/data/' \
  "https://backupaccount.blob.core.windows.net/backups/$DATE/" \
  --recursive \
  --log-level WARNING

# Crontab: 0 2 * * * /opt/scripts/backup-to-azure.sh

12.4 Amazon S3 to Azure migration with AzCopy

# Configure AWS credentials
export AWS_ACCESS_KEY_ID="<access-key>"
export AWS_SECRET_ACCESS_KEY="<secret-key>"

# Copy entire S3 bucket to Azure container
azcopy copy \
  'https://s3.amazonaws.com/my-bucket/' \
  'https://myaccount.blob.core.windows.net/mycontainer/?<SAS>' \
  --recursive \
  --s2s-preserve-access-tier false

13. Final Summary

Essential concepts:

Azure Storage Explorer is a desktop graphical tool for visually managing Storage Accounts. Ideal for exploration, punctual operations, and SAS token generation.
AzCopy is a command-line tool optimized for high-performance transfers. Ideal for automation, migration, and large data volumes.
Both tools complement each other and cover different use cases.

Critical differences:

Copy vs Sync in AzCopy: copy copies all specified files; sync copies only differences and can delete at destination with --delete-destination true.
Server-side copy: When AzCopy copies between two Storage Accounts, data doesn't pass through the local machine. This eliminates internet connection bandwidth limitations.
Authentication: Azure AD is the recommended method for both tools. SAS tokens are for temporary and shared access.
Transfer resumption: AzCopy maintains journals that allow resuming interrupted operations with azcopy jobs resume.

What needs to be remembered:

AzCopy requires azcopy login before using with Azure AD.
Minimum role for reading is Storage Blob Data Reader; for writing is Storage Blob Data Contributor.
AzCopy journals and logs are stored in ~/.azcopy (Linux/macOS) or %USERPROFILE%\.azcopy (Windows).
Storage Explorer is not suitable for automation; AzCopy doesn't manage Queue and Table Storage.
In CI/CD pipelines, use Service Principal or Managed Identity to authenticate AzCopy, never account keys.
The sync command compares by modification date and size, not by content hash.
AzCopy supports direct copy from Amazon S3 and Google Cloud Storage to Azure Storage.

1. Initial Intuition​

2. Context​

2.1 Why these tools exist​

2.2 Position in the Azure ecosystem​

3. Concept Construction​

3.1 Azure Storage Explorer​

What it is​

Main capabilities​

Authentication forms in Storage Explorer​

3.2 AzCopy​

What it is​

Main capabilities​

Server-side copy: AzCopy's differentiator​

3.3 Authentication in AzCopy​

4. Structural View​

5. Practical Operation​

5.1 Azure Storage Explorer: essential operations​

Connecting a Storage Account​

File upload​

Generating SAS tokens in Storage Explorer​

5.2 AzCopy: fundamental commands​

Upload (local to Azure)​

Download (Azure to local)​

Copy between Storage Accounts (server-side)​

Synchronization (sync)​

Advanced filters​

5.3 Job management in AzCopy​

6. Implementation Methods​

6.1 Storage Explorer: when to use​

6.2 AzCopy: when to use​

6.3 Direct comparison​

7. Control and Security​

7.1 Recommended authentication by scenario​

7.2 Principle of least privilege with SAS​

7.3 Environment variable for SAS in AzCopy​

7.4 Logging and auditing in AzCopy​

8. Decision Making​

8.1 Tool selection by scenario​

8.2 copy vs sync​

9. Best Practices​

10. Common Errors​

11. Operation and Maintenance​

11.1 Checking AzCopy version​

11.2 AzCopy performance settings​

11.3 Monitoring ongoing transfers​

11.4 Relevant limits​

12. Integration and Automation​

12.1 AzCopy in Azure DevOps pipelines​

12.2 AzCopy with GitHub Actions​

12.3 Automated backup with cron (Linux)​

12.4 Amazon S3 to Azure migration with AzCopy​

13. Final Summary​

1. Initial Intuition

2. Context

2.1 Why these tools exist

2.2 Position in the Azure ecosystem

3. Concept Construction

3.1 Azure Storage Explorer

What it is

Main capabilities

Authentication forms in Storage Explorer

3.2 AzCopy

What it is

Main capabilities

Server-side copy: AzCopy's differentiator

3.3 Authentication in AzCopy

4. Structural View

5. Practical Operation

5.1 Azure Storage Explorer: essential operations

Connecting a Storage Account

File upload

Generating SAS tokens in Storage Explorer

5.2 AzCopy: fundamental commands

Upload (local to Azure)

Download (Azure to local)

Copy between Storage Accounts (server-side)

Synchronization (`sync`)

Advanced filters

5.3 Job management in AzCopy

6. Implementation Methods

6.1 Storage Explorer: when to use

6.2 AzCopy: when to use

6.3 Direct comparison

7. Control and Security

7.1 Recommended authentication by scenario

7.2 Principle of least privilege with SAS

7.3 Environment variable for SAS in AzCopy

7.4 Logging and auditing in AzCopy

8. Decision Making

8.1 Tool selection by scenario

8.2 `copy` vs `sync`

9. Best Practices

10. Common Errors

11. Operation and Maintenance

11.1 Checking AzCopy version

11.2 AzCopy performance settings

11.3 Monitoring ongoing transfers

11.4 Relevant limits

12. Integration and Automation

12.1 AzCopy in Azure DevOps pipelines

12.2 AzCopy with GitHub Actions

12.3 Automated backup with cron (Linux)

12.4 Amazon S3 to Azure migration with AzCopy

13. Final Summary