-
Notifications
You must be signed in to change notification settings - Fork 3.9k
Microsoft Azure Storage Cmdlets
Johann Dirry edited this page Mar 7, 2017
·
2 revisions
One of the frequent requests we receive is to allow IT administrator use script to automate some most common operations against storage resources such as containers and blobs. By using Windows Azure Storage Cmdlets, we enable IT administrator to create, list or delete containers and blobs, upload or download blobs, transfer blobs across containers, storage accounts or azure subscriptions.
To use windows azure storage cmdlets you need supply the credential to access the storage resources, you can setup one of the following
(1) Import your Azure Subscription
To configure a default Windows Azure subscription, and set a default storage account please follow the instructions at Get Started with Windows Azure Cmdlets
(2) Configure the environment variable AZURE_STORAGE_CONNECTION_STRING
Sample of Connection String - DefaultEndpointsProtocol=https;AccountName=<MyAccountName>;AccountKey=<MyAccountKey>
(3) Use New-AzureStorageContext cmdlet to supply your credential in runtime
SYNTAX
New-AzureStorageContext [-StorageAccountName] <String> [-StorageAccountKey] <String> [-Protocol <String>] [<CommonParameters>]
New-AzureStorageContext [-StorageAccountName] <String> -Anonymous [-Protocol <String>] [<CommonParameters>]
New-AzureStorageContext -ConnectionString <String> [<CommonParameters>]
New-AzureStorageContext –Local [<CommonParameters>]
* Parameters
–StorageAccount Name: specify the resource owner’s storage account name
-Anonymous: specify if you want to access the storage resources as anonymous user, only public available resources can be accessed by anonymous user
–Local: specify for storage emulator
-<CommonParameters> This cmdlet supports the common parameters: -Verbose, -Debug, -ErrorAction, -ErrorVariable, -OutBuffer, and -OutVariable. For more information, see about_CommonParameters
* Sample Usage
New-AzureStorageContext -StorageAccountName <MyAccountName> -StorageAccountKey <MyAccountKey> -Protocol https
New-AzureStorageContext -ConnectionString " DefaultEndpointsProtocol=https;AccountName=<MyAccountName>;AccountKey=<MyAccountKey>"
New-AzureStorageContext -Local
New-AzureStorageContext -StorageAccountName <MyAccountName> -AnonymousSYNTAX
New-AzureStorageContainer [-Name] <String> [[-Permission] <String>] [-Context <AzureStorageContext>] [<CommonParameters>]
* Parameters
–Name: specify the name of container you plan to create
–Permission: defines the Public Access level of this container, by default, the container and any blobs within it may be accessed only by the owner of the storage account. If you want to give anonymous users read permissions to a container and its blobs, you can set the container permissions to allow public access. Anonymous users can read blobs within a publicly accessible container without authenticating the request.
You have following options for managing container access:
Full public read access: Use the value of “Container”, this means container and blob data can be read via anonymous request. Clients can enumerate blobs within the container via anonymous request, but cannot enumerate containers within the storage account.
Public read access for blobs only: Use the value of “Blob”, this means blob data within this container can be read via anonymous request, but container data is not available. Clients cannot enumerate blobs within the container via anonymous request.
No public read access: By default Container and blob data can be read by the account owner only.
Please refer to this blog post on how to restrict access to Container and Blob
* Sample Usage
Create a new full public read access container
New-AzureStorageContainer –Name testcontainer –Permission Container
Create a new public read access for blobs only container
New-AzureStorageContainer –Name testcontainer –Permission Blob
Create a private container, no public read access
New-AzureStorageContainer –Name testcontainerSYNTAX
Get-AzureStorageContainer [[-Name] <String>] [-Context <AzureStorageContext>] [<CommonParameters>]
Get-AzureStorageContainer -Prefix <String> [-Context <AzureStorageContext>] [<CommonParameters>]
* Parameters
–Name: specify the name of the container
-Prefix: specify the prefix to use to list containers that match the prefix
* Sample Usage
List all the containers where the name starts with “test”
Get-AzureStorageContainer -prefix testSYNTAX
Remove-AzureStorageContainer [-Name] <String> [-Force] [-Context <AzureStorageContext>] [-WhatIf] [-Confirm] [<CommonParameters>]
* Parameters
–Name: specify the name of the container
–Force: Execute the command without prompting the user
–WhatIf: Describes what would happen if you executed the command without actually executing the command
–Confirm Prompts you for confirmation before executing the command
* Note: remove one container will also remove all blobs in this container including snapshots
* Sample Usage
Remove all containers including the blobs from all container
Get-AzureStorageContainer | Remove-AzureStorageContainerSYNTAX
Set-AzureStorageContainerAcl [-Name] <String> [-Permission] <String> [-PassThru] [-Context <AzureStorageContext>] [<CommonParameters>]
* Parameters
–Name: specify the name of the container
–Permission: specify the PublicAccess level of this container, Off means no public access allowed, you can also specify the value as Blob or Container
–Passthru: Specify if you want see the output of this Set cmdlet, if not specified, there will be no output by default
* Sample Usage
Set all the container’s public access level as Off so that anonymous user won’t be able to access
Get-AzureStorageContainer | Set-AzureStorageContainerAcl –Permission OffSYNTAX
Get-AzureStorageBlob [[-Blob] <String>] [-Container] <String> [-Context <AzureStorageContext>] [<CommonParameters>]
Get-AzureStorageBlob [-Prefix <String>] [-Container] <String> [-Context <AzureStorageContext>] [<CommonParameters>]
* Parameters
–Name: specify the name of the container
-Prefix: specify the prefix to use to list containers that match the prefix
* Sample Usage
List all blobs from all containers
Get-AzureStorageContainer | Get-AzureStorageBlobSYNTAX
Remove-AzureStorageBlob [-Blob] <String> [-Container] <String> [-DeleteSnapshot] [-Force] [-Context <AzureStorageContext>] [<CommonParameters>]
Remove-AzureStorageBlob -ICloudBlob <ICloudBlob> [-DeleteSnapshot] [-Force] [-Context <AzureStorageContext>] [<CommonParameters>]
Remove-AzureStorageBlob -CloudBlobContainer <CloudBlobContainer> [-Blob] <String> [-DeleteSnapshot] [-Force] [-Context <AzureStorageContext>] [<CommonParameters>]
* Parameters
-Blob: specify the name of the blob you’d like to remove
-DeleteSnapshot: specify if you’d like to delete all snapshots together with the base blob, by default blob snapshots won’t be deleted even the base blob has been removed
* Sample Usage
Remove all blobs from all containers including snapshots
Get-AzureStorageContainer | Get-AzureStorageBlob | Remove-AzureStorageBlob –DeleteSnapshotSYNTAX
Set-AzureStorageBlobContent [-File] <String> [-Container] <String> [-Blob <String>] [-BlobType <String>] [-Properties <Hashtable>] [-Metadata <Hashtable>] [-ConcurrentTaskCount <Int32>] [-Force] [-Context <AzureStorageContext>] [<CommonParameters>]
Set-AzureStorageBlobContent [-File] <String> -ICloudBlob <ICloudBlob> [-BlobType <String>] [-Properties <Hashtable>] [-Metadata <Hashtable>] [-ConcurrentTaskCount <Int32>] [-Force] [-Context <AzureStorageContext>] [<CommonParameters>]
Set-AzureStorageBlobContent [-File] <String> [-Blob <String>] -CloudBlobContainer <CloudBlobContainer> [-BlobType <String>] [-Properties <Hashtable>] [-Metadata <Hashtable>] [-ConcurrentTaskCount <Int32>] [-Force] [-Context <AzureStorageContext>] [<CommonParameters>]
* Parameters
–File: specify the location of local file you’d like to upload
–Container: specify the destination container name
–Blob: specify the blob name, if not specify it will use the source file name
–BlobType: specify the blob type, it can be Block or Page, default is block blob
–Properties: specify the property of the blob
–Metadata: specify the meta data of the blob
–ConcurrentTaskCount: specify the concurrent network calls. By default, when you upload files from local computer to Windows Azure Storage, this cmdlet will initiate network calls up to eight times the number of cores this local computer had to execute concurrent tasks. For example, if your local computer has four cores, this cmdlet will initiate up to 32 (eight times of 4) network calls at one time. However, if you want to limit the concurrency to throttle local CPU and bandwidth usage, you can specify the maximum concurrent network calls by using -ConcurrentTaskCount. The value specified here is the absolute count and will not be multiplied by the core count
* Sample Usage
Upload c:\foo\1.txt to testcontainer and rename it to new.txt
Set-AzureStorageBlobContent -File C:\foo\1.txt -Container testcontainer -Blob new.txt
Upload all files from c:\foo to testcontainer with 16 concurrent network calls
ls c:\foo | Set-AzureStorageBlobContent –Container testcontainer –ConcurrentTaskCount 16SYNTAX
Get-AzureStorageBlobContent [-Blob] <String> [-Container] <String> [-Destination <String>] [-CheckMd5] [-ConcurrentTaskCount <Int32>] [-Force] [-Context <AzureStorageContext>] [<CommonParameters>]
Get-AzureStorageBlobContent -ICloudBlob <ICloudBlob> [-Destination <String>] [-CheckMd5] [-ConcurrentTaskCount <Int32>] [-Force] [-Context <AzureStorageContext>] [<CommonParameters>]
Get-AzureStorageBlobContent -CloudBlobContainer <CloudBlobContainer> [-Blob] <String> [-Destination <String>] [-CheckMd5] [-ConcurrentTaskCount <Int32>] [-Force] [-Context <AzureStorageContext>] [<CommonParameters>]
* Parameters
-Blob: specify the source blob name you’d like to download
–Container: specify the source container name
–CheckMd5: enable MD5 check when blob downloaded to destination, by default it is off
* Sample Usage
Download all files from testcontainer to c:\foo
Get-AzureStorageBlob -Container testcontainer | Get-AzureStorageBlobContent -Destination c:\gooSYNTAX
Start-CopyAzureStorageBlob [-SrcBlob] <String> [-SrcContainer] <String> [-Context] <AzureStorageContext> -DestContainer <String> [-DestBlob <String>] [-Context <AzureStorageContext>] [-DestContext <AzureStorageContext>] [-Force] [<CommonParameters>]
Start-CopyAzureStorageBlob -ICloudBlob <ICloudBlob> -DestContainer <String> [-DestBlob <String>] [-Context <AzureStorageContext>] [-DestContext <AzureStorageContext>] [-Force] [<CommonParameters>]
Start-CopyAzureStorageBlob -ICloudBlob <ICloudBlob> -DestICloudBlob <ICloudBlob> [-Context <AzureStorageContext>] [-DestContext <AzureStorageContext>] [-Force] [<CommonParameters>]
Start-CopyAzureStorageBlob -CloudBlobContainer <CloudBlobContainer> [-SrcBlob] <String> -DestContainer <String> [-DestBlob <String>] [-Context <AzureStorageContext>] [-DestContext <AzureStorageContext>] [-Force] [<CommonParameters>]
Start-CopyAzureStorageBlob -SrcUri <String> -DestContainer <String> [-DestBlob <String>] [-DestContext <AzureStorageContext>] [-Force] [<CommonParameters>]
* Parameters
–SrcBlob: specify the source blob name
–SrcContainer: specify the source container name
–SrcContext: supply the source context if the source URI pointed to a blob in Azure Storage, it won’t be needed if the source URI pointed to a public accessible blob or file out of Azure Storage
–DestBlob: specify the destination blob name
–DestContainer: specify the destination container name
–SrcUri: specify the source URL to copy file from non-Azure repository to Azure Storage
* Sample Usage
Copy 1.txt from container1 to container2
Start-CopyAzureStorageBlob -srcblob 1.txt -srccontainer container1 -destcontainer container2 –destblob 1.txt
Copying all disks to another subscription
$destContext = New-AzureStorageContext –StorageAccountName name -StorageAccountKey key
Get-AzureDisk | foreach
{
$src = $_.MediaLink
$fileName = $_.DiskName + '.vhd'
$destContainer = 'copiedvhds/’
Start-CopyAzureStorageBlob -SrcUri $source -destContainer $destContainer -destblob $fileName –destContext $destContext
if($_.OS -ne $null) {
Add-AzureDisk -DiskName $fileName -OS $.OS -MediaLocation $dest
}
else {
Add-AzureDisk -DiskName $fileName -OS $.OS -MediaLocation $dest
}
}
Copy file from public URI to Azure Storage
Start-CopyAzureStorageBlob -srcuri <URI of a Public File> -destcontainer testcontainer -destblob MyFile
* IaaS Samples
Piping output from a disk to be copied
(Get-Azuredisk -DiskName 'myosdisk').MediaLink.ToString() | Start-CopyAzureStorageBlob -Context $sourcecontext -DestContainer $destcontainer –DestContext $destcontext -DestBlob $destblobnameSYNTAX
Stop-CopyAzureStorageBlob [-Blob] <String> [-Container] <String> [-Force] [-CopyId <String>] [-Context <AzureStorageContext>] [<CommonParameters>]
Stop-CopyAzureStorageBlob -ICloudBlob <ICloudBlob> [-Force] [-CopyId <String>] [-Context <AzureStorageContext>] [<CommonParameters>]
Stop-CopyAzureStorageBlob -CloudBlobContainer <CloudBlobContainer> [-Blob] <String> [-Force] [-CopyId <String>] [-Context <AzureStorageContext>] [<CommonParameters>]
* Parameters
–Blob: specify the destination blob name of the blob copy operation
–Container: specify the destination container name of the blob copy operation
* Sample Usage
Stop the ongoing blob copy
Stop-CopyAzureStorageBlob -Blob test.vhd –Container vhdcollectionSYNTAX
Get-AzureStorageBlobCopyState [-Blob] <String> [-Container] <String> [-WaitForComplete] [-Context <AzureStorageContext>] [<CommonParameters>]
Get-AzureStorageBlobCopyState -ICloudBlob <ICloudBlob> [-WaitForComplete] [-Context <AzureStorageContext>] [<CommonParameters>]
Get-AzureStorageBlobCopyState -CloudBlobContainer <CloudBlobContainer> [-Blob] <String> [-WaitForComplete] [-Context <AzureStorageContext>] [<CommonParameters>]
* Parameters
–Blob: specify the destination blob name of the blob copy operation
–Container: specify the destination container name of the blob copy operation
–WaitForComplete: wait until the copy complete
* Sample Usage
Copying multiple blobs in async mode and monitor their copy state
$copiedBlobs = Get-AzureStorageBlob -Container container1 | Start-CopyAzureStorageBlob -DestContainer container2
$copiedBlobs | Get-AzureStorageBlobCopyState –WaitForCompleteNote - if you hit timeout exception while uploading or downloading file from Azure Storage by using storage cmdlets, please try set -ConcurrentTaskCount 1 in your command line and this may fix the issue.
This is because we are using multithread to utilize the network bandwidth, but sometimes the network would be restricted in customer’s local environment, So you may see the timeout. If this works, the next step you can try is increasing this value from 1 to higher number (2,4,6,8) to stress your bandwidth and get the best result.