Skip to content

Useful AWS CLI Commands

Jump to bottom
shamik edited this page Apr 25, 2024 · 1 revision

Install AWS CLI

https://docs.aws.amazon.com/cli/latest/userguide/getting-started-install.html

curl "https://awscli.amazonaws.com/awscli-exe-linux-x86_64.zip" -o "awscliv2.zip" 
unzip awscliv2.zip 
sudo ./aws/install

Configure AWS CLI

aws configure

Enable CLI autocomplete

aws --cli-auto-prompt

Disable CLI autocomplete

aws --no-cli-auto-prompt

Listing all S3 buckets

aws s3 ls

Listing all files in a s3 dir

https://awscli.amazonaws.com/v2/documentation/api/latest/reference/s3/ls.html

aws s3 ls <s3 dir>  --summarize --human-readable
#aws s3 ls s3://a250588-amplify-research-eu-west-1/research/ESG/numeric_attributes/  --summarize --human-readable

Downloading a file from S3

https://awscli.amazonaws.com/v2/documentation/api/latest/reference/s3/cp.html

aws s3 cp <s3 filepath> <local dir path>
# aws s3 cp s3://esg-anlp/amplify/predictions/preds_test.csv ~/Repos/iTOP-semantic-apply-model/notebooks/metrics/

Downloading all the content of a folder in s3, this will automatically sync files from s3

https://awscli.amazonaws.com/v2/documentation/api/latest/reference/s3/sync.html

aws s3 sync <s3 dir> <local dir path>
# aws s3 sync s3://esg-anlp/amplify/predictions/ Repos/iTOP-semantic-apply-model/notebooks/metrics/

Another option is to copy recursively all files from the s3 dir.

aws s3 cp <s3 dir path> <local dir path> --recursive

Adding the --dryrun flag will check what the command is going to do.

Uploading all the content of a folder in s3, this will automatically sync files from s3

https://awscli.amazonaws.com/v2/documentation/api/latest/reference/s3/sync.html

aws s3 sync <local dir path> <s3 dir>
# aws s3 sync Repos/iTOP-semantic-apply-model/notebooks/metrics/ s3://esg-anlp/amplify/predictions/

Another option is to copy recursively all files from the s3 dir.

aws s3 cp <local dir path>  <s3 dir path> --recursive

Adding the --dryrun flag will check what the command is going to do.

Enabling bucket acceleration for faster upload speeds

https://docs.aws.amazon.com/AmazonS3/latest/userguide/transfer-acceleration-examples.html

aws s3api put-bucket-accelerate-configuration --bucket esg-anlp --accelerate-configuration Status=Enabled

Python API to download files from S3

Connecting to an EC2 instance

ssh -i {key pair} <machine name:path>
# ssh -i shamik.pem [email protected]:~

Copying a file to EC2

scp -i shamik.pem -rC <file path> <machine name:path>
# scp -i shamik.pem -rC shamik/Finetuning-transformers-lightning/shap_test.ipynb [email protected]:~

Copying a file from EC2 to local dir

scp -i shamik.pem -rC <machine name:path/to/file> <local path>
# scp -i shamik.pem -rC [email protected]:~/bambino.txt shamik/

https://linuxhandbook.com/scp-command/#:~:text=Copying%20an%20entire%20directory%20with%20scp%20command%20Copying,source%20path.%20This%20is%20called%20the%20recursive%20mode.

https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/connect-linux-inst-ssh.html

Transfering an entire directory to EC2

scp -i shamik.pem -rC <local dir> <machine name:path>
# scp -i shamik.pem -rC ../Codes/summ_eval/ [email protected]:~/Summarisation/Codes

SCP between EC2 instances with pem file

scp -r -i shamik.pem <dir>  <machine name with private IP:dir path> 
# scp -r -i shamik.pem roberta-large-iter2-V2/  [email protected] 6.63://data/yamini/3scale_experiments/section_wise_models/models/ 

# The IP address used here for connecting to another EC2 instance is private
Clone this wiki locally