Amazon AWS S3 List and download files with Python

If you want to list and download files from Amazon AWS S3 you can use python code to achieve it. The example code below demonstrate how to list and download files from your amazon bucket:

from boto3.session import Session
import boto3

ACCESS_KEY = 'MY_KEY'
SECRET_KEY = 'MY_SECRET'

session = Session(aws_access_key_id=ACCESS_KEY,
              aws_secret_access_key=SECRET_KEY)
s3 = session.resource('s3')
your_bucket = s3.Bucket('bucket_name')

for s3_file in your_bucket.objects.all():
print(s3_file.key)

my_bucket = boto3.resource('s3').Bucket('my.bucket.name')
my_bucket.download_file('server_file.txt', 'local_name.txt')

Amazon AWS S3 Errors and Solutions

An error occurred (404) when calling the HeadObject operation: Not Found

This error is related to donwloading files with AWS BOTO3 S3 python code. Usually the problems is that you target wrong path on the amazon server. The golden rule for using amazon AWS S3 is:

The relative path to S3 file for download from your bucket

example:

File is:

bucket.download_file(name, file)

* where name is: projectX/2018/myfile.txt
* file - is your local folder and file where is going to be donwloaded the amazon file

NoCredentialsError: Unable to locate credentials

This error is related to bad or not configured amazon client. The solution is simple - just run:

aws configure

or:

  • Create a file in ~/.aws/credentials
  • Add below content and save:
[default]
aws_access_key_id = YOUR_ACCESS_KEY
aws_secret_access_key = YOUR_SECRET_KEY

AttributeError: 'S3' object has no attribute 'Bucket'

This error is a bit different and is related to wrong usage of amazon buckets. You should use:

boto3.resource('s3').Bucket('mybucket')
or
s3 = boto3.resource('s3')

and not

boto3.client('s3').Bucket('mybucket')
# or
s3 = boto3.client('s3')

which will raise the error above.