AWS Integration

Flask-Philo supports basic Amazon Web Service (AWS) integration using Amazon’s boto3 AWS SDK.

More specifically, Flask-Philo provides methods for integration with two main services within Amazon’s AWS family:

  • S3 Storage - Integration with Amazon’s S3 file storage buckets, providing a number of useful methods for storing and retrieving data
  • SQS Queuing - Integration with Amazon’s Simple Queuing Service (SQS), with a number of methods for creating and managing queuing systems

Settings AWS Credentials

Flask-philo supports two means of authentication for AWS. Firstly, via environment variables:

$ export AWS_SECRET_ACCESS_KEY=your_secret_key
$ export AWS_ACCESS_KEY_ID=your_access_key

…or via configuration in the settings file:

<your_app>/config/development.py

AWS = {
    'credentials': {
        'aws_access_key_id': 'your_access_key',
        'aws_secret_access_key': 'your_secret_key'
    }
}

Amazon S3 Bucket

Flask-Philo supports the use of Amazon’s S3 file storage buckets, and provides a number of useful methods for storing and retrieving data.

Each S3 method may be called in two ways:

  • Imported and called as a standard Python function
  • Invoked from the command line using manage.py

…examples for both are included for every method documented below

Retrieving available Bucket contents

To list all available items within a specified S3 Bucket, we use the list_objects_v2 method

list_objects_v2(bucket_name, region_name)

  • bucket_name : Name of Amazon S3 Bucket
  • region_name : Name of Amazon S3 Region

Example Python calling code :

from flask_philo.cloud.aws.s3 import list_objects_v2

# Retrieve bucket content
bucket_name = 'my_data_bucket'
region_name = 'us-west-2'
bucket_content = list_objects_v2(bucket_name, bucket_region)['Contents']

# Print all bucket items
print("Bucket contents : ")
for bucket_item in bucket_content:
    print(bucket_item['Key'])

##########################
This code yields the following printed output :
Bucket contents :
readme.txt
13167621.mp3
18776371.mp3

This method may also be directly invoked from the command line as follows:

$ python3 manage.py aws s3 list_objects_v2 --bucket my_data_bucket --region us-west-2
+----+--------------+---------------------------+---------------+---------------+
|    |     Key      |       Last Modified       | Size (bytes)  | Storage Class |
+----+--------------+---------------------------+---------------+---------------+
| 1  | readme.txt   | 2018-04-11 15:56:52+00:00 |    1238522    |    STANDARD   |
| 2  | 13167621.mp3 | 2018-04-11 15:56:52+00:00 |    7110629    |    STANDARD   |
| 3  | 18776371.mp3 | 2018-04-11 15:57:07+00:00 |    8877935    |    STANDARD   |
+----+--------------+---------------------------+---------------+---------------+

Downloading a file from an S3 Bucket

Individual S3 data items may be retrieved using the download_file method

download_file(destination_filename, bucket_name, source_key, bucket_region)

  • destination_filename : Local, writable file location for downloaded file
  • bucket_name : Name of Amazon S3 Bucket
  • source_key : Amazon S3 key for the desired bucket item
  • region_name : Name of Amazon S3 Region

Example Python calling code :

from flask_philo.cloud.aws.s3 import download_file, list_objects_v2

# Retrieve first bucket item
bucket_name = 'my_data_bucket'
region_name = 'us-west-2'
bucket_item = list_objects_v2(bucket_name, region_name)['Contents'][0]

# Download bucket item to new file location "dest/my_new_local_file.txt"
download_file('dest/my_new_local_file.txt', bucket_name, bucket_item['Key'], region_name)

This method may also be directly invoked from the command line as follows:

$ python3 manage.py aws s3 download_file --bucket my_data_bucket --region us-west-2 --fname my_new_local_file.txt --key my_bucket_file.txt

Uploading a file to an S3 Bucket

Individual files may be uploaded to an S3 bucket using the upload_file method

upload_file(source_filename, bucket_name, destination_key, bucket_region)

  • source_filename : Local, readable file location as source of upload
  • bucket_name : Name of Amazon S3 Bucket
  • destination_key : New Amazon S3 key for the uploaded bucket item
  • region_name : Name of Amazon S3 Region

Example Python calling code :

from flask_philo.cloud.aws.s3 import upload_file, list_objects_v2

bucket_name = 'my_data_bucket'
region_name = 'us-west-2'

# Upload new file to S3 Bucket using Key 'My_New_File_Key'
upload_file('dest/my_new_local_file.txt', bucket_name, 'My_New_File_key', region_name)

This method may also be directly invoked from the command line as follows:

$ python3 manage.py aws s3 upload_file --bucket my_data_bucket --region us-west-2 --fname my_local_file.txt --key My_New_File_key.txt

Uploading a folder to an S3 Bucket

Bulk uploads of an entire directory’s contents is possible using the upload_dir method

upload_dir(source_dir, bucket_name, region_name)

  • source_dir : Local, readable directory containing all files for upload
  • bucket_name : Name of Amazon S3 Bucket
  • region_name : Name of Amazon S3 Region

Example Python code :

from flask_philo.cloud.aws.s3 import upload_dir

bucket_name = 'my_data_bucket'
region_name = 'us-west-2'
source_dir = './my_files/for_upload'
upload_dir(source_dir, bucket_name, region_name)

This method may also be directly invoked from the command line as follows:

$ python3 manage.py aws s3 upload_dir --bucket my_data_bucket --region us-west-2 --root_folder ./my_files/for_upload

Amazon Simple Queuing Service (SQS)

To facilitate task queueing between software components (e.g. between multiple decoupled microservices), Flask-Philo Integrates with Amazon’s Simple Queuing Service (SQS), with a number of methods for creating and managing message queuing systems.

For more detail on SQS message queuing, visit the SQS Introduction

Each SQS methods may be called in two ways:

  • Imported and called as a standard Python function
  • Invoked from the command line using manage.py

…examples for both are included for every method documented below

Sending a Message

Send a single message to a queue using the send_message method

send_message(queue_url, message_body, region_name)

  • queue_url : URL for SQS queue
  • message_body : Body of queue message
  • region : Name of Amazon S3 Region

Example Python code :

from flask_philo.cloud.aws.sqs import send_message

queue_url = 'https://us-west-2.queue.amazonaws.com/523525905522/new_test_queue'
message_body = 'My new test message'
region = 'us-west-2'
data = send_message(queue_url, message_body, region)

This method may also be directly invoked from the command line as follows:

$ python3 manage.py aws sqs send_message --queue_url https://us-west-2.queue.amazonaws.com/523525905522/new_test_queue --message_body "New Queue Message" --region us-west-2

execute_command :  aws
{
  'MD5OfMessageBody': '0ff14a859b7243d32b52126afe82eb68',
  'MessageId': '83dcc0b9-81bf-488d-a8a8-0941228dcec6',
  'ResponseMetadata': {
    'HTTPStatusCode': 200,
    'RetryAttempts': 0,
    'RequestId': '5a78a04b-e4b1-51e1-afa4-f3bc7dd7ac9e',
    'HTTPHeaders': {
      'x-amzn-requestid': '5a78a04b-e4b1-51e1-afa4-f3bc7dd7ac9e',
      'server': 'Server',
      'connection': 'keep-alive',
      'content-type': 'text/xml',
      'date': 'Wed, 22 Aug 2018 17:02:30 GMT',
      'content-length': '378'
    }
  }
}

Sending a Message Batch

Send multiple messages to a queue using the send_message_batch method

send_message_batch(queue_url, entries, region)

  • queue_url : URL for SQS queue
  • entries : List of message objects, in dictionary form
  • region : Name of Amazon S3 Region

Example Python code :

from flask_philo.cloud.aws.sqs import send_message_batch

url = 'https://us-west-2.queue.amazonaws.com/523525905522/new_test_queue'
region = 'us-west-2'
message_batch = [
    {"Id": "1", "MessageBody": "Test Message One"},
    {"Id": "2", "MessageBody": "Test Message Two"}
]

data = send_message_batch(queue_url=url, entries=message_batch, region=region)

This method may also be directly invoked from the command line as follows:

$ python3 manage.py aws sqs send_message_batch --queue_url https://us-west-2.queue.amazonaws.com/523525905522/new_test_queue --region us-west-2 --entries "[{\"Id\":\"1\",\"MessageBody\":\"[message one]\"},{\"Id\":\"2\",\"MessageBody\":\"[message two]\"}]"

execute_command :  aws
{
  'ResponseMetadata': {
    'RetryAttempts': 0,
    'RequestId': '9066cfe7-4504-5e4a-ac68-9c10dfa212cb',
    'HTTPHeaders': {
      'connection': 'keep-alive',
      'server': 'Server',
      'x-amzn-requestid': '9066cfe7-4504-5e4a-ac68-9c10dfa212cb',
      'date': 'Wed, 22 Aug 2018 17:07:50 GMT',
      'content-type': 'text/xml',
      'content-length': '734'
    },
    'HTTPStatusCode': 200
  },
  'Successful': [
    {
      'Id': '30b7b2c9-ac68-4541-af5b-64b24935c4c4',
      'MD5OfMessageBody': 'abd51c84e005843b6bffeb7a2d6526c7',
      'MessageId': 'a26c01a4-39d3-4381-91e4-17b4e3f04177'
    },
    {
      'Id': '1caac440-f795-4294-979b-ea9759ae6a47',
      'MD5OfMessageBody':
      'f0ca119551cf7532da54a23477fd39ff',
      'MessageId': '8c5cd9d4-6fae-4a2c-a5de-05fc4a6334a6'
    }
  ]
}

Retrieving Messages

To retrieve a single message from a queue, use the receive_message method

receive_message(queue_url, max_number_of_messages, region)

  • queue_url : URL for SQS queue
  • max_number_of_messages : Optional Specify number of message to be retrieved
  • region : Name of Amazon S3 Region

Example Python code :

from flask_philo.cloud.aws.sqs import receive_message

url = 'https://us-west-2.queue.amazonaws.com/523525905522/new_test_queue'
region_name = 'us-west-2'

retrieved_messages = receive_message(queue_url=url, region=region_name)
print("Body of message retrieved :", retrieved_messages['Messages'][0]['Body'])

##########################
This code yields the following printed output :
Body of message retrieved : My new test message

Optionally, we may retrieve more than one message at a time using the max_number_of_messages attribute

from flask_philo.cloud.aws.sqs import receive_message

url = 'https://us-west-2.queue.amazonaws.com/523525905522/new_test_queue'
region_name = 'us-west-2'

retrieved_messages = receive_message(queue_url=url, region=region_name, max_number_of_messages=2)

# Print all retrieved messages
print("Messages : ")
for message in retrieved_messages['Messages']:
    print(message['Body'])

##########################
This code yields the following printed output :
Messages :
Test Message One
Test Message Two

This method may also be directly invoked from the command line as follows:

$ python3 manage.py aws sqs receive_message --queue_url https://us-west-2.queue.amazonaws.com/523525905522/new_test_queue --region us-west-2

execute_command :  aws
{
  "ResponseMetadata": {
    "RetryAttempts": 0,
    "HTTPHeaders": {
      "date": "Wed, 22 Aug 2018 17:13:05 GMT",
      "connection": "keep-alive",
      "content-length": "863",
      "x-amzn-requestid": "45dbb41c-3cb1-599f-bc19-e7eba750f0e2",
      "server": "Server",
      "content-type": "text/xml"
    },
    "RequestId": "45dbb41c-3cb1-599f-bc19-e7eba750f0e2",
    "HTTPStatusCode": 200
  },
  "Messages": [
    {
      "ReceiptHandle": "AQEBptTr8Wns1LqPFVicLxUwobj2Nlk8nheH7r/f4H47KsP61iL7pwB6B1za9iCvHMooFB/JqX1dXEjd+JlGE9NYTGWhKz9dL8GqSrdmpJfLQmCxtprX83pNO3Xxrei8q81kQAhp0813/X8bxkxCboU/+c9xp83cJ/29fiQfP5nQTURHPMShysSoZ6UmEyvF5tAlrE28mr7WhpVGLs7birZPGEJFLB9cfESTXqqSAWNIbw+xDIXKS6E53hj37XktQ6juL+xKFUpEnIWympGCrFW09pigrotiA2Ysri3pjsc2ra4VY5UTk/L7EkTd0CymiyLdKcDVrYfW9bBYea9Jy2zCmi2tv7Bm/lpO1/ZLOdl/lU8X5uN7APGCMW0a5ba3RNHV/zbUWUmXpyqA6X3jaZvvGw==",
      "MessageId": "619b1306-fe8b-4289-9ef4-aa07577b407a",
      "Body": "Queue_item 1",
      "MD5OfBody": "4b83caab69a87b19e2566f1c4d0710d5"
    }
  ]
}

Listing Available Queues

To obtain a list of all available SQS queues grouped by region, use the list_queues method. Note that this method may take some time to return, given that it must iteratively poll all accessible Amazon SQS regions

list_queues()

Example Python code :

from flask_philo.cloud.aws.sqs import list_queues

queues_by_region = list_queues()
queue_url_list = queues_by_region['us-west-2']['QueueUrls']

print("Queue URLs :")
for queue_url in queue_url_list:
    print(queue_url)

##########################
This code yields the following printed output :
Queue URLs :
https://us-west-2.queue.amazonaws.com/523525905522/test_queue
https://us-west-2.queue.amazonaws.com/523525905522/new_test_queue
https://us-west-2.queue.amazonaws.com/523525905522/my-priority-list

This method may also be directly invoked from the command line as follows:

$ python3 manage.py aws sqs list_queues
+-----------------------------------------------------------------------------+-----------+
|                                    Queue Url                                |   Region  |
+-----------------------------------------------------------------------------+-----------+
|         https://us-west-2.queue.amazonaws.com/523522205522/test_queue       | us-west-2 |
|        https://us-west-2.queue.amazonaws.com/523525905522/new_test_queue    | us-west-2 |
|         https://us-west-2.queue.amazonaws.com/523525905522/my-priority-list | us-west-2 |
+-----------------------------------------------------------------------------+-----------+

Create a New Queue

To create new SQS queue, use the create_queue method

create_queue(queue_name, region)

  • queue_name : Name for new SQS queue
  • region : Name of Amazon S3 Region
from flask_philo.cloud.aws.sqs import create_queue

# Create new SQS queue
region_name = 'us-west-2'
new_queue = create_queue("my_test_queue", region_name)
sqs_url = new_queue['QueueUrl']

# Send a message to the new queue
data = send_message(sqs_url, 'Queue_item 1', region_name)

This method may also be directly invoked from the command line as follows:

$ python3 manage.py aws sqs create_queue --region us-west-2 --queue_name my-new-sqs-queue

execute_command :  aws
{
  "ResponseMetadata": {
    "RequestId": "426395bd-9e0d-5aee-b0f9-c64a02dd38ed",
    "RetryAttempts": 0,
    "HTTPHeaders": {
      "connection": "keep-alive",
      "content-type": "text/xml",
      "server": "Server",
      "date": "Wed, 22 Aug 2018 17:17:04 GMT",
      "x-amzn-requestid": "426395bd-9e0d-5aee-b0f9-c64a02dd38ed",
      "content-length": "338"
    },
    "HTTPStatusCode": 200
  },
  "QueueUrl": "https://us-west-2.queue.amazonaws.com/523525905522/my-new-sqs-queue"
}

Purge a Queue of all Messages

To purge an SQS queue of all messages, use the purge_queue method

purge_queue(queue_url, region)

  • queue_url : URL of an existing SQS queue
  • region : Name of Amazon S3 Region
from flask_philo.cloud.aws.sqs import purge_queue

# Purge the SQS queue
sqs_url = 'https://us-west-2.queue.amazonaws.com/523525901222/existing_sqs_queue'
region_name = 'us-west-2'
purge_queue(queue_url=sqs_url, region=region_name)

This method may also be directly invoked from the command line as follows:

$ python3 manage.py aws sqs purge_queue --region us-west-2 --queue_url https://us-west-2.queue.amazonaws.com/523525905522/my-new-sqs-queue

execute_command :  aws
{
  "ResponseMetadata": {
    "RetryAttempts": 0,
    "HTTPStatusCode": 200,
    "RequestId": "b3f869d0-cab1-54c9-9f76-3a2757f6d6ba",
    "HTTPHeaders": {
      "content-length": "209",
      "x-amzn-requestid": "b3f869d0-cab1-54c9-9f76-3a2757f6d6ba",
      "server": "Server",
      "connection": "keep-alive",
      "content-type": "text/xml",
      "date": "Wed, 22 Aug 2018 17:19:04 GMT"
    }
  }
}

Delete a Queue

To entirely delete an SQS queue and its messages, use the delete_queue method.

delete_queue(queue_url, region)

  • queue_url : URL of an existing SQS queue for deletion
  • region : Name of Amazon S3 Region
from flask_philo.cloud.aws.sqs import delete_queue

# Delete the SQS queue
sqs_url = 'https://us-west-2.queue.amazonaws.com/523525901222/existing_sqs_queue'
region_name = 'us-west-2'
delete_queue(queue_url=sqs_url, region=region_name)

This method may also be directly invoked from the command line as follows:

$ python3 manage.py aws sqs delete_queue --region us-west-2 --queue_url https://us-west-2.queue.amazonaws.com/523525905522/my-new-sqs-queue
execute_command :  aws
{
  "ResponseMetadata": {
    "RetryAttempts": 0,
    "HTTPHeaders": {
      "connection": "keep-alive",
      "date": "Wed, 22 Aug 2018 17:20:35 GMT",
      "server": "Server",
      "content-length": "211",
      "x-amzn-requestid": "947d5da5-6995-5cb0-b828-0e52f4368c8a",
      "content-type": "text/xml"
    },
    "HTTPStatusCode": 200,
    "RequestId": "947d5da5-6995-5cb0-b828-0e52f4368c8a"
  }
}

External Resources