Upload Files to AWS S3 with AWS API Gateway and AWS Lambda

In a world where serverless architecture is becoming more prevalent, automating workflows like uploading files to AWS S3 buckets through APIs is a common requirement. Let’s walk through the process of setting up a serverless application using AWS Lambda and API Gateway to upload binary files to an S3 bucket.This tutorial will guide you on how to achieve automating workflows and we’ll use Postman to test our endpoint.

Step 1: Create an IAM Role

Before we begin, ensure that AWS Lambda has the necessary permissions to interact with other AWS services such as S3 and API Gateway. Create an IAM role with the AWSLambdaExecute policy, which grants the Lambda function permission to access S3 and CloudWatch Logs.

1. Go to the IAM console.
2. Choose “Roles” and then “Create role”.
3. Select “Lambda” for the service that will use this role.
4. Attach the AWSLambdaExecute policy.
5. Name the role and create it.

Step 2: Create an AWS S3 bucket

Next, we need a destination for our file uploads:

1. Navigate to the S3 console.
2. Click “Create bucket”.
3. Provide a unique name for your bucket, such as news3api.
4. Uncheck the “Block all public access” option if you want the bucket to be public (be cautious with this setting in a production environment).

Step 3: Create a Lambda Function

The Lambda function will handle the file upload logic:

1. In the Lambda console, click “Create function”.
2. Choose “Author from scratch”.
3. Enter a name for your function, select Python as the runtime.
4. Change the default execution role to the IAM role you created earlier.
5. Click “Create function”.

With the function created, it’s time to add the code. Here’s a Python script that uploads a binary file to your S3 bucket:

Step 4: Create API Gateway

Now, let’s set up the API Gateway:

1. In the API Gateway console, click “Create API”.
2. Select “REST API” and click “Build”.
3. Create a new API or use an existing one.
4. Set up your new API with the ‘Regional’ endpoint type.

5. Create a new resource within your API named “Upload”.

6. Create a POST method for the “Upload” resource.
7. Select “Lambda Function” as the Integration type and link it to the Lambda function you’ve created.

8. Add permission to the Lambda function when prompted

9. Now edit the Post method that was created
10. Add mapping template content type to “application/pdf” and for template type add as follows:

For the API to handle binary data, you’ll need to set up binary support:

1. Under the API settings, add application/pdf to the binary media types.
2. Configure the content handling for the integration request to convert binary files to base64-encoded strings.
3. Deploy your API to a new or existing stage.

Testing the Endpoint with Postman

With the API deployed, you can test the file upload using Postman:

1. Open Postman and create a new request.
2. Set the method to POST and input the invoke URL of the API Gateway endpoint.
3. Append the resource path /upload to the URL.
4. In the Headers section, set Content-Type to application/pdf.
5. In the Body section, select ‘binary’ and choose the PDF file you want to upload.
6. Send the request.

If everything is set up correctly, you’ll receive a 200 status code in response, and the file will be uploaded to the specified S3 bucket.

Remember to secure your Lambda function and S3 bucket appropriately for production use and avoid making the bucket public unless necessary. This guide provides a basic overview, but you should follow AWS best practices for automating workflows and security and permissions.