Configuring S3 bucket in another account to notify SQS or SNS

For customers who have access to a data bucket and want an event-driven model to catch new files in a data delivery bucket.

An important consideration is there is no UI in the AWS console to manage this. You will need to use the CLI, code, CloudFormation, etc, to implement this action.

In this case, we want to publish s3:ObjectCreated:* events in a Synoptic data bucket to a SQS in your account.

This guide largely covers this requirement https://docs.aws.amazon.com/AmazonS3/latest/userguide/ways-to-add-notification-config-to-bucket.html

Prerequisites

In establishing access, you give Synoptic the AWS account ID of your account, and we grant both access to read/list objects, as well as permissions to put and get notifications.

Steps required

1. Create the queue - you can do this in the console. There is no need to grant access to our provider account, as publishes come from the S3 service, not our account. This is addressed in the next step
   

2. Update the queue policy to permit s3 from the source bucket to publish to your queue (in addition to any other access policies on your queue or topic). Following least privilege principles you may want to limit actions to post messages only.

   CODE

   {
         "Sid": "s3_publisher_statement",
         "Effect": "Allow",
         "Principal": {
           "Service": "s3.amazonaws.com"
         },
         "Action": "SQS:*",
         "Resource": "YOUR_QUEUE_ARN",
         "Condition": {
           "ArnLike": {
             "aws:SourceArn": "SYNOPTIC_BUCKET_ARN"
           }
         }
       }

3. Use the CLI to put a bucket notification/queue event on the Synoptic-owned bucket https://docs.aws.amazon.com/cli/latest/reference/s3api/put-bucket-notification-configuration.html
   With sqs-config.json being the SQS configuration specification

   JSON

   {"QueueConfigurations":[
     {"QueueArn":"YOUR_QUEUE_ARN",
     "Events":["s3:ObjectCreated:*"]}
   ]}

4. from the CLI use this command to place the bucket notification rule on our bucket

   CODE

   aws s3api put-bucket-notification-configuration \
   --bucket [SYNOPTIC_BUCKET_NAME] \
   --notification-configuration file://sqs-config.json

With these steps completed, your SNS or SQS should begin to receive notifications immediately when new files are published to the bucket, and you can invoke lambdas or other events to consume and act on the data.

If your access is disabled in the future, notifications may persist for after access is withdrawn, but you will lack the ability to capture the objects indicated in the messages.