> ## Documentation Index
> Fetch the complete documentation index at: https://docs.stigg.io/llms.txt
> Use this file to discover all available pages before exploring further.

# Ingestion from AWS S3

Stigg is able to automatically collect events for processing as they arrive into an S3 bucket.

Using an S3 bucket as the data source of events can be useful in the following cases:

1. Unloading data from a data warehouse into an S3 bucket.
2. Report events in very large volume.

There are many ways to write to S3. You can use AWS S3 SDK, AWS Glue, Logstash, Fluentd and other tools.

<Note>
  Ingestion from S3 available in Stigg’s Scale plan. See [**Stigg’s pricing**](https://www.stigg.io/pricing) for more details.
</Note>

## Event format

This S3 integration requires that events are in the shape of the event API [schema](./index#event-schema), the format of the files can be either `.jsonl` or `.csv`.

For fields that you want to include as part of the `dimensions` object, just add the `dimensions.` prefix, for example `dimensions.storage_bytes`.

### JSONL Example

<CodeGroup>
  ```json JSONL theme={null}
  {"idempotencyKey": "8c1cc9f1-5a04-4405-9eb9-daffdcf96e83", "eventName": "storage", "customerId": "customer-demo-1", "timestamp": "2023-04-03T09:56:50.902462Z", "dimensions.region": "us-east-2", "dimensions.storage_bytes": 1000}
  {"idempotencyKey": "9d14d2b7-48a7-4e89-9aa0-fe5a66801e16", "eventName": "storage", "customerId": "customer-demo-1", "timestamp": "2023-04-04T09:56:50.902462Z", "dimensions.region": "us-east-1", "dimensions.storage_bytes": 300}
  {"idempotencyKey": "f3b4973d-0d5f-4097-92c6-b673e0bcd0a6", "eventName": "storage", "customerId": "customer-demo-1", "timestamp": "2023-04-05T09:56:50.902462Z", "dimensions.region": "us-west-1", "dimensions.storage_bytes": 450}
  ```
</CodeGroup>

### CSV Example

<CodeGroup>
  ```sql CSV theme={null}
  "idempotencyKey","eventName","customerId","timestamp","dimensions.region","dimensions.storage_bytes"
  "8c1cc9f1-5a04-4405-9eb9-daffdcf96e83","storage","customer-demo-1","2023-04-03T09:56:50.902462Z","us-east-2","1000"
  "9d14d2b7-48a7-4e89-9aa0-fe5a66801e16","storage","customer-demo-1","2023-04-04T09:56:50.902462Z","us-east-1","300"
  "f3b4973d-0d5f-4097-92c6-b673e0bcd0a6","storage","customer-demo-1","2023-04-05T09:56:50.902462Z","us-west-1","450"
  ```
</CodeGroup>

## Integration setup

Create 2 separate S3 buckets:

1. S3 bucket for raw events - This bucket will include the raw events and Stigg automatically pull events from.
2. S3 Dead-letter bucket - in case of ingestion failure due to parsing or validation error, Stigg will write the events back to this S3 bucket.

Grant Stigg [cross-account access](https://repost.aws/knowledge-center/cross-account-access-s3) to S3 bucket - our team will provide you with the IAM role which will be used for Stigg to access the S3 buckets.

#### Policy for raw events S3 bucket

<CodeGroup>
  ```json JSON theme={null}
  {
      "Version": "2012-10-17",
      "Statement": [
          {
              "Effect": "Allow",
              "Principal": {
                  "AWS": "arn:aws:iam::<STIGG_ACCOUNT_ID>:role/<STIGG_PROVISIONED_ROLE>"
              },
              "Action": "s3:ListBucket",
              "Resource": "arn:aws:s3:::<RAW_EVENTS_BUCKET>"
          },
          {
              "Effect": "Allow",
              "Principal": {
                  "AWS": "arn:aws:iam::<STIGG_ACCOUNT_ID>:role/<STIGG_PROVISIONED_ROLE>"
              },
              "Action": [
                  "s3:GetObject",
                  "s3:GetObjectAcl"
              ],
              "Resource": "arn:aws:s3:::<RAW_EVENTS_BUCKET>/*"
          }
      ]
  }
  ```
</CodeGroup>

#### Policy for dead-letter S3 bucket

<CodeGroup>
  ```json JSON theme={null}
  {
      "Version": "2012-10-17",
      "Statement": [
          {
              "Effect": "Allow",
              "Principal": {
                  "AWS": "arn:aws:iam::<STIGG_ACCOUNT_ID>:role/<STIGG_PROVISIONED_ROLE>"
              },
              "Action": "s3:ListBucket",
              "Resource": "arn:aws:s3:::<RAW_EVENTS_DLQ_BUCKET>"
          },
          {
              "Effect": "Allow",
              "Principal": {
                  "AWS": "arn:aws:iam::<STIGG_ACCOUNT_ID>:role/<STIGG_PROVISIONED_ROLE>"
              },
              "Action": [
                  "s3:GetObject",
                  "s3:GetObjectAcl",
                  "s3:PutObject",
                  "s3:PutObjectAcl"
              ],
              "Resource": [
                  "arn:aws:s3:::<RAW_EVENTS_DLQ_BUCKET>/*"
              ]
          }
      ]
  }
  ```
</CodeGroup>

Stigg uses [S3 event notifications](https://docs.aws.amazon.com/AmazonS3/latest/userguide/EventNotifications.html) in order to receive notifications when objects are created in the S3 bucket.

Once you finish setting up the policy above, Stigg will add permissions to an SQS queue to allow S3 to write to it.

In order to get notifications from S3 bucket, the `s3:objectCreated:*` event type must be configured via the [AWS console](https://docs.aws.amazon.com/AmazonS3/latest/userguide/enable-event-notifications.html).

This SQS queue will be in the same region as your bucket, so Stigg will provide you with the provisioned SQS ARN once the bucket region is known.