> ## Documentation Index
> Fetch the complete documentation index at: https://docs.stigg.io/llms.txt
> Use this file to discover all available pages before exploring further.

# Delta Lake

## Overview

Stigg's integration with Delta Lake allows you to export your product catalog, customer, and subscription data as Delta Lake tables in your own cloud storage — AWS S3, Google Cloud Storage, or Azure — for use with Spark, Databricks, Snowflake, and other lakehouse engines.

<Note>
  Stigg's native data export integrations are included in the Scale plan, and are also available as an optional add-on to the Growth plan. See Stigg's pricing for more details.
</Note>

<Note>
  Delta Lake is currently a beta destination for sync volumes exceeding 1 billion rows per month. [Contact Stigg support](mailto:support@stigg.io) if you expect to exceed that volume.
</Note>

<Card title="View the full entity schema" icon="sitemap" href="./schema">
  See every table and column exported to your destination, organized by entity group.
</Card>

## Setting up the integration

### Prerequisites

* Cloud storage you control — an AWS S3 bucket, a Google Cloud Storage bucket, or an Azure storage account with hierarchical namespace enabled
* Depending on your provider: an IAM role or access key (AWS), a service account (Google Cloud), or a storage access key (Azure) with read/write access to the target bucket or container

<Note>
  This guide covers **AWS S3** as the primary flow. If your storage is on **Google Cloud** or **Azure**, see [Other storage backends](#other-storage-backends) below instead.
</Note>

<Steps>
  <Step title="Create the destination S3 bucket">
    In the AWS S3 console, click **Create bucket**. Enter a **Bucket name**. AWS recommends setting **Object Ownership** to **ACLs disabled** and **Block Public Access settings for this bucket** to **Block all public access**.

    Click **Create bucket**, and note the bucket name and AWS Region.
  </Step>

  <Step title="Create an IAM policy and role">
    In the AWS IAM console, create a policy that grants access to the bucket, replacing `BUCKET_NAME` with the name from the previous step:

    <CodeGroup>
      ```json Access policy theme={null}
      {
        "Version": "2012-10-17",
        "Statement": [
          {
            "Effect": "Allow",
            "Action": "s3:ListBucket",
            "Resource": "arn:aws:s3:::BUCKET_NAME"
          },
          {
            "Effect": "Allow",
            "Action": [
              "s3:GetObject",
              "s3:PutObject",
              "s3:DeleteObject"
            ],
            "Resource": "arn:aws:s3:::BUCKET_NAME/*"
          }
        ]
      }
      ```
    </CodeGroup>

    <Note>
      If your bucket uses KMS encryption (CMK), also add a statement granting `kms:GenerateDataKey` and `kms:Decrypt` on the key, replacing `REGION_NAME`, `ACCOUNT_ID`, and `KEY_ID`:

      ```json theme={null}
      {
        "Effect": "Allow",
        "Action": ["kms:GenerateDataKey", "kms:Decrypt"],
        "Resource": "arn:aws:kms:REGION_NAME:ACCOUNT_ID:key/KEY_ID"
      }
      ```

      SSE-C encryption isn't currently supported.
    </Note>

    Then create a role for Stigg to use:

    * **IAM Role (recommended)** — create a role with a **Custom trust policy**, pasting in the trust policy shown in the Stigg connect form to allow `AssumeRole` access. Attach the policy created above, then note the role's **ARN**.
    * **HMAC Access Key** — alternatively, create an IAM user with **Programmatic access**, attach the policy created above, and note the generated **Access key ID** and **Secret access key**.
  </Step>

  <Step title="Connect Stigg with Delta Lake">
    In [Stigg](https://app.stigg.io/), navigate to **Integrations > Apps > Delta Lake**.

    Enter the following information in the connection form:

    | Field         | Description                                                                                                |
    | ------------- | ---------------------------------------------------------------------------------------------------------- |
    | Bucket name   | The name of your S3 bucket                                                                                 |
    | Bucket region | The AWS region where the bucket is located                                                                 |
    | Role ARN      | The ARN of the IAM role created above (or the HMAC access key ID and secret, if using that method instead) |

    Click **Test & connect**.
  </Step>

  <Step title="Select entities to export">
    After entering your connection details, expand the **Entities to export** section to choose which entity groups to include in the sync. All groups are selected by default.

    See [Exported entities](./overview#exported-entities) for a description of each group.
  </Step>
</Steps>

## Other storage backends

### Google Cloud

**Prerequisites:** A Google Cloud Storage bucket, and a GCP service account Stigg can access — either via impersonation (recommended) or a service account key.

1. In the GCP console, go to **IAM & Admin > Service Accounts** and create a service account. In **Grant users access to this service account**, enter the service account shown in the Stigg connect form, then note the email of the service account you just created.
2. Grant access using one of:
   * **Service account impersonation (recommended)** — on the **Permissions** tab, edit the principal from step 1 and add the **Service Account Token Creator** role.
   * **Service account key** — under **Keys**, create and download a JSON key instead.
3. In **Cloud Storage**, create a bucket and note the bucket name. On its **Permissions** tab, grant your service account the **Storage Legacy Bucket Writer**, **Storage Legacy Bucket Reader**, and **Storage Legacy Object Reader** roles.
4. In [Stigg](https://app.stigg.io/), navigate to **Integrations > Apps > Delta Lake** and provide:

   | Field                 | Description                                                  |
   | --------------------- | ------------------------------------------------------------ |
   | Bucket name           | The name of your GCS bucket                                  |
   | Folder name           | The folder path within the bucket where data will be written |
   | Service account email | The email of the service account you created in step 1       |

### Azure

**Prerequisites:** An Azure storage account with hierarchical namespace enabled (Data Lake Storage Gen2), and an access key.

1. In the Azure portal, create a storage account. Under **Advanced**, keep **Enable storage account key access** on and enable **Enable hierarchical namespace** under Data Lake Storage Gen2. Under **Data protection**, turn off **Enable soft delete for blobs**, **Enable soft delete for containers**, and **Enable soft delete for file shares**.
2. Create a container within the storage account.
3. Under **Security + networking > Access keys**, note the **Key** value.
4. In [Stigg](https://app.stigg.io/), navigate to **Integrations > Apps > Delta Lake** and provide:

   | Field                | Description                                                     |
   | -------------------- | --------------------------------------------------------------- |
   | Storage account name | The name of your Azure storage account                          |
   | Container name       | The name of the Blob Storage container                          |
   | Folder name          | The folder path within the container where data will be written |
   | Access key           | The access key generated above                                  |

Whichever backend you use, finish by clicking **Test & connect**, then expand **Entities to export** on the connect form to choose which entity groups to include in the sync — see [Exported entities](./overview#exported-entities) for a description of each group.

## Configuration options

<Warning>
  Changing these settings on an existing destination table won't take effect until you perform a full refresh of the table.
</Warning>

| Protocol MinReader Version | Column mapping mode | Deletion vectors         | Change data feed |
| -------------------------- | ------------------- | ------------------------ | ---------------- |
| 1                          | `NONE`              | Disabled (copy-on-write) | Disabled         |
| 2                          | `NONE`              | Disabled (copy-on-write) | Disabled         |
| 3                          | `ID`                | Enabled (merge-on-read)  | Enabled          |

Match your configuration to the protocol version required by your downstream readers (for example, Athena requires version 1, Snowflake requires version 2, and BigQuery requires version 3).

* **`retention_window_days`** — the number of days historical table versions (used for time travel or auditing) are retained. Set according to your organization's data retention policy.
* **`column_mapping_mode`** — controls how columns are mapped between storage and the table schema. Recommended: `ID`, for robust identifier-based mapping. Use `NONE` or `NAME` only if you need to support a lower protocol reader version.
* **`deletion_vectors_disabled`** — enables merge-on-read (deletion vectors mark rows as deleted without rewriting Parquet files) when `false`, or copy-on-write (entire files are rewritten on modification) when `true`. Recommended: `false`, unless you need to support a lower protocol reader version.
* **`change_data_feed_disabled`** — controls whether row-level change tracking is active, for incremental processing or auditing. Recommended: `false`, unless you need to support a lower protocol reader version.


## Related topics

- [Exported entities](/documentation/importing-and-exporting-data/export/overview#exported-entities.md)
- [Sync process, schedule, manual sync, and sync history](/documentation/importing-and-exporting-data/export/overview#sync-process.md)