Delta Lake

​

Overview

​

Setting up the integration

​

Prerequisites

• Cloud storage you control — an AWS S3 bucket, a Google Cloud Storage bucket, or an Azure storage account with hierarchical namespace enabled
• Depending on your provider: an IAM role or access key (AWS), a service account (Google Cloud), or a storage access key (Azure) with read/write access to the target bucket or container

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": "s3:ListBucket",
      "Resource": "arn:aws:s3:::BUCKET_NAME"
    },
    {
      "Effect": "Allow",
      "Action": [
        "s3:GetObject",
        "s3:PutObject",
        "s3:DeleteObject"
      ],
      "Resource": "arn:aws:s3:::BUCKET_NAME/*"
    }
  ]
}

{
  "Effect": "Allow",
  "Action": ["kms:GenerateDataKey", "kms:Decrypt"],
  "Resource": "arn:aws:kms:REGION_NAME:ACCOUNT_ID:key/KEY_ID"
}

​

Other storage backends

​

Google Cloud

1. In the GCP console, go to IAM & Admin > Service Accounts and create a service account. In Grant users access to this service account, enter the service account shown in the Stigg connect form, then note the email of the service account you just created.
2. Grant access using one of:
   • Service account impersonation (recommended) — on the Permissions tab, edit the principal from step 1 and add the Service Account Token Creator role.
   • Service account key — under Keys, create and download a JSON key instead.
3. In Cloud Storage, create a bucket and note the bucket name. On its Permissions tab, grant your service account the Storage Legacy Bucket Writer, Storage Legacy Bucket Reader, and Storage Legacy Object Reader roles.
4. In Stigg, navigate to Integrations > Apps > Delta Lake and provide:

   Field	Description
   Bucket name	The name of your GCS bucket
   Folder name	The folder path within the bucket where data will be written
   Service account email	The email of the service account you created in step 1

​

Azure

1. In the Azure portal, create a storage account. Under Advanced, keep Enable storage account key access on and enable Enable hierarchical namespace under Data Lake Storage Gen2. Under Data protection, turn off Enable soft delete for blobs, Enable soft delete for containers, and Enable soft delete for file shares.
2. Create a container within the storage account.
3. Under Security + networking > Access keys, note the Key value.
4. In Stigg, navigate to Integrations > Apps > Delta Lake and provide:

   Field	Description
   Storage account name	The name of your Azure storage account
   Container name	The name of the Blob Storage container
   Folder name	The folder path within the container where data will be written
   Access key	The access key generated above

​

Configuration options

• retention_window_days — the number of days historical table versions (used for time travel or auditing) are retained. Set according to your organization’s data retention policy.
• column_mapping_mode — controls how columns are mapped between storage and the table schema. Recommended: ID, for robust identifier-based mapping. Use NONE or NAME only if you need to support a lower protocol reader version.
• deletion_vectors_disabled — enables merge-on-read (deletion vectors mark rows as deleted without rewriting Parquet files) when false, or copy-on-write (entire files are rewritten on modification) when true. Recommended: false, unless you need to support a lower protocol reader version.
• change_data_feed_disabled — controls whether row-level change tracking is active, for incremental processing or auditing. Recommended: false, unless you need to support a lower protocol reader version.