Skip to main content

Azure Data Lake Storage Gen2

Export data to an existing Azure Data Lake Storage Gen2 container as CSV, JSON Lines, Parquet, or Delta Lake tables. File formats support Append only. Delta Lake supports Append and Replace.

Prerequisites​

  • An Azure storage account with hierarchical namespace (HNS) enabled and an existing container.
  • A Microsoft Entra application/service principal with a tenant ID, client ID, and client secret.
  • Assign Storage Blob Data Contributor on the target container. Allow Extract's workers through the account's network restrictions. Both Blob and DFS endpoints must be reachable.

See Microsoft's service-principal guidance and Azure access control.

Setup​

  1. Add Azure Data Lake Storage Gen2 as a destination.
  2. Enter the storage account name (without a URL), container name, tenant UUID, client UUID, and client secret.
  3. Validate the account. Login checks HNS and container access without creating resources; a successful login does not prove write permission.
  4. Create a connection and choose its output format and path.

File output​

Choose Parquet (the default), CSV, or JSON Lines. CSV includes a header. Optional gzip compression applies only to CSV and JSON Lines.

The default file path is extract/{connection_name}/{stream_name}/{run_time}.{extension}, for example extract/Daily reports/events/2026-10-08T07-03-54.123456Z.parquet. {connection_name} uses the optional connection name, falling back to <source account name> -> <destination account name> when unset or empty. {stream_name} uses the stream title. {run_time} is the stream run creation timestamp in UTC, with microseconds, and stays unchanged when that run retries. This timestamp describes the run, not the dates inside its data; use cursor placeholders when organizing by data date. Include {extension} in custom paths to get the appropriate suffix, including .gz. Paths are relative to the container and also support the standard date and cursor placeholders. Keep connection and stream names distinct: identical names and timestamps share a path. Path syntax inside connection names is percent-encoded; stream titles keep the existing path behavior (a slash creates another directory level). Renaming a connection or stream changes its directory. Previously saved templates remain unchanged, and ID placeholders are still supported.

Files use block-blob uploads and become visible after the complete upload commits. Reusing a path replaces that object unless the custom run disables Should override existing files; protected writes fail if the object already exists. Append describes the connection's loading mode, not byte appending to an existing file.

Delta Lake output​

Choose Delta Lake to create one table per stream. The default table directory is extract/{connection_name}/{stream_name}, using the same connection-name fallback as file output. Only {connection_name}, {stream_name}, {connection_id}, and {stream_id} are supported in table paths; table locations must remain stable across runs. Keep names distinct and unchanged: renaming a connection, fallback account name, or stream points to a different table directory. To keep an existing table after a rename, set its previous directory explicitly. Saved ID-based templates remain supported. Use an empty directory for a new table and a different directory for every stream.

Extract writes Parquet data and commits it through the Delta transaction log after the source finishes successfully. Readers see a complete committed batch. Retrying the same stream run does not apply it again. Append adds rows; a separately requested Append run can repeat source records. Empty Append input creates no data commit.

Replace is available only for Delta and defaults for newly discovered Partition and Full Refresh streams. Existing saved load modes remain unchanged. Full Refresh Replace replaces the entire table, including clearing it when a successful source run returns no rows. Partition Replace replaces incoming partition-key tuples and preserves other history. For a single date or timestamp partition key with explicit starting and ending cursors, it replaces that whole window, including dates that return no rows. Date boundaries are inclusive; timestamp boundaries are start-inclusive/end-exclusive. Without explicit bounds, only keys present in incoming data are replaced; an absent date is retained. Replace requires complete source data for the scope and does not support diff mode.

New tables with a single date partition key are partitioned by that date. Existing unpartitioned tables remain supported: Extract writes preserved rows and new data to new files, then changes the active file references in one transaction. Original files are not edited or immediately deleted. Other key shapes use unpartitioned files. Concurrent changes can require retrying a replacement from a fresh snapshot.

Files under _extract_staging/<run>/<attempt> can be active table data after commit. The name describes their initial upload stage; do not delete these folders manually. Read the table using a Delta-aware reader at the table root, rather than reading all Parquet files recursively.

Add New Fields allows new nullable columns. Existing columns remain in the table when deselected, with nulls in subsequent rows. Incompatible type changes fail. Dates retain their date type, timestamps use UTC microseconds, and arbitrary JSON values are stored as JSON strings.

Only basic Extract-owned Delta tables are supported. This connector does not support arbitrary partition configuration, advanced Delta features, upserts, external catalog registration, compaction, or vacuum. File-overwrite controls do not apply to Delta. Do not change the output format to ordinary files inside a Delta table directory.

Failed runs may leave unreferenced Parquet files. Extract does not automatically delete these because a commit response can be lost after the transaction succeeds. Use Delta-aware maintenance with an appropriate retention period; never delete files solely because a run reported an error.

Troubleshooting​

  • HNS required: enable hierarchical namespace on the account or choose the Azure Blob Storage destination.
  • Access denied: check the service principal's role, client-secret expiry, container name, ACLs, and network rules.
  • Non-Delta files at table path: select a new empty table directory.
  • Schema mismatch: restore compatible source types or choose a new table path. Replace retains the existing schema and rejects incompatible type changes.
  • Uncertain commit: retry the same stream run so Extract can reconcile its transaction identifier.