Skip to main content

Azure Data Lake Storage Gen2

Overview​

The Azure Data Lake Storage Gen2 destination connector lets you authenticate to an ADLS Gen2 storage account and validate access to a target container (filesystem).

Note: Destination loading (writing data) is still in development. At this time, the connector validates credentials and container access during setup/login, but does not yet deliver records.

Setup​

Prerequisites​

  • An Azure Storage account with ADLS Gen2 enabled (hierarchical namespace).
  • A target container (filesystem) in that storage account.
  • An Azure AD application (service principal) with permission to access the storage account.

Azure permissions​

Create or use an existing Azure AD app registration (service principal), then grant it access to the storage account/container. Common approaches include:

  • Assign an Azure RBAC role such as Storage Blob Data Contributor (or a more restrictive role that still allows the required access) scoped to the storage account or container.

You will need:

  • Tenant ID
  • Client ID
  • Client Secret

Authentication​

This connector uses OAuth 2.0 Client Credentials against Microsoft Entra ID (Azure AD).

During login, the connector requests a token from:

  • https://login.microsoftonline.com/{tenant_id}/oauth2/v2.0/token

with scope:

  • https://storage.azure.com/.default

It then validates the target container (filesystem) by calling the ADLS Gen2 DFS endpoint:

  • https://{storage_account}.dfs.core.windows.net/{container_name}?resource=filesystem

Configuration​

Provide the following fields when configuring the destination:

FieldTypeRequiredDescription
storage_accountstring✅Azure storage account name. Must be lowercase letters and digits only.
container_namestring✅The ADLS Gen2 container (filesystem) name to validate/use.
tenant_idstring✅Azure AD tenant ID that owns the app registration.
client_idstring✅Application (client) ID for the service principal.
client_secretstring✅Client secret for the service principal.

Hashing and normalization​

This destination connector does not currently expose field-level hashing options.

Data contract​

Destination loading is not yet implemented, so there is no published data contract (object naming, file formats, schemas, or write semantics) at this time.

Streams​

Not applicable yet. The connector does not currently write any streams to ADLS Gen2.