Skip to main content

MinIO

MinIO is an S3-compatible object store. The MinIO destination writes each stream run to an object in a MinIO bucket.

Prerequisites​

  • A reachable MinIO endpoint URL (for example, https://minio.example.com or http://localhost:9000)
  • A MinIO access key + secret key with permission to read and write to the target bucket (the connector validates access by performing a HeadBucket and a small test PutObject/DeleteObject during login)

Setup​

  1. Create a MinIO user/access key that has at least:
    • s3:ListBucket on the bucket
    • s3:PutObject, s3:DeleteObject on the target prefix
  2. In Extract, configure the connector:
    • Endpoint URL
    • Access Key ID / Secret Access Key
    • Bucket Name
    • Bucket Key template (see below)

Connection Settings​

The Key (object path) supports the same macros as the S3 destination (for example: {stream_name}, {timestamp}, {connection_id}, {connection_run_id}, {stream_run_id}, {extension}, and cursor-derived placeholders).

TLS / self-signed certificates​

If your MinIO uses HTTPS with a self-signed/invalid certificate, enable Insecure TLS (Skip Certificate Verification).

This disables TLS certificate verification and should only be used in trusted networks.

Configuration​

Provide the following settings when configuring the MinIO destination:

  • Endpoint URL: The MinIO server URL (for example, https://minio.example.com).
  • Region: The S3 region to use (often us-east-1 for MinIO unless your deployment requires a different value).
  • Access key ID / Secret access key: Credentials for an account with access to the target bucket.
  • Bucket: The bucket to write to. The connector validates access on connect by:
    • calling HeadBucket (read/metadata access), and
    • uploading and deleting a small test object (write/delete access).
  • Key: The object key (path) to write to in the bucket. This is a template and can include placeholders (see the Notes section for available variables).
  • Output format: jsonl, csv, or parquet.
  • Compress: When enabled, output is gzip-compressed for jsonl and csv (files end in .gz). Parquet is not additionally gzip-compressed.
  • Force path style: Enable if your MinIO deployment requires path-style addressing.
  • Insecure TLS: Enable only if you need to connect with TLS verification disabled (for example, when using self-signed certificates). See TLS / self-signed certificates.

If your workspace/run is configured to prevent overwriting existing files, the destination will fail if the resolved Key already exists in the bucket.

Notes​

  • During connection testing, the destination validates both read and write access to the target bucket by:

    • Calling HeadBucket on the configured bucket.
    • Uploading a small test object under extract_login_test/<uuid>.txt, then attempting to delete it. Ensure the credentials/policy allow s3:ListBucket, s3:GetBucketLocation, s3:PutObject, and s3:DeleteObject (or equivalent MinIO permissions) for the configured bucket/prefix.

  • When using output_format: csv, the connector writes a header row.

Troubleshooting​

  • Login fails with head_bucket errors

    • Verify the bucket name exists and the credentials have permission to access it.
    • Confirm the endpoint URL is correct and reachable from where Extract is running.
    • If you’re using a non-AWS S3-compatible endpoint, ensure the connection settings (such as region and path-style addressing) match your MinIO configuration.

  • Login fails with put_object failed during login test

    • The connector validates write access during login by uploading a small test object and then deleting it. If this step fails:
      • Ensure the credentials have s3:PutObject permission for the bucket (and s3:DeleteObject if your policy restricts deletes).
      • Check bucket policies that block writes (for example, requiring specific prefixes, encryption headers, or object lock settings).

  • Upload fails because the object/key already exists

    • Some runs may be configured to prevent overwriting existing files. In that case, uploads will fail if the resolved key already exists in the bucket.
    • Fix by changing the destination key template to produce unique paths per run (for example, include a run identifier or timestamp), or remove/rename the existing object in MinIO before re-running.

  • TLS / certificate errors

    • If you’re using self-signed certificates, follow the guidance in TLS / self-signed certificates to trust the certificate chain used by your MinIO endpoint.