S3 Data Lake
danger
This connector is in early access, and SHOULD NOT be used for production workloads. This connector is subject to breaking changes without notice.
We're interested in hearing about your experience! See Github for more information.
Reference
Config fields reference
Field
Type
Property name
string
s3_bucket_name
string
s3_bucket_region
string
warehouse_location
string
main_branch_name
object
catalog_type
string
access_key_id
string
secret_access_key
string
s3_endpoint
Changelog
Expand to review
| Version | Date | Pull Request | Subject |
|---|---|---|---|
| 0.2.20 | 2025-01-32 | #52068 | Add support for default namespace (/database name) |
| 0.2.19 | 2025-01-16 | #51595 | Clarifications in connector config options |
| 0.2.18 | 2025-01-15 | #51042 | Write structs as JSON strings instead of Iceberg structs. |
| 0.2.17 | 2025-01-14 | #51542 | New identifier fields should be marked as required. |
| 0.2.16 | 2025-01-14 | #51538 | Update identifier fields if incoming fields are different than existing ones |
| 0.2.15 | 2025-01-14 | #51530 | Set AWS region for S3 bucket for nessie catalog |
| 0.2.14 | 2025-01-14 | #50413 | Update existing table schema based on the incoming schema |
| 0.2.13 | 2025-01-14 | #50412 | Implement logic to determine super types between iceberg types |
| 0.2.12 | 2025-01-10 | #50876 | Add support for AWS instance profile auth |
| 0.2.11 | 2025-01-10 | #50971 | Internal refactor in AWS auth flow |
| 0.2.10 | 2025-01-09 | #50400 | Add S3DataLakeTypesComparator |
| 0.2.9 | 2025-01-09 | #51022 | Rename all classes and files from Iceberg V2 |
| 0.2.8 | 2025-01-09 | #51012 | Rename/Cleanup package from Iceberg V2 |
| 0.2.7 | 2025-01-09 | #50957 | Add support for GLUE RBAC (Assume role) |
| 0.2.6 | 2025-01-08 | #50991 | Initial public release. |