Create and replace Apache Iceberg tables with partitions within the AWS Glue Knowledge Catalog utilizing the AWS SDK and AWS CloudFormation

In recent times, we’ve witnessed a big shift in how enterprises handle and analyze their ever-growing knowledge lakes. On the forefront of this transformation is Apache Iceberg, an open desk format that’s quickly gaining traction amongst large-scale knowledge shoppers.

Nevertheless, as enterprises scale their knowledge lake implementations, managing these Iceberg tables at scale turns into difficult. Knowledge groups usually must handle desk schema evolution, its partitioning, and snapshots variations. Automation streamlines these operations, gives consistency, reduces human error, and helps knowledge groups deal with higher-value duties.

The AWS Glue Knowledge Catalog now helps Iceberg desk administration utilizing the AWS Glue API, AWS SDKs, and AWS CloudFormation. Beforehand, customers needed to create Iceberg tables within the Knowledge Catalog with out partitions utilizing CloudFormation or SDKs and later add partitions from Amazon Athena or different analytics engines. This prevents the desk lineage from being tracked in a single place and provides steps outdoors automation within the steady integration and supply (CI/CD) pipeline for desk upkeep operations. With the launch, AWS Glue prospects can now use their most popular automation or infrastructure as code (IaC) instruments to automate Iceberg desk creation with partitions and use the identical instruments to handle schema updates and kind order.

On this submit, we present the right way to create and replace Iceberg tables with partitions within the Knowledge Catalog utilizing the AWS SDK and CloudFormation.

Resolution overview

Within the following sections, we illustrate the AWS SDK for Python (Boto3) and AWS Command Line Interface (AWS CLI) utilization of Knowledge Catalog APIs—CreateTable() and UpdateTable()—for Amazon Easy Storage Service (Amazon S3) primarily based Iceberg tables with partitions. We additionally present the CloudFormation templates to create and replace an Iceberg desk with partitions.

Conditions

The Knowledge Catalog API adjustments are made accessible within the following variations of the AWS CLI and SDK for Python:

• AWS CLI model of two.27.58 or above
• SDK for Python model of 1.39.12 or above

AWS CLI utilization

Let’s create an Iceberg desk with one partition, utilizing CreateTable() within the AWS CLI:

aws glue create-table --cli-input-json file://createicebergtable.json

The createicebergtable.json is as follows:

{
    "CatalogId": "123456789012",
    "DatabaseName": "bankdata_icebergdb",
    "Identify": "transactiontable1",
    "OpenTableFormatInput": { 
      "IcebergInput": { 
         "MetadataOperation": "CREATE",
         "Model": "2",
         "CreateIcebergTableInput": { 
            "Location": "s3://sampledatabucket/bankdataiceberg/transactiontable1/",
            "Schema": {
                "SchemaId": 0,
                "Kind": "struct",
                "Fields": [ 
                    { 
                        "Id": 1,
                        "Name": "transaction_id",
                        "Required": true,
                        "Type": "string"
                    },
                    { 
                        "Id": 2,
                        "Name": "transaction_date",
                        "Required": true,
                        "Type": "date"
                    },
                    { 
                        "Id": 3,
                        "Name": "monthly_balance",
                        "Required": true,
                        "Type": "float"
                    }
                ]
            },
            "PartitionSpec": { 
                "Fields": [ 
                    { 
                        "Name": "by_year",
                        "SourceId": 2,
                        "Transform": "year"
                    }
                ],
                "SpecId": 0
            },
            "WriteOrder": { 
                "Fields": [ 
                    { 
                        "Direction": "asc",
                        "NullOrder": "nulls-last",
                        "SourceId": 1,
                        "Transform": "none"
                    }
                ],
                "OrderId": 1
            }  
        }
      }
   }
}

The previous AWS CLI command creates the metadata folder for the Iceberg desk in Amazon S3, as proven within the following screenshot.

You possibly can populate the desk with values as follows and confirm the desk schema utilizing the Athena console:

SELECT * FROM "bankdata_icebergdb"."transactiontable1" restrict 10;
insert into bankdata_icebergdb.transactiontable1 values
    ('AFTERCREATE1234', DATE '2024-08-23', 6789.99),
    ('AFTERCREATE5678', DATE '2023-10-23', 1234.99);
SELECT * FROM "bankdata_icebergdb"."transactiontable1";

The next screenshot exhibits the outcomes.

After populating the desk with knowledge, you’ll be able to examine the S3 prefix of the desk, which is able to now have the knowledge folder.

The knowledge folders partitioned in line with our desk definition and Parquet knowledge information created from our INSERT command can be found beneath every partitioned prefix.

Subsequent, we replace the Iceberg desk by including a brand new partition, utilizing UpdateTable():

aws glue update-table --cli-input-json file://updateicebergtable.json

The updateicebergtable.json is as follows.

{
  "CatalogId": "123456789012",
  "DatabaseName": "bankdata_icebergdb",
  "Identify": "transactiontable1",
  "UpdateOpenTableFormatInput": {
    "UpdateIcebergInput": {
      "UpdateIcebergTableInput": {
        "Updates": [
          {
            "Location": "s3://sampledatabucket/bankdataiceberg/transactiontable1/",
            "Schema": {
              "SchemaId": 1,
              "Type": "struct",
              "Fields": [
                {
                  "Id": 1,
                  "Name": "transaction_id",
                  "Required": true,
                  "Type": "string"
                },
                {
                  "Id": 2,
                  "Name": "transaction_date",
                  "Required": true,
                  "Type": "date"
                },
                {
                  "Id": 3,
                  "Name": "monthly_balance",
                  "Required": true,
                  "Type": "float"
                }
              ]
            },
            "PartitionSpec": {
              "Fields": [
                {
                  "Name": "by_year",
                  "SourceId": 2,
                  "Transform": "year"
                },
                {
                  "Name": "by_transactionid",
                  "SourceId": 1,
                  "Transform": "identity"
                }
              ],
              "SpecId": 1
            },
            "SortOrder": {
              "Fields": [
                {
                  "Direction": "asc",
                  "NullOrder": "nulls-last",
                  "SourceId": 1,
                  "Transform": "none"
                }
              ],
              "OrderId": 2
            }
          }
        ]
      }
    }
  }
}

UpdateTable() modifies the desk schema by including a metadata JSON file to the underlying metadata folder of the desk in Amazon S3.

We insert values into the desk utilizing Athena as follows:

insert into bankdata_icebergdb.transactiontable1 values
    ('AFTERUPDATE1234', DATE '2025-08-23', 4536.00),
    ('AFTERUPDATE5678', DATE '2022-10-23', 23489.00);
SELECT * FROM "bankdata_icebergdb"."transactiontable1";

The next screenshot exhibits the outcomes.

Examine the corresponding adjustments to the knowledge folder within the Amazon S3 location of the desk.

This instance has illustrated the right way to create and replace Iceberg tables with partitions utilizing AWS CLI instructions.

SDK for Python utilization

The next Python scripts illustrate utilizing CreateTable() and UpdateTable() for an Iceberg desk with partitions:

CloudFormation utilization

Use the next CloudFormation templates for CreateTable() and UpdateTable(). After the CreateTable template is full, replace the identical stack with the UpdateTable template by creating a brand new changeset in your stack and executing it.

Clear up

To keep away from incurring prices on the Iceberg tables created utilizing the AWS CLI, delete the tables from the Knowledge Catalog.

Conclusion

On this submit, we illustrated the right way to use the AWS CLI to create and replace Iceberg tables with partitions within the Knowledge Catalog. We additionally offered the SDK for Python and CloudFormation pattern code and templates. We hope this helps you automate the creation and administration of your Iceberg tables with partitions in your CI/CD pipelines and manufacturing environments. Strive it out in your personal use case and share your suggestions within the feedback part.

In regards to the authors

Acknowledgements: A particular because of everybody who contributed to the event and launch of this function – Purvaja Narayanaswamy, Sachet Saurabh, Akhil Yendluri and Mohit Chandak.