Analyzing your information catalog: Question SageMaker Catalog metadata with SQL

As your information and machine studying (ML) property develop, monitoring which property lack documentation or monitoring asset registration traits turns into difficult with out customized reporting infrastructure. You want visibility into your catalog’s well being, with out the overhead of managing ETL jobs. The metadata function of Amazon SageMaker offers this functionality to customers. Changing catalog asset metadata into Apache Iceberg tables saved in Amazon S3 Tables removes the necessity to construct and preserve customized ETL pipelines. Your group can then question asset metadata straight utilizing commonplace SQL instruments. Now you can reply governance questions like asset registration traits, classification standing, and metadata completeness utilizing commonplace SQL queries by means of instruments like Amazon Athena, Amazon SageMaker Unified Studio notebooks, and BIsystems.

This automated method reduces ETL improvement time and provides your group visibility into catalog well being, compliance gaps, and asset lifecycle patterns. The exported tables embrace technical metadata, enterprise metadata, venture possession particulars, and timestamps, partitioned by snapshot date to allow time journey queries and historic evaluation. Groups can use this functionality to proactively monitor catalog well being, establish gaps in documentation, monitor asset lifecycle patterns, and guarantee that governance insurance policies are persistently utilized.

How metadata export works

After you allow the metadata export function, it runs routinely on a day by day schedule:

1. SageMaker Catalog creates the infrastructure — An Amazon Easy Storage Service (Amazon S3) desk bucket named aws-sagemaker-catalog is created with an asset_metadata namespace and an empty asset desk.
2. Each day snapshots are captured — A scheduled job runs as soon as per day round midnight (native time per AWS Area) to export up to date asset metadata.
3. Metadata is structured and partitioned — The export captures technical metadata (resource_id, resource_type), enterprise metadata (asset_name, business_description), venture possession particulars, and timestamps, partitioned by snapshot_date for question efficiency.
4. Information turns into queryable — Inside 24 hours, the asset desk seems in Amazon SageMaker Unified Studio beneath the aws-sagemaker-catalog bucket and turns into accessible by means of Amazon Athena, Studio notebooks, or exterior BI instruments.
5. Groups question utilizing commonplace SQL — Information groups can now reply questions like “What number of property had been registered final month?” or “Which property lack enterprise descriptions?” with out constructing customized ETL pipelines.

The export evaluates catalog property and their metadata properties within the area, changing them into Apache Iceberg desk format. The information flows into downstream analytics operations instantly, with no separate ETL or batch processes to take care of. The exported metadata turns into a part of a queryable information lake that helps time-travel queries and historic evaluation.

On this publish, we exhibit easy methods to use the metadata export functionality in Amazon SageMaker Catalog and carry out analytics on these tables. We discover the next particular use-cases.

• Audit historic modifications to research what an asset regarded like at a particular time limit.
• Monitor asset development view how the information catalog has grown over the past 30 days.
• Monitor metadata enhancements to see which property gained descriptions or possession over time.

Answer overview

Determine 1 – SageMaker catalog export to S3 Tables

The structure consists of three key elements:

1. Amazon SageMaker Catalog exports asset metadata day by day to Amazon S3.
2. S3 Tables shops metadata as Apache Iceberg tables within the aws-sagemaker-catalog bucket with ACID compliance and time journey.
3. Question engines (Amazon Athena, Amazon Redshift, and Apache Spark) entry metadata utilizing commonplace SQL from the asset_metadata.asset desk.

What metadata is uncovered?

SageMaker Catalog exports metadata within the asset_metadata.asset desk:

The snapshot_time column helps point-in-time evaluation and question of historic catalog states.

Stipulations

To observe together with this publish, it’s essential to have the next:

For SageMaker Unified Studio area setup directions, check with the SageMaker Unified Studio Getting began information.

After you full the stipulations, full the next steps.

1. Add this coverage to our IAM consumer or function to allow metadata export. If utilizing SageMaker Unified Studio to question the catalog, add this coverage to the AmazonSageMakerAdminIAMExecutionRole managed function.

{ "Model": "2012-10-17", 
"Assertion": [ 
{
 "Effect": "Allow",
 "Action": [ "datazone:GetDataExportConfiguration",
 "datazone:PutDataExportConfiguration"
 ],
 "Useful resource": "*"
 },
 {
 "Impact": "Enable",
 "Motion": [
 "s3tables:CreateTableBucket",
 "s3tables:PutTableBucketPolicy"
 ],
 "Useful resource": "arn:aws:s3tables:*:*:bucket/aws-sagemaker-catalog" 
} 
]
}

2. Grant describe and choose permissions for SageMaker Catalog with AWS Lake Formation. This step could be carried out within the AWS Lake Formation console.
   1. Choose Permissions -> Information permissions and select Grant.
      
      
      

      Determine 2 – AWS Lake Formation grant permission

   2. Below Principal sort, choose Principals, IAM customers and roles and the AWS managed AmazonSageMakerAdminIAMExecutionRole execution function.
   3. Select Named Information Catalog assets.
   4. Below Catalogs, seek for and choose :s3tablecatalog/aws-sagemaker-catalog.
   5. Below Databases, choose asset_metadata database.
      

      Determine 3 – AWS Lake Formation catalog, database, and desk

      

      Determine 4 – AWS Lake Formation grant permission

   6. For Desk, choose asset.
   7. Below Desk permissions, test Choose and Describe.
   8. Select Grant to avoid wasting the permissions.

Allow information export utilizing the AWS CLI

Configure metadata export utilizing the PutDataExportConfiguration API. The Amazon DataZone service routinely creates an S3 desk bucket named aws-sagemaker-catalog with an asset_metadata namespace, and schedules a day by day export job. Asset metadata is exported as soon as day by day round midnight native time per AWS Area.

The SageMaker Area identifier is out there on area element web page within the AWS Administration Console. Accessing the asset desk by means of the S3 Tables console or the Information tab in SageMaker Unified Studio can require as much as 24 hours.

AWS CLI command to allow SageMaker catalog export:

aws datazone put-data-export-configuration --domain-identifier  --region  --enable-export

Use this AWS CLI command to validate the configuration is enabled:

aws datazone get-data-export-configuration --domain-identifier  --region 
{
    "isExportEnabled": true,
    "standing": "COMPLETED",
    "s3TableBucketArn": "arn:aws:s3tables:::bucket/aws-sagemaker-catalog",
    "createdAt": "2025-11-26T18:24:02.150000+00:00",
    "updatedAt": "2026-02-23T19:33:40.987000+00:00"
}

Entry the exported asset desk

1. Navigate to Amazon SageMaker Domains within the AWS Administration Console.
2. Choose your area and choose Open.
   

   Determine 5 – Open Amazon SageMaker Unified Studio

3. In SageMaker Unified Studio, select a venture from the Choose a venture dropdown record.
4. To question SageMaker catalog information, choose Construct within the menu bar after which select Question Editor. To create a brand new venture, observe the directions within the Amazon SageMaker Unified Studio Consumer Information.
   

   Determine 6 – Open SageMaker Unified Studio Question Editor

The asset_metadata.asset desk is out there in Information explorer. Use Information explorer to view the schema and question information to carry out analytics from.

5. Increase Catalogs in Information explorer. Then, choose and develop s3tablecatalog, aws-sagemaker-catalog, asset_metadata, and asset.
6. Take a look at querying the catalog with SELECT * FROM asset_metadata.asset LIMIT 10;.

Determine 7 – Question SageMaker catalog

Queries for observability and analytics

With setup full, execute queries to achieve insights on catalog utilization and modifications. To observe asset development, and think about how the information catalog has grown over the past 5 days:

SELECT 
    DATE (snapshot_time) as date,
    COUNT (*) as total_assets
FROM asset_metadata.asset
WHERE 
     DATE (snapshot_time) >= CURRENT_DATE - INTERVAL '5' DAY
GROUP BY DATE (snapshot_time)
ORDER BY date DESC;

Determine 8 – Question asset development

Use the catalog to trace metadata modifications to find out which property gained descriptions or possession over time. Use this question to establish property that gained enterprise descriptions over the previous 5 days by evaluating at this time’s snapshot with the sooner snapshot.

SELECT
    t.asset_id,
    t.resource_name,
    p.business_description as description_before,
    t.business_description as description_now
FROM asset_metadata.asset t
JOIN asset_metadata.asset p ON t.asset_id = p.asset_id
WHERE DATE(t.snapshot_time) = CURRENT_DATE
    AND DATE(p.snapshot_time) = CURRENT_DATE - INTERVAL '5' DAY
    AND p.business_description IS NULL
    AND t.business_description IS NOT NULL;

Examine asset values at a particular time limit utilizing this question to retrieve metadata from any snapshot date.

SELECT
     asset_id,
     resource_name,
     business_description,
     extended_metadata['owningEntityId'] as proprietor,
     snapshot_time
FROM asset_metadata.asset
WHERE asset_id = 'your-asset-id'
     AND DATE(snapshot_time) = DATE('2025-11-26');

Clear up assets

To keep away from ongoing prices, clear up the assets created on this walkthrough:

1. Disable metadata export:

Disable the day by day metadata export to cease new snapshots:

aws datazone put-data-export-configuration 
  --domain-identifier 

2. Delete S3 Tables assets:

Optionally, delete the S3 Tables namespace containing the exported metadata to take away historic snapshots and cease storage prices. For directions on easy methods to delete S3 tables, see Deleting an Amazon S3 desk within the Amazon Easy Storage Service Consumer Information.

Conclusion

On this publish, you enabled the metadata export function of SageMaker Catalog and used SQL queries to achieve visibility into your asset stock. The function converts asset metadata into Apache Iceberg tables partitioned by snapshot date, so you’ll be able to carry out time-travel queries, monitor catalog development, monitor metadata completeness, and audit historic asset states. This offers a repeatable, low-overhead technique to preserve catalog well being and meet governance necessities over time.

To study extra about Amazon SageMaker Catalog, see the Amazon SageMaker Catalog documentation. To discover Apache Iceberg desk codecs and time-travel queries, see the Amazon S3 Tables documentation.

Concerning the Authors