AWS S3 adds queryable annotations for object metadata

AWS has launched a new metadata capability for Amazon S3 called annotations, designed to simplify how organizations manage rich, evolving context for their stored objects. Unlike existing metadata options, annotations allow users to attach up to 1,000 named entries per object, each up to 1 MB in size, totaling 1 GB per object. This metadata can be stored in JSON, XML, YAML, or plain text formats and modified or deleted at any time without rewriting the underlying object. The feature is available in all AWS Regions, including AWS China Regions, and is billed at standard S3 rates regardless of the object’s storage class.

The introduction of annotations addresses a long-standing challenge for organizations managing large-scale datasets, particularly those building AI-driven workflows or autonomous agents. Traditional metadata systems often require separate databases or sidecar files, which can complicate synchronization and increase costs. Annotations eliminate this need by embedding metadata directly alongside objects, ensuring it moves automatically during operations like replication or cross-region transfers and is deleted when the object is removed.

How annotations work

Annotations are attached to objects using the PutObjectAnnotation API, which supports both new and existing objects. Each annotation is identified by a unique name, allowing multiple teams to add or modify metadata independently. For example, a media company could attach technical specifications in JSON format to a video file while another team adds a plain-text AI-generated summary. Users can retrieve, update, or delete specific annotations using the GetObjectAnnotation, PutObjectAnnotation, and DeleteObjectAnnotation APIs, respectively. For objects uploaded via multipart upload, annotations must be added after the upload is complete.

To enable large-scale querying, AWS offers S3 Metadata annotation tables, which automatically index annotations into a managed Apache Iceberg table. These tables can be queried using Amazon Athena or other Iceberg-compatible engines. Enabling annotation tables requires configuring the bucket’s metadata settings via the AWS Console or the CreateBucketMetadataConfiguration API. Once enabled, annotations appear in the table within approximately one hour, though backfilling existing annotations may take longer for buckets with large datasets. Queries can filter annotations by name, content, or timestamp, enabling use cases like identifying all video assets with specific technical attributes or tracking annotation changes in near real time.

Use cases and industry impact

The flexibility of annotations makes them suitable for a range of industries. In media and entertainment, organizations can use annotations to track transcripts, licensing details, or content moderation results without relying on external media asset management systems. Financial services firms can attach AI-generated summaries or sentiment analysis to research documents, allowing autonomous agents to discover relevant datasets through natural-language queries. Life sciences companies can annotate clinical trial data with regulatory status or patient cohort details, streamlining compliance audits even for archived data in S3 Glacier storage classes.

Limitations and considerations

While annotations offer significant advantages, they are not a replacement for all existing metadata options. System-defined metadata (e.g., object size or storage class) and object tags (e.g., for access control or lifecycle management) remain better suited for their specific purposes. Annotations are designed for rich, mutable context rather than operational tasks. Additionally, annotation tables require enabling S3 Metadata, which may involve additional configuration and IAM role setup. Users should also note that annotation storage is billed separately from the object itself, though the cost remains consistent across storage classes.

AWS has positioned annotations as a solution for organizations seeking to scale metadata management without the overhead of maintaining separate systems. The feature is particularly relevant for AI-driven workflows, where the ability to query metadata at scale can significantly reduce the time and cost of data discovery. As adoption grows, annotations could become a standard tool for industries reliant on large, evolving datasets.