SEATTLE — In a major architectural leap for enterprise data management, Amazon Web Services (AWS) has announced a powerful new capability for Amazon Aurora PostgreSQL that allows developers and data engineers to directly query operational data alongside massive analytical data lakes. Powered by an embedded DuckDB engine, the new feature eliminates the historic friction of Extract, Transform, and Load (ETL) pipelines, empowering single-interface queries across live transactional tables and open table formats like Apache Iceberg and Apache Parquet.

The announcement marks one of the first major structural integrations following Amazon’s recent acquisition of DuckLabs, the core maintainers of the high-performance analytical database project DuckDB. By weaving DuckDB directly into the fabric of Aurora PostgreSQL, AWS is offering a streamlined path for modern applications—ranging from real-time customer dashboards and transactional history enrichment to complex, reasoning AI agents—to interact with disparate data stores without the overhead of data duplication.

Main Facts: Unifying Operational and Analytical Worlds

The core of this release centers on bridging the traditional divide between transactional databases (OLTP) and analytical data stores (OLAP). Historically, organizations maintained strict silos: operational data lived in databases like Aurora, while historical archives, data warehouses, and data lakes lived in cloud object storage like Amazon S3, formatted as Parquet or managed through Apache Iceberg.

To bridge this gap, teams routinely deployed complex reverse ETL pipelines. These pipelines periodically copied, transformed, and synchronized data, driving up infrastructure costs, introducing latency, and creating administrative headaches. Furthermore, as organizations increasingly deploy generative AI applications and autonomous agents, predicting and pre-replicating every possible dataset an agent might need has become practically impossible.

The new Aurora PostgreSQL capability solves this by embedding DuckDB directly into the database engine. Key structural highlights include:

• Zero-ETL Overhead: Queries run directly against data in Amazon S3, S3 Tables, and the AWS Glue Data Catalog. No data movement, copying, or replication is required.
• Open Format Support: Native compatibility with Apache Iceberg, Apache Parquet, and Iceberg REST Catalog (IRC)-compatible external catalogs via AWS Glue federation.
• Familiar Interface: Developers can use standard PostgreSQL syntax, tools, and client connections (such as psql or the Amazon RDS console) to query external data lakes.
• Advanced Query Optimizations: Aurora applies intelligent execution planning, including predicate pushdown and column pruning, to ensure only relevant bytes are read from S3.
• Read Replicas and Caching: Frequently accessed data is cached locally within the Aurora instance, and analytical scans can be offloaded to read replicas to protect operational workloads.

Chronology: The Road to Embedded Analytical Efficiency

The journey toward this release reflects a broader industry trend toward converged database architectures—systems capable of handling both high-throughput operational workloads and complex analytical queries concurrently.

• The Pre-Integration Era: For years, combining recent operational data with multi-year historical archives required engineering teams to build custom data pipelines. These pipelines used tools like AWS Glue, Apache Airflow, or third-party orchestrators to stream database logs or night-time dumps into Amazon S3, converting them into columnar formats for analytical querying.
• The Rise of DuckDB: DuckDB gained massive popularity in the data engineering community as an in-process analytical database designed for extreme performance on single machines. Its vectorized query execution engine made it exceptionally fast at scanning large analytical files like Parquet.
• The DuckLabs Acquisition: Recognizing the potential of DuckDB’s lightweight, high-performance architecture, Amazon brought the DuckLabs team onboard.
• Integration into Aurora PostgreSQL: AWS engineers set to work embedding the core efficiency of DuckDB directly into Aurora PostgreSQL. By compiling DuckDB into the database engine, AWS bypassed the need for additional network hops or external middleware.
• General Availability (Today): AWS has rolled out the feature globally across all commercial and AWS GovCloud (US) Regions for Aurora PostgreSQL major versions 17 (starting with version 17.11) and 18 (starting with version 18.6).

Supporting Data & Technical Implementation

Implementing the new capability requires minimal configuration, relying on standard PostgreSQL extensions and AWS Identity and Access Management (IAM) controls.

Setup and Configuration

To begin querying external data lakes from Aurora PostgreSQL, database administrators must complete a straightforward onboarding sequence:

1. Version Requirement: Provision or upgrade an Aurora PostgreSQL cluster to version 17.11+ or 18.6+.
2. IAM Permissions: Attach an IAM role featuring the AuroraAnalytics policy to the Aurora cluster, granting read access to Amazon S3 and the AWS Glue Data Catalog.
3. Extension Activation: Connect via any PostgreSQL client and run:

   CREATE EXTENSION aurora_analytics;

4. Schema Definition: Define foreign tables pointing directly to the target S3 storage locations or Glue database tables.

Automated Schema Inference

One of the most notable developer-experience features is automated schema inference. By utilizing an empty parentheses structure in the CREATE FOREIGN TABLE command, Aurora automatically reads the underlying metadata of Parquet or Iceberg files:

CREATE FOREIGN TABLE transaction_history ()
SERVER aurora_analytics_server
OPTIONS (
    location 's3://<my-bucket>/finance/transaction_history.parquet',
    format 'parquet'
);

For large-scale data lakes containing hundreds or thousands of tables, engineers can bypass manual table creation entirely using the IMPORT FOREIGN SCHEMA command, which bulk-registers an entire AWS Glue Data Catalog database instantly.

Executing Unified Queries

Once tables are registered, developers can execute standard SQL queries that perform complex joins and unions across operational and historical datasets. For instance, a financial application can merge the last seven days of live transactional data (recent_transactions) with five years of archived history stored in S3 (transaction_history):

SELECT merchant, category, amount, transaction_date, 'recent' AS source
FROM recent_transactions
WHERE customer_id = 'C-1001'
UNION ALL
SELECT merchant, category, amount, transaction_date, 'historical' AS source
FROM transaction_history
WHERE customer_id = 'C-1001'
  AND transaction_date >= CURRENT_DATE - INTERVAL '5 years'
ORDER BY transaction_date DESC
LIMIT 15;

Under the hood, DuckDB handles the heavy analytical scan of the columnar Parquet file in S3, while Aurora manages the live operational table writes.

For workloads demanding single-digit-millisecond latency, engineers retain the flexibility to materialize frequently accessed data lake segments into native Aurora tables using standard commands like CREATE TABLE AS SELECT or MERGE INTO, combining the best of both virtualized and physical storage strategies.

Official Responses and Strategic Vision

AWS leadership has emphasized that this feature represents a foundational shift in how database engines will evolve in the era of distributed cloud storage and artificial intelligence.

"We’re excited to bring the speed and simplicity of DuckDB directly into Amazon Aurora PostgreSQL," said AWS engineering representatives in the official release notes. "Whether you’re powering real-time dashboards, enriching transactions with historical context, or building AI agents that reason over both live and archived data, you can now do it all through a single, familiar interface."

Industry analysts note that as AI agents become more prevalent in enterprise software stacks, architectures that require rigid, predefined ETL pipelines are increasingly becoming bottlenecks. AI agents require fluid, ad-hoc access to vast repositories of contextual data—both structured and unstructured, recent and historical. By embedding DuckDB into Aurora, AWS is providing a mechanism where agents can query data on-demand without database administrators needing to anticipate every access pattern in advance.

Furthermore, AWS confirmed that this integration is designed to be extensible. As the open-source DuckDB project evolves, future performance upgrades and feature additions will flow naturally into Aurora and potentially other AWS data services, compounding the long-term return on investment for users.

Implications for the Enterprise

The introduction of direct data lake querying within Amazon Aurora PostgreSQL carries significant implications for database administrators, data engineers, and corporate finance departments alike:

• Drastic Cost Reductions: By eliminating reverse ETL pipelines, organizations save on compute cycles used for data replication, intermediate staging storage, and pipeline maintenance engineering hours.
• Simplified Data Governance: Because queries leverage existing AWS Glue Data Catalogs and fine-grained IAM permissions, security teams can maintain centralized governance over data lakes without replicating permission models down to operational databases.
• Enhanced Operational Performance: By allowing analytical scans to execute via DuckDB and offloading queries to read replicas, primary transactional databases experience reduced CPU and memory contention, ensuring mission-critical write operations remain lightning-fast.
• No Additional Licensing Fees: AWS has made the capability available at no additional charge across all commercial and GovCloud regions. Customers pay only for the incremental Aurora compute consumed during query execution and standard Amazon S3 request and data retrieval fees.

Getting Started

The new capability is available immediately. Organizations can explore the feature set by visiting the Amazon RDS console, reviewing the updated Amazon Aurora PostgreSQL documentation, or testing queries via standard PostgreSQL administration tools. Feedback can be channeled through AWS re:Post or traditional AWS Support channels.