Igor Suhorukov

My PostgreSQL is faster than yours and has almost caught up with ClickHouse

2026-10-21T00:00:00+00:00

A friend of mine is an excellent architect but a complete snob. He has four different databases in production: PostgreSQL for OLTP, ClickHouse for OpenTelemetry data and web analytics, Elasticsearch for full-text search in the application, and MongoDB, “for historical reasons.”

In between, there are ETL and CDC pipelines, plus on-call developers hunting for where exactly the results don’t match. To him, it’s all a well-thought-out architecture; to me, it’s needless complexity in maintenance and development.

The other day I told him what I’ve been working on lately.

“I’ve managed to plug ClickHouse’s MergeTree table engine into PostgreSQL as a table access method, on a vanilla database core, without a fork, and with a vectorized query executor reading from it.”

“What for? Just use ClickHouse and spare yourself the pain.”

“Because I need JOINs across lots of tables, consistent data throughout the system, and one platform that’s familiar to develop and maintain.”

“That’s not how it works. It’s either reliable or fast!”

“Well, I’ve managed to combine the best of both worlds.”

Vectorization hit a wall in heap

“Wait a minute. Didn’t you just port Greenplum into a Postgres extension and then write a vectorized executor on top of it? What do you need MergeTree for?”

“Vectorization grinds through columns fast, but it chokes the moment it has to read the data.”

The backstory began when I ported Apache Cloudberry to PostgreSQL 19 as a set of extensions, which got me ORCA, Greenplum’s cost-based planner, on PostgreSQL 19. On top of that query optimizer I developed pg_vexec, a vectorized executor without a single core patch.

Its nodes, VecScan, VecHashJoin, VecAgg and VecSort, get into a query plan through the database’s standard extension mechanisms and compete with the ordinary nodes on cost. ORCA prices the vector nodes with its own cost model. Inside, the executor processes batches of 1,024 rows in columnar form with kernels bound to function OIDs. Where there is no dedicated kernel, the stock Postgres expression interpreter evaluates the expression right inside the node.

The first ClickBench measurement didn’t make me happy: vectorization didn’t speed things up much. With the standard planner on heap, pg_vexec cut the geometric mean query time by only 19%, from 625 to 506 ms, and under ORCA by 18%, from 1,647 to 1,349 ms.

“18%, seriously? Was that worth building all the vectorization for?”

“Well, I ran a profiler and saw that the function kernels took only 0–2% of the total query execution time. Depending on the query, deforming wide rows ate 51–86% of the cycles of a GROUP BY, and hash table lookups 5%. Another 26–47% went to copying pages.”

When a row weighs about 1.4 KB, 5–6 rows fit on a page, and a batch of 1,024 rows is smeared across roughly 180 pages. To get three of the benchmark table’s 105 columns, heap dutifully pulls whole pages into memory. The Postgres table API hands out rows one at a time through a slot, and scan_begin() doesn’t even take a list of the columns the query needs.

Nor is there any way to skip the pages you don’t need: heap offers no extension point for that. A simulation of such skipping made the same ClickBench queries 8–100× faster (Q20 went from 510 to 5.3 ms). But it was only a simulation.

“So use a columnar store. PostgreSQL has a dozen of them.”

“Thanks for the advice, Captain Obvious :) On unmodified PostgreSQL 19, vexec couldn’t find a single source that would hand it data column by column. Citus columnar and Hydra are columnar inside, but through the table AM API they hand out the very same slots, one row at a time. Greenplum’s PAX and AOCO live only on the port’s patched Postgres core: besides the extensions, the port also has a series of core patches. Sure, on 16 segments of gp_ao with ORCA + vexec the port squeezed out 105 ms, but that’s sixteen segments against a single node, and still a fork of the database core, which is exactly what I wanted to get away from.”

That’s how my plan got a new task: bring in the MergeTree family of table engines and try to catch up with ClickHouse on ClickBench.

How I cut the mean query time from 715 to 49 milliseconds

The very first, still unoptimized version of MergeTree together with pg_vexec immediately brought the geometric mean query time down to 148 ms from the 715 ms of heap without vectorization in the October 5 measurement. After that came a programmer’s usual routine: profiling, hypotheses, prototyping, and rolling changes back.

GCC’s auto-vectorization at -O2, which PGXS builds with, gave almost nothing: exactly one loop got vectorized, and not a single comparison. Once the code was rewritten in SSE2 with movemask, with an LLM’s help, comparing 1,024 rows took 82 ns instead of 426.

The code that decides whether an expression can throw an error had no branch for T_TargetEntry. As a result, VecHashJoin read its row-based child one row at a time, and because of that TPC-DS Q2 assembled 2,153,556 single-row batches. A one-line fix gave 1,246 ms instead of the original 3,447.

In auto mode the planner wouldn’t put a VecSort above the scan node, and in force mode VecSort’s cost went negative: -46403! After the fix, SELECT * FROM hits ORDER BY CounterID LIMIT 10 runs in 67–76 ms instead of the original 2.1 seconds.

The scan first asks the data source for the columns of the first condition and reads the rest only for the batches where rows remain after that condition. On Q23, single-threaded, this cut the execution time from 471 to 247 ms.

Columns with few distinct values now arrive in pg_vexec as indexes into a LowCardinality dictionary, and expressions are computed once per dictionary value. Q10, single-threaded: 85.6 ms instead of 148.2.

PostgreSQL’s regular expression engine builds its DFA from scratch on every call, and in Q28 its miss() function ate 36% of the cycles. A dedicated kernel in the vectorizer plus partitioned aggregation in shared memory cut Q28’s execution time more than fivefold, from 1,090 to 213 ms. Across the whole of ClickBench, that saved 16%.

To be honest, not every step led to an improvement. A table with LowCardinality, small granules and no compression suddenly got slower and took 11.8 GB on disk instead of 4.8 GB. The disabled compression ate the space, but the slowdown had nothing to do with the options: the test bench restarted the server before the merges had finished, and the queries read 35 parts instead of three. Even the original table gave 106 ms on 35 parts, against 71. The biggest speedup for pg_vexec came with the initial work on MergeTree; after that, progress was slow and painstaking, with many forks in the road about where to take the project next.

Yet another reinvented columnar wheel

That’s exactly how I started, with a wheel of my own, but I quickly sent it to the scrap heap. MergeTree has been tested for years in user analytics, under real load and real business queries. And of course, you can’t win at ClickBench without the technologies the benchmark was written for: ClickHouse’s data formats, and background sorting and merging in storage.

Now you can run ch_table_engines in Postgres and get storage tailored to analyzing server logs:

-- postgresql.conf: shared_preload_libraries = 'ch_table_engines'
CREATE EXTENSION ch_table_engines;
CREATE TABLE hits (...) USING mergetree
    WITH (order_by = 'CounterID, EventDate, UserID', storage = 'pages');

“So you just linked ClickHouse into a Postgres backend?”

“It’s not that simple. MergeTree doesn’t build on its own: it’s part of ClickHouse’s dbms library and drags the interpreter, the parsers and the functions along with it. Of about a hundred and fifty submodules, only 28 were needed. Static linking produces libchte_engine.so. Nor can you take the ready-made StorageMergeTree, with its multithreaded model of a single server process. When a second process tries to get into its directory, you get ‘Another server instance in same directory is already running’, and Postgres creates a separate process for every connection. So it took some duct tape of my own to strap it onto PostgreSQL.”

But MergeTree is of little use with the standard Postgres executor. Without the vectorized executor, MergeTree lost even to heap: under ORCA, the TPC-H SF1 queries took 61.8 s in total against 17.2 s for heap with the stock executor, and TPC-DS 380.5 s against 58.8 s. With pg_vexec, the same queries on MergeTree ran in 6.9 s and 48.6 s. Without that piece, you’d get a columnar store that hands out data row by row through a slot, like a slow heap.

UPDATE doesn’t bloat the table

“Fine, so much for analytics. But admit it: updating data in Postgres while long transactions are running is a pain. Every UPDATE is a new row version, dead tuples pile up, and autovacuum huffs and puffs. When we first put our system into production, table bloat was a problem for us. We had to shorten our transactions and move the analytical queries out of Postgres.”

“I do admit it! That’s exactly why I made iheap, a PostgreSQL 19 table access method that overwrites a row in place, at the same TID. Its old version goes to an undo log, and only a reader whose snapshot needs it rebuilds it from there.”

The idea is similar to zheap, a project that people spent years trying to get into the core before abandoning it. I, however, managed to do something similar without a core patch. In PostgreSQL 19, the database’s built-in mechanisms are enough to implement it, from a custom WAL resource manager to visibility horizons.

-- postgresql.conf: shared_preload_libraries = 'pg_inplace_update_am'
CREATE EXTENSION pg_inplace_update_am;
CREATE TABLE accounts (id int PRIMARY KEY, balance int) USING iheap;
ALTER TABLE old_table SET ACCESS METHOD iheap;  -- convert an existing table

Undo lives in extents of the table’s own main fork, so WAL, crash recovery, replicas, backups and logical replication work as they do for any table. Row locks are undo records too, so there are no MultiXacts and no freezing. And so that an update of an indexed column doesn’t touch the other indexes, the extension carries dmbtree, a fork of PG19’s btree that can mark an old entry as deleted: a new entry appears only in the indexes whose keys changed, not in every index of the table.

My PostgreSQL

“All right. But why all this if ClickHouse is faster anyway?”

“Because it’s a great database for a huge metrics table with hundreds of columns. And what if the database also has to hold transactional data from different systems, with dozens of joins between tables? You can get all of it in PostgreSQL, in a single transaction. And a long-running transaction is no longer such a big problem under frequent updates: iheap tables don’t bloat the way heap tables do. Unless you’re Big Tech, why pile up the complexity of maintaining several systems?”

On my laptop, PostgreSQL 19 ran the ClickBench queries on 10 million rows only 3 times slower than ClickHouse with the same data on the same machine: 49.9 ms against 16.8 ms by geometric mean. Vanilla Postgres was 39 times slower than ClickHouse in those runs. The result is my own set of PostgreSQL extensions, pg_vexec, ch_table_engines and iheap, and not a single core patch. You simply choose the storage with the standard CREATE TABLE ... USING syntax: mergetree for metrics and wide tables, iheap for update-heavy tables without table and index bloat. Replicas, pg_dump and MVCC work as they do with ordinary heap, because PostgreSQL still takes care of them.

A specialized database means extra complexity in development and maintenance, unless your data is on the scale of the Large Hadron Collider’s telemetry or Google’s. Good old PostgreSQL can still hold its own against modern databases in everyday projects!

Vectorized Query Execution in PostgreSQL 19: Arrow, ORCA and Flight SQL in a PostgreSQL Extension

2026-10-07T00:00:00+00:00

Last week I ported Apache Cloudberry to PostgreSQL 19 as a set of extensions and described where the port falls behind the original fork. On ClickBench, columnar engines beat it by an order of magnitude. The culprit isn’t the MPP implementation but PostgreSQL’s executor, which works on rows. Each row travels up the node tree on its own, every value goes through fmgr, and the cluster’s segments merely add more processors running the very same loop over table rows.

The logical next step is a vectorized executor. Apache Cloudberry’s open-source code doesn’t have one: only traces of the closed-source engine remain in the repository, namely the create_vectorization_plan flag, which the open-source planner always passes as false, a WindowHashAgg node with no executor behind it, and a PAX adapter under VEC_BUILD that hasn’t compiled for a long time. I would have to design it myself and, of course, as an extension once again.

That is how pg_vexec came about, a family of extensions for PostgreSQL 19:

vexec: a vectorized planner and executor in one module;
vexec_flight: an Arrow Flight SQL endpoint that serves and accepts Arrow while the vectorized executor is on;
vexec_pgvector and vexec_postgis: kernel packs that let pgvector’s and PostGIS’s functions compute over batches.

The project doesn’t require modifying PostgreSQL, even with the ORCA planner. The gp_orca module from the Cloudberry port now builds on its own, without gp_core and without core patches, and vexec turns ORCA into a vectorized engine: ORCA’s search weighs its alternatives with the costs of vector nodes, and its translator builds those nodes directly. Core patches are needed only if Cloudberry itself is loaded into the server. Then vexec can also read the columnar PAX and ao_column tables column by column and write to them the same way, without turning the data into rows, and between segments the Motions carry data as Arrow IPC frames.

Once again Claude Code wrote the code for me, in parallel sessions, each in its own git worktree, while I set the tasks, made the architectural decisions and checked the results. I started on the plan for a vectorized PostgreSQL extension on October 2, the code started on October 5, and by the morning of October 7 thirteen iterations had been built, from the first ClickBench run for the baseline to sending columnar data as Arrow from the database’s memory between servers through Motions.

It started with DataFusion

It all began a month and a half ago with the idea of embedding a ready-made vectorized engine, Apache DataFusion, in PostgreSQL. This had already been done in open source before me, but I wanted the new solution to be distributed and faster. In the plan for pg_arrow, a Rust core was to sit behind a C ABI, with everything that touches PostgreSQL’s headers kept in a thin C layer. DataFusion would live in a single background worker per cluster, with its own Tokio thread pools. A backend would offer the planner ArrowScan CustomPaths through the path hooks, translate the chosen subtree from PlannerInfo into a DataFusion logical plan and send it to the service. Producer processes would encode heap rows into Arrow pages in shared memory, results would come back as pages in PostgreSQL’s layout, Flight SQL would serve Arrow to the outside world straight from the service, and columnar tables would be stored in Iceberg on object storage.

The more closely I worked through this plan with the agents, the longer the list of compromises grew. DataFusion’s semantics aren’t PostgreSQL’s: collations, numeric scale, the ordering of NaN, time zones, and even now(), which DataFusion pins when it hands out a session state, so two statements of one transaction would see different times. jsonb and geometries would travel between processes as text and EWKB and be rebuilt on every row, and every data path added a copy: heap into Arrow pages, results back into PostgreSQL pages. The service would be a child of the postmaster, so an abort, a potential segfault in zstd or a visit from the OOM killer would restart the whole cluster. DataFusion 55.1’s hash join couldn’t spill to disk, datafusion-proto silently dropped plan fields, and until SubPlans, InitPlans, CTEs, partitions and eager aggregation were translated, such queries would stay on PostgreSQL’s plan.

And the gain was questionable: the fastest parts of such an engine, vectorized scans, filters and aggregates, can be had inside the backend too, with PostgreSQL’s semantics and without copies between processes. So pg_arrow remained a plan without a single line of code. When my port of Cloudberry to extensions gave me ORCA, PAX and gp_ao as modules, I came up with a plan for vectorizing query execution: not one more engine next to PostgreSQL, but a vectorized executor inside the PostgreSQL DBMS itself. It kept several things from pg_arrow: the mapping table for Arrow types, pages in PostgreSQL’s layout as the prototype of the postgres format, the corpus of function semantics, Flight SQL logins through pg_hba.conf, and even the definition of done for Flight SQL, which is that TPC-H Q1 through it must be faster than binary COPY.

So, as in the refactorings of legacy systems at my previous jobs, I decided to clean out the Augean stables starting from the main dependency, the planner. I chose ORCA for that role and began preparing PostgreSQL for the idea: vectorized execution, and data transfer over Flight SQL in addition to pgwire.

Why an extension?

Vectorizing PostgreSQL isn’t a new idea, and every way of doing it has had its price. openGauss built a vectorized executor of about 82,000 lines into a fork of PostgreSQL 9.2 that can no longer be merged with upstream. pg_duckdb hands the query to a second engine, DuckDB, right inside the backend, and it raises the same questions about semantics that pg_arrow did. And openGauss, vectorize_engine, Hydra and TimescaleDB’s VectorAgg insert vector nodes into a finished plan, so vectorization never competes on cost. In TimescaleDB this approach has led to wrong results (#9902, for example) and crashes.

I gave the agents the same conditions as for the Cloudberry port: “plug the executor in only through PostgreSQL’s existing hooks, and without the extension loaded the server must behave like vanilla.” Six research reports checked every boundary of the design against the code, from planning to the cluster, and not one of them needed a new hook. Hence the rules for every phase:

no core changes. On the whole, I got lucky! The vendors who built vectorized executors for PostgreSQL apparently got all the extension points and hooks this takes into the upstream core, except for vectorized reading and writing of data in the table AM. The core had all the extension points needed;
vector execution nodes are available to the planner and compete with row-based alternatives on cost. Nobody rewrites a finished plan;
one model for both planners: one capability oracle, one cost model in PostgreSQL’s units, one set of node builders;
PostgreSQL’s semantics down to the last SQLSTATE: NULLs, unevaluated CASE branches, collations, NaN, the order of float summation. A kernel only speeds things up, and completeness comes from the fallback to PostgreSQL’s expression evaluator;
Cloudberry’s sources (other than pg19/) don’t change, and no code is copied from openGauss, Hydra or TimescaleDB;
the backend stays single-threaded and in C: palloc within work_mem, spills to BufFile, errors through ereport.

The extension’s vanilla behaviour is checked at every step of implementation, and checked hard, as in my Cloudberry port. PostgreSQL’s 239 regression tests pass against unchanged expected files with vexec installed, preloaded with mode off, and in explain mode. A differential runner compares the answers of force sessions in the postgres format, in the arrow format and with random column layouts against off, and every difference, such as plans that the tests print themselves, has been examined and recorded. My Cloudberry port’s suites likewise compare force with off, and the 121 TPC-H and TPC-DS queries are checked against DuckDB’s answers. And if a plan with a vector node reaches a segment where the library has been removed, the segment answers with a clean error, ExtensibleNodeMethods "VecScan" was not registered, and the cluster stays up.

Architecture: what this does better than plain PostgreSQL and Cloudberry

PostgreSQL and Cloudberry execute a query the same way, row by row. vexec replaces the rows in memory with a buffer of up to 1,024 rows laid out by column: that many rows of a fragment fit in the L1/L2 caches while they are being filtered, projected and aggregated. But the in-memory columnar structure itself is less interesting than where vector queries over this data are planned, and how data gets into this structure and out of it.

	PostgreSQL 19	Cloudberry without vexec	pg_vexec on vanilla PostgreSQL 19	pg_vexec in the Cloudberry port
Unit of execution	a row	a row: the open-source code has no vectorized executor	a batch of up to 1,024 rows by column; kernels by function OID, fallback inside the same node	the same, on the coordinator and every segment
Who decides what to vectorize	-	judging by PAX’s tests, the closed-source engine converts a finished plan node for node, with the row plan’s costs	both planners, by cost: PostgreSQL’s path hooks, and ORCA	ORCA, costing vector nodes, for MPP plans; PostgreSQL’s planner on the segments
Table reads	heap through slots	AOCO and PAX return slots	heap page by page, with batched MVCC checks; other AMs through slots	PAX and ao_column return batches; aggregates from PAX’s statistics
Inserts	ModifyTable row by row; `multi_insert` only in COPY FROM	the same; PAX and gp_ao split rows into columns themselves	VecInsert; into heap 1,000 rows at a time through `table_multi_insert`	VecInsert writes columns into PAX and ao_column without rows
Between processes	Gather: a MinimalTuple per row	Motion: a MinimalTuple per row and an fmgr hash of the key per row	as in PostgreSQL	Arrow IPC frames, vectorized cdbhash, shared memory for segments on one host
Arrow for clients	no: ADBC assembles Arrow from binary COPY	no	Flight SQL: results served from batches, inserts into VecInsert	the same, on the coordinator
PostGIS and pgvector functions	row by row	row by row	a batch’s rows at a time through fmgr, by the packs’ declarations	the same, on the segments
Core changes	-	a fork; the port has 24 hook patches	none	only those the port itself needs

Now in order, starting with what it is all built from.

Components

vexec is a single module in shared_preload_libraries, built with PGXS both for vanilla PostgreSQL 19 and for the Cloudberry extension port. The vectorized planner (plan/) is built into both planners and decides which operators become vector nodes. The executor is a set of CustomScan nodes (exec/) on top of a shared batch layer (batch/) and an expression compiler (expr/). Alongside them live the Arrow IPC codec (ipc/), the egress API for Flight SQL (egress/) and the Motion frames (motion/).

External links go through PostgreSQL’s rendezvous variables, with the major version in their names, the same way the port’s modules find gp_core’s API. Storage modules register batch readers in vexec/source_v1 and sinks in vexec/sink_v1, vexec_flight finds vexec/egress_v1, the kernel packs find vexec/kernels_v1, and vexec itself finds ORCA through Cloudberry/gp_orca_vec_v1. The order of the modules in shared_preload_libraries doesn’t matter, and without the library loaded, a variable stays an empty entry in the server’s memory and nothing is registered.

A vectorized planner inside the database, not next to it

In PostgreSQL’s planner, vexec adds paths through the planner’s own hooks: VecScan in set_rel_pathlist_hook, just as Citus’s columnar storage adds its scan, VecHashJoin in set_join_pathlist_hook, and VecAgg and VecSort in create_upper_paths_hook. From there they compete with row paths in the ordinary add_path. A vector path’s cost is derived from the row path’s, with multipliers for the kernels’ work, a columnar source estimates only the bytes of the columns it needs, and the transitions between rows and batches are chosen by cost. The strategy masks are honoured as well, so enable_seqscan = off and pg_hint_plan’s hints apply to vector paths too. PostgreSQL calls no hook for partial aggregation, so vexec builds the pair of partial and final VecAggs around a Gather itself:

SET vexec.mode = force;
EXPLAIN (COSTS OFF)
SELECT hw.k, count(*), sum(hp.b) FROM hp JOIN hw ON hp.id = hw.id GROUP BY hw.k;

 Vec Finalize HashAggregate
   Group Key: k
   ->  Gather
         Workers Planned: 3
         ->  Vec Partial HashAggregate
               Group Key: k
               ->  Vec Hash Join
                     Hash Cond: (hp.id = hw.id)
                     ->  Parallel Vec Seq Scan on hp
                     ->  Vec Seq Scan on hw

The path hooks can’t reach ORCA’s plans: ORCA builds its own PlannerInfo and returns a finished plan. So the port uses a small API in gp_orca, through which vexec registers its oracle, its cost estimates and its node builders:

CCostModelVec, a subclass of CCostModelGPDB, evaluates ORCA’s formulas twice, as they are and with vector multipliers, and blends the two by the share of steps that go to kernels. ORCA thus chooses join orders, aggregation stages and Motions with vectorized execution already in the cost;
the translator offers vexec every finished node, and vexec replaces it with a vector node if the oracle agrees, before the Motions are checked, the slice table is made and M8’s passes run, so every later step of the Cloudberry port sees the final tree;
ORCA’s hashed window aggregation, which PostgreSQL 19 has nothing to execute with, becomes a VecWindowHashAgg under an ordinary WindowAgg.

In vexec.mode = explain, vexec costs the vector alternatives but doesn’t choose them, and EXPLAIN (VEXEC) explains what was considered and why it wasn’t taken:

SET vexec.mode = explain;
EXPLAIN (VEXEC, COSTS OFF)
SELECT b, count(*), sum(d) FROM vt WHERE a > 10 GROUP BY b ORDER BY b;

 Sort
   Sort Key: b
   ->  HashAggregate
         Group Key: b
         ->  Seq Scan on vt
               Filter: (a > 10)
 Vexec: mode explain, format postgres
   VecScan on vt: not chosen (explain mode)
     source: heap's pages; quals: 1 kernel step, 0 fallback steps; target: 0 kernel steps, 0 fallback steps
   VecAgg on GROUP BY: not chosen (explain mode)
     hashed, 1 grouping column, 2 aggregates, 2 with vector transitions (count(*), numeric sum)
   VecSort on ORDER BY: not chosen (explain mode)
     1 sort key

Planning doesn’t get any more expensive: EXPLAIN of all 121 TPC queries on four segments under ORCA took 9,715 ms without vexec and 9,696 ms with it, and under PostgreSQL’s planner 161 and 166 ms.

In phase V6 I brought ORCA to vanilla PostgreSQL 19. The -Dorca_single_node build replaces gp_core with a stub of about 400 lines: it answers calls to gp_core’s API, and to the 14 of its functions that ORCA calls on a single node, the way gp_core would answer on a server without segments. With ORCA on vanilla REL_19_STABLE, all 239 regression tests pass, 27 of them with examined differences, and the 121 TPC queries in force mode answer the way DuckDB does. Of the 652 hash joins in their plans, 651 became VecHashJoin: all but a full join, which VecHashJoin can’t do yet.

Two in-memory data formats

The batch format is chosen by a setting; that was my requirement. vexec.batch_format = postgres, the default, stores values the way PostgreSQL defines them, and arrow stores them in standard Arrow types:

Type	postgres format	arrow format
bool	a byte per value	a bit per value
date, timestamp	PostgreSQL’s epoch, 2000-01-01	the Unix epoch
interval	PostgreSQL’s 16 bytes	`month_day_nano`
text, bytea	a Datum per value, with the varlena header	`utf8_view` or `binary_view`
int, float, uuid, numeric with a typmod of up to 38 digits	the same in both formats: values at the type’s width, numeric as scaled int64 or int128

PostgreSQL’s own boundaries cost the postgres format nothing: fmgr, slots, tuplesort, hash functions, and the heap, ao_column and porc storage. Arrow’s boundaries cost the arrow format nothing: export, Flight SQL, porc_vec and frames between processes. Conversions happen only at these named boundaries, and the cost model accounts for their price. The validity bitmap is shared by both: it has the same polarity and bit order as the NULL bitmap of a heap tuple. Kernels are generated for each layout, and four settings change the layout separately for strings, bool, dates and numeric. On ClickBench the two formats differed by no more than 2.6%.

The scaled numeric was borrowed from openGauss as an idea but written from scratch to PostgreSQL’s numeric rules, and the gain on TPC-H Q1 is almost entirely its doing: with the varlena layout Q1 got 2–9% faster, with the scaled numeric 1.8–2.8× faster. The epochs have an edge case, though: the timestamp 294247-01-10 04:00:54.775807, shifted to the Unix epoch, gives exactly the int64 maximum, which PostgreSQL uses for +infinity, so a batch holding that timestamp keeps its column in PostgreSQL’s epoch.

Columnar data structures from storage

The source contract answers the question “why would the table AM need new hooks at all?” A storage module publishes begin, next, rescan, end and estimate functions through a rendezvous variable, and the scan is opened by the ordinary table_beginscan, so the snapshot, predicate locks, parallel scan ranges, MVCC, pruning and deletes all stay with the access method. A deleted row goes into the selection bitmap, not into validity; otherwise count(*) would count it.

heap is read by vexec itself, page by page, through heap_prepare_pagescan, the way TABLESAMPLE does it: page pruning, locking, PostgreSQL 19’s batched visibility check HeapTupleSatisfiesMVCCBatch, and the needed columns deformed straight into the batch. vexec doesn’t decide visibility with code of its own: in the port the snapshot is distributed.
PAX returns groups of up to 131,072 rows, sliced into batches without copying, and porc_vec stores almost Arrow’s layout. For aggregate(), count, min, max, sum and avg without conditions are taken straight from the files’ and groups’ statistics, and count(*) over 10 million rows takes 0.33 ms instead of 265. And through set_keys(), a VecSort with a LIMIT passes the scan a “running bound”, so PAX skips groups that can no longer make it into the answer.
ao_column returns blocks of up to 16,384 rows: a fixed-width column without NULLs becomes a slice of the decompressed buffer, and a varlena column becomes Datums pointing into it.

Inserts mirror this. VecInsert takes the place of ModifyTable’s loop and writes a batch into the storage’s sink or, where there is none, through table_multi_insert 1,000 rows at a time, as COPY does. NOT NULL is checked against validity, and the first row that fails the check goes to ExecConstraints() so that the error is the same. Partitions are chosen by PostgreSQL’s functions, indexes get their entries through ExecInsertIndexTuples(), and once again the core didn’t need touching: COPY FROM inserts without ModifyTable too. On a cluster, ModifyTable stays on the coordinator, because gp_core dispatches the write through it, and VecInsert works under it on the segments. Triggers, foreign keys, ON CONFLICT, RETURNING and WITH CHECK keep the ordinary ModifyTable, and EXPLAIN names the reason.

Everything vexec required of my Cloudberry port is module code in pg19/: gp_orca’s API and CCostModelVec, the entry points of GpCoreApi 1.15, the vexec.* names in the list of settings that the coordinator sends to the segments, PAX’s and gp_ao’s batch readers and sinks, and shm, the transport within one host. ORCA’s four core libraries compile from the Cloudberry tree unmodified. vexec doesn’t call the port’s core patches: it may use them indirectly, but only the ones PAX and gp_ao use, plus the memory hook, which counts its batches in resource groups like any module’s memory.

Arrow IPC through Motions

Between segments, Cloudberry sends rows: for every row a Motion forms a MinimalTuple and calls the key’s hash function through fmgr, and rows go to the coordinator through libpq with a send and a receive for every value. If there are vector nodes on both sides of a Motion, a batch is taken apart into rows only to be put back together again.

I asked whether the distribution keys could be read straight from Arrow buffers, and batches passed between nodes over Arrow Flight SQL. The first is done, the second isn’t: Flight would bring C++, gRPC and threads into the backends; flow control, the receiver’s STOP and the statement token, all of which the Cloudberry port’s interconnect already has, would have to be built again; and it would copy no less. So a batch travels over the port’s transports as an Arrow IPC frame, and the Motion code itself doesn’t change:

When the translator offers vexec a Motion whose sending fragment has a vector node at its top, vexec puts a VecMotionSend under it and a VecMotionReceive over it, and rebuilds the Motion itself in the same place through gp_core’s setters: a Redistribute becomes an Explicit Redistribute by segment number. A row of such a Motion is the fragment’s columns, all NULL, the segment number and the frame as a bytea.
A frame is 16 bytes of vexec’s own (VXF1, a reserved field, the schema hash) followed by an Arrow IPC message from V7_0’s codec. In the arrow format the buffers go out as they are; in the postgres format a varlena is written whole, and the receiver’s Datums point straight into the frame. The schema goes to each receiver once and is kept there by its hash, since the frames of all the senders arrive interleaved.
The vectorized cdbhash reproduces GpHashSegment bit for bit, otherwise rows would go to the wrong segments: kernels for integers, float, text, uuid and dates, hash_numeric reproduced from the scaled integer, and a row-by-row path through gp_core for legacy keys.
A VecHashJoin that no longer needs its side with a Motion stops that side’s senders through squelch_subtree(), and the vector nodes’ counters come back from the segments into EXPLAIN ANALYZE.
Frames go to the coordinator through the same binary libpq cursor, without a send and a receive for every value.

For segments on one host, the port gained the shm transport: the sender creates an anonymous file with memfd_create, passes the descriptor to the receiver through SCM_RIGHTS after the statement token and writes frames into a ring, and the receiver reads them in place, so writing into the ring is the only copy. Both sides sleep on an eventfd in their WaitEventSet, so cancellation and timeouts work as they do over tcp, and the anonymous mapping disappears along with the last process, even one that crashed. Between hosts, a slice goes over tcp.

Both switches take effect from the next statement: vexec.enable_motion_frames is read at planning time and calls ResetPlanCache() when its value changes, so prepared statements and PL/pgSQL are planned again, and the transport is chosen by gp.interconnect_type. A sorted Gather, which merges rows, a write’s Motion and a Motion over a row node are not framed. Here is a plan from a test cluster of three segments:

 Vec Finalize Aggregate
   ->  Vec Motion Receive
         ->  Gather Motion 3:1  (slice1; segments: 3)
               ->  Vec Motion Send
                     Frames To: the one gathering
                     ->  Vec Partial Aggregate
                           ->  Vec Hash Join
                                 Hash Cond: (h.n2 = x.n)
                                 ->  Vec Motion Receive
                                       ->  Explicit Redistribute Motion 3:3  (slice2; segments: 3)
                                             ->  Vec Motion Send
                                                   Frames To: the segments their keys hash to
                                                   Hash Key: n2
                                                   ->  Vec Seq Scan on lg_h h
                                 ->  Vec Seq Scan on lg_n x
 Optimizer: GPORCA

The cluster tests pass both over tcp and under shm, where all 2,777 senders and 2,777 receivers went through rings, and after a receiver is terminated, neither a mapping nor a file in /dev/shm is left behind. On wide rows of 1.8 KB, shm came out faster: about 80 ms against 93 over tcp.

Arrow both in the database’s memory and on the wire with Flight SQL

Today, Arrow clients read PostgreSQL through ADBC and binary COPY: the server sends rows, and the driver assembles columns from them. vexec_flight serves Arrow straight from batches, and only while the vectorized executor is on: the acceptor starts only if vexec_flight.listen_addresses is set and vexec is loaded, with vexec.mode = off a statement gets FAILED_PRECONDITION, and the extension has no row path to the client at all.

For BI and ML, Flight SQL is more than just another transport. Over pgwire a result travels as rows: every value goes through its type’s send function or is turned into text, a driver such as psycopg or JDBC assembles rows, and then pandas, Polars or DuckDB lay them out by column all over again. Over Flight SQL the client receives Arrow record batches, the very layout these libraries keep their data in anyway: pyarrow hands over a fixed-width column without NULLs as a NumPy array without copying. The schema with its types arrives before the first row, decimals with their precision and scale, timestamptz as UTC time, so a BI tool doesn’t have to guess types from text. The drivers are standard ones: ADBC Flight SQL for Python, Go, Java and C, and Flight SQL JDBC for anything that takes a JDBC driver. The way back is columnar too: adbc_ingest() writes a DataFrame straight into a table through VecInsert. Flight served a million lineitem rows 4.7× faster than binary COPY. Flight SQL only complements PostgreSQL’s familiar pgwire clients; it doesn’t replace them.

The process model here is PostgreSQL’s, not a service with threads: an acceptor without threads asks the postmaster to start a dynamic background worker for each connection and hands it the socket through SCM_RIGHTS. The session itself runs TLS, HTTP/2 on nghttp2 and gRPC, and waits on its socket and its latch the way a backend does, so pg_cancel_backend() and CancelFlightInfo work on it. A login is checked against pg_hba.conf with the client’s real address, the role’s password and the connection limits, and statements run through portals, so hooks, privileges, RLS, transactions and pg_stat_statements behave as they do over PostgreSQL’s usual protocol.

The result is written by vexec’s DestReceiver. If there is a vector node at the top of the plan, the ExecutorRun hook hands over its batches without a single row, and the columns whose layout matches their Arrow type go to the socket in one writev straight from the batch’s buffers. A slow client simply slows the session down: while a 20 GB result was read at a leisurely pace, the session’s memory grew by less than a megabyte.

Inserts take the reverse path. CommandStatementIngest becomes INSERT INTO t ... SELECT ... FROM vexec.ingest_stream(handle), and VecIngest takes the next DoPut message from the socket only when the statement needs the next batch. The client’s Arrow is checked as input (offsets, UTF-8, date ranges, decimal precision), and where a column’s type matches the Arrow type, the message’s buffers become the buffers of the batch that VecInsert hands to the PAX or ao_column sink.

PostGIS and pgvector functions

vexec calls other extensions’ functions through fmgr row by row, inside a vector node. But pgvector doesn’t mark its functions leakproof, and PostGIS marks only its btree comparisons and geometry_hash, so a condition with a distance or ST_Intersects became lazy as a whole. A kernel pack doesn’t rewrite functions or introduce an ABI: on top of fmgr and PostgreSQL’s soft errors, it declares which functions may be called, and on which rows, earlier than PostgreSQL would call them:

never raises: the function is called on the batch’s active rows; these are PostGIS’s twelve 2-D box operators and ST_SRID;
check: the function is called where the pack’s check passes, and PostgreSQL evaluates the remaining rows itself, with its own error; these are pgvector’s distances when the dimensions are equal, and ST_X and ST_Y for a point;
prefilter: the pack’s answer is taken where it has one, and an undecided row (a soft error from ereturn) goes to the function itself; these are eleven PostGIS predicates, by their boxes.

A declaration is bound by the extension, its version, and the function’s signature and C symbol, and it is dropped on ALTER EXTENSION UPDATE. vexec_postgis reads the geometry format with its own code, written from gserialized.txt, without a single line of PostGIS. pgvector’s tests (14) and PostGIS’s core tests (143) give the same answers with vexec as without it, and a server without a pack evaluates the same calls row by row: slower, but correctly.

Implementation caveats

Inherited from PostgreSQL 19:
- Gather and Gather Merge stay row-based: the tuple queue carries one MinimalTuple per row;
- ExecSetTupleBound doesn’t reach CustomScan, so VecSort gets its LIMIT bound at planning time;
- there are no vector nodes under UPDATE, DELETE, MERGE, LockRows or WHERE CURRENT OF cursors: EvalPlanQual re-reads a row into the scan node’s slot, and for heap that has to be a buffer slot.
Compared with Cloudberry:
- on vanilla PostgreSQL, ORCA’s plans are serial: its parallelism is switched on by gp_core’s gp.enable_parallel setting;
- a sorted Gather Motion doesn’t carry frames; it merges rows;
- PAX has no index_delete_tuples, so after an aborted insert into a table with a btree index the next insert may fail, exactly as it does through ModifyTable.
Vector-specific:
- COUNT(DISTINCT) grouped by strings is 3.1–3.6× slower under PostgreSQL’s planner in auto mode: the cost model prices a serial vector plan below a parallel row plan;
- text columns come out of a heap scan more slowly than out of a row scan;
- force gives up index paths, except for nearest-neighbour search, so on a table with a primary key auto is faster than force.

Measurements

All the figures come from Docker images built from committed branches, on my 16-core Ryzen 9 9955HX3D with 60 GB of memory, without assertions, and every answer was checked against DuckDB’s.

ClickBench was run by its own protocol on vanilla PostgreSQL 19 with ORCA, but on 10 million rows, so these figures can’t be compared with the published results on 100 million. The geometric mean of the hot-run timings of the 43 queries, in ms:

Configuration	heap	heap with a primary key
PostgreSQL’s planner	625	402
PostgreSQL’s planner + vexec, auto	506	311
PostgreSQL’s planner + vexec, force	498	365
ORCA	1,647	924
ORCA + vexec, auto	1,349	745

vexec takes 19–24% off PostgreSQL’s planner in auto mode and 16–19% off ORCA, and the best queries get 8.8–9.5× faster: Q15 and Q16 under PostgreSQL’s planner, Q29 under ORCA. Under PostgreSQL’s planner, Q10, Q11 and Q13 with COUNT(DISTINCT) got slower; under ORCA, Q25 and Q39, where the scan returns text columns. ORCA trails PostgreSQL’s planner by 2.3–2.6×, above all because of its serial plans.

Flight SQL, TPC-H Q1 at SF1, the median of five runs from execute to an Arrow table in the client:

	vanilla PostgreSQL 19	the port’s coordinator, 2 segments
`adbc_driver_postgresql`, vexec off	0.743 s	1.115 s
`adbc_driver_postgresql`, vexec auto	0.400 s	0.567 s
`adbc_driver_flightsql`, vexec auto	0.389 s	0.555 s
the first million lineitem rows, `adbc_driver_postgresql`, vexec off	0.917 s	1.012 s
the same through `adbc_driver_flightsql`, vexec auto	0.194 s	0.748 s

Q1’s time is the executor’s time, and vexec cuts it almost in half, whichever driver reads the result’s four rows. The transport shows on a million rows: Flight serves them 4.7× faster than binary COPY. On the port, the measurement was taken before V7, and the result still went through the Gather Motion as rows.

Inserting 200,000 lineitem rows, the median of three loads, in rows per second:

Path	porc_vec, one node	ao_column, one node	heap, one node	porc_vec, 2 segments
COPY CSV from a server file	955,491	1,176,792	1,294,406	568,093
binary COPY through ADBC	547,494	635,661	651,960	399,412
Flight SQL before VI: DoPut, a portal per row	27,466	-	-	771
Flight SQL `adbc_ingest` through VecIngest and VecInsert	1,580,626	1,826,448	1,545,523	647,592

Into porc_vec, Flight loads 1.65× faster than COPY CSV and 2.9× faster than binary COPY. On the cluster the gain is only 1.14× so far: the stream crosses a Motion as rows and is waiting for V7’s frames.

To be fair: the first TPC measurement on four segments after V2, when only scans and aggregation were vectorized, averaged out at about 1×. Q1 got 1.8–2.8× faster, but wherever a vector scan fed a row hash join, up to half the speed was lost, which is why VecHashJoin and heap’s page reader came next. Their measurement is waiting for a quiet host.

The implementation process

It all started with a question to Claude Code: how to build a vectorized executor inside PostgreSQL, by reusing ORCA with vectorized planning on top of Cloudberry’s columnar formats, or by adding columnar-access hooks to the core. Nine research reports went through the code of PostgreSQL 19, the Cloudberry port, PAX, AOCO, Arrow and the predecessors. The plan grew to 4,700 lines and 113,000 words; every claim in it is a file:lines reference or a number measured on this host, and its header keeps a log of my questions and of what changed after them.

About fifteen of my questions changed the architecture before the first line of code was written: two batch formats, keys read from Arrow buffers, shared memory for segments on one host, Flight SQL as a separate extension, columnar inserts without core changes, kernel packs, a ClickBench measurement before implementation began. I turned down nanoarrow’s IPC: its writer copies every buffer of a batch into one body, doesn’t understand string views and allocates memory with malloc.

Phase	Date	Result
VB	Oct 5	ClickBench baseline before the first line of vexec
V0	Oct 5	groundwork: the oracle, the cost model, two batch formats, the source contract
V1	Oct 5	VecScan, VecResult, the expression compiler, both planners, segments, PAX and gp_ao readers
V2	Oct 6	VecAgg, aggregates from PAX’s statistics, the first measurement
V3	Oct 6	VecHashJoin
V4	Oct 6	heap’s page reader, parallel paths, VecSort, VecRepartition, VecBitmapHeapScan
V5	Oct 6	CCostModelVec, VecWindowHashAgg
V6	Oct 6	ORCA on vanilla PostgreSQL 19
VK	Oct 6	kernel packs for pgvector and PostGIS
V7_0, V10	Oct 6	the Arrow IPC codec, the egress API, vexec_flight
VI	Oct 7	sinks, VecInsert, inserts from Flight SQL
V7	Oct 7	Arrow frames through Motions, the shm transport

Here is how it went over time:

Each phase gets its own branch and its own worktree, and a commit happens only after I approve it, after which the task statuses in the plan are updated. On October 6, V4, V5, V6, VK and V10 were built by parallel sessions, while the Cloudberry port’s session took the defects they found in Cloudberry and fixed them in its own session: there were about ten, in PAX, gp_core and ORCA, plus one more in PostGIS 3.7.0rc2. The sessions take turns with the 40 GB runs: two of them side by side once hung the host.

There were surprises this time too:

A test that passes without the fix. Since V1, the boolean register’s type had lived in the statement’s memory, and the next statement read freed memory. In a single query string the check passed, because an identical type took up the same memory, but sent as separate queries it crashed the backend every time. Now every regression check is first run without its fix.
Unknown geometry type: 2139062143. That’s 0x7F7F7F7F, a debug build’s freed memory: vexec evaluated a lazy target in the query’s memory, that is, in the functions’ fn_mcxt, and PostGIS’s caches considered that memory their own. PostGIS’s own test caught it;
ORCA pruned the best plan. ORCA uses a partial plan as a lower bound for pruning, and CCostModelVec costed it above the plans it bounds, so ORCA threw away cheaper aggregations and join orders;
OID 7100. ORCA hard-codes Cloudberry’s hash OIDs as constants, while gp_core assigns them at installation, so a GROUP BY on a table keyed by cdbhash_int4_ops failed with could not find hash function for type 23 in operator family 7100. I ran into it while implementing Arrow data transfer through Motions.

The outcome

In three days of implementation I built a vectorized query executor and planner as a PostgreSQL 19 extension that:

runs on a vanilla server without a single core patch, ORCA included, and goes back to vanilla PostgreSQL behaviour with one setting;
turns ORCA into a vectorized engine, on a single node just as on every segment of an MPP cluster;
moves data in columnar form from PAX and ao_column into the executor’s memory and back without needless serialization and deserialization, and passes batches between segments as Arrow frames;
serves and accepts data as Arrow through Flight SQL, straight from the planner’s and the storage’s data structures;
processes PostGIS and pgvector data types with their own functions over columnar buffers in the query executor’s memory;
behaves like PostgreSQL: 239 regression tests pass without edits to the expected files, and the TPC and ClickBench answers match DuckDB’s reference answers.

That’s about 43,000 lines of C in vexec, 7,900 in vexec_flight, a thousand in the kernel packs and about 13,600 in the Cloudberry port’s modules in pg19/, the shm transport and copies of PAX files included. Next up are further optimizations and performance measurements, and running ClickBench on columnar tables in Cloudberry. Of course the project still has room to grow and things to optimize, but even at this stage it is far more than a prototype for speeding up PostgreSQL. The result is an open-source database that integrates better with the AI/ML ecosystem by speaking Apache Arrow over Flight SQL to the outside world, while still supporting the existing PostgreSQL drivers.

In a few days of implementation I proved that vectorizing query execution in PostgreSQL doesn’t need a fork. PostgreSQL 19 already has everything needed to plan vector nodes inside the planner rather than patch plans after the planning phase. And PostgreSQL’s semantics were preserved without sacrificing query speed. ORCA, inherited from Greenplum, now plans vector nodes even on vanilla PostgreSQL, without gp_core and without a single core patch. A modular approach in an open-source database is worth far more than walling a fork off behind a high barrier to entry and the exclusivity of its maintenance. I believe that in the long run, modularity and an open ecosystem pay off more for the developer community and for those who run PostgreSQL themselves.

The results are available here: pg_vexec (vexec), pg_vexec_flight, pg_vexec_pgvector and pg_vexec_postgis; the Cloudberry port’s changes needed for vectorization are in pg19/.

Greenplum without the fork: Apache Cloudberry as PostgreSQL 19 Extensions

2026-09-30T00:00:00+00:00

This is my personal experiment, not an official Apache Cloudberry release.

Since 2017 I have been keeping an eye on the PostgreSQL ecosystem for analytics and data warehousing. At work I had used AWS Redshift, with all the drawbacks and inconveniences a developer gets from a fork of an old PostgreSQL. To my mind, the ideal PostgreSQL-based solution should run on the latest PostgreSQL release, be built as an extension, be easy to start locally in a container for tests, and work with the latest versions of existing extensions such as PostGIS and pgvector.

Citus came closest to what I wanted, but I wouldn’t call it a full-fledged massively parallel processing (MPP) system. It is more about sharding data for OLTP workloads.

Greenplum, an MPP fork of PostgreSQL, has a lot going for it, and plenty of people run it for real work, but the PostgreSQL it is built on lags behind the current one. So I set out to turn Apache Cloudberry, the most up-to-date open-source version of Greenplum, into a set of extensions for PostgreSQL 19. For now this is my personal experiment, not an official Apache Cloudberry release. With an LLM’s help, more than 90% of Cloudberry’s database code has been ported, and 59% of the original project’s tests now run on the new version. It took me 12 days…

Greenplum has always trailed PostgreSQL by a few releases. Greenplum 6 shipped on PostgreSQL 9.4 and Greenplum 7 on 12, and Apache Cloudberry, which carries the Greenplum project on, got its main branch up to 16.9 only in May 2026. The reason is that this massively parallel database is a fork: Cloudberry has changed 882 of PostgreSQL’s source files and added 646 new ones, not counting the 1.66 million lines of its ORCA optimizer, test data included. Every new PostgreSQL major version means months of merging, and by the time the developers finish, upstream has moved on.

I decided to go the other way around and cross a hedgehog with a snake, there is a saying: instead of pouring PostgreSQL’s sources into Greenplum, fit Cloudberry on top of PostgreSQL 19 as a set of extensions. The most important principle was that PostgreSQL 19 may get a few small changes for new extension points, but a server with my core patches and without Cloudberry’s extensions loaded must be indistinguishable from vanilla PostgreSQL 19, both to its users and to other extensions, and must perform almost exactly like the unmodified core, within a fraction of a percent.

Claude Code wrote the code for me, in many sessions, with agents working in parallel git worktrees. I discussed the initial plan with the LLM, made the architectural decisions and checked the results. The last time I wrote serious C/C++ was almost 20 years ago, and yet I consider this experiment a success!

All told, it took 12 days and 629 commits to get an MPP engine that works like the Greenplum fork, not just another FDW that drags all the data to the coordinator. Plans are cut into slices, rows travel between segments while a query runs, and transactions use distributed snapshots and two-phase commit. PostgreSQL itself needed only 22 small patches that add new extension points. (I confess there were 28 at first; I brought the number down later, at the cost of a few compromises.)

TL;DR

What	Number
PostgreSQL 19 core changes	22 commits, 47 files, +1,578/−41 lines
The port	one new directory, `pg19/`: 1,153 files, +432,978 lines. Zero lines changed in Cloudberry’s own files
ORCA	920 core files (~380k lines) compiled from Cloudberry’s tree unmodified
Cloudberry database code ported	≈93% by lines of code
Cloudberry’s own tests passing	691 of the 1,180 test files in its schedules (59%)
TPC-H + TPC-DS, scale factor 1	ORCA plans all 121 queries; answers match DuckDB’s; 3.1× and 4.0× faster than the non-ORCA fallback
Cost of the dormant hooks	−0.005% to +0.155% executed instructions

Code: core series, port, build instructions.

Why an extension

Pivotal open-sourced Greenplum more than a decade ago, in 2015, but some time after Broadcom bought VMware, the project’s repositories were archived. The same code now lives on in forks: open-gpdb, EDB’s WarehousePG and Apache Cloudberry. Cloudberry is based on Greenplum 7 and entered the Apache Incubator. PostgreSQL 19 isn’t stable yet: it is at Beta 4, with a release candidate expected any week now.

How big is the thing being ported? SonarCloud counts 2.6M lines of code on Apache Cloudberry’s main branch (analysis of September 28, 2026, commit 3f4c688). That is 4.0M lines with comments and blank lines, in 10,189 files:

Language	Lines of code
C	1,396,894
SQL (SonarCloud’s PL/SQL analyzer)	731,180
C++	337,098
XML	82,263
Python	65,961
Shell	7,310
YAML	6,670
HTML, CSS, JSON, Docker, other	1,869
Total	2,629,245

Nine-tenths of the SQL is tests: 438k lines in src/test and 218k in PAX’s test suites. The count covers the whole repository, tests and tools included, so it is not the base of the ≈93% figure, which weighs the database code module by module.

The main advantage of an extension is that you get the latest PostgreSQL and almost every tool in its ecosystem unmodified, as ready-built packages. Moving to the next major version is also far less painful than it is for a fork: rebasing means carrying 22 small patches, about 1,600 lines in total, over to the new release, not months of merging.

For current Greenplum and Cloudberry users, the extension brings:

A current PostgreSQL, from MERGE and the latest SQL/JSON goodies to asynchronous I/O and parallel autovacuum.
Community extensions. The nodes can run upstream PostGIS 3.7 instead of Cloudberry’s fork of PostGIS 3.3.2, and pgvector 0.8.6 runs unpatched. Binary modules built for vanilla PostgreSQL 19 load as they are, because the core’s ABI hasn’t changed.
The familiar bits: DISTRIBUTED BY and the classic PARTITION BY ... EVERY, gpfdist, gp_toolkit, gpMgmt’s tools from gpinitsystem to gpexpand, and EXPLAIN output with Motion nodes.

There is no free lunch, of course. You pay with a logical migration (dump and restore) instead of pg_upgrade. The 438 configuration settings are renamed with a gp. prefix, although gpconfig still accepts the old names. Much of Cloudberry’s DDL is translated under the hood into standard PostgreSQL statements, and that is what command tags and dumps show. And database-level encryption gives way, for now, to volume encryption in the operating system.

For newcomers to Greenplum, the extension means:

The patched PostgreSQL looks like any ordinary PostgreSQL as long as Cloudberry isn’t loaded into it.
Even on a single node, loading the modules gives you incremental materialized views, a task scheduler, append-optimized tables and PAX columnar storage, and, most useful of all, the ORCA optimizer.
Going MPP on a cluster doesn’t mean trading the PostgreSQL you know for another ecosystem, other drivers or other BI tools.
Your PL/pgSQL, PostGIS and pgvector stay with you.

Cloudberry as a PostgreSQL-compatible MPP warehouse

A cost-based distributed optimizer plans each query. The plan itself, serialized, is shipped to the nodes and runs on all segments at once, and the segments pass the data they need to each other through Motions. Through all of this, transactions stay ACID across the whole cluster, and compressed columnar storage is there when you want it. On top of it all sit workload management and failover.

The typical use is a data warehouse of tens or hundreds of terabytes in star and snowflake schemas, with reports that join many large tables. It is also often used to run ELT in SQL and PL/pgSQL.

Here is how it compares with PostgreSQL extensions as of September 2026:

	My Cloudberry port	Citus 14.2	pg_duckdb 1.1.1	pg_lake 3.5.3	PG-Strom 6.1
Scale	cluster of segments	cluster of shards	one node	one node + DuckDB in a sidecar process	one GPU server
Execution	PostgreSQL executor; plan slices on every segment, Motions between them	PostgreSQL executor; SQL fragments per shard	vectorized DuckDB in each backend	vectorized DuckDB in `pgduck_server`	GPU code generated from SQL
Large joins	ORCA, cost-based distributed plans	join order “relatively naive” (its docs); repartition joins off by default	one machine	pushed to DuckDB, otherwise run by PostgreSQL	inner side must fit in GPU memory
Consistency	2PC + distributed snapshots	2PC, no global snapshot	no transaction writing to both PostgreSQL and DuckDB tables	Iceberg UPDATE/DELETE run one at a time	PostgreSQL’s
Core	22 core patches	stock	stock	stock	stock

pg_mooncake belongs to the same single-node lakehouse family, but Databricks bought Mooncake Labs in October 2025 and version 0.2 never shipped.

Of all of these, only my Cloudberry port runs a complex query over large tables as a single distributed, pipelined plan in which every node reads the same distributed snapshot. It also lets you keep PostGIS, pgvector and other types in its columnar tables and run UPDATE and MERGE on them. Where it loses, for now, is speed per CPU core: PostgreSQL’s executor is an engine built around rows in memory. And in its current form the extension is unlikely to make it into managed PostgreSQL services: it needs a patched core, and the likes of AWS and Azure won’t go for that.

Among the popular open-source engines for analytical queries, ClickHouse 26.9 and StarRocks 4.1.5 are entirely different beasts, with vectorized columnar executors. StarRocks has its own Cascades optimizer and lakehouse catalogs, while its multi-statement transactions are still in beta. ClickHouse has its own SQL dialect and experimental transactions.

ClickBench shows the gap between the executors: one flat table of 100 million rows, 43 queries, no joins, on a c6a.4xlarge machine. Each figure is the total over all 43 queries of the hot run, meaning the better of runs 2 and 3, as ClickBench defines it; each column header links to its published result file:

ClickHouse	DuckDB	StarRocks	pg_duckdb (Parquet)	Cloudberry 1.5.3 (AOCO)	Greenplum 7.1 (AOCO)	Citus columnar	PostgreSQL heap
17.6 s	26.3 s	44.5 s	52.5 s	294.7 s	768.6 s	1,745.1 s	11,875.4 s

Source data: the ClickBench repository at commit ad5c59b (2026-09-28); the same results are charted at benchmark.clickhouse.com.

On scans and aggregates the vectorized engines are an order of magnitude ahead, because Cloudberry’s executor processes rows in memory without vectorization, and so does my port’s, which is simply PostgreSQL’s. But this benchmark has no joins of many large tables, and that is exactly where MPP engines justify their architecture and optimizations.

So my rule of thumb is:

Cloudberry for complex SQL over large tables with transactions, PL/pgSQL, PostGIS and pgvector, on your own servers;
Citus for multi-tenant workloads routed by key;
pg_duckdb and pg_lake for single-node speed and data in Iceberg;
ClickHouse for logs and events, where it shines;
StarRocks for joins combined with lakehouse elasticity.

The ground rules

It all started with a plan. My two prompts to Claude Opus 5 were “research cloudberry project directory and explore how to use hooks and macros to move it with minimal code changes to PostgreSQL v19” and “Plan the approach so that modifications to the PostgreSQL core—made without the Greenplum extension loaded—do not alter the operation or functionality of the PostgreSQL server.” The result was six rules that the coding agents followed strictly:

Use PostgreSQL’s existing extension points first: hooks, table access methods, CustomScan, custom WAL resource managers, background workers, security labels, FDWs.
Where they aren’t enough, add a minimal hook of your own. Here, a hook of your own means a function pointer that is NULL by default, an exported function, a registration list or a flag that is off by default. No existing signature changes, no existing struct gains a field, and the catalog, WAL, page and protocol formats stay as they are.
Without the extension loaded, the server behaves like vanilla PostgreSQL.
Cloudberry’s existing code isn’t edited. The whole port is one new directory, pg19/, and the upstream project’s files are left alone, which keeps merges simple. If a file does have to change, a copy of it goes into the extension’s directory with a note on what was modified.
ORCA sits behind planner_hook. What it can’t handle is rewritten in the query before it gets there; the optimizer itself is not patched.
Third-party extensions stay as upstream ships them, unedited. Where an adaptation is needed, it lives in pg19/, as it did for a couple of PostGIS features.

Vanilla behaviour is checked by tests that compare two Docker images built from the same PostgreSQL commit, one with the patch series and one without:

PostgreSQL’s own meson test: the same 371 tests pass;
abidiff: 32 symbols added, 0 changed;
catalogs, settings, keywords and stored parse trees: byte-identical;
query IDs: identical;
data directories and streaming replication: interchangeable in both directions;
cachegrind: from −0.005% to +0.155% instructions, against a 0.5% failure threshold (the check);
a script checks that every statement added inside an existing function sits behind a hook, a registry or a flag; there is one documented exception;
a probe module installs every hook and checks what it does.

The checks were worth it: the very first test of the new combo-CID hook crashed the server.

Where the existing hooks fell short: 22 new ones

On a completely unmodified core, Greenplum could only work as a different product: segments would receive SQL text instead of ready-made plans, and each would have its own OIDs and snapshots, the way Citus does it. An MPP executor needs the whole cluster to share one plan and one transaction. PostgreSQL 19 has no extension points for that, so I had to cobble together new core patches:

Need	Patches	Why no existing hook works
Identical OIDs on every node	R1 `new_oid_hook`	OID allocation has no hook
All slices at once, readers seeing the writer’s uncommitted rows	R2 combo-CID hooks, R4 `XactAdoptTransactionState()`	only PostgreSQL’s own parallel workers share a transaction
2PC in Cloudberry’s order; table locks without upgrade deadlocks	O33, O30	`XACT_EVENT_COMMIT` fires once the transaction is already visible; the parser picks lock modes first
Cloudberry’s syntax, `gp_segment_id`, `SELECT *` hiding matview counters	O26, O10, O31, O28	there is no parser hook; system columns change catalogs
Distributed ANALYZE, incremental matviews	O3, O27	heap sampling is hard-wired; maintenance mode is static
Append-optimized and PAX storage, diskquota, checksums	O13–O18, O21, O23	`TableAmRoutine` cannot grow without breaking the ABI; the storage manager has no hooks
Tablespaces per node, memory protection, fault tests	O32, O25, O29	redo has fixed paths built in; memory-context methods are `static const`

At first the core got 28 new commits. Later, after a careful look at what could be removed or replaced, five patches moved into the modules (the query cancel while waiting for synchronous replication, node labels in EXPLAIN, the size functions, UPDATE’s old row version and the unlink hook), and two more were merged into one. The main compromise: full-table UPDATEs on PAX got 70% slower, though UPDATEs on append-optimized tables got 10–25% faster.

Four of the new hooks are general enough to propose upstream: the table AM registry, smgr file events, the parser hook and the OID hook.

Changes to formats and the protocol, however, can’t be carried by any custom hook:

private libpq messages;
20 shared catalogs;
81 keywords;
217 fixed OIDs;
cluster-wide TDE.

Greenplum’s metadata moved into PostgreSQL’s security labels and extension tables, and instead of TDE I decided that, for now, it is better to encrypt the volume at the operating-system level. With somewhat deeper core changes, though, TDE could be made to work exactly as it does in Cloudberry.

Architecture

From the outside it is Greenplum: a coordinator that holds the catalog and plans queries, and segments on other hosts, each a full PostgreSQL 19 instance with its own share of the rows. The only new thing is how it is all wired together.

SQL parsing works without a forked grammar (Cloudberry forks it). The gp_sql module splits each statement into tokens with PostgreSQL’s own scanner and replaces Greenplum’s syntax with the standard one. Its ProcessUtility_hook then picks up the options that PostgreSQL would otherwise reject with an error:

CREATE TABLE sales (id bigint, dt date)
DISTRIBUTED BY (id)
PARTITION BY RANGE (dt)
  (START (date '2026-01-01') END (date '2027-01-01')
   EVERY (interval '1 month'));

-- reaches PostgreSQL 19's grammar as (simplified):
CREATE TABLE sales (id bigint, dt date)
PARTITION BY RANGE (dt)
WITH (gp.distributed_by = '(id)',
      gp.partition_by = $gp$PARTITION BY RANGE (dt) (START ...)$gp$);

One statement stays one statement after the rewrite. An early version of the port that turned it into two broke every driver that uses prepared statements. Error positions still point into the original, unrewritten text. The grammar is one of the places a fork has to merge, with conflicts, at every new release.

ORCA is written so that its four core libraries don’t include a single PostgreSQL header, which is why its 920 files compile from Cloudberry’s source tree without any changes. It talks to the database through the port’s Query ↔ DXL ↔ plan translator (27 files, about 33k lines) and a wrapper layer. Of the wrapper layer’s 198 functions, 170 needed nothing but #include changes, and real API drift from the older PostgreSQL showed up in only six places. Whatever ORCA can’t digest has to be rewritten in the query before it gets there: every PostGIS spatial predicate, for example. If ORCA still declines a query, the reason is counted and the query takes the “gather route”: PostgreSQL’s own planner, with each distributed table gathered to the coordinator.

Cloudberry’s private protocol messages would break a PostgreSQL 19 session, so the dispatcher is an ordinary libpq client, with SCRAM or TLS, that sends each slice inside SELECT gp_internal.exec_fragment(...), and the segment’s planner_hook puts that plan fragment in place of the query.

The distributed snapshot arrives as a SET LOCAL. On each segment a writer process runs one slice and reader processes run the rest: they adopt the writer’s transaction through R4’s new core function XactAdoptTransactionState(), join its lock group, and resolve combo command IDs (combo CIDs) through the R2 hooks. Everything else is PostgreSQL’s standard parallel-worker machinery.

Motions stream rows over TCP, over Cloudberry’s UDP with flow control, over UDP2, or through a proxy that multiplexes all the Motions between two nodes.

Here is how a commit runs across the cluster:

Transactions work without a snapshot patch. Distributed snapshots looked impossible without core patches until the angle changed: make each segment’s snapshot agree with the coordinator’s, not the reverse. A segment waits for a transaction that the distributed snapshot says has committed but that the segment holds only as prepared, and it hides one that it has committed locally but that is still running globally. A replication slot keeps VACUUM away from the rows such transactions deleted. The coordinator’s own commit record decides every outcome, and a transaction that only one segment wrote commits in one phase: single-row INSERTs went from 371 to 950 transactions per second. Cloudberry’s global deadlock detector runs as it is, in a background worker.

As for storage and failover: append-optimized tables are table access methods whose blocks are ordinary 8K pages, logged by a custom WAL resource manager, so standbys, base backups and pg_checksums see normal pages. PAX’s C++ code compiles in place, with 65 of its 91 files unchanged. External tables are an FDW. Directory tables are now WAL-logged, which Cloudberry’s are not. FTS is a background worker that probes segments over SQL and promotes mirrors with pg_promote(), and gpfts instances elect a leader through etcd to fail over the coordinator.

The components

Module	Role	Plugs in via
`gp_core`	cluster file, dispatcher, Motions, interconnect, DDL on every node, 2PC, distributed snapshots, deadlock detector, FTS, `gp_toolkit`	executor, utility and planner hooks; background workers; custom WAL rmgr; security labels; CustomScan; R1, R2, R4, O3, O10, O29–O33
`gp_orca`	ORCA, translator, fallback counters, pre-ORCA rewrites	`planner_hook`, CustomScan, EXPLAIN hooks
`gp_sql`	Cloudberry syntax, tags, directory tables, extension scripts on every node	O26, `ProcessUtility_hook`, a WAL rmgr
`gp_ao`, `pax`	append-optimized row and column tables, PAX, bitmap index	table AM, O13–O18, O21, O23, WAL rmgrs
`gp_exttable`, `pxf_fdw`, `gpcloud`, `datalake_fdw`	external data: file, gpfdist, http, s3, PXF, Iceberg	FDW, table AM
`gp_resource`, `gp_security`, `gp_matview`, `gp_task`, `diskquota`	resource groups on cgroups, memory protection, profiles, incremental matviews, task scheduler, quotas	labels, O25, O27, O28, O21, background workers
`gpfts`, gpMgmt	coordinator failover; Cloudberry’s management tools	programs working over SQL and PostgreSQL 19’s own `initdb`, `pg_ctl`, `pg_basebackup`, `pg_rewind`

What the administrator sees:

gp_core loads first, and the modules that can only be preloaded refuse LOAD, so a server never runs half-initialized.
The port has no shared catalogs of its own. Role attributes live in shared security labels, secrets in a maintenance database, and the topology in a cluster file. gp_segment_configuration and its relatives are views.

Limitations

Inherited from PostgreSQL 19:
- extension settings must have a dot in their names, hence gp.*;
- command tags are a fixed list;
- CHECK_FOR_INTERRUPTS() takes no callbacks, so a running query moves to another resource group only while it is waiting for something;
- a cursor fetched in batches never runs in parallel.
The port compared with Cloudberry:
- Cloudberry’s MPP variant of the PostgreSQL planner, “Route B”, isn’t ported, so whatever ORCA declines takes the slower gather route;
- planning with ORCA costs more on short queries: 13.7 ms against 0.25 ms for a one-row point lookup, as measured by the pgorca project;
- parallelism inside a segment covers only the writer’s slice;
- SERIALIZABLE on a cluster is REPEATABLE READ, as in Greenplum.
The nature of MPP:
- the coordinator is the only planner and the single entry point;
- joins that aren’t on the distribution key move data over the network;
- unique constraints must include the distribution key;
- adding nodes means redistributing data.

How it was built

On September 17, six research agents each studied their own part of Cloudberry’s code. A script checked all 1,440 of their file:lines references, and the same day 14 implementation decisions were made in a conversation with the LLM. The plan grew into a journal of more than 16,000 lines, where Built <date> is reserved for code that was written, run and tested, and a corrections section records where reality disagreed with it. One entry reads “a test that passes may check nothing”: an interconnect test had been passing since M2 while a bare except swallowed its failures.

The agents settled implementation questions on their own and wrote down why they chose what they did. Anything that touched the PostgreSQL core, a shared branch or a deletion came to me. Large milestones were split between parallel agents, each with its own branch and worktree: nine agents built M8, and seven more cleaned up bugs and skipped tests.

Milestone	Dates	Result
M0 skeleton	Sept 18	all modules build and load; vanilla checks pass
M1 one node	Sept 18–22	Cloudberry SQL, ORCA, PostGIS; PostgreSQL’s 239 regression tests line for line
M2 cluster	Sept 22–24	dispatcher, Motions, streaming interconnect
M3 transactions	Sept 23–24	2PC, distributed snapshots, deadlock detector
M4–M6	Sept 24–26	mirrors and FTS; AO, PAX, external tables; resource groups
M7 parity and tools	Sept 26–27	gpMgmt on PostgreSQL 19’s tools, PostGIS on a cluster, TPC measured
M8	Sept 27–28	UDP2, proxy, gpfts, gpexpand, four add-on extensions

Commits per day: 10, 9, 15, 26, 31, 29, 87, 51, 87, 186 and 98. The first days went into ORCA and its translator, where understanding what was going on mattered more than speed; that was where the foundation was laid. From September 24, once the cluster worked and tests ran in parallel, the milestones started to overlap.

There were a few surprises along the way:

Zero segments. At first, single-node mode reported zero segments. A debug build of ORCA quietly handed every query to the fallback planner, while a release build divided by zero and clamped infinity to 1e+250, producing plausible plans by pure chance.
Bit 0x0400. Cloudberry decides whether to call ORCA by cursor flag 0x0400, which in PostgreSQL 19 means CURSOR_OPT_CUSTOM_PLAN. A literal port would have compiled without a single error and turned ORCA off for every custom plan.
Lock upgrades. When Cloudberry’s table lock was taken after the parser’s weaker lock, 87% of concurrent UPDATEs in pgbench failed. That’s where core patch O30 came from.
My Linux desktop kept dying. Forty-three parallel test jobs used up 60 GB of RAM, and the OOM killer took my desktop session down along with them. Now the runner starts a job only if the memory declared for it fits into what is free.

Non-functional requirements and ORCA on TPC

The requirements I measured were vanilla behaviour, hook overhead, ABI stability, reproducible builds and correct answers. Every figure comes from Docker images built from committed branches. A full run of the port’s test suites is 58–64 jobs and takes 500–900 seconds.

The deciding measurement concerned Cloudberry’s MPP planner (Route B). My instruction was to fix the ORCA fallbacks caused by the port itself first, and then, before deciding anything, to measure a real workload.

The setup:

TPC-H’s 22 and TPC-DS’s 99 queries as DuckDB ships them, on scale factor 1 data;
one coordinator and four segments in one container, on a 16-core host with 60 GB of memory;
no assertions, data in tmpfs, JIT and parallel query off;
DuckDB’s answers as the reference, compared to the cent.

	TPC-H	TPC-DS
Planned by ORCA	22/22	99/99
Answers equal to DuckDB’s	all	all
ORCA vs the gather route, geometric mean	3.1× (4.4 s vs 123 s, 20 queries)	4.0× (25 s vs 386 s, 96 queries)
Not finished in 120 s on the gather route	q17, q20	04, 14, 64

Source data:

The benchmark harness is pg19/test/tpc in the port: it generates the data, loads the cluster, runs every query under ORCA and on the gather route, and compares the rows.
The queries and reference answers come from DuckDB 1.5.5’s own kits: TPC-H queries and SF1 answers, TPC-DS queries and SF1 answers.
A recorded run of the harness is in the message of commit beaf7379d12: 121 of 121, 3.13× and 3.94×, and the same five queries unfinished.

The totals in the table come from the first measurement, which ran three rounds.

Under ORCA those five queries take 0.2–1.6 s, and all 121 run in 34 s. On six short queries (0.1–0.75 s) ORCA is more than a fifth slower. On Cloudberry’s regression suite, the port’s own reasons for falling back from ORCA went down from 563 to 52; what remains is the list ORCA inherited, non-default collations among them. The agent suggests concentrating on cutting ORCA’s fallbacks to the PostgreSQL planner rather than porting Route B’s 30–45k lines. I haven’t made my own decision about porting Cloudberry’s MPP planner yet, but the plan for implementing it is already written.

The outcome

Twelve days ago it wasn’t clear whether Greenplum could be built as an extension of a modern PostgreSQL without turning into a completely different product. Now there is code, and almost all of the functionality has been ported:

a PostgreSQL core with 22 small patches of mine, provably vanilla in behaviour and compatibility when the Greenplum extension isn’t loaded;
Cloudberry’s and ORCA’s sources, unmodified;
stock PostGIS and pgvector work, and other extensions should too; for pgvector’s nearest-neighbour search there is even a planner optimization that pushes ORDER BY ... LIMIT ... down to the segments;
more than 90% of the database code ported as extensions: 93%, or 97% if you leave out Cloudberry’s MPP planner (Route B in my plan). ORCA’s 380k lines are reused as they are, through a wrapper in the extension’s code, and are counted on neither side.

So far only 691 of Cloudberry’s 1,180 test files run, and that coverage has to grow. Next on the list are the rest of isolation2 and greenplum_schedule. I may also port Cloudberry’s MPP planner, as a fallback for ORCA.

The most important lesson of these twelve days: porting Greenplum as a PostgreSQL extension worked not because an LLM simply writes code fast, but because the development process was set up the way a good engineering team works.

Research into the code came with references to the exact lines it relied on, with prototypes, and with measurements of them. Invariants were checked by tests during coding and refactoring, and “done” meant that the new functionality passes the original system’s tests. Where a result differed, the difference was recorded the same way: tied to facts about where development stood in that phase, and to the later phase in which the unmodified test was expected to pass. To speed up iterations, the tests ran in parallel in Docker containers, making the most of the machine’s 32 threads and 64 GB of RAM.

In the last stages the agents worked in parallel, in isolated worktrees, while every serious call was mine: which path development should take, or what to do when the architecture had to deviate from the plan. The Greenplum fork took years to move from PostgreSQL 9.4 to 12; for the extension, a new major version means editing 22 small core patches and the extension code that goes with them.

Still running an old PostgreSQL? Then we’re coming to you!

The project: core patch series, port, build instructions.