Two engines, one Iceberg catalog: StarRocks beside Trino, neither holding a key

The claim a REST catalog is actually making

Parts one to three built a lakehouse on Trino and never left it. That is a fine way to test Iceberg, and a poor way to test the catalog, because the argument for Polaris over Hive Metastore is not that it looks tables up faster.

The argument is that the catalog is the system of record, so a second engine should see the tables the first one wrote — without an export, a copy, a CREATE EXTERNAL TABLE per table, or a second set of credentials.

That last clause is the part worth checking. Part one went to some trouble to make Trino hold no standing S3 key. If adding StarRocks means pasting the MinIO keys into a second config file, the boundary is gone and the whole exercise was decoration.

So: attach StarRocks 4.1.4 to the same catalog, read the same tables, and see. The original 2024 build ran StarRocks 3.2 beside Trino 437 against Hive Metastore — same question, and the answer has changed.

Step 1 — StarRocks on the same network

  starrocks:
    image: starrocks/allin1-ubuntu:4.1.4
    profiles: ["engines"]
    depends_on:
      polaris:
        condition: service_healthy
    ports:
      - "9030:9030"   # MySQL protocol
      - "8030:8030"   # FE HTTP
    healthcheck:
      test: ["CMD-SHELL", "mysql -h127.0.0.1 -P9030 -uroot -e 'SELECT 1' || exit 1"]
      interval: 15s
      timeout: 10s
      retries: 40
      start_period: 90s

allin1 bundles the frontend and backend in one container. Production splits them; a lab that splits them spends 2 GB teaching nothing this post is about.

ERROR 1064 (HY000): Backend node not found. Check if any backend node is down.

Verify. SELECT current_version() returns 4.1.4-… rather than the backend error.

Step 2 — One catalog definition, no table definitions

CREATE EXTERNAL CATALOG lakehouse PROPERTIES (
  'type' = 'iceberg',
  'iceberg.catalog.type' = 'rest',
  'iceberg.catalog.uri' = 'http://polaris:8181/api/catalog',
  'iceberg.catalog.warehouse' = 'lakehouse',
  'iceberg.catalog.security' = 'oauth2',
  'iceberg.catalog.oauth2.credential' = 'root:s3cr3t-dev-only',
  'iceberg.catalog.oauth2.scope' = 'PRINCIPAL_ROLE:ALL',
  'iceberg.catalog.vended-credentials-enabled' = 'true',
  'aws.s3.endpoint' = 'http://minio:9000',
  'aws.s3.enable_path_style_access' = 'true'
);

That is the entire integration. No table DDL, no schema mirroring, no sync job.

Read it back:

SHOW DATABASES FROM lakehouse;

bronze
gold
information_schema
silver
silver_gold
silver_silver

Every namespace Trino created, immediately. Including silver_gold and silver_silver — the debris from the schema-naming bug in part two, which I had fixed in dbt but never cleaned out of the catalog.

That is a small thing that says something real: the catalog is shared, so mistakes are shared too. A second engine is also a second pair of eyes on your namespace hygiene.

SELECT count(*) FROM lakehouse.silver.orders_enriched;

1645100

Trino reports the same number for the same table.

Verify. Row counts match between engines, and SHOW DATABASES FROM lakehouse lists the namespaces you created in Trino.

Step 3 — Does the second engine reintroduce standing keys?

The catalog above declares vended-credentials-enabled and — deliberately — no aws.s3.access_key or aws.s3.secret_key at all. If StarRocks did not support vending, this would fail at first read.

It does not fail. 1,645,100 rows, with StarRocks obtaining short-lived credentials from Polaris exactly as Trino does: authenticate as an OAuth2 principal, receive a session token scoped to the table’s prefix, use it, discard it.

This is the result that justifies the architecture. The authorisation boundary built in part one is engine independent. Adding a query engine does not add a place where a long-lived S3 key has to live, which is the failure mode that quietly undoes most “we use a catalog” stories.

Step 4 — Measure both, fairly

Same query, same table, same catalog:

SELECT count(*), sum(totalprice)
FROM lakehouse.silver.orders_enriched
WHERE orderdate >= '1995-03-01' AND orderdate < '1995-04-01';

Getting a comparable number took two attempts. Wall-clock through each engine’s CLI is useless: the Trino CLI starts a JVM, the MySQL client does not. Subtracting a SELECT 1 baseline from each was worse — it gave Trino a negative query time, because JVM startup varies by more than the query costs.

Both engines report their own timings, so use those. Trino from system.runtime.queries; StarRocks from its FE audit log:

grep 'orders_enriched' /data/deploy/starrocks/fe/log/fe.audit.log \
  | grep -o 'Time=[0-9]*' | tail -7

Step 5 — Clean up the shared debris

Since the catalog is shared, so is the tidying:

DROP TABLE IF EXISTS lakehouse.silver_silver.orders_enriched;
DROP TABLE IF EXISTS lakehouse.silver_gold.daily_revenue;
DROP SCHEMA IF EXISTS lakehouse.silver_silver;
DROP SCHEMA IF EXISTS lakehouse.silver_gold;

bronze
gold
information_schema
silver
system

Note this needs DROP_WITH_PURGE_ENABLED from part two, or the drops fail with a Trino error that never mentions purge.

Failure modes

Clean up and operating consequence

docker compose --profile engines down

The rule: a catalog earns its place when the second engine costs one DDL statement and no new credentials.

That is a testable bar, and it is worth applying before committing to a catalog rather than after. Attach a second engine to a throwaway namespace, define it with no storage keys, and read a table the first engine wrote. If that works, the catalog is doing the job the architecture diagram claims. If it needs a copy, a manual table registration, or its own set of long-lived keys, you have a metadata service — which may still be worth running, but not for the reasons usually given.

The rest of this series is on GitHub as a single docker compose file: MinIO, Polaris, Trino, dbt, Airflow and StarRocks, with every figure in these four posts reproducible from it.