SeaweedFS catalog

ClickHouse supports integration with multiple catalogs (Unity, Glue, REST, Polaris, etc.). This guide will walk you through the steps to query your data using ClickHouse and the SeaweedFS catalog.

SeaweedFS is an open-source distributed file and object store with an S3-compatible gateway. Its S3 Table Buckets provide both halves of an Iceberg deployment: the embedded Iceberg REST catalog serves the table metadata, and the table bucket stores the table data as Parquet files behind the same S3 endpoint:

• Single service - catalog metadata and Parquet data are served by one process, with no separate metadata database
• REST API compliance with the Iceberg REST catalog specification
• Server-side maintenance - automatic Parquet compaction and snapshot expiration, with no external maintenance service

Local development setup

For local development and testing, you can run SeaweedFS and ClickHouse with Docker Compose. This approach is ideal for learning, prototyping, and development environments.

Prerequisites

1. Docker and Docker Compose: Ensure Docker is installed and running
2. Versions: SeaweedFS 4.42 or later; ClickHouse 26.8 or later (versions back to 25.8 can read and insert, but creating tables through the catalog requires 26.8)
3. Python with PyIceberg (optional): used below to seed sample data

Setting up the local SeaweedFS catalog

Step 1: Create a new folder in which to run the example, then create a file s3config.json with the credentials for the S3 gateway and the catalog:

{
  "identities": [
    {
      "name": "analyst",
      "credentials": [
        {
          "accessKey": "tutorialkey",
          "secretKey": "tutorialsecret"
        }
      ],
      "actions": ["Admin", "Read", "Write", "List", "Tagging"]
    }
  ]
}

Step 2: Create a file docker-compose.yml with the following configuration:

services:
  seaweedfs:
    image: chrislusf/seaweedfs:latest
    command: mini -dir=/data -s3.config=/etc/seaweedfs/s3config.json -tableBucket=analytics -admin.port=12646
    ports:
      - "8333:8333"   # S3 endpoint
      - "8181:8181"   # Iceberg REST catalog
    volumes:
      - ./s3config.json:/etc/seaweedfs/s3config.json
      - seaweedfs_data:/data
    networks:
      - iceberg_net

  clickhouse:
    image: clickhouse/clickhouse-server:latest
    container_name: seaweedfs-clickhouse
    ports:
      - "8123:8123"
      - "9000:9000"
    depends_on:
      - seaweedfs
    networks:
      - iceberg_net

volumes:
  seaweedfs_data:

networks:
  iceberg_net:
    driver: bridge

The mini command starts the whole SeaweedFS stack in a single container. The -tableBucket=analytics flag pre-creates an S3 Tables bucket named analytics, which serves as the Iceberg warehouse. -admin.port=12646 keeps the admin gRPC port that SeaweedFS derives from it below the Linux ephemeral port range, where a startup connection could otherwise claim it first.

Step 3: Run the following command to start the services:

docker compose up -d

Seeding sample data

The catalog starts out empty. Create a table and append a few rows with PyIceberg (pip install pyiceberg pyarrow):

import pyarrow as pa

from pyiceberg.catalog.rest import RestCatalog

catalog = RestCatalog(
    "seaweedfs",
    uri="http://localhost:8181",
    warehouse="s3://analytics",
    credential="tutorialkey:tutorialsecret",
    **{
        "s3.endpoint": "http://localhost:8333",
        "s3.access-key-id": "tutorialkey",
        "s3.secret-access-key": "tutorialsecret",
        "s3.region": "us-east-1",
        "s3.path-style-access": "true",
    },
)

rows = pa.table({
    "id": pa.array([1, 2, 3, 4, 5, 6], pa.int64()),
    "region": ["NA", "EU", "EU", "APAC", "NA", "EU"],
    "amount": pa.array([12.5, 40.0, 7.25, 99.9, 3.5, 61.0], pa.float64()),
})

catalog.create_namespace("sales")
table = catalog.create_table("sales.orders", schema=rows.schema)
table.append(rows)

Connecting to the local SeaweedFS catalog

Connect to your ClickHouse container:

docker exec -it seaweedfs-clickhouse clickhouse-client

Then create the database connection to the SeaweedFS catalog:

SET allow_experimental_database_iceberg = 1;

CREATE DATABASE lake
ENGINE = DataLakeCatalog('http://seaweedfs:8181/v1', 'tutorialkey', 'tutorialsecret')
SETTINGS catalog_type = 'rest',
    warehouse = 's3://analytics',
    storage_endpoint = 'http://seaweedfs:8333/analytics',
    catalog_credential = 'tutorialkey:tutorialsecret',
    oauth_server_uri = 'http://seaweedfs:8181/v1/oauth/tokens'

The engine arguments carry the S3 credentials ClickHouse uses to read table data, while catalog_credential and oauth_server_uri authenticate to the catalog itself through the OAuth2 client-credentials flow. SeaweedFS accepts the same access key and secret key for both.

Querying SeaweedFS catalog tables using ClickHouse

Now that the connection is in place, you can start querying via the SeaweedFS catalog. For example:

USE lake;

SHOW TABLES;

┌─name─────────┐
│ sales.orders │
└──────────────┘

To query a table:

SELECT region, sum(amount) AS total
FROM `sales.orders`
GROUP BY region
ORDER BY total DESC;

┌─region─┬──total─┐
│ EU     │ 108.25 │
│ APAC   │   99.9 │
│ NA     │     16 │
└────────┴────────┘

Creating tables and writing data from ClickHouse

You can also create tables in the SeaweedFS catalog and write to them directly from ClickHouse:

SET allow_experimental_database_iceberg = 1;
SET allow_experimental_insert_into_iceberg = 1;
SET write_full_path_in_iceberg_metadata = 1;

CREATE TABLE lake.`sales.returns` (id Int64, reason String)
ENGINE = IcebergS3('http://seaweedfs:8333/analytics/sales/returns/', 'tutorialkey', 'tutorialsecret');

INSERT INTO lake.`sales.returns` VALUES (1, 'damaged'), (2, 'wrong size');

SELECT * FROM lake.`sales.returns` ORDER BY id;

┌─id─┬─reason─────┐
│  1 │ damaged    │
│  2 │ wrong size │
└────┴────────────┘

The IcebergS3 engine clause names the storage path for the new table, and write_full_path_in_iceberg_metadata makes ClickHouse register the full table location with the catalog.

When ClickHouse commits an insert, the SeaweedFS catalog repairs metadata the experimental writer does not yet produce: it fills in missing field IDs in manifests, rewrites bucket-relative file paths as absolute locations, and stamps a default name mapping on the table. Strict readers such as PyIceberg and Spark can then read the rows ClickHouse wrote. This requires SeaweedFS 4.42 or later.

Loading data from your Data Lake into ClickHouse

If you need to load data from the SeaweedFS catalog into ClickHouse, start by creating a local ClickHouse table:

CREATE TABLE default.orders
(
    `id` Int64,
    `region` String,
    `amount` Float64
)
ENGINE = MergeTree()
ORDER BY (region, id);

Then load the data from your SeaweedFS catalog table via an INSERT INTO SELECT:

INSERT INTO default.orders
SELECT * FROM lake.`sales.orders`;