Cloudflare PipelinesOpen Beta

Streaming Data Ingestion

Ingest high volumes of real-time data, transform it with SQL, and write to Apache Iceberg tables or files on R2.

Streams

Durable, buffered data streams that receive events via HTTP or Worker bindings

SQL Transforms

Filter, normalize, and transform data in-flight with standard SQL

Sinks

Write to R2 Data Catalog (Iceberg) or raw files (JSON/Parquet)

5 MB/s

per stream, 5 MB max payload

Exactly-once

delivery

Zero

infrastructure to manage

pipeline.sql

 1-- Transform and filter clickstream events
 2INSERT INTO events_table
 3SELECT
 4  user_id,
 5  lower(event) AS event_type,
 6  to_timestamp_micros(ts_us) AS event_time,
 7  regexp_match(url, '^https?://([^/]+)')[1] AS domain,
 8  url,
 9  referrer,
10  user_agent
11FROM events_json
12WHERE event = 'page_view'
13  AND NOT regexp_like(user_agent, '(?i)bot|spider');

Set up via CLI or Dashboard:

$ npx wrangler pipelines setup

3 / 12