Cloudflare Pipelines: Streaming Ingestion

2026-01-27

Cloudflare PipelinesOpen Beta

Streaming Data Ingestion

Ingest high volumes of real-time data, transform it with SQL, and write to Apache Iceberg tables or files on R2.

Streams

Durable, buffered data streams that receive events via HTTP or Worker bindings

SQL Transforms

Filter, normalize, and transform data in-flight with standard SQL

Sinks

Write to R2 Data Catalog (Iceberg) or raw files (JSON/Parquet)

Up to 100 MB/s

per pipeline during open beta

Exactly-once

delivery

Zero

infrastructure to scale or manage

pipeline.sql

 1-- Transform and filter clickstream events
 2INSERT INTO events_table
 3SELECT
 4  user_id,
 5  lower(event) AS event_type,
 6  to_timestamp_micros(ts_us) AS event_time,
 7  regexp_match(url, '^https?://([^/]+)')[1] AS domain,
 8  url,
 9  referrer,
10  user_agent
11FROM events_json
12WHERE event = 'page_view'
13  AND NOT regexp_like(user_agent, '(?i)bot|spider');

Set up everything in one command:

$ npx wrangler pipelines setup