R2 SQL in ActionStatistical Analysis

Linear regression on AI Gateway telemetry — does token count predict latency?

latency-regression.sql

SELECT model,
  ROUND(corr(
    CAST(total_tokens AS DOUBLE),
    CAST(duration_ms AS DOUBLE)), 4
  ) AS token_latency_corr,
  ROUND(regr_slope(
    CAST(duration_ms AS DOUBLE),
    CAST(total_tokens AS DOUBLE)), 4
  ) AS ms_per_token,
  ROUND(regr_r2(
    CAST(duration_ms AS DOUBLE),
    CAST(total_tokens AS DOUBLE)), 4
  ) AS r_squared,
  regr_count(
    CAST(duration_ms AS DOUBLE),
    CAST(total_tokens AS DOUBLE)
  ) AS sample_size
FROM ai_agents.invocations
WHERE duration_ms > 0 AND total_tokens > 0
GROUP BY model
HAVING regr_count(CAST(duration_ms AS DOUBLE),
  CAST(total_tokens AS DOUBLE)) > 20
ORDER BY r_squared DESC

corrregr_sloperegr_r2regr_count

7 rows — top models by R²ms_per_token × tokens + base = predicted latency

model	corr	ms_per_token	r²	samples
HIGH R²@cf/google/gemma-7b-it-lora	0.9499	65.85	0.9022	688
HIGH R²@cf/mistral/mistral-7b-v0.2-lora	0.7972	54.96	0.6355	570
@cf/baai/bge-large-en-v1.5	0.4604	0.03	0.2119	815
mistral-small-3.1-24b	-0.2662	-48.64	0.0709	329,637
llama-3.3-70b	0.0566	3.11	0.0032	147,443
gemma-3-12b	0.0083	0.26	0.0001	329,828
llama-3.1-8b	0.0041	0.11	0.0000	883,512

Insight: LoRA fine-tuned models show strong token-latency correlation (R² > 0.6), while large general models do not — latency is dominated by other factors.

8 / 12