Scaling and big data¶
FastProof starts as one process with PostgreSQL and object storage, and that is the right setup for most teams. This page explains the optional parts for large installations: Kafka between the HTTP API and the database, separate roles, and ClickHouse or OpenSearch for search over large volumes of logs.
Start simple¶
The default (ingest.mode: direct, one Deployment, search.backend: postgres) handles a busy CI farm:
- uploads are spooled, queued in PostgreSQL and parsed by ingest workers on any replica;
- live logs reach the UI through PostgreSQL
LISTEN/NOTIFY(or Valkey with several replicas); - log search and similar failures run in PostgreSQL over compressed log chunks.
Add replicas (with S3 or a ReadWriteMany volume) before adding anything else. Move on only when one of the signals below appears.
When to enable what¶
| Signal | What helps |
|---|---|
| Upload bursts make the API slow, or the ingest queue grows during peak hours | roles.split: true, more roles.ingest replicas |
| Many agents stream live logs at once and PostgreSQL write load dominates | ingest.mode: kafka |
| PostgreSQL storage grows mostly from logs, or log search gets slow | A sink (ClickHouse or OpenSearch) and search.backend |
| Long-term trends over many months and projects | ClickHouse sink (daily aggregates are kept after raw rows expire) |
| An OpenSearch cluster already runs logs for your platform | OpenSearch sink |
Direct or Kafka¶
direct |
kafka |
|
|---|---|---|
| Path | HTTP, spool, PostgreSQL job queue, ingest workers | HTTP gateway, Kafka topics, writer consumers |
| Extra moving parts | none | Kafka |
| Back pressure | PostgreSQL queue depth | Kafka retention absorbs bursts; writers catch up |
| Live logs to the UI | PostgreSQL LISTEN/NOTIFY |
Each API replica reads ingest.logs from the latest offset |
| Ordering | per upload | per launch (records are keyed by launch id) |
In Kafka mode only the gateway produces, after token authentication, unless
FASTPROOF_KAFKA_TRUSTED_PRODUCERS=true. Consumer groups are coordinated by FastProof in PostgreSQL (Kafka
consumer groups are not used), and the writer commits its offset in the same transaction as the data, so
retries never duplicate results. FastProof exports consumer lag on /metrics.
Topics (after FASTPROOF_KAFKA_TOPIC_PREFIX, default fastproof.): ingest.results, ingest.logs,
ingest.uploads and events. Create them yourself with the partition count you want, or let FastProof create
them (kafka.topics.create, on for the bundled Kafka).
Roles¶
| Role | Does |
|---|---|
api |
HTTP API, UI, live event streams; in Kafka mode the gateway that produces |
ingest |
Parses uploaded archives (allure-results, JUnit, CTRF, Playwright) and stores attachments |
writer |
Kafka mode: consumes results and logs and writes them to PostgreSQL |
sink |
Copies results, attempts, steps and logs to ClickHouse or OpenSearch |
scheduler |
Periodic jobs (partitions, retention); safe on any number of replicas |
FASTPROOF_ROLES=all (the default) runs everything in one process. With roles.split: true the Helm chart
runs api (with scheduler), ingest, writer and sink as separate Deployments, each with its own
replicas, resources, probes, HPA and PDB. See Helm chart.
ClickHouse or OpenSearch¶
| ClickHouse | OpenSearch | |
|---|---|---|
| Best at | Aggregates and long-term trends, compact storage of large log volumes | Relevance-ranked full-text search, existing OpenSearch platforms |
| FastProof stores | Tables results, attempts, steps, logs (MergeTree, partitioned by month, TTL) and series_daily aggregates |
Data streams such as <prefix>logs-*, expired by an ISM policy |
| Log search | Text index on messages (ClickHouse 26.2 or newer; older servers use bloom-filter indexes) | Full-text queries |
| Similar failures | Same fingerprint, then n-gram distance of normalized messages | Same fingerprint, then full-text match of normalized messages |
| Memory for a small node | About 2 GiB | About 1.5 GiB (768 MiB heap) |
| Supported versions | 26.8 LTS recommended; 26.2 or newer for the text index | 2.19 and 3.x |
Both sinks are fed from FastProof's event outbox, never from the API path, so a slow or unavailable sink does
not slow uploads; it catches up later. PostgreSQL stays the source of truth. Both can run at once
(FASTPROOF_SINKS=clickhouse,opensearch); search.backend picks which one answers search.
ClickHouse for serious use¶
The chart's bundled ClickHouse is one node without replication, sized for a lab or a small team. For
production, run ClickHouse with the Altinity ClickHouse operator
(chart altinity-clickhouse-operator 0.27.x) and point FastProof at it with sinks.clickhouse.external.*:
- a
ClickHouseInstallationwith at least two replicas per shard and aClickHouseKeeperInstallation(three Keeper nodes) for replication; - set the pod and container security context in the CHI
podTemplateyourself (the operator's defaults are empty): non-root UID 101,RuntimeDefaultseccomp, all capabilities dropped; - expose only the HTTPS interface (8443) to FastProof; keep the native port 9000 closed with a NetworkPolicy;
- create a
fastproofuser withGRANT ALL ON fastproof.*and a profile withmax_memory_usage; - pin the image by digest and review patch notes for "native protocol", "privilege" and "crash" entries.
FastProof creates its tables on start. On a replicated cluster, create the database with the Replicated
engine so DDL reaches every replica.
OpenSearch for serious use¶
Use a managed OpenSearch or the official chart with three or more nodes, real certificates (cert-manager) and
the security plugin on. Give FastProof a user whose role has indices_all on fastproof-* and
.ds-fastproof-*, index template and ISM policy management, and cluster_monitor. Never disable the security
plugin.
Sizing¶
Start from these and measure with fastproof bench:
| Setup | FastProof | PostgreSQL | Kafka | ClickHouse | OpenSearch |
|---|---|---|---|---|---|
| Small team, direct | 1 replica, 1 CPU, 1 GiB | 1 CPU, 1 GiB | none | none | none |
| CI farm, direct | 2-4 replicas, 2 CPU, 1 GiB each | 2-4 CPU, 4-8 GiB | none | none | none |
Lab big data (ci/bigdata-values.yaml) |
api, ingest, writer, sink at 0.5-1 CPU, 384-768 MiB | 1 CPU, 768 MiB | 1 broker, 768 MiB (384 MiB heap) | 1 node, 2 GiB | none |
| Large, Kafka | api 2+, ingest 2+, writer 2+, sink 1-2 | HA cluster | 3 brokers | 2+ replicas | 3+ nodes |
- Kafka partitions bound writer parallelism: run at most as many writer replicas as partitions of
ingest.results. - Size the PostgreSQL connections for all Deployments: replicas ×
database.maxConnectionssummed over roles. - Run one heavy backend at a time on small nodes: ClickHouse, OpenSearch and Kafka together need about 4.5 GiB.
TODO
Throughput and lag numbers from the big-data benchmark (G9) will be added here.
Benchmark¶
fastproof bench generates launches, results and log lines with realistic failure patterns, pushes them
through the HTTP API (or straight to Kafka), and reports throughput and server-side lag. Run it as a
Kubernetes Job next to the installation you want to measure.