LogForge

Most log collectors write straight to a database on the ingest path — under burst load that creates backpressure and blocks request threads. LogForge separates the write path (Kafka, fire-and-forget, acks=all) from the storage path (batch-inserted by a separate consumer), so the ingestion endpoint's latency stays flat regardless of database load.

Kafka · fire-and-forget ingest PostgreSQL full-text search (GIN) Redis-cached queries Batch insert, ON CONFLICT DO NOTHING DLQ for failed records

Search the query API — live

This queries the real query_service — the same asyncpg code, the same full-text-search SQL (to_tsvector('english', message)), running against a real Postgres database with 400 seeded, clearly-synthetic log rows across 5 services. See "What's real in this deploy" below for what that means and doesn't mean.

Time	Service	Level	Message

Architecture

Client HTTP POST /logs
    |
    v
Ingestion Service (:8001)      FastAPI, Redis rate limit, acks=all, retries=5, lz4
    |
    v
Kafka (logs-topic)             durability layer, 168h retention
    |
    v
Processor Service              confluent-kafka consumer, batch of 100 or 500ms
    |                           ON CONFLICT DO NOTHING for idempotent re-delivery
    |                           failed records -> logs-topic-dlq
    v
PostgreSQL (logs table)        GIN index on to_tsvector(message), timestamp index
    |
    v
Query Service (:8002)          <- you're querying this one, live, above
    |                           filter/search + cursor pagination + Redis cache
    v
Alerting Service               polls every 60s, dedupes per rule window

What's real in this deploy

The Query Service above is genuinely live against a real Postgres database — every search you run executes the actual SQL. What's not deployed here: Kafka, the processor consumer, and the alerting poll loop are long-running processes by design (that's the whole point of the architecture — decoupling ingest from storage), which doesn't fit a stateless serverless function. The 400 log rows were inserted directly against the schema rather than arriving through the Kafka pipeline shown above; run docker compose up from the repo to see the complete four-service pipeline end to end, ingest path included.