↓ Skip to main content
Side Effects
  1. Home projects/

Siedzieje: finding local events with a pipeline of agents

··Index: 5·
Llm Go Python Nats
Table of Contents

In one town, Nowy Sącz, cultural events get announced on web pages, in RSS feeds, Instagram and Facebook posts, on ticketing sites, on posters and even in audio, and Siedzieje (Polish for roughly “what’s going on”) tries to collect all of them. Every event is supposed to end up as a structured record with what, when, where, price, organizer and category. The repository also holds a database of about 725 local venues and sources.

The pipeline
#

I built the system as a set of independent agents that talk to one another only through NATS JetStream:

Siedzieje pipeline: nine agents connected only through NATS JetStream, from discovery to storeweb pages, RSS, social media, posters, audioMinIObinary files, kept only for a whilediscoveryfetchconvertOCR, speechnormalizeclassifyLLMNATS JetStream: the only link between agentsextractLLMdedupvalidatestorePostgreSQLevent recordsagents in Go, Python and TypeScript; each one reads a stream and writes the next
discovery → fetch → convert → normalize → classify → extract → dedup → validate → store
  • convert turns everything into text: HTML to text, OCR for posters, speech recognition for audio;
  • classify and extract use LLMs to decide whether a text describes an event and to pull out the fields;
  • dedup and validate catch the same event announced in several places, and records with missing or impossible data;
  • store writes the result to PostgreSQL, while binary files sit in MinIO only for a while, because text is the source of truth.

The agents are written in Go, Python and TypeScript, whichever suits the job.

Debugging one stage at a time
#

To debug an agent on real data without running the whole pipeline, I wrote a snapshot tool that records the input events of any stage and replays them later (the agent has no idea it’s being fed a replay). Planned outputs include RSS, iCal, Telegram, Discord, ntfy push notifications, Bluesky and Mastodon.

Related