Siedzieje (Polish for roughly “what’s going on”) collects cultural events in one town, Nowy Sącz, from wherever they are announced: web pages, RSS feeds, Instagram and Facebook posts, ticketing sites, posters and audio. Each event ends up as a structured record with what, when, where, price, organizer and category. The repository also has a database of about 725 local venues and sources.
The pipeline#
The system is a set of independent agents. They talk to each other only through NATS JetStream:
discovery → fetch → convert → normalize → classify → extract → dedup → validate → store
- Convert turns everything into text: HTML to text, OCR for posters, speech recognition for audio.
- Classify and extract use LLMs to decide whether a text is an event and to pull out its fields.
- Dedup and validate catch the same event announced in several places, and records with missing or impossible data.
- The result goes into PostgreSQL. Binary files stay in MinIO only for a while, because text is the source of truth.
The agents are written in Go, Python and TypeScript, whichever suits the job.
Debugging one stage at a time#
A snapshot tool records the input events of any stage and replays them later. I can debug an agent on real data without running the whole pipeline, and the agent does not know it is being fed a replay.
Planned outputs are RSS, iCal, Telegram, Discord, ntfy push notifications, Bluesky and Mastodon.