All case studies
Case 03, Insight StreamPublished4 min read

Building an AI content intelligence pipeline

The engineering questionHow do you turn content from many sources into relevant, searchable insights without blocking the application?

System flow in Insight Stream
  1. 01RSS and Reddit sources
  2. 02Celery + Redis workers
  3. 03LLM sentiment
  4. 04Embeddings in pgvector
  5. 05Feed and search API
Diagram: a fast request path from the Vue client through FastAPI and a Redis cache to PostgreSQL with pgvector, and a background Celery path that fetches RSS and Reddit content, removes duplicates, labels sentiment and stores embeddings.
Figure 03The request path reads prepared data while Celery workers fetch, de-duplicate, classify and embed content in the background.

01 Context

The problem

Staying current on a topic means reading hundreds of articles and Reddit threads to find the few that matter. Insight Stream collects that content, organises it by the topics each person follows, classifies sentiment and makes it searchable by meaning. The catch is that every item needs slow, unreliable work before it is useful: fetching, cleaning, de-duplicating, a model call for sentiment and another for embeddings. Doing any of that while someone waits for their feed would make the app slow and fragile.

  • The feed and search must stay fast, whatever the sources are doing.
  • One slow or broken source must not hold up the others.
  • The same story must not be processed or paid for twice.
  • Search should work on meaning, not only on exact words.

02 Design

The architecture and decisions

The system has two paths. The request path is FastAPI reading data that is already prepared, with Redis caching hot feeds. The background path is Celery workers, with Redis as the broker, that ingest sources, remove duplicates, classify sentiment and create embeddings. PostgreSQL with pgvector holds both the content and the vectors.

  • FastAPI, SQLAlchemy and Alembic, with JWT auth and rate limiting
  • Celery workers with Redis as broker and cache
  • PostgreSQL with pgvector for content and embeddings
  • LLM providers behind one interface: OpenAI, Gemini, DeepSeek and Groq
  • Vue 3, TypeScript and Pinia on the client
ChoseCelery workers with Redis
OverRunning work inside API requests
Because

Retries, scheduling and separate worker processes mean slow sources and model calls never touch request latency.

Chosepgvector in PostgreSQL
OverA separate vector database
Because

Articles, topics, sentiment and embeddings can be filtered and joined in one query, with one backup and one source of truth.

ChoseThree sentiment labels from an LLM
OverA classic sentiment library
Because

Good, Bad or Neither is simple to show and filter on, and an LLM handles context and sarcasm better than word lists.

A source task that retries on its own (simplified)python
@celery.task(bind=True, autoretry_for=(httpx.HTTPError,),
             retry_backoff=True, max_retries=5)
def ingest_source(self, source_id: int):
    items = fetch_and_normalise(source_id)
    for item in skip_duplicates(items):
        enrich_item.delay(save(item).id)  # sentiment + embedding

03 Implementation

The difficult parts

Ingesting from many sources

RSS feeds and Reddit look nothing alike, so each source is normalised into one item shape with a title, text, URL, source and time, then tagged with the topic it belongs to. Each source is its own task, so one failing feed can’t stop the rest.

Duplicates before model calls

The same story often arrives from several feeds. URLs are normalised and the content is hashed, and duplicates are skipped before any model call. That keeps the feed clean and stops paying twice for the same item.

Unreliable APIs and failed jobs

Sources and model providers time out, rate-limit and return errors. Tasks retry with backoff, and enrichment runs separately from fetching, so a failed model call is retried on its own without fetching the source again. Tasks are written so running one twice is harmless, which is what makes retries safe.

Keyword versus vector search

Keyword search is precise for names and exact terms, but it misses articles that say the same thing in different words. Embedding search finds those by meaning. Because both live in PostgreSQL, the API can filter by topic and sentiment in the same query as the similarity search.

04 Validation

How I check it

The pipeline is mostly background work, so the checks focus on what happens when things go wrong, and on whether the results are useful.

  • Tests for the de-duplication rules, with the same story arriving through different URLs and feeds.
  • Tasks are run against sources that fail or time out, to check retries and that nothing is stored twice.
  • Sentiment labels are spot-checked by hand on a sample of items after any prompt or model change.
  • Next step: a small set of queries with expected results, to compare keyword and vector search with numbers.

05 Reflection

What I learned and what comes next

Retries are only safe if repeats are harmless

Making every task safe to run twice mattered more than any retry setting.

Spend model calls last

Filtering and de-duplicating before calling a model cut cost and noise at the same time.

What comes next

Combining keyword and vector search in one ranking, and publishing measured search quality.