The Concierge answers design-system questions — what a component does, which token to use, how something is meant to be composed — without a human in the loop.
It ran as a hosted service on kickstartds.com, with the release written up on the blog. Most of the interesting work was not the model but the context: chunking component documentation, tokens and examples so the answers stayed grounded in the actual system.
| Fact | Detail |
|---|---|
| Hosted at | kickstartds.com/concierge |
| Approach | Retrieval-grounded chat over component docs |
| Stack | Jupyter/Python service |
| Started | March 2023 |
What's in it
Grounded in the component documentation, the token catalog and real usage examples.
The blog post covers the architecture and the design decisions behind it.
In detail
What is actually inside it
Questions are embedded and matched against a knowledge base built from component documentation, tokens and examples. Grounding came from the corpus, not from prompt wording.
quality_analysis.ipynb and quality_calculation.ipynb exist next to the ingestion notebooks — the system was evaluated as a data problem before it was tuned as a chat problem.
fly.toml plus a separate dashboard/ app: the service ran somewhere small and observable, with a UI to look at what it was doing.
| Detail | Value |
|---|---|
| Architecture | embed → retrieve → answer (Flask + sentence-transformers) |
| Model | msmarco-distilbert-cos-v5 |
| Data pipeline | 8 notebooks incl. BERTopic and quality metrics |
| Deployed to | Fly.io (`fly.toml`) |
| Started | March 2023 |
Technical notes
The decisions behind it
Why a bi-encoder instead of calling a language model directly?
What did BERTopic do here?
What actually runs, and where
The boundary between the model and the index
The Concierge was three processes with one HTTP hop between them, and most of the design decisions show up at that hop.
The browser talks to a Supabase Edge Function written in TypeScript. That function owns the conversation: it moderates the question through OpenAI’s moderation endpoint, asks for an embedding, runs the vector search, assembles the prompt and streams the completion back as Server-Sent Events. It holds no model of its own.
Embedding is the one thing that could not live in the Edge Function, because Supabase functions run on Deno and a PyTorch sentence-transformer does not. So it moved into a separate Flask service (app.py) on Fly.io, deployed as question-embedding in ams. The container loads msmarco-distilbert-cos-v5 once at import, caps it at max_seq_length = 256, and exposes a single route: POST / with {"question": "..."} in, {"embedding": [768 floats]} out. No auth, no batching, CORS(app) wide open — the file still carries the TODO to restrict it.
On the database side there is nothing but Postgres and pgvector: a sections table with embedding vector(768) and an ivfflat index using vector_cosine_ops, plus a match_sections() SQL function that returns rows ordered by 1 - (embedding <=> query_embedding). Retrieval is that one RPC with similarity_threshold: 0.4 and match_count: 15. The notebooks carry a cross-encoder re-ranker that was used to inspect results offline; it is not in the deployed path — all fifteen vector hits go straight into the prompt.
The prompt is then trimmed with a GPT-3 tokenizer against a hard 2100-token budget, and generation runs text-davinci-003 at temperature: 0 and max_tokens: 1200. The first SSE event the client receives is not text but the retrieved sections themselves, so the source list can render before the answer finishes. Every completed answer is written back to Postgres: questions keeps the question, the full prompt, its token length, the answer and the question embedding, while question_answer_sections keeps which sections were used and how similar they scored. The Concierge is also a record of what people ask.
msmarco-distilbert-cos-v5 vectors, stored as vector(768) in Postgres.| Piece | Detail |
|---|---|
| Embedding service | Flask + `sentence-transformers` on Fly.io (`question-embedding`, `ams`) |
| Endpoint contract | `POST /` · `{ question }` in → `{ embedding: 768 floats }` out |
| Encoder | `msmarco-distilbert-cos-v5`, `max_seq_length = 256` |
| Vector store | Supabase Postgres + `pgvector`, `ivfflat` / `vector_cosine_ops` |
| Retrieval | `match_sections()` RPC — threshold `0.4`, `match_count 15` |
| Context budget | ≤ 2100 GPT-3 tokens of retrieved sections per prompt |
| Generation | `text-davinci-003` · `temperature 0` · `max_tokens 1200` · SSE stream |
| Ingestion | Playwright/Chromium + trafilatura; split on Markdown headings, 300-token chunks / 50 overlap |
Judging the corpus before generating anything
Three audits, none of them a gate
The release post promises that hallucinations are minimised because all context is fed from our own corpus. That promise only holds if the corpus is worth retrieving from, so four of the eight notebooks exist to decide what belongs in it.
The first pass is mechanical: keep sections with more than 250 tokens, 4–15 newlines and more than 120 spaces, and drop exact duplicates. On the committed run that cut 180,783 extracted passages down to roughly 6,000 — the crawl is full of navigation, cookie banners and API tables that are not answers to anything.
The remaining notebooks then treat quality as something to measure, not assume. All three lean on the NLTK Brown corpus as a stand-in for ordinary English prose.
quality_analysis.ipynb embeds 300-token chunks of the Brown corpus, scores every section by its mean cosine similarity to all of them, and splits at the 10th percentile. Sections that read nothing like prose — nav dumps, attribute tables, transcripts — land at the bottom; the committed run put that threshold at 0.0973.
quality_calculation.ipynb trains a Kneser-Ney bigram language model on Brown and averages each section’s bigram entropy under it. Text that a general English model finds predictable is generic boilerplate; infinite entropy from out-of-vocabulary words is clamped to 100 so one unseen token cannot dominate the score.
cleanup_data.ipynb runs cleanlab’s OutOfDistribution over the section embeddings, splits at the 2.5th percentile, then re-clusters the outlier, rest and anomaly buckets with UMAP + HDBSCAN and colours them by domain — which shows which sources produce content that does not fit the rest.
Where it broke, and what it could not do
Limits, as observed in the code and today
Could it answer any design-system question?
similarity_threshold 0.4, the edge function throws instead of calling the model, and the prompt forbids answering outside the retrieved context — the fallback text is literally “Sorry, I don’t know how to help with that.” The blog frames this as the point: if there is no relevant content in the database, no answer is generated. The intended failure mode is a refusal or a bad pointer, not an invention.What is broken in the committed code?
create_knowledge_base.ipynb builds paraphrase-multilingual-MiniLM-L12-v2 vectors (384 dimensions) while the table is vector(768) and the deployed encoder is msmarco-distilbert-cos-v5 (768) — a model switch that was left half-finished in the notebooks. The edge function assembles its request body by string interpolation ({"question": "${sanitizedQuery}"}), so a question containing a double quote emits invalid JSON and the embedding call fails before retrieval. And dashboard/src/.env ships a live database password, where the adjacent .env.example is empty.What is still alive today?
https://question-embedding.fly.dev still returns a 768-float vector in about half a second (checked October 2026). Almost everything around it is gone: the Supabase project the code hard-codes (pzdzoelitkqizxopmwfg.supabase.co) no longer resolves, and OpenAI retired text-davinci-003, so the answer path as committed cannot run. The cheap, self-hosted half outlived the generation half.What did the crawler miss?
#, ? or /tag/, so tabbed content and tag archives were never reached; the ingestion notes call out the Volkswagen Frontify docs as partly lost for exactly this reason. Each domain also carries a manual exclusion list — /octicons/* on primer.style, /blog/tag/* on bradfrost.com, /components/examples/ on duetds.com — so the corpus reflects one person’s judgement about what a section should look like.Links
Where to find this one