Reddit data for retrieval-augmented generation

Two designs

Live lookup in one function

def reddit_context(question, threads=3):
    hits = get("/search", q=question, sort="relevance", timeRange="year", limit=10)
    docs = []
    for post in hits["data"]["posts"][:threads]:
        thread = get(f"/posts/{post['id']}", commentLimit=30)["data"]
        docs.append(thread)                      # post + its top comments, one chunk
    return docs

That is 1 search and 3 thread requests per question. get is the small client from the Python guide. An agent can do the same through the MCP server with no code.

Syncing an index

for sub in ["LocalLLaMA", "MachineLearning", "rust"]:
    for post in get(f"/r/{sub}/top", limit=100)["data"]["posts"]:
        thread = get(f"/posts/{post['id']}", commentLimit=100)["data"]
        vector_store.upsert(id=post["id"], text=render(thread), metadata={"sub": sub})

What it costs

Thread size does not change the price. A post with 200 comments is one request, the same as a post with none. Listing 100 threads to sync costs ten requests, and each thread read is one more.

Three things that improve answers

RAG questions