Reddit data for research
Short answer: Use an archive for historical data and a live API for anything recent. Record exactly how you collected it so the dataset can be rebuilt, and settle the ethics and terms questions with your institution before you collect anything.
Pick the source by time period
| Period you study | Source | Notes |
|---|---|---|
| Past years | Public archives such as Arctic Shift and the Academic Torrents dumps | Complete and citable. Large downloads |
| The last few weeks | A live API | Archives have not caught up yet |
| An ongoing event | A live API on a schedule | Start early. You cannot go back for what was deleted |
| Small, non-commercial samples | The official API | Free within 100 queries per minute |
More on the split between history and live data is on the Pushshift alternatives page.
Make the dataset reproducible
- Fix the sampling rule. Which communities, which sort order, which dates, which keywords.
- Log every request. Store the path, the parameters and the time it was made.
- Keep ids. Post and comment ids let another team fetch the same items.
- Record what was missing. Removed and deleted items are a finding, not noise.
- Publish the code. A script and an id list are easier to share lawfully than raw text.
Estimate the collection cost
Apixstar charges $0.0007 per request, or per 10 results of a listing or search. A post with up to 200 comments is one request.
| Study design | What it takes | Cost |
|---|---|---|
| 10,000 posts from listings | 100 pages of 100 | $0.70 |
| Those 10,000 posts with their comment threads | 100 pages plus 10,000 thread reads | $7.70 |
| 20 subreddits, newest 10 posts every 10 minutes for a month | 86,400 polls | $60 |
Ethics and terms to settle first
- Review board. Ask whether your study needs approval, even for public posts.
- Identifiability. Quoted text can be searched. Paraphrase or aggregate where people could be found.
- Sensitive communities. Health, sexuality and crisis forums deserve extra care.
- Reddit's terms. They restrict redistribution and model training. Read the Data API wiki.
- Our terms. Apixstar forbids using data to re-identify or profile individuals. See acceptable use.
Research questions
Is the Reddit API free for academic research?
Reddit allows free non-commercial use of its Data API within rate limits, and it has run a separate program for researchers. Check Reddit's current terms, because eligibility and conditions have changed over time.
Can I publish a dataset of Reddit posts?
Be careful. Reddit's terms restrict redistribution of its content, and privacy law may apply to usernames and text. Many researchers publish post ids and code so others can rebuild the dataset, not the raw text.
Do I need ethics approval to study public Reddit posts?
Ask your institution. Public availability does not settle the question. Review boards consider whether users would expect the use and whether individuals could be identified or harmed.
How do I handle deleted posts?
Decide the rule before you collect and state it in your methods. A common approach is to remove content from your working data when the author deletes it.
Can I use Reddit data to train a model?
Reddit's terms restrict using its content to train machine learning models without permission. Get that question answered before you start, not after.