Social Media Data Scripts

Social Media Data Scripts
Alternatives To Social Media Data Scripts
Project NameStarsDownloadsRepos Using ThisPackages Using ThisMost Recent CommitTotal ReleasesLatest ReleaseOpen IssuesLicenseLanguage
Rsshub23,832212 hours ago1,820September 23, 2022268mitJavaScript
🍰 Everything is RSSible
5 days ago3mitRust
🤱🏻 Turn any webpage into a desktop app with Rust. 🤱🏻 很简单的用 Rust 打包网页生成很小的桌面 App
Hitomi Downloader11,860
20 days ago2,096Python
:cake: Desktop utility to download images/videos/music/text from various websites, and more.
Alternative Front Ends3,765
4 days ago37agpl-3.0
Overview of alternative open source front-ends for popular internet platforms (e.g. YouTube, Twitter, etc.)
5 days ago14gpl-3.0JavaScript
A web extension that redirects popular sites to alternative frontends and backends
Privacy Redirect1,604
a month ago196gpl-3.0JavaScript
A simple web extension that redirects Twitter, YouTube, Instagram & Google Maps requests to privacy friendly alternatives.
13 hours ago27February 28, 20226mitPython
Make everything a GitHub svg poster and Skyline!
Alternative Frontends1,284
2 months ago10gpl-3.0
🔐🌐 Privacy-respecting web frontends for popular services
Russia It Podcast1,128
a year ago11
Список русскоязычных подкастов на тему информационных технологий
6 years ago17C
c 语言实现的一个最小的http代理,支持翻墙
Alternatives To Social Media Data Scripts
Select To Compare

Alternative Project Comparisons

Mining data from social media platforms


At present, most journalists treat social sources like they would any other — individual anecdotes and single points of contact. But to do so with a handful of tweets and Instagram posts is to ignore the potential of hundreds of millions of others.

Many stories lay dormant in the vast amounts of data produced by everyday consumers. Here's a guide and tool box that may help you. What you find below are a number of scripts developed to mine data from APIs.

Slides that explain the work process can be found here. I'm currently in the process of writing more thorough resources on the subject of social media data mining. Feel free to reach out with questions on Twitter @lamthuyvo!

How to get the data

What data you can get with the scripts

This is a growing list of scripts we've put together to make social data mining easier.

There are broadly three different ways to harvest data from the social web:

  • APIs
  • Personal archives
  • Scraping

The kind of data that official channels like API data streams provide is very limited. Despite harboring warehouses of data on consumers’ behavior, social media companies only provide a sliver of it through their APIs (for Facebook, developers can only get data for public pages and groups, and for Twitter, this access is often restricted to a set number of tweets from a user’s timeline or to a set time frame for search).

Scripts and instructions related to APIs can be found in the 01-apis directory of this repository.

Personal Archives

There are ways for users of social media platforms to request and download archives of their own online persona and behavior. Some services like Facebook or Twitter will allow users to download a history of the data that constitutes their posts, their messaging, or their profile photos.

Scripts and instructions related to personal archives can be found in the 02-personal-archives directory of this repository.


While there's plenty of social media data on display on the sites you browse, extracting social media data from the platforms through scraping is often against the terms of service. Scraping a social media platform can get users booted from a service and potentially even result in a lawsuit.

If you end up wanting to look into harvesting data from the social web, there is information information related in the 03-scraping directory of this repository.


Below is a set of instructions you can follow to get your machine ready to run any of the Python scripts in this repository. While Python is one of the most powerful languages for data gathering and analysis, it can take a few tries to get it installed and running properly. If you're a beginner, don't despair though, these growing pains are normal and can vary from machine to machine. We promise the payoff is worth it!

Before you begin

  1. If you don’t already have Python installed, start by getting Python up and running. There are two Python versions — Python 2 and Python 3. Please install Python 3, as it handles modern Internet language and characters related to it better. Also have git installed. A helpful guide to getting a brand new machine set up can be found here, courtesy of NPR's Visuals Team.
  2. You should also make sure you have pip.

Installing all the libraries you need

  1. You need to get developer oauth credentials from the social media platforms you want to tap into. Oauth credentials are like an ID and password (often referred to as an app ID and secret respectively) that you create for an app or a script to access the data stream that a social media company provides. This data stream — also known as a company's Application Program Interface, or API — is often accessible using these credentials through a link (for example, this is what one of these queries could look like|YOURSECRET). Here's where you can get them: Twitter: Facebook:

Setting up your system

  1. Open up your Terminal and go to the folder where you want to clone this repository of code using the cd bash command.
git clone
cd social-media-data-scripts
  1. Then install all the dependencies, i.e. the Python libraries we are using for these scripts by running the following command:
pip install -r requirements.txt


sudo pip install -r requirements.txt

If you have problems with installing the dependencies through

pip install requests
pip install tweepy --ignore-installed six
pip install beautifulsoup4


sudo pip install requests
sudo pip install tweepy --ignore-installed six
pip install beautifulsoup4

Getting your data

Hooray! You're ready to get your data now. We have created a directory for scripts that you can use to get data from each data source.

You can follow the directions for each script in its sub-folders:

  • To gather data from APIs, you can use the scripts in this directory: 01-apis
  • To gather data from personal archives, you can use the scripts in this directory: 02-personal-archives
  • To gather data from live web sites, you can use the scripts and instructions in this directory: 03-scraping

Further Reading

There are numerous useful resources and tools out on the web for social media data gathering. Find an incomplete list that I'll continue to update below.



Academic research

Technical resources

Popular Twitter Projects
Popular Youtube Projects
Popular Social Media Categories
Related Searches

Get A Weekly Email With Trending Projects For These Categories
No Spam. Unsubscribe easily at any time.