Web scraper for scraping, tracking and visualizing prices of products on various websites.
Alternatives To Scraper
Project NameStarsDownloadsRepos Using ThisPackages Using ThisMost Recent CommitTotal ReleasesLatest ReleaseOpen IssuesLicenseLanguage
3 months ago5mitPython
Web scraper for scraping, tracking and visualizing prices of products on various websites.
84 years ago92August 09, 20182otherPython
Python3 API wrapper and web scraper for
Media Crawler22
6 years ago12Python
Web scraper for generating a graph of media connections via articles, twitter, reddit, and more
5 years ago4January 18, 20172JavaScript
Node web scraper that extracts metadata static & javascript websites using HTML, Open Graph, Twitter metadata
2 years ago13mitJupyter Notebook
A utility to log, visualize, and highlight climbing strengths, weaknesses, and trends. Built using Python, ElasticSearch, Kibana, and Docker
6 years agomitPython
A simple web crawler developed as coursework for Algorithms on Graph Theory - PUC Minas
8 months agomitPython
Python 3 graph implementation designed to be turned into a web scraper for graph data.
10 years agomitRuby
A web crawler that creates a graph based on the pages, static assets and how they are interlinked.
6 years agoC++
System that finds the critical path of the graph of the disciplines of the Computer Science course at UnB (University of Brasília).
2 years agogpl-3.0Jupyter Notebook
Extract Tables from Wikipedia into an excel file
Alternatives To Scraper
Select To Compare

Alternative Project Comparisons

Table of contents


With this program you can easily scrape and track prices on product at multiple websites.
This program can also visualize price over time of the products being tracked. That can be helpful if you want to buy a product in the future and wants to know if a discount might be around the corner.

Requires python 3.10+


Feel free to fork the project and create a pull request with new features or refactoring of the code. Also feel free to make issues with problems or suggestions to new features.


In version v1.1, I have changed how data is stored in records.json: dates under each product have been changed to datapoints and now a list containing dictionaries with date and price keys.
If you want to update your data to be compatible with version v1.1, then open an interactive python session where this repository is located and run the following commands:

>>> from scraper.format_to_new import Format
>>> Format.format_old_records_to_new()


In version v2.3.0, I have add the column short_url to products.csv. If you have add products before v2.3.0, then run the following commands in an interactive python session to add the new column:

>>> from scraper.format_to_new import Format
>>> Format.add_short_urls_to_products_csv()


Requires python 3.10+

Clone this repository and move into the repository:

git clone
cd scraper

Then make sure you have the modules, run this in the terminal:

pip3 install -r requirements.txt

Add products

To add a single product, use the following command, where you replace <category> and <url> with your category and url:

python3 -a -c <category> -u <url>


python3 -a -c vr -u

This adds the category (if new) and the product to the records.json file, and adds a line at the end of the products.csv file so the script can scrape price of the new product.

To add multiple products at once, just add specify another category and url with -c <category> and -u <url>. E.g. with the following command I add two products:

python3 -a -c <category> -u <url> -c <category2> -u <url2>

This is equivalent to the above:

python3 -a -c <category> <category2> -u <url> <url2>

OBS: The url must have a schema like: https:// or http://.
OBS: If an error occures when adding a product, then the error might happen because the url has a & in it, when this happens then just put quotation marks around the url. This should solve the problem. If this doesn't solve the problem then summit a issue.

Websites to scrape from

This scraper can (so far) scrape prices on products from:

*OBS these Amazon domains should work: .com, .ca, .es, .fr, .de and .it
The listed Amazon domains is from my quick testing with one or two products from each domain.
If you find that some other Amazon domains works or some of the listed doesn't please create an issue.

Scrape products

To scrape prices of products run this in the terminal:

python3 -s

To scrape with threads run the same command but with the --threads argument:

python3 -s --threads

Delete data

If you want to start from scratch with no data in the records.json and products.csv files, then just run the following command:

python3 --delete --all

You can also just delete some products or some categories:

python3 --delete --id <id>
python3 --delete --name <name>
python3 --delete --category <category>

Then just add products like described here.

If you just want to reset your data for every product, deleting all datapoints inside every product, then run this command:

python3 --reset --all

This deletes the data inside each product, such as id, url and all datapoints.

You can also just reset some products or all products in some categories:

python3 --reset --id <id>
python3 --reset --name <name>
python3 --reset --category <category>

User settings

User settings can be added and changed in the file settings.ini.


Under the category ChangeName you can change how the script changes product names, so similar products will be placed in the same product in records.json file.

When adding a new setting under the category ChangeName in settings.ini, there must be a line with key<n> and a line with value<n>, where <n> is the "link" between keywords and valuewords. E.g. value3 is the value to key3.

In key<n> you set the keywords (seperated by a comma) that the product name must have for to be changed to what value<n> is equal to. Example if the user settings is the following:

key1 = asus,3080,rog,strix,oc
value1 = asus geforce rtx 3080 rog strix oc

The script checks if a product name has all of the words in key1, it gets changed to what value1 is.


You can change the time between each time a url is being request by changing the field request_delay in the file scraper/settings.ini under the Scraping section.

Default is 0 seconds, but to avoid the website you scrape products from thinking you are DDOS attacting them or you being restricted from scraping on their websites temporarily, set the request_delay in settings.ini to a higher number of seconds, e.g. 5 seconds.

Clean up data

If you want to clean up your data, meaning you want to remove unnecessary datapoints (datapoints that have the same price as the datapoint before and after it), then run the following command:

python3 --clean-data

Search products and categories

You can search for product names and categories you have in your records.json by using the argument --search [<word> ...]. The search is like a keyword search, so e.g. if you enter --search logitech all product names and categories that contains the word "logitech" are found.

You can search with multiple keywords, just seperate them with a space: --search logitech corsair. Here all the product names and categories that contains the words "logitech" or "corsair" are found.

View the latest datapoint of product(s)

If you want to view the latest datapoint of a product, you can use the argument --latest-datapoint with --id and/or --name.


python3 --name "logitech z533" --latest-datapoint

The above command will show the latest datapoint for all the websites the specified product, in this case "logitech z533", has been scraped from and will show something like this:

> Komplett - 849816
  - DKK 999.0
  - 2022-09-12
> Proshop - 2511000
  - DKK 669.0
  - 2022-09-12
> Avxperten - 25630
  - DKK 699.0
  - 2022-09-12

View all products

To view all the products you have scraped, you can use the argument --print-all-products.


python3 --print-all-products

This will print all the products in the following format:


Visualize data

To visualize your data, just run with the -v or --visualize argument and then specify which products you want to be visualized. These are your options for how you want to visualize your products:

  • --all to visualize all your products
  • -c [<category> [<category> ...]] or --category [<category> [<category> ...]] to visualize all products in one or more categories
  • --id [<id> [<id> ...]] to visualize one or more products with the specified id(s)
  • -n [<name> [<name> ...]] or --name [<name> ...]] to visualize one or more products with the specified name(s)
  • --compare to compare two or more products with the specified id(s), name(s) and/or category(s) or all products on one graph. Use with --id, --name, --category and/or --all

Example graph

Command examples

Show graphs for all products

To show graphs for all products, run the following command:

python3 -v --all

Show graph(s) for specific products

To show a graph for only one product, run the following command where <id> is the id of the product you want a graph for:

python3 -v --id <id>

For multiple products, just add another id, like so:

python3 -v --id <id> <id>

Show graphs for products in one or more categories

To show graphs for all products in one category, run the following command where <category> is the category you want graph from:

python3 -v -c <category>

For multiple categories, just add another category, like so:

python3 -v -c <category> <category>

Show graps for products with a specific name

To show graphs for product(s) with a specific name, run the following command where <name> is the name of the product(s) you want graphs for:

python3 -v --name <name>

For multiple products with different names, just add another name, like so:

python3 -v --name <name> <name2>

If the name of a product has multiple words in it, then just add quotation marks around the name.

Only show graph for products that are up to date

To only show graphs for the products that are up to date, use the flag --up-to-date or -utd, like so:

python3 -v --all -utd

The use of the flag -utd is only implemented when visualizing all products like the example above or when visualizing all products in a category:

python3 -v -c <category> -utd

Compare two products

To compare two products on one graph, use the flag --compare with flag --id, --name, --category and/or --all, like so:

python3 -v --compare --id <id>
python3 -v --compare --name <name>
python3 -v --compare --category <category>
python3 -v --compare --id <id> --name <name> --category <category>
python3 -v --compare --all

OBS when using --name or --category multiple products can be visualized

Popular Web Crawler Projects
Popular Graph Projects
Popular Data Processing Categories
Related Searches

Get A Weekly Email With Trending Projects For These Categories
No Spam. Unsubscribe easily at any time.
Web Crawler