Project Overview
All Deals of Today is an innovative web platform designed to filter the best deals from leading retailers into a single, accessible website. This project uses advanced web scraping techniques to navigate and bypass the sophisticated scraping detectors used by Amazon, Walmart, and others, presenting users with Amazon Deals of the Day, Bestbuy's top deals, Walmart, and Target offers all in one place.
Key Features
- Comprehensive Deals: Offers a selection of the day's best deals from Amazon, Bestbuy, Walmart, and Target.
- Advanced Web Scraping: Utilizes multiple scraping methods to successfully scrape data without triggering anti-scraping mechanisms.
- Daily Updates: Deals are refreshed daily thanks to automated GitHub CRON jobs, ensuring latest offers.
- Affiliate Marketing: Product links include affiliate codes, generating revenue with each purchase made from links on the site.
How It Works
Data Collection
The backbone of All Deals of Today is a script that scrapes major retailers' deals pages. Through using advanced techniques and algorithms, my script is able to extract deal information while bypassing retailers' countermeasures against scraping.
Data Processing
Scraped data is processed and formatted into a clean JSON structure, making it easily accessible for the front end to display. Each deal entry contains essential information such as the product name, description, discounted price, and a direct link to the purchase page.
Frontend Display
The frontend dynamically unpacks the JSON data to present each deal in a user-friendly interface. Deals are organized by retailer and category, allowing users to effortlessly browse and find offers.
Automation with GitHub CRON Jobs
To ensure that All Deals of Today always features the latest deals, the project is deployed with GitHub CRON jobs to run the scraping script once every day. This automation keeps the site's content fresh and accurate.
Example JSON Format
{
"retailer": "Amazon",
"title": "Echo Dot (3rd Gen)",
"description": "Smart speaker with Alexa - Charcoal",
"price": "$24.99",
"originalPrice": "$49.99",
"link": "https://www.amazon.com/dp/product-link?&tag=ManitsAffiliateID"
}