#xbyte

[juveria-dalvi]

Web Scraping 103 : Scrape Amazon Product Reviews With Python –

Amazon is a well-known e-commerce platform with a large amount of data available in various formats on the web. This data can be invaluable for gaining business insights, particularly by analyzing product reviews to understand the quality of products provided by different vendors.

In this guide we will look into web scraping steps to extract amazon reviews of a particular product and save it in excel or csv format. Since manually copying information online can be tedious, in this guide we’ll focus on scraping reviews from Amazon. This hands-on experience will enhance our practical understanding of web scraping techniques.

Before we start, make sure you have Python installed in your system, you can do that from this link: python.org. The process is very simple, just install it like you would install any other application.

Now that everything is set let’s proceed:

How to Scrape Amazon Reviews Using Python

Install Anaconda using this link: https://www.anaconda.com/download . Be sure to follow the default settings during installation. For more guidance, please click here.

We can use various IDEs, but to keep it beginner-friendly, let’s start with Jupyter Notebook in Anaconda. You can watch the video linked above to understand and get familiar with the software.

Steps for Web Scraping Amazon Reviews:

Create New Notebook and Save it. Step 1: Let’s start importing all the necessary modules using the following code:

import requests from bs4 import BeautifulSoup import pandas as pd

Step 2: Define Headers to avoid getting your IP blocked. Note that you can search my user agent on google to get your user agent details and replace it below “User-agent”: “here goes your useragent below”.

custom_headers = { "Accept-language": "en-GB,en;q=0.9", "User-agent": "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/17.1 Safari/605.1.15", }

Step 3: Create a python function, to fetch the webpage, check for errors and return a BeautifulSoup object for further processing.

# Function to fetch the webpage and return a BeautifulSoup object def fetch_webpage(url): response = requests.get(url, headers=headers) if response.status_code != 200: print("Error in fetching webpage") exit(-1) page_soup = BeautifulSoup(response.text, "lxml") return page_soup

Step 4: Inspect Element to find the element and attribute from which we want to extract data, Lets Create another function to select the div and attribute and set it to variable , extract_reviews identifies review-related elements on a webpage, but it doesn’t yet extract the actual review content. You would need to add code to extract the relevant information from these elements (e.g., review text, ratings, etc.).

Function to extract reviews from the webpage def extract_reviews(page_soup): review_blocks = page_soup.select('div[data-hook="review"]') reviews_list = []

Step 5: Below code processes each review element and extracts the customer’s name (if available), and stores it in the customer variable. If no customer information is found, customer remains none.

#for review in review_blocks: author_element = review.select_one('span.a-profile-name') customer = author_element.text if author_element else None rating_element = review.select_one('i.review-rating') customer_rating = rating_element.text.replace("out of 5 stars", "") if rating_element else None title_element = review.select_one('a[data-hook="review-title"]') review_title = title_element.text.split('stars\n', 1)[-1].strip() if title_element else None content_element = review.select_one('span[data-hook="review-body"]') review_content = content_element.text.strip() if content_element else None date_element = review.select_one('span[data-hook="review-date"]') review_date = date_element.text.replace("Reviewed in the United States on ", "").strip() if date_element else None image_element = review.select_one('img.review-image-tile') image_url = image_element.attrs["src"] if image_element else None

Step 6: The purpose of this function is to process scraped reviews. It takes various parameters related to a review (such as customer, customer_rating, review_title, review_content, review_date, and image URL), and the function returns the list of processed reviews.

review_data = { "customer": customer, "customer_rating": customer_rating, "review_title": review_title, "review_content": review_content, "review_date": review_date, "image_url": image_url } reviews_list.append(review_data) return reviews_list

Step 7: Now, Let’s initialize a search_url variable with an Amazon product review page URL

def main(): review_page_url = "https://www.amazon.com/BERIBES-Cancelling-Transparent-Soft-Earpads-Charging-Black/product- reviews/B0CDC4X65Q/ref=cm_cr_dp_d_show_all_btm?ie=UTF8&reviewerType=all_reviews" page_soup = fetch_webpage(review_page_url) scraped_reviews = extract_reviews(page_soup)

Step 8: Now let’s print(“Scraped Data:”, data) scraped review data (stored in the data variable) to the console for verification purposes.

# Print the scraped data to verify print("Scraped Data:", scraped_reviews)

Step 9: Next, Create a dataframe from the data which will help organize data into tabular form.

# create a DataFrame and export it to a CSV file reviews_df = pd.DataFrame(data=scraped_reviews)

Step 10: Now exports the DataFrame to a CSV file in current working directory

reviews_df.to_csv("reviews.csv", index=False) print("CSV file has been created.")

Step 11: below code construct acts as a protective measure. It ensures that certain code runs only when the script is directly executed as a standalone program, rather than being imported as a module by another script.

# Ensuring the script runs only when executed directly if __name__ == '__main__': main()

Result:

Why Scrape Amazon Product Reviews?

Scraping Amazon product reviews can provide valuable insights for businesses. Here’s why you should consider it:

● Feedback Collection: Every business needs feedback to understand customer requirements and implement changes to improve product quality. Scraping reviews allows businesses to gather large volumes of customer feedback quickly and efficiently.

● Sentiment Analysis: Analyzing the sentiments expressed in reviews can help identify positive and negative aspects of products, leading to informed business decisions.

● Competitor Analysis: Scraping allows businesses to monitor competitors’ pricing and product features, helping to stay competitive in the market.

● Business Expansion Opportunities: By understanding customer needs and preferences, businesses can identify opportunities for expanding their product lines or entering new markets.

Manually copying and pasting content is time-consuming and error-prone. This is where web scraping comes in. Using Python to scrape Amazon reviews can automate the process, reduce manual errors, and provide accurate data.

Benefits of Scraping Amazon Reviews

● Efficiency: Automate data extraction to save time and resources.

● Accuracy: Reduce human errors with automated scripts.

● Large Data Volume: Collect extensive data for comprehensive analysis.

● Informed Decision Making: Use customer feedback to make data-driven business decisions.

I found an amazing, cost-effective service provider that makes scraping easy. Follow this link to learn more.

Conclusion

Now that we’ve covered how to scrape Amazon reviews using Python, you can apply the same techniques to other websites by inspecting their elements. Here are some key points to remember:

● Understanding HTML: Familiarize yourself with HTML structure. Knowing how elements are nested and how to navigate the Document Object Model (DOM) is crucial for finding the data you want to scrape.

● CSS Selectors: Learn how to use CSS selectors to accurately target and extract specific elements from a webpage.

● Python Basics: Understand Python programming, especially how to use libraries like requests for making HTTP requests and BeautifulSoup for parsing HTML content.

● Inspecting Elements: Practice using browser developer tools (right-click on a webpage and select “Inspect” or press Ctrl+Shift+I) to examine the HTML structure. This helps you find the tags and attributes that hold the data you want to scrape.

● Error Handling: Add error handling to your code to deal with possible issues, like network errors or changes in the webpage structure.

● Legal and Ethical Considerations: Always check a website’s robots.txt file and terms of service to ensure compliance with legal and ethical rules of web scraping.

By mastering these areas, you’ll be able to confidently scrape data from various websites, allowing you to gather valuable insights and perform detailed analyses.

[xbytecrawling]

This blog gives you information about scraping Amazon product data like names, pricing, ASIN, etc. and how professional services like X-Byte enterprise Crawling helps you in that.

Amazon provides many services to the Prime members. There’s presently no way of just exporting product data from Amazon to the spreadsheet for business requirements that you may have. Either to do comparison shopping, competitor research, or building an API for the app projects.

It’s evident that web scraping can easily solve this problem.

Amazon product data scraping will permit you to choose particular data that you’d wish from Amazon site into a JSON or spreadsheet file. You may even make an automated procedure, which runs on the daily, weekly, monthly basis for constantly updating your data.

Know More:  Scrape Amazon Product Data

[xbytecrawling]

X-Byte Enterprise Crawling provides the best Data scraping Google Hotels by Region services to extract or scrape Google hotels by region data.

The room sharing and rentals business are expected to stand out in the hotel business by 2020. Having superior profit margins with progressively growing markets, the vacation rentals might stay for a very long time. This race has become harder and it is the main reason why vacation rentals companies are looking for newer price monitoring services techniques. Scrape Google Hotels by region is among the maximum return activities that offer businesses benefits in the competition.

Click Here:  https://www.xbyte.io/scrape-google-hotels-by-region.php

[xbytecrawling]

How to Extract different Prices from an E-commerce Website?

We have to convert all these elements into easy JavaScript objects that stores the XY position value, font size and text content that looks anything like {'text':'Tennis Ball', 'fontSize':'14px', 'x':100,'y':200}. Therefore, we need to write some functions for that like given below:

Writing new scrapers for different e-commerce websites can be very tedious and expensive. This tutorial will demonstrate a few advanced methods to create a simple scraper, which could recognize patterns as well as bring data from an e-commerce page.

Let’s take a quick look at some product pages as well as identify some design patterns about how the product prices get displayed on different sites.

Sephora.com

Amazon.com

Patterns and Observations

Certain patterns, which we recognized by searching at the product pages include:

Price generally comes above further currency figures

Price is a currency figure having the biggest font sizes

Prices look like currency figures (never like words)

Prices comes within initial 600 pixels of height

Certainly, there might be exemptions to these comments, we’ll chat how to cope with these exemptions later in the blog. We can use all the observations to make an effective and general scraper.

Execution of General E-commerce Scrapers

1st Step: Installation

Here, the tutorial utilizes Google Chrome as a web browser. In case, you are not using it, you can just install it and follow the instruction.

Rather than Google Chrome, the developers use programmable versions of the Google Chrome named Puppeteer. It will eliminate the requirement of running GUI apps to run a scraper. Though, it is outside the range of the tutorial.

2nd Step: Chrome Developer Tool

Different codes presented here are designed in as easy as possible manner so it can’t fetch the prices from all product pages available there.

For the meantime, we’ll visit any Sephora or Amazon product pages in the Google Chrome browser.

Visit that product pages in the Google Chrome

Then right-click anyplace on a page to choose ‘Inspect’ option and open Chrome DevTools

Then click on a DevTools’ Console tab

Within a Console tab, enter some JavaScript codes and browser will accomplish the codes in context of a web page, which have been loaded. Also, you can study more about the DevTools through the official documentation.

3rd Step: Running a Javascript snippet

You need to copy this JavaScript snippet given below and paste that in a console.

let elements = [ ...document.querySelectorAll(' body *') ]

function createRecordFromElement(element) {

const text = element.textContent.trim()

var record = {}

const bBox = element.getBoundingClientRect()

if(text.length <= 30 && !(bBox.x == 0 && bBox.y == 0)) {

record['fontSize'] = parseInt(getComputedStyle(element)['fontSize']) }

record['y'] = bBox.y

record['x'] = bBox.x

record['text'] = text

return record

}

let records = elements.map(createRecordFromElement)

function canBePrice(record) {

if( record['y'] > 600 ||

record['fontSize'] == undefined || !record['text'].match(/(^(US ){0,1}(rs\.|Rs\.|RS\.|\$|₹|INR|USD|CAD|C\$){0,1}(\s){0,1}[\d,]+(\.\d+){0,1}(\s){0,1}(AED){0,1}$)/) )

return false

else return true

}

-

let possiblePriceRecords = records.filter(canBePrice)

let priceRecordsSortedByFontSize = possiblePriceRecords.sort(function(a, b) {

if (a['fontSize'] == b['fontSize']) return a['y'] > b['y']

return a['fontSize'] < b['fontSize']

})

console.log(priceRecordsSortedByFontSize[0]['text']);

Press the ‘Enter’ key and you will see the product price displayed on a console.

If you don’t do that, you have perhaps visited the product page that is an exemption to our explanations. It is completely common, we’ll chat how we can increase our script for covering more product pages about these types. You can try any sample pages given in the step 2.

This animated GIF given below indicates how we extract the prices from Amazon.com

How Does It Work?

First, we need to draw all the HTML DOM elements in a page

let elements = [ ...document.querySelectorAll(' body *') ]

We have to convert all these elements into easy JavaScript objects that stores the XY position value, font size and text content that looks anything like {'text':'Tennis Ball', 'fontSize':'14px', 'x':100,'y':200}. Therefore, we need to write some functions for that like given below:

function createRecordFromElement(element) {

const text = element.textContent.trim() // Brings content of an element

var record = {} // Starts an easy JavaScript object

const bBox = element.getBoundingClientRect()

// getBoundingClientRect is the function given by Google Chrome, this returns

// an object that comprises x,y values, width and height

if(text.length <= 30 && !(bBox.x == 0 && bBox.y == 0)) {

record['fontSize'] = parseInt(getComputedStyle(element)['fontSize'])

}

// getComputedStyle is the function given by Google Chrome, this returns an

// object having all the style data. As this function is fairly

// time-consuming, we only collect the font sizes of elements those

// length of text content is nearly 30 and whose x as well as y coordinates are not 0

record['y'] = bBox.y

record['x'] = bBox.x

record['text'] = text

return record

}

Now, transform all the collected elements to JavaScript objects through applying the functions on all the elements through JavaScript map functions.

let records = elements.map(createRecordFromElement)

Think about the explanations we made about how the price gets displayed. Now, we can filter those records that match with the design observations. Therefore we require a function, which says whether the given records match with the design observations.

function canBePrice(record) {

if(record['y'] > 600 ||

record['fontSize'] == undefined || !record['text'].match(/(^(US ){0,1}(rs\.|Rs\.|RS\.|\$|₹|INR|USD|CAD|C\$){0,1}(\s){0,1}[\d,]+(\.\d+){0,1}(\s){0,1}(AED){0,1}$)/) )

return false

else return true

}

We use Regular Expression option for checking if the provided text is the currency figures or not. Also, you may modify that regular expression if it doesn’t include any pages, which you’re testing with.

Currently, we may filter only the records, which are perhaps pricing records

let possiblePriceRecords = records.filter(canBePrice)

To conclude, as we’ve witnessed, prices come as a currency figure getting the maximum font size. In case, there are several currency figures having equally higher font sizes, then price perhaps corresponds to one residing with the higher positions. We sort out all our records depending on the conditions, through JavaScript’s sort functions.

let priceRecordsSortedByFontSize = possiblePriceRecords.sort(function(a, b) {

if (a['fontSize'] == b['fontSize']) return a['y'] > b['y']

return a['fontSize'] < b['fontSize']

})

Currently, we just have to show that on a console

console.log(priceRecordsSortedByFontSize[0]['text'])

Take that Further

Affecting to the GUI-less-dependent Scalable Programs

You may replace the Google Chrome having the headless variety of that named Puppeteer. It is perhaps the quickest option for web rendering. This works completely depending on the similar ecosystem given in the Google Chrome. When the Puppeteer is all set, you can programmatically insert our script into a headless browser as well as have the pricing returned to the function in a program.

Improve and Enhance the Scripts

You will immediately notice that a few product pages won’t work with a script as they don’t trail the expectations we have fulfilled about how product prices are displayed as well as the patterns that we have recognized.

Unfortunately, there are no “holy grails” or perfect solutions for that problem. This is quite possible to produce more pages and recognize more patterns as well as improve this scraper.

Another important step, which you would utilize to deal with other pages include employing Artificial Intelligence or Machine Learning dependent methods to recognize and categorize patterns as well as automate the procedure to a bigger amount. This sector is a growing field we at X-Byte are using these methods already with variable degrees of attainment.

If you want any help in Amazon price scraping, you can investigate our tutorial specially intended for Amazon:

We Can Assist With Data and Automation Requirements

Convert the Internet to structured, meaningful, and practical data

Your Name

[email protected]

Please enter data sources, details, requests - everything relevant

You SHOULD NOT contact X-Byte for all help with the Tutorials as well as Codes using a form or through calling us, in its place please add the comments to the end of this tutorial page to get help.

Disclaimer

Any codes given in the tutorials are for learning objectives and illustration. We aren’t accountable for how this is used as well as undertake no liabilities for any harmful usage of source codes. The mere occurrence of these codes on our website does not indicate that we inspire scraping or scraping the sites referenced in a code as well as supplementary tutorial. This tutorial only helps in illustrating the method of programming the web scraper for general internet sites. We aren’t thankful to offer any help for a code, though, in case you are adding your questions within the comment section, we might occasionally address them.

[xbytecrawling]

Strengthen and Guard Your Brand with Brand Monitoring

X-Byte offers Brand and Reputation Monitoring Services For eCommerce, Retail Sector, Social Media and Online forums. We manage the reputation of your brand.

Nowadays, every business needs numerous data to derive its value. Be it structured or unstructured data. Business data plays a vital role in driving growth, development, and innovation for your dream venture. But the data that enterprise possesses come handy only if they are structured and the company can derive value from it. The increasing importance of big data gives birth to the concept of web scraping. With the help of web scraping services, you can extract unrecognized and unmanaged data and convert them into organized and structured formats. This strategy will help you out if your brand is being talked about on multiple platforms like social media, expert forums, comments, etc. Whereas, with the help of scraping tool algorithm, you can fetch only data that contains reference about the brand.

In an online world, “Brand Monitoring” is essential for organization. And the organizations face serious negative consequences as a result, if they do NOT invest in Brand Monitoring services.

Visit :  Brand and Reputation Monitoring Services