#scrapy framework

[olegvolodin]

Harvesting Craigslist's users emails, web scraping of their phones numbers, contact names from Craigslist. I wrote a bot using python, scrapy, webdriver to web-scrape phones numbers, emails, contact names. http://blablup.com Crawling sites, parsing pages, scraping information from different types sites, storing data in files and databases. Please do not hesitate to contact me, if you need to find, enter, extract, web scraping some data from any sites: [email protected] [email protected] or https://twitter.com/volodin_o https://www.facebook.com/blablupcom https://www.linkedin.com/pub/oleg-vol... https://www.tumblr.com/blog/olegvolodin http://inheritor.elance.com

[olegvolodin]

Scraping Movie Data From Justdial Using Scrapy.

Oleg Volodin

2015-03-05 20:07

           Comments                                    

This is the second post about extracting movie information from the site Justdial. The first one was devoted to scraping the same data using Selenium. This post describes the abilities of the Scrapy framework to extract data including dynamically loaded Javascript objects.

Scrapy is the great framework for crawling both the whole site and the only part of it. Along with crawling, Scrapy can be used for extracting, processing and saving data. Moreover, Scrapy uses Twisted asynchronous networking Python library.

Unfortunately, Scrapy cannot download the part of a webpage that is loaded by means of JSON. In this post I'll show how to circumvent this flaw.

In the beginning, we should create a spider in the Scrapy standart spider folder. Read more

http://blablup.com/posts/scraping-movie-data-from-justdial-using-scrapy.html

[olegvolodin]

[olegvolodin]

Scraping Movies Information From http://www.justdial.com.

Oleg Volodin

2015-02-25 20:07

           Comments                                    

I have created this blog for making notes to myself and other persons who are interested in scraping information from the Internet sites, automating routine operations using different programmable tools.

The first task I decided to tackle was scraping some information from the site Justdial. I have to grab all movies, showtimes, information on Chennai cinema theaters like name, address and telephone number. To get this information, I need analyze the first page of the cinema theaters that justdial.com return us. The address of this page is http://www.justdial.com/Chennai/Cinema-Halls/ct-7451/page-1. Using Firebug tools of the Firefox browser I can get all necessary html tags that contain useful information.

Unfortunately, when I examined the page of a particular cinema theater thoroughly, I noted that movies titles and showtimes appeared on the page by means of JavaScript. Luckily, pythonists have Selenium, a great framework for getting, testing and analyzing web pages. So, before parsing a web page with information loaded by JavaScript, we have to use a webdriver Selenium suggests. For my task I chose the PhantomJS headless browser. BeautifulSoup is the next great python library that helps me to parse the DOM tree the Selenium webdriver returned.

Armed with this information, let's start coding.

http://blablup.com/posts/scraping-movies-information-from-justdial.html

[olegvolodin]

crawling, scraping data by python, scrapy framework, selenium, flask and django