Excellent Scraping/Form Posting/Mining Mechanize Examples/Scripts (that Work!!!))) hehe
WWW::Mechanize examples Google require 'rubygems' require 'mechanize' agent = WWW::Mechanize.new agent.user_agent_alias = 'Mac Safari' page = agent.get("http://www.google.com/") search_form = page.forms.with.name("f").first search_form.q = "Hello" search_results = agent.submit(search_form) puts search_results.body Rubyforge require 'mechanize' agent = WWW::Mechanize.new page = agent.get('http://rubyforge.org/') link = page.links.text(/Log In/) page = agent.click(link) form = page.forms[1] form.form_loginname = ARGV[0] form.form_pw = ARGV[1] page = agent.submit(form, form.buttons.first) puts page.body File Upload This example uploads one image as two different images to flickr. require 'rubygems' require 'mechanize' agent = WWW::Mechanize.new # Get the flickr sign in page page = agent.get('http://flickr.com/signin/flickr/') # Fill out the login form form = page.forms.name('flickrloginform').first form.email = ARGV[0] form.password = ARGV[1] page = agent.submit(form) # Go to the upload page page = agent.click page.links.text('Upload') # Fill out the form form = page.forms.action('/photos_upload_process.gne').first form.file_uploads.name('file1').first.file_name = ARGV[2] agent.submit(form) Pluggable Parsers Lets say you want html pages to automatically be parsed with Rubyful Soup. This example shows you how: require 'rubygems' require 'mechanize' require 'rubyful_soup' class SoupParser < WWW::Mechanize::Page attr_reader :soup def initialize(uri = nil, response = nil, body = nil, code = nil) @soup = BeautifulSoup.new(body) super(uri, response, body, code) end end agent = WWW::Mechanize.new agent.pluggable_parser.html = SoupParser Now all HTML pages will be parsed with the SoupParser class, and automatically give you access to a method called ‘soup’ where you can get access to the Beautiful Soup for that page. Using a proxy require 'rubygems' require 'mechanize' agent = WWW::Mechanize.new agent.set_proxy('localhost', '8000') page = agent.get(ARGV[0]) puts page.body The transact method transact runs the given block and then resets the page history. I.e. after the block has been executed, you’re back at the original page; no need count how many times to call the back method at the end of a loop (while accounting for possible exceptions). This example also demonstrates subclassing Mechanize. require 'mechanize' class TestMech < WWW::Mechanize def process get 'http://rubyforge.org/' search_form = page.forms.first search_form.words = 'WWW' submit search_form page.links.with.href( %r{/projects/} ).each do |link| next if link.href =~ %r{/projects/support/} puts 'Loading %-30s %s' % [link.href, link.text] begin transact do click link # Do stuff, maybe click more links. end # Now we're back at the original page. rescue => e $stderr.puts "#{e.class}: #{e.message}" end end end end TestMech.new.process
