technote: wget
wget -r -k -p -c -np -e robots=off http://www.website.com/directory/
-r : recursive
-k : convert links to local folder
-p : get all files including images and data files
-c : continue/retry if errors
-np : don’t recursively go above the parent directory
-e robots=off : ignore robots files
(optional: -N to only get files with newer timestamps than on local drive)










