Pipelines for spatial data visualization in Python and R
Pipelines for spatial data visualization in Python and R
I have written various posts covering spatial data visualization for supply chain analysis and management, in both R and Python. Relevant libraries comprise e.g. Leaflet in Python, leaflet in R, deckgl in R and ggmap in R. I have also demonstrated how one can create spatial data animations in R, using a framework build around ggmap and gganimate.
In this post I want to introduce some exemplary…
Spatial data animation with ggmap and gganimate: A use case example
Spatial data animation with ggmap and gganimate: A use case example
Following a series of introductionary posts on spatial data visualization in R, covering packages such as ggmap, leaflet, tidygeocoder, geopy, deckgl and osmdata, I here present a framework utilizing some relevant packages. The framework was developed in R. It was implemented in a real-world project and was used to visualize the spread of defined groups. Members of these groups were registerered…
Map-based point and density plots in R, using ggmap
Map-based point and density plots in R, using ggmap
I want to provide an example for how to visualise spatial attributes of a dataset using the ggmap package in R. I start by loading the packages that I will want to use for my analysis.
#devtools::install_github("dkahle/ggmap", ref = "tidyup") # since currently ggmap is not on CRAN library(ggmap)
On each corner is a door into a small common room with a few paintings and a washer/dryer set. This opens up into a small, carpeted circular room with doors to six personal rooms. Each bedroom has a similar layout, but they’re still rather heavily personalized.
Hobby Room
This little cove has a variety of odds and ends for entertainment, from board games to fiction novels to sheet music and instruments. Everything is in carefully sorted shelves, and there’s no room to actually perform the hobbies in there.
Arcade
Although the arcade has a few classic game cabinets, it’s mostly a repository of consoles new and old for multiplayer gaming. A large cabinet houses an impossible selection of games, most requiring quick reflexes. A few tables for air hockey and the like are also available.
Theatre
The majority of this space is seating, though there is a large oval stage at the front. There’s a very small backstage space, but the curtains can be pushed aside to show a large movie screen. A projector can be used to watch movies, with the movie selection on a digital screen in the back of the theatre.
Diner
This 50′s-style diner is decked out with red plush seats and checkerboard tiles. A bar winds around a good portion of the area, though there’s of course no alcohol. A digital order-it-yourself console takes your orders from the menu, and your meal is rolled out onto the bar a few minutes later. You are, however, able to fill and refill your drinks yourself, unless you want a shake.
Stairs
A spiral staircase reaches down to the floor below. It appears to stop here.
Elevator
The elevator won’t open despite anyone’s best efforts. Apparently it is only available if someone is physically unable to take the stairs. Just to the left is a small trash chute, lockable with a key.
This area contains four full-size tennis courts. Around the sides of these courts are tennis balls, rackets of different brands and sizes, and a few pitching machines.
Arm Machines
A variety of presses and other machines focusing on the arms and shoulders. There’s also a station with a water cooler, clean towels, disinfectant spray, and a disposal for used towels.
Leg Machines
A variety of presses and other machines focusing on the legs and hips. There’s also a station with a water cooler, clean towels, disinfectant spray, and a disposal for used towels.
Core Machines
A variety of machines and exercise balls focusing on the chest, abdomen, and back. There’s also a station with a water cooler, clean towels, disinfectant spray, and a disposal for used towels.
Racquetball Courts
In this area are four racquetball courts separated by glass. An area at the front contains basic equipment for the sport from goggles to racquets and balls. The doors to each court lock from the inside, with no keys required.
Basketball Courts
This open area hosts four full-size basketball courts with stands around the edges for onlookers. Cupboards and large baskets in the back hold basketballs, air pumps, spare nets, and other supplies. The door can be locked from the inside with a key. And every hour on the hour, a strange song comes over the speakers...
Free Weights
This area is flush with dumbbells of all weights and constructions, racked neatly. There are also exercise balls and jump ropes, as well as benches and other seats for use during lifting.
Stairs
A spiral staircase goes up and around to make a full circuit of the building before reaching the next floor—except it appears to just run into the ceiling. Odd. It, of course, also reaches down to the floor below.
Elevator
The elevator won’t open despite anyone’s best efforts. Apparently it is only available if someone is physically unable to take the stairs. Just to the left is a small trash chute, lockable with a key.
This room houses a variety of equipment both for pool upkeep (chlorine tablets, pool skimmers, etc.) and pool toys (floating noodles, inner tubes, etc.). There’s also a small selection of swimsuits that aren’t precisely high-end designer but are in all of the students’ sizes.
Wading Pool
An offshoot of the main pool, the wading pool is an area probably meant for those who can’t swim. A few small, simple spurting fountains add a little variety to the mostly two-foot-deep pool. There are a few benches and even tables if someone wants to dine with feet in the water.
Poolside Cafe
This lovely area has several tables and chairs that extend all the way into the water. There’s a counter for serving and recipes for a number of tasty treats, especially sandwiches. Equipment is also there, including a panini press, but it seems you’ll have to go downstairs for ingredients.
Pool
In the center of the floor is a large, triangular pool. The corners are the shallowest, while the depth increases towards the center to 9 feet. It has a few little jets for circulating the water or shooting it through pool noodles at high velocity. There’s also a large drain at the bottom. Hopefully it doesn’t go straight down.
Showers/Changing Rooms
There are two large rooms for showers and changing. Each has five rather large, almost luxurious shower stalls and ten smaller stalls for changing into and out of swimwear. They also include ten full-length lockers, each with a locked combination lock.
Hot Tubs
This nook is dotted with hot tubs of different sizes, though they all seem to be about the same at the core. The temperatures and jets are adjustable at a little panel by the edge of each tub.
Stairs
A spiral staircase goes up and around to make a full circuit of the building before reaching the next floor—except it appears to just run into the ceiling. Odd. It, of course, also reaches down to the floor below.
Elevator
The elevator won’t open despite anyone’s best efforts. Apparently it is only available if someone is physically unable to take the stairs. Just to the left is a small trash chute, lockable with a key.
by Robert Del Vicario I've been in the market to buy a house lately as I think it would be nice for the dog to have a back yard to run around in. One of my considerations in purchasing a home is internet speed. As a general principle I want the fastest internet possible assuming that it is not outrageously expensive, and as luck would have it Cincinnati Bell is now offering 1Gbps internet speeds for ~$70. However, Cincinnati Bell does not offer a coverage map which makes it a gigantic hassle to determine which houses do and do not have 1Gbps internet speeds. Additionally, the coverage seems quite poor. Given my obsessive compulsive nature I decided to make my own coverage map. To create my own coverage map I scraped all of the home listings that meet my home buying criteria (2 or more bathrooms, $100k-$300k, single family or multi-family home) in the Cincinnati area off of realtor.com and then ran them through Cincinnati Bell's internet speed lookup tool. The process is annoyingly slow, and the imperfect nature of my scripting means that I drop many observations. However, I do think that the coverage map provides a general sense regarding 1Gbps coverage in Cincinnati.
require(data.table) require(stringr) require(XML) require(rvest) require(RSelenium) require(ggmap) setwd("C:/Users/rdelvicario/Dropbox/Realtor Scraper") trunk_url <- "http://www.realtor.com/realestateandhomes-search/Cincinnati_OH/beds-2/baths-2/type-single-family-home,multi-family-home/price-100000-300000/show-hide-pending/pg-" branch_url <- "?ml=4" extract_text <- function(i trunk_url branch_url){ try({ #extract data from the McKinsey address <- data.table() page <- html(paste0(trunk_url i branch_url)) x <- page %>% html_nodes(".listing-location .ellipsis span") %>% html_text() for(z in 1:10){ street_number <- str_sub(x[1 + (z - 1) * 4] start = 1 end = str_locate(x[1 + (z - 1) * 4] " ")-1) street <- str_sub(x[1 + (z- 1) * 4] start = str_locate(x[1 + (z - 1) * 4] " ")+1 str_length(x[1 + (z - 1) * 4])) city <- x[2 + (z - 1) * 4] state <- x[3 + (z - 1) * 4] zip <- x[4 + (z - 1) * 4] df_temp <- data.table(street_number street city state zip) address <- rbind(df_temp address fill = T) } address <- address[!duplicated(address) ] return(address) } silent = T) } #address collection addresses <- data.table() for(i in 1:429){ df_temp <- extract_text(i trunk_url branch_url) addresses <- rbind(addresses df_temp) flush.console() print(i) } #clear obviously bad values addresses <- addresses[!duplicated(addresses) ] addresses$street_number <- as.numeric(addresses$street_number) addresses$zip <- as.numeric(addresses$zip) addresses <- addresses[complete.cases(addresses) ] write.csv(address "150420 realtor addresses.csv" row.names = F) #Start Cincinnati Bell web scrape #tutorial http://cran.r-project.org/web/packages/RSelenium/vignettes/RSelenium-basics.html #css selector is: div:nth-child(3) .blue RSelenium::startServer() # remDr <- remoteDriver(remoteServerAddr = "localhost" # , port = 4444 # , browserName = 'firefox') eCap <- list(phantomjs.binary.path = "C:/Users/rdelvicario/Desktop/phantomjs.exe") remDr <- remoteDriver(browserName = "phantomjs" extraCapabilities = eCap) extractSpeed <- function(addresses i wait_time = 5){ tryCatch({ #create new instance of web browser remDr$open(silent = T) #navigate to cincinnati bell wbsite remDr$navigate("http://www.cincinnatibell.com/internet/") #remove landing page webElem <- remDr$findElement(using = "id" "x-region") webElem$clickElement() #find and click the check avaliability link webElem <- remDr$findElement(using = "css selector" "p > a") # webElem$highlightElement() # to visually check what elemnet is selected webElem$clickElement() #sleep for a few seconds Sys.sleep(5) #input address data webElem <- remDr$findElement(using = "id" "street_num") webElem$sendKeysToElement(list(as.character(addresses$street_number[i]))) webElem <- remDr$findElement(using = "id" "street_name") webElem$sendKeysToElement(list(addresses$street[i])) webElem <- remDr$findElement(using = "id" "zip") webElem$sendKeysToElement(list(as.character(addresses$zip[i]))) #click check address button webElem <- remDr$findElement(using = "css selector" "img.b_check_address.float_right") webElem$clickElement() #sleep for a few seconds Sys.sleep(wait_time) #extract speed webElem <- remDr$findElement(using = "css selector" "div:nth-child(3) .blue") out <- webElem$getElementText()[[1]] #close browser session remDr$close() #return value out } error = function(e){ remDr$close() NA }) } #test extractSpeed(addresses 3 wait_time = 20) #loop through all the addresses to determine internet speed addresses$speed <- NA for(i in 1:nrow(addresses)){ addresses$speed[i] <- extractSpeed(addresses i 10) flush.console() print(addresses$speed[i]) flush.console() print(i) } # write.csv(addresses, "data locations.csv", row.names = F) #create addresses addresses$address <- paste0(addresses$street_number " " addresses$street ", " addresses$city ", " addresses$state " " addresses$zip) #remove locations with no data addresses_2 <- addresses addresses_2 <- addresses_2[!is.na(addresses_2$speed) ] #get lattitude and longitude from google latlon <- t(sapply(addresses_2$addressgeocode USE.NAMES=F)) latlon <- as.data.frame(latlon) #attach addresses to data.frame addresses_2 <- cbind(addresses_2 latlon) addresses_2$lon <- as.numeric(addresses_2$lon) addresses_2$lat <- as.numeric(addresses_2$lat) #write out data write.csv(addresses_2 "data locations with lattitude and longitude.csv" row.names = F)
Map Quest: #Nashville Residential Building Permits #realestate #rstats #dataviz
Strap on your tool belt, we’re talkin’ ’bout building permits! Specifically, we’re dipping our toes into the building of new homes in Nashville, Tennessee, where we mourn the loss of former resident Swizzle Stick to the Big Apple. Bye, Tay-Tay!
Nashville’s real estate market has been on fire the last few years, and there’s much discussion in the local mediaabout how the rising sales prices and…
