The Swarm Lab

http://www.theswarmlab.com We are an interdisciplinary research lab that studies the mechanisms underlying the coordination of large animal groups, such as ant colonies or human crowds, and their applications to complex problems such the organization of pedestrian traffic or the control of robotic swarms. We study how information is exchanged and transformed during interactions between the members of a group, and how this can lead to the emergence of "intelligent" group behaviors. In short, we study Swarm Intelligence.

[theswarmlab-blog]

Simon to speak at the Rutgers Entomology Seminar Series

On October 24, 2014 I will give a talk as part of the Rutgers Entomology Seminar Series. I will present our latest work on army ants’ living architecture. My talk will start at 11am in room 101 of Blake Hall on Rutgers New Brunswick campus. Hereafter is the abstract of my talk.

Title: An ant bridge too far: living architectures in New World army ants

Like the Roman Empire at its peak, a successful ant colony relies on an effective network of roads that facilitate the movement of its powerful army and industrious population across a vast territory. Fifty years ago, E. O. Wilson discovered the chemical nature of these transportation networks comprised of pheromone trails laid by the colony’s workers. His work paved the way for five decades of study on the incredibly efficient organization of ant colonies, based on simple behaviors, multiple interactions and powerful scents.

In this talk, I will briefly review recent discoveries from field, experimental and theoretical works on the construction and functioning of ant transportation networks. I will then focus more specifically on the latest work we have been doing in my group to understand how some species of ants (army ants in particular) build dynamic support structures out of their own bodies to facilitate the traffic along their very active trails. I will talk about what we have discovered so far on the construction mechanisms of these living architectures, present preliminary results of field experiments that we have recently performed, and discuss our plans for future research on this subject.

[theswarmlab-blog]

The SwarmLab at the 2nd Workshop on Biological Distributed Algorithms

Postdoctoral researcher Chris Reid will represent the SwarmLab at the 2nd Workshop on Biological Distributed Algorithms October 11-12, 2014 in Austin, Texas, USA. He will present our latest work on slime mold decision-making on October 12 at 2pm. The abstract of his talk is copied below.

Title: Cellular Decision-making: How an Amoeboid Organism Solves the Two-Armed Bandit Problem

Chris Reid, Hannelore MacDonald, Tanya Latty, Richard Mann and Simon Garnier

Abstract: A fundamental conundrum in decision making is the exploration-exploitation tradeoff; do I exploit well-known but potentially sub-optimal options, or do I risk further exploration for potentially more rewarding ones? The problem faces casino gamblers and foraging organisms alike, but there remains no known generally optimal solution. Several studies in humans and other animals have examined the tradeoff using the 2-Armed Bandit problem, where a player aims to maximize their gain when faced with two slot machines, each with a distinct but unknown reward rate. Studies thus far have only been undertaken in organisms with brains, yet the exploration-exploitation tradeoff also applies to unicellular foragers, which must tackle the problem without the aid of neurons. Also, solutions offered by collective systems have never been investigated. We tested the slime mold Physarum polycephalum, which behaves as a self-organized collective system, with the 2-Armed Bandit problem by assessing the effect of sampling on foraging patch choice in a T-maze. We generate insight into the basic processes of decision making in a unicellular organism, including the use of relative vs absolute reward criteria (in both the frequency of reward, and the combination of frequency and magnitude), and the effect of static vs dynamic exploration environments – factors demonstrated to affect human decision making processes. We then propose several biologically plausible decision criteria the slime mould may be using, and use Bayesian inference to determine which of these models best explains the empirical data. These results can directly inform new models of slime mould decision making and behaviour, improving on the existing, largely biomechanical framework, by incorporating new insight into slime mould ‘psychology’. Our study challenges the common view that neurological hardware is required to solve complex problems, and provides insight into basic processes of decision making, beyond phylogenetic boundaries and orders of biological organization.

[theswarmlab-blog]

Postdoctoral opportunity in modeling biological supply chains

We are currently seeking a Senior Research Analyst to lead the development of mathematical models of biological supply chains on the ARC-funded “Risky Business” project. This is a 2-year position based out of Sydney, Australia.

The successful candidate will report primarily to the two Chairs in the Institute of Transport & Logistics Studies at the University of Sydney (Dr. Michael Bell and Dr. Michiel Bliemer). It will also work closely with experimental biologists in the Behaviour and Genetics of Social Insects Laboratory of the School of Biological Sciences at the University of Sydney (Dr. Tanya Latty) and in the Department of Biological Sciences at the New Jersey Institute of Technology (Dr. Simon Garnier). The successful candidate will develop and analyze mathematical supply chain models, participate in the design of experiments, contribute to the interpretation of experimental results, and prepare papers for presentation at conferences and for publication in international peer-reviewed journals. The role will be to support research being undertaken in the aforementioned units on supply chain vulnerability and resilience with a view to, on the one hand, achieving insights into the behavior of biological systems (slime molds and social insects in particular), and on the other, formulating methods for analyzing, “hardening” and repairing human supply chains.

More information about the application process can be found here. The full description of the position can be found here.

For more information, please contact:

Dr. Michael Bell: [email protected]

Dr. Michiel Bliemer: [email protected]

Dr. Tanya Latty: [email protected]

Dr. Simon Garnier: [email protected]

[theswarmlab-blog]

The SwarmLab at ANTS 2014

The 2014 ANTS conference will take place September 10-12 in Brussels, Belgium. This conference dedicated to the field of Swarm Intelligence will be a very exciting event for everyone interested in collective behaviors in both natural and artificial systems.

I will be there this year to give the plenary talk on the first day of the conference (Wednesday, September 10 at 9.30am). I will present our latest work on Argentine ant trail networks and on army ant living architectures. Hereafter is the abstract of my talk.

Abstract

Like the Roman Empire at its peak, a successful ant colony relies on an effective network of roads that facilitate the movement of its powerful army and industrious population across a vast territory. Fifty years ago, E. O. Wilson discovered the chemical nature of these transportation networks comprised of pheromone trails laid by the colony’s workers. His work paved the way for five decades of study on the incredibly efficient organization of ant colonies, based on simple behaviors, multiple interactions and powerful scents.

In this talk, I will review recent discoveries from field, experimental and theoretical works on the construction and functioning of ant transportation networks. I will highlight the latest findings on traffic organization along ant chemical trails, the topology and geometry of these networks, and the development of support structures on uneven terrains. Finally, I will discuss prospective applications of the ants’ “chemical logic” in operation research and collective robotics, with a particular focus on the flexibility of ant trail systems and their ability to integrate and process multiple sources of information.

[theswarmlab-blog]

The SwarmLab at ANTS 2014

[theswarmlab-blog]

PhD opportunity in Complex Mechanisms of Fish Schooling Behavior

Applications are now being accepted for a PhD opportunity in the Federated Department of Biological Sciences at Rutgers-Newark and NJIT studying the Complex Mechanisms of Fish Schooling Behavior. Research in this area will integrate functional ecology, behavior, fluid dynamics, and computational methods to determine the effect of social cues and presence of locomotor advantages governing collective behavior of fishes.

Applications should be submitted online at http://gradstudy.rutgers.edu. Use program name “Biology” and program code “26120″. It is strongly recommended that you contact Dr. Brooke Flammang or Dr. Simon Garnier in advance of applying to discuss the position in more detail. Applications are being considered now and due by December 1, 2014.

For more information, please contact:

Brooke Flammang, PhD – Fluid Locomotion Lab: [email protected]

Simon Garnier, PhD – Swarm Lab: [email protected]

[theswarmlab-blog]

The SwarmLab at Sci Foo

I will be present at the 9th Annual Science Foo Camp (“Sci Foo”) in Palo Alto, California, from August 8 to 10. Sci Foo is an event co-organized by Google, O’Reilly Media, Digital Science and Nature that “brings together between 200 and 300 leading scientists, technologists, writers and other thought-leaders at the Googleplex for a weekend of unbridled discussion, demonstration and debate”.

Sci Foo is an informal conference without predefined agenda. Participants collaboratively create one from start to finish the first evening. Since it will be my first participation, I am not sure what to expect but these videos from previous years suggest that it will be a lot of fun!

[theswarmlab-blog]

SwarmLab's Shiny webapps for teaching collective behavior and swarm intelligence

R is a programming language and software environment mostly known for its exhaustive suite of statistics tools. But it is much more than a Swiss army knife for data analysis. It is also a fantastic environment to develop quickly programs and apps to support your lecture material and create interactive demos for instance.

If you have not followed closely the news in the R community, there have been multiple important developments pushed by a new generation of very active R programmers, in particular by the terrific team behind the RStudio IDE. One of these developments is the creation of ‘Shiny’, a web application framework for R. ‘Shiny’ allows anybody with intermediate R skills – beginners might struggle a bit – to create in a few lines of code a complete web-application entirely written with the R language.

And this changes everything for people like me who like to provide their students with interactive tools to explore questions and manipulate concepts covered during lectures. Using the ‘Shiny’ framework and with minimum effort, I can now create interactive web apps that showcase classic models of collective behaviors and swarm intelligence. The students can then manipulate the parameters of these models using simple sliders and buttons, and observe directly the consequences on the models’ predictions. This is a fantastic way to create interactive assignments to complement the content of a lecture or to prepare the students for the next one.

Once you have created a web application (I strongly recommend using RStudio for this as it makes the process very smooth), you can host them on your own server or deploy them on ShinyApps.io (in one click from within RStudio), a dedicated service provided by the good folks at RStudio. It is free for small apps, which is largely enough for my class for instance. Larger apps that require heavy data processing for instance might require a paid account.

After testing this approach successfully last semester during my class on “Collective Intelligence in Biological Systems” (NJIT BIOL 337), I decided to use it more systematically this year. I started updating the test apps from last semester and building new ones. I will release them little by little throughout the semester as I find the time to work on them, but you can already test out the first 3 of them and download their source code on my GitHub:

Collective food source selection in ants:

Source code: https://github.com/sjmgarnier/Shiny/tree/master/ant_collective_decision

App: http://sjmgarnier.shinyapps.io/ant_collective_decision/

Collective nest selection in honeybees:

Source code: https://github.com/sjmgarnier/Shiny/tree/master/bee_house_hunting

App: http://sjmgarnier.shinyapps.io/bee_house_hunting/

Opinion dynamics (spreading and competition):

Source code: https://github.com/sjmgarnier/Shiny/tree/master/opinion_dynamic

App: http://sjmgarnier.shinyapps.io/opinion_dynamic/

The 3 apps (as well as those that will follow) are distributed under the GPL-3 license. Feel free to download, use, modify and redistribute the source code. Comments and suggestions of new models to include in this list are very welcome!

[theswarmlab-blog]

The SwarmLab at ABS2014

Just after ISBE2014, the SwarmLab will head to Princeton, NJ for the 51st Annual Conference of the Animal Behavior Society. The conference will be held at Princeton University from Saturday, August 9th to Thursday, August 14th, 2014.

On the evening of the opening day (Sunday, August 10th), PI Simon Garnier will co-organize a workshop “Quantitative analysis of collective behavior – from the lab to the wild” with Prof. Iain Couzin, Dr. Andrew King, Dr. Nicolas Perony and Dr. Damien Farine. This workshop will conclude a day started with the keynote address of Prof. Iain Couzin on “Collective sensing and decision making in animal groups”.

The same day, Princeton PhD candidate Matthew Lutz will present the work he has performed on the living architectures of New World army ants, in collaboration with PI Simon Garnier and post-doc Chris Reid from the SwarmLab. His presentation will be at 10.45 am in the Taplin Auditorium (Fine Hall, Princeton University). In the afternoon, post-doc Chris Reid will present his work on the problem-solving abilities of the slime mold Physarum polycephallum at 5.45pm in Jadwin Hall (room 10).

Hereafter is the abstract for Chris’ presentation:

How slime mould cracks the Two-Armed-Bandit problem; insights into unicellular decision making

Should I exploit well-known options, or do I risk further exploration for potentially higher reward? This is the exploration-exploitation tradeoff, and while it faces casino gamblers and foraging organisms alike, the optimal solution is unknown. The tradeoff has been studied using the 2-Armed Bandit problem, where a player aims to maximize their gain when faced with two slot machines, each with a distinct but unknown reward rate. Studies thus far have only used organisms with brains. We tested the slime mold Physarum polycephalum with the 2-Armed Bandit problem by assessing the effect of sampling on foraging patch choice in a T-maze.We generate insight into the basic processes of decision making in a cell, including the use of relative vs absolute reward criteria (in both the frequency of reward, and the combination of frequency and magnitude), and the effect of static vs dynamic exploration environments. We propose several biologically plausible decision criteria the organism may be using and, through Bayesian inference, determine which of these models best explains the empirical data. We challenge the common view that neurological hardware is required to solve complex problems.

[theswarmlab-blog]

The SwarmLab at ISBE2014

The 2014 Conference of the International Society for Behavioral Ecology (ISBE2014) is about to kick off in New York City. It will take place from July 31 to August 5 at Hunter College. The SwarmLab will be represented there by post-doc Chris Reid who will present his latest work on the living architectures built by New World army ants. His presentation will be on Saturday, August 2 at 4.30pm in the ‘Social Behavior’ session (room HN 1036 at Hunter College). Hereafter his the abstract for his talk:

Investigating the self-organized living architectures of Eciton army ants.

Individual workers of the army ant Eciton hamatum use their bodies to form adaptable and dynamic collective structures to assist the passage of nestmates. Our study is the first to induce the formation of army ant bridges in the field. By redirecting an ant trail over our modular platform setup, we impose a gap for the ants to bridge with their interlinked bodies, and then increase the width of this gap at a constant rate. This allows us to determine the rules for bridge construction, as well as how the structure is modified in response to changing length and associated physical forces. We also examined deconstruction rules, as we closed the gap on an established bridge. We investigated the effect of angular deviation from the trail on the dynamics of bridge construction, by making the ants traverse a kinked bridge that could be set to a variety of acute angles. Above and below a critical range of deviation, stable bridges formed that provided only a slight improvement in path length for the ants. However within the critical range, dynamic bridges formed that slowly moved perpendicularly toward the trail, lengthening the bridge but decreasing the required deviation for travelling ants. We propose a simple model to account for these self-organised shortcuts. Finally, we describe and investigate a novel architecture, “support platforms”, which form on steep substrates with low tractability and presumably enhance nestmates locomotion speed.

[theswarmlab-blog]

Public lecture in Montreal, Canada: "Des Fourmis aux Hommes - La Science des Foules" [April 24, 2014]

On Thursday April 24, I will give a public lecture in Montreal, Canada, organized by Coeur des Sciences UQAM in partnership with the French consulate in Montreal. I will give an overview of the current knowledge on the self-organization of large animal and human groups. The lecture will be in French and will start at 7pm.

More information (in French) and registration link can be found at the following address: https://coeurdessciences.uqam.ca/component/eventlist/details/409-la-science-des-foules.html.

[theswarmlab-blog]

R vs Python - Round 3

Document title: R vs Python – Round 3

Date: February 5, 2014

Text by: Simon Garnier (www.theswarmlab.com / @sjmgarnier)

R code by: Simon Garnier (www.theswarmlab.com / @sjmgarnier)

Python code by: Randy Olson (www.randalolson.com / @randal_olson)

Document generated with RStudio (www.rstudio.com), knitr (www.yihui.name/knitr/) and pandoc (www.johnmacfarlane.net/pandoc/). Python figures generated with iPython Notebook (www.ipython.org/notebook.html).

Foreword

My friend Randy Olson and I got into the habit to argue about the relative qualities of our favorite languages for data analysis and visualization. I am an enthusiastic R user (www.r-project.org) while Randy is a fan of Python (www.python.org). One thing we agree on however is that our discussions are meaningless unless we actually put R and Python to a series of tests to showcase their relative strengths and weaknesses. Essentially we will set a common goal (e.g., perform a particular type of data analysis or draw a particular type of graph) and create the R and Python codes to achieve this goal. And since Randy and I are all about sharing, open source and open access, we decided to make public the results of our friendly challenges so that you can help us decide between R and Python and, hopefully, also learn something along the way.

Today’s challenge: walk the line

1 – Introduction

Linear regression is one of the most popular statistical tools, in particular when it comes to detect and measure the strength of trends in time-series (but not only). Today we will show you how to perform a linear regression in R and Python, how to run basic diagnostic tests to make sure the linear regression went well, and how to nicely plot the final result.

The data we will use today are provided by Quandl. Quandl is a company that collects and organizes time-series datasets from hundred’s of public sources, and can even host your datasets for free. It is like the YouTube of time-series data. Besides having a huge collection of datasets, the nice thing about Quandl is that it provides packages for R and Python (and other languages as well) that make it very easy – one line of code! – to retrieve time-series datasets directly from your favorite data analysis tool. Yeah Quandl!

The dataset that we will explore contains information about the average contributions of individuals and corporations to the income taxes, expressed as a fraction of the Gross Domestic Produce (GDP). We will use linear regression to estimate the variation of these contributions from 1945 to 2013.

In the following, we will detail the different steps of the process and provide for each step the corresponding code (red boxes for R, green boxes for Python). You will also find the entire codes at the end of this document.

If you think there’s a better way to code this in either language, leave a pull request on our GitHub repository or leave a note with suggestions in the comments below.

2 – Step by step process

First things first, let’s set up our working environment by loading some necessary libraries.

# Load libraries require("Quandl") # Functions to access Quandl's database require("lattice") # Base graph functions require("latticeExtra") # Layer graph functions

import Quandl

Now, let’s load the dataset from Quandl. You can find it on Quandl’s website here. Among other things, this dataset contains the contributions of individuals and corporations to the income taxes, expressed as a fraction of the GDP. Its original source is the Tax Policy Center. The dataset code at Quandl is “TPC/HIST_RECEIPT” and we will retrieve data from December 31, 1945, to December 31, 2013.

# Load data from Quandl my.data <-Quandl("TPC/HIST_RECEIPT", start_date = "1945-12-31", end_date = "2013-12-31") # Display first lines of the data frame head(my.data)

quandl_data = Quandl.get("TPC/HIST_RECEIPT", trim_start="1945-12-31", trim_end="2013-12-31") quandl_data.head()

The column names from Quandl’s data frame contain spaces. We need to correct that before we can proceed with the rest of the analysis.

# The columns' names contain spaces, make them syntactically valid names(my.data) <-make.names(names(my.data))

%matplotlib inline import matplotlib.pyplot as plt # Rename the columns we're using with underscores -- required for the linear regression quandl_data["Individual_Income_Taxes"] = quandl_data["Individual Income Taxes"] quandl_data["Corporation_Income_Taxes"] = quandl_data["Corporation Income Taxes"] quandl_data["Fiscal_Year"] = range(len(quandl_data.index))

The dataset contains three columns of interest for us: individuals’ and corporations’ income taxes as a fraction of the GDP, and fiscal year. We will first plot the individuals’ and corporations’ income taxes as a function of the fiscal year to see what the data look like.

# Create graph object graph <-xyplot(Individual.Income.Taxes ~Fiscal.Year, data = my.data, type = c("g", "b"), col = "#00526D") graph <-graph +xyplot(Corporation.Income.Taxes ~Fiscal.Year, data = my.data, type = c("b"), col = "#AD3333") # Create pretty theme, because who doesn't like a nice looking graph my.theme <-within(trellis.par.get(), plot.line <-within(plot.line, lwd <-3 ) axis.text <-within(axis.text, cex <-1.25 ) par.xlab.text <-within(par.xlab.text, cex <-1.5 ) par.ylab.text <-within(par.ylab.text, cex <-1.5 ) add.text <-within(add.text, cex <-1.25 ) superpose.line <-within(superpose.line, col <-c("#00526D", "#AD3333") lwd <-3 ) ) # Compute upper and lower limits of the y axes max.val <-max(my.data$Corporation.Income.Taxes, my.data$Individual.Income.Taxes) ylim <-c(0 -0.05 *max.val, max.val *1.15) # Prepare a legend key <-within(list(), text <-c("Individuals", "Corporations") corner <-c(0.05, .95) lines <-TRUE points =FALSE ) # Update graph object graph <-update(graph, xlab = "Fiscal year", ylab = "Income taxes (% of GDP)", ylim = ylim, par.settings = my.theme, auto.key = key) # Print graph object print(graph)

plt.figure(figsize=(12, 7)) plt.plot(quandl_data["Fiscal_Year"], quandl_data["Individual_Income_Taxes"], marker="o", color="#00526D", lw=2) plt.plot(quandl_data["Fiscal_Year"], quandl_data["Corporation_Income_Taxes"], marker="o", color="#AD3333", lw=2) plt.xlabel("Fiscal Year", fontsize=16) plt.ylabel("Income taxes (% of GDP)", fontsize=16) plt.xlim(-1, 69) plt.xticks(quandl_data["Fiscal_Year"][5::10], quandl_data.index[5::10].year) plt.grid()

The first thing one can notice in this graph is the clear decreasing trend of the corporations’ income taxes: since 1945, they have dropped from about 7% of the GDP to less than 2%. On the contrary, the individuals’ contribution to the income taxes seems to have remained stable around 8% of the GDP, with maybe a slight general increase over time. Also there seems to be correlated fluctuations, with several successive years of increase or decrease probably due to the prolonged action of successive governments. The intensity of these fluctuations seems uneven along the considered time period. Finally the corporations’ income taxes seem to decrease linearly (more or less) until the mid-80’s but do not seem to change much after that. These three elements (autocorrelation, heteroscedasticity and change in slope) will certainly raise flags in diagnostic tests but we will deal with them in another challenge. Today we want to focus on making and plotting simple linear regressions instead.

First, let’s fit a linear model to the individual and corporation income taxes data.

# Fit individuals' income taxes data model.ind <-lm(Individual.Income.Taxes ~Fiscal.Year, data = my.data) # Fit corporations' income taxes data model.corp <-lm(Corporation.Income.Taxes ~Fiscal.Year, data = my.data)

# Fit a linear regression to the individual income taxes time series individual_regression = sm.OLS.from_formula("Individual_Income_Taxes ~ Fiscal_Year", quandl_data).fit() # Fit a linear regression to the corporation income taxes time series corporation_regression = sm.OLS.from_formula("Corporation_Income_Taxes ~ Fiscal_Year", quandl_data).fit()

Now let’s have a look to the summary statistics of each fitted model. We will display the R output only to keep things simple. However if you decide to look at the Python output, the result of the regression might look a bit different. Randy had to transform the dates into years because the fitting function did not seem capable of handling dates like the fitting function in R does (it converts the dates in days in this case).

First, the model for individuals’ income taxes:

# Summary statistics for individual income taxes model summary(model.ind)

## ## Call: ## lm(formula = Individual.Income.Taxes ~ Fiscal.Year, data = my.data) ## ## Residuals: ## Min 1Q Median 3Q Max ## -1.9522 -0.3065 0.0217 0.3101 2.0674 ## ## Coefficients: ## Estimate Std. Error t value Pr(>|t|) ## (Intercept) 7.84e+00 1.09e-01 72.18 <2e-16 *** ## Fiscal.Year 2.58e-05 1.33e-05 1.93 0.058 . ## --- ## Signif. codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1 ## ## Residual standard error: 0.806 on 67 degrees of freedom ## Multiple R-squared: 0.0528, Adjusted R-squared: 0.0387 ## F-statistic: 3.73 on 1 and 67 DF, p-value: 0.0575

print individual_regression.summary()

Then, the model for corporations’ income taxes:

# Summary statistics for corporation income taxes model summary(model.corp)

## ## Call: ## lm(formula = Corporation.Income.Taxes ~ Fiscal.Year, data = my.data) ## ## Residuals: ## Min 1Q Median 3Q Max ## -1.535 -0.478 -0.150 0.394 2.394 ## ## Coefficients: ## Estimate Std. Error t value Pr(>|t|) ## (Intercept) 3.43e+00 9.93e-02 34.6 <2e-16 *** ## Fiscal.Year -1.56e-04 1.22e-05 -12.8 <2e-16 *** ## --- ## Signif. codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1 ## ## Residual standard error: 0.737 on 67 degrees of freedom ## Multiple R-squared: 0.711, Adjusted R-squared: 0.706 ## F-statistic: 164 on 1 and 67 DF, p-value: <2e-16

print corporation_regression.summary()

What do we see? For the individual income taxes, we see a small, positive increase of +2.579e-05% of GDP per day (R treats dates as days). Over the whole 1945 to 2013 period, this corresponds to a total increase of +0.64% of GDP, i.e. a +8.4% global increase in the contribution of individuals to income taxes. However this increase is only marginally significant (p = 0.0575), therefore we can consider that the contribution of individuals to the income taxes has remained fairly stable during the 7 decades since WWII.

We have a very different story for the corporation income taxes. The linear model shows a strongly significant (p < 2e-16) decrease of -1.564e-04% of GDP per day. Over the 1945-2013 period, this corresponds to a total decrease of -3.88% of GDP, i.e. a -80.1% global decrease in the contribution of corporations to income taxes.

Of course, we need to be very cautious in the interpretation of these results because the linear models we have fitted to the data do not take into account the potential sources of error that we have identified earlier. We will see next time how we can evaluate them, and correct them if necessary.

For now, we will assume that everything is good and we will focus on plotting the two linear models and their confidence interval over the data.

To do this, we will first compute the value and confidence interval as predicted by each model for different dates. We will choose dates that cover the whole studied period (1945-2013), plus a little extra on both ends (10%) to make the graph more aesthetically pleasing (i.e., the models’ predictions will cover a larger time period than the x-axis limits of the graph).

# Create new dates for model predictions difference <-as.Date("2013-12-31") -as.Date("1945-12-31") new.data <-data.frame(Fiscal.Year = seq.Date(from = as.Date("1945-12-31") -0.1 *difference, to = as.Date("2013-12-31") +0.1 *difference, by = "years")) # Compute model predictions (values + confidence intervals) for new dates # Model of individual income taxes predict.ind <-within(new.data, Predictions <-as.data.frame(predict(model.ind, newdata = new.data, interval = "confidence")) ) # Model of corporation income taxes predict.corp <-within(new.data, Predictions <-as.data.frame(predict(model.corp, newdata = new.data, interval = "confidence")) )

# Predict the 95% confidence bands for the regression individual_prstd, individual_iv_l, individual_iv_u = wls_prediction_std(individual_regression) individual_st, individual_data, individual_ss2 = summary_table(individual_regression, alpha=0.05) individual_fitted_values = individual_data[:, 2] individual_predict_mean_ci_low, individual_predict_mean_ci_upper = individual_data[:, 4:6].T # Predict the 95% confidence bands for the regression corporation_prstd, corporation_iv_l, corporation_iv_u = wls_prediction_std(corporation_regression) corporation_st, corporation_data, corporation_ss2 = summary_table(corporation_regression, alpha=0.05) corporation_fitted_values = corporation_data[:, 2] corporation_predict_mean_ci_low, corporation_predict_mean_ci_upper = corporation_data[:, 4:6].T

And finally, we will add to the data plot the predicted values as a line and the confidence intervals of the predicted values as a polygon.

# Add predicted values for model of individual income taxes graph <-graph +xyplot(Predictions$fit ~Fiscal.Year, data = predict.ind, type = c("l"), col = "#00526D") # Add predicted values for model of corporation income taxes graph <-graph +xyplot(Predictions$fit ~Fiscal.Year, data = predict.corp, type = c("l"), col = "#AD3333") # Add confidence polygon for model of individual income taxes graph <-graph +layer_(lpolygon(x = c(predict.ind$Fiscal.Year, rev(predict.ind$Fiscal.Year)), y = c(predict.ind$Predictions$upr, rev(predict.ind$Predictions$lwr)), col = "#00526D25", border = "#00526D75")) # Add confidence polygon for model of corporation income taxes graph <-graph +layer_(lpolygon(x = c(predict.corp$Fiscal.Year, rev(predict.corp$Fiscal.Year)), y = c(predict.corp$Predictions$upr, rev(predict.corp$Predictions$lwr)), col = "#AD333325", border = "#AD333375")) # Reprint graph object print(graph)

plt.plot(quandl_data["Fiscal_Year"], individual_fitted_values, color="#00526D", lw=2) plt.fill_between(quandl_data["Fiscal_Year"], individual_predict_mean_ci_low, individual_predict_mean_ci_upper, color="#00526D", alpha=0.2) plt.plot(quandl_data["Fiscal_Year"], corporation_fitted_values, color="#AD3333", lw=2) plt.fill_between(quandl_data["Fiscal_Year"], corporation_predict_mean_ci_low, corporation_predict_mean_ci_upper, color="#AD3333", alpha=0.2) ;

And that’s it for today! We hope that you enjoyed it and that you’ll come back next time to find out more about running diagnotic tests and correcting potential errors on today’s linear models.

3 – Source code

R and Python source codes are available here.

[theswarmlab-blog]

Simon's talk at Swansea University (Wales)

On February 20, 2014, Simon will give a talk at the Department of Biosciences of Swansea University. His talk is part of the Biosciences seminar series. It will be held at 1pm in the Zoology Museum. Hereafter are the title and abstract of the talk.

Title: All roads lead to the mound

Abstract: Like the Roman Empire at its peak, a successful ant colony relies on an effective network of roads that facilitate the movement of its powerful army and industrious population across a vast territory. Fifty years ago, E. O. Wilson discovered the chemical nature of these transportation networks comprised of pheromone trails laid by the colony’s workers. His work paved the way for five decades of study on the incredibly efficient organization of ant colonies, based on simple behaviors, multiple interactions and powerful scents. In this talk, I will review recent discoveries from field, experimental and theoretical works on the construction and functioning of ant transportation networks. I will highlight the latest findings on traffic organization along ant chemical trails, the topology and geometry of these networks, and the development of support structures on uneven terrains. Finally, I will discuss prospective applications of the ants’ “chemical logic” in operation research and collective robotics, with a particular focus on the flexibility of ant trail systems and their ability to integrate and process multiple sources of information.

[theswarmlab-blog]

R vs Python – Round 2 (2/2)

Document title: R vs Python – Round 2 (2/2)

Date: February 2, 2014

Text by: Simon Garnier (www.theswarmlab.com / @sjmgarnier)

R code by: Simon Garnier (www.theswarmlab.com / @sjmgarnier)

Python code by: Randy Olson (www.randalolson.com / @randal_olson)

Document generated with RStudio (www.rstudio.com), knitr (www.yihui.name/knitr/) and pandoc (www.johnmacfarlane.net/pandoc/). Python figures generated with iPython Notebook (www.ipython.org/notebook.html).

Foreword

My friend Randy Olson and I got into the habit to argue about the relative qualities of our favorite languages for data analysis and visualization. I am an enthusiastic R user (www.r-project.org) while Randy is a fan of Python (www.python.org). One thing we agree on however is that our discussions are meaningless unless we actually put R and Python to a series of tests to showcase their relative strengths and weaknesses. Essentially we will set a common goal (e.g., perform a particular type of data analysis or draw a particular type of graph) and create the R and Python codes to achieve this goal. And since Randy and I are all about sharing, open source and open access, we decided to make public the results of our friendly challenges so that you can help us decide between R and Python and, hopefully, also learn something along the way.

Today’s challenge: a data thief manual for honest scientists (Part 2 of 2)

1 – Introduction

Last time we showed you how to scrape data from www.MovieBodyCounts.com. Today, we will finish what we started by retrieving additional information from www.imdb.com. In particular, we will attempt to recover the following pieces of information for each of the movies we collected last time: MPAA rating, genre(s), director(s), duration in minutes, IMDb rating and full cast. We will detail the different steps of the process and provide for each step the corresponding code (red boxes for R, green boxes for Python). You will also find the entire codes at the end of this document.

If you think there’s a better way to code this in either language, leave a pull request on our GitHub repository or leave a note with suggestions in the comments below.

2 – Step by step process

First things first, let’s set up our working environment by loading some necessary libraries.

# Load libraries # No additional libraries needed here. Yeah!

# String parsing libraries from imdb import IMDb import pandas as pd import re

Randy is lucky today. Someone else has already written a package (‘IMDbPY’) to scrape data from IMDb. Unfortunately for me, R users are too busy working with serious data sets to take the time to write such a package for my favorite data processing language. Hadley Wickham has included a ‘movie’ data set in the ggplot2 package that contains some of the information stored on IMDb, but some of the pieces we need for today’s challenge are missing.

Since I am not easily discouraged, I decided to write my own IMDb scraping function (see below). It is not as sophisticated as the Python package Randy is using today, but it does the job until someone else decides to write a more complete R/IMDb package. As you will see, I am using the same scraping technique (XPath) as the one I used in the first part of the challenge.

# Create IMDB scraper IMDb <-function(ID) # Retrieve movie info from IMDb.com. # # Args: # ID: IDs of the movies. # # Returns: # A data frame containing one line per movie, and nine columns: movie ID, # film title, year of release, duration in minutes, MPAA rating, genre(s), # director(s), IMDb rating, and full cast. # Load required libraries require(XML) require(pbapply) # Apply functions with progress bars!!! # Wrap core of the function in do.call and pblapply in order to # pseudo-vectorize it (pblapply) and return a data frame (do.call) info <-do.call(rbind, pblapply(ID, FUN = function(ID) ') data.frame(ID = ID, Film = Film, Year = Year, Length_Minutes = Length_Minutes, MPAA_Rating = MPAA_Rating, Genre = Genre, Director = Director, IMDB_rating = IMDB_rating, Full_Cast = Full_Cast)) ))

imdb_access = IMDb()

Randy and I now have a working IMDb scraper. We can start collecting and organizing the data that we need.

First, let’s load the data we collected last time.

# Load data from last challenge data <-read.csv("movies-R.csv")

movie_data = pd.read_csv("movies.csv")

Then, we will extract the movie IMDb ID from the IMDb URL we collected last week. It’s easy, it’s the only number in the URL.

# For each movie, extract IMDb info and append it to the data data <-within(data, # Extract ID number IMDB_ID <-gsub("[^0-9]", "", IMDB_URL)

# Grab only the movie number out of the IMDB URL movie_data["Movie_Number"] = movie_data["IMDB_URL"].apply(lambda x: re.sub("[^0-9]", "", x))

Now that this is done, we will simply let the IMDb scraper collect the data we want and we will append it to the data from the first part of the challenge.

# Download IMDb info into a temporary variable IMDB_Info <-IMDb(IMDB_ID) # Save MPAA rating MPAA_Rating <-IMDB_Info$MPAA_Rating # Save genre(s) Genre <-IMDB_Info$Genre # Save director(s) Director <-IMDB_Info$Director # Save duration in minutes Length_Minutes <-IMDB_Info$Length_Minutes # Save IMDb rating IMDB_rating <-IMDB_Info$IMDB_rating # Save full cast Full_Cast <-IMDB_Info$Full_Cast # Delete IMDb info IMDB_Info <-NULL )

with open("film-death-counts-Python.csv", "wb") as out_file: out_file.write("Film,Year,Body_Count,MPAA_Rating,Genre,Director,Actors,Length_Minutes,IMDB_Rating\n") for movie_entry in movie_data.iterrows(): # Use a try-catch on the loop to prevent temporary connection-related issues from stopping the scrape try: movie = imdb_access.get_movie(movie_entry[1]["Movie_Number"]) movie_fields = [] # Remove non-ASCII character encodings and commas from movie titles movie_fields.append(movie["title"].encode("ascii", "replace").replace(",", "")) movie_fields.append(str(movie["year"])) movie_fields.append(str(movie_entry[1]["Body_Count"])) # Some movies don't have MPAA Ratings on IMDB try: movie_fields.append(str(movie["mpaa"].split(" ")[1])) except: movie_fields.append("") # For movies with multiple genres/directors/actors, join them with bars | movie_fields.append(str("|".join(movie["genres"]))) movie_fields.append(str("|".join([str(x) for x in movie["director"]]))) movie_fields.append(str("|".join([str(x) for x in movie["cast"]]))) movie_fields.append(str(int(movie["runtime"][0].split(":")[-1]))) movie_fields.append(str(float(movie["rating"])))

And finally, all what is left to do is to save the complete data set into a .csv file and close the script.

write.csv(data, file = "movies-R-full.csv")

# All entries are comma-delimited out_file.write(",".join(movie_fields) + "\n") except Exception as e: print "Error with", str(movie)

That’s it! You should now have a .csv file somewhere on your computer containing all the information we just scraped in both parts of this challenge.

Sorry it took us so long to complete this part, but beginnings of semesters are always very busy times at the university.

Stay tuned for our next challenge! It will be about making a linear regression, running basic diagnostic tests and plotting the resulting straight line with its confidence interval.

3 – Source code

R and Python source codes are available here.

[theswarmlab-blog]

R vs Python – Round 2

Document title: R vs Python – Round 2

Date: January 12, 2014

Text by: Simon Garnier (www.theswarmlab.com / @sjmgarnier)

R code by: Simon Garnier (www.theswarmlab.com / @sjmgarnier)

Python code by: Randy Olson (www.randalolson.com / @randal_olson)

Document generated with RStudio (www.rstudio.com), knitr (www.yihui.name/knitr/) and pandoc (www.johnmacfarlane.net/pandoc/). Python figures generated with iPython Notebook (www.ipython.org/notebook.html).

Foreword

My friend Randy Olson and I got into the habit to argue about the relative qualities of our favorite languages for data analysis and visualization. I am an enthusiastic R user (www.r-project.org) while Randy is a fan of Python (www.python.org). One thing we agree on however is that our discussions are meaningless unless we actually put R and Python to a series of tests to showcase their relative strengths and weaknesses. Essentially we will set a common goal (e.g., perform a particular type of data analysis or draw a particular type of graph) and create the R and Python codes to achieve this goal. And since Randy and I are all about sharing, open source and open access, we decided to make public the results of our friendly challenges so that you can help us decide between R and Python and, hopefully, also learn something along the way.

Today’s challenge: a data thief manual for honest scientists (Part 1 of 2)

1 – Introduction

Last week we started our challenge series with a rather simple task: plot a pretty barchart from some data collected by Randy for his recent post on the “Top 25 most violence packed films” in the history of the movie industry. Today we will try to up our game a bit with a more complex task. We will show you how you can collect the data that Randy used for his post directly from the website they originate from (www.MovieBodyCounts.com). This is called data scraping, or the art of taking advantage of the gigantic database that is the Internet.

The basic principle behind the scraping of website data is simple: a website is a like database, and each page of the website is like a table of this database. All we want is find in the database the tables that contain information that we would like to acquire, and then extract this information from within these relevant tables. This task can be relatively easy if all the pages of a website have a similar structure (i.e., if the database is clean and well maintained). In this ideal situation, all we have to do is identify one or more stable markers that delimit the desired information and use them to tell R or Python what to save in memory. Unfortunately not all websites have a similar structure across all of their pages and it can quickly become a nightmare to identify such markers. Worse, sometimes you will have to resign yourself to scrape or correct part or all of the data manually.

For this challenge, we will attempt to recover the following pieces of information for each movie listed on www.MovieBodyCounts.com: title, release year, count of on-screen deaths and link to the movie page on www.imdb.com (this will help us for part 2 of this challenge next week). We will detail the different steps of the process and provide for each step the corresponding code (red boxes for R, green boxes for Python). You will also find the entire codes at the end of this document.

2 – Step by step process

First things first, let’s set up our working environment by loading some necessary libraries.

# Load libraries library(RCurl) # Everything necessary to grab webpages on the Web library(XML) # Everything necessary to parse XML and HTML code library(pbapply) # Progress bars!!! Just because why not # Create curl handle which can be used for multiple HHTP requests. # followlocation = TRUE in case one of the URLs we want to grab is a redirection # link. curl <- getCurlHandle(useragent = "R", followlocation = TRUE)

# String parsing libraries import string import re # urllib2 reads web pages if you provide it an URL import urllib2 # html2text converts HTML to Markdown, which is much easier to parse from html2text import html2text

Now a word about the organization of www.MovieBodyCounts.com. To be perfectly honest, it is a bit messy Movies are organized in a series of alphabetically ordered lists (by the first letter of each movie’s title), each letter having its own page (http://www.moviebodycounts.com/movies-[A-Z].htm). There is also a list for movies which title starts with a number (http://www.moviebodycounts.com/movies-numbers.htm). Finally, all category letters are capitalized in the lists’ URLs, except for letters v and x. Annoying, right? This is just one of the many little problems one can encounter when dealing with messy databases

With all this information in mind, our first task is to create a list of all these lists.

# Prepare URLs of the movie lists alphabetically ordered by first letter of # movie title (capital A to Z, except for v and y) + "numbers" list (for movies # which title starts with a number) urls.by.letter <- paste0("http://www.moviebodycounts.com/movies-", c("numbers", LETTERS[1:21], "v", "W" , "x", "Y", "Z"), ".htm")

# Generate a list of all letters for the Movie pages (+ a "numbers" page) # MovieBodyCount's actor pages are all with capital letters EXCEPT v and x letters = ["numbers"] + list(string.letters[26:52].upper().replace("V", "v").replace("X", "x"))

Our next task is to go through the HTML code of all these lists and gather the URLs of all the movie webpages. This is where the data scraping really starts.

As you will quickly notices by reading the following code, Randy and I have decided to use a different approach to identify and collect the desired URLs (and of all the data in the rest of this challenge). I have decided to rely on the XML Path Language (XPath), a language that makes it easy to navigate through elements and attributes in an XML/HTML document. Randy has decided to use an approach based on more “classical” string parsing and manipulation functions. Note that these are just personal preferences. XPath interpreters are also available in Python, and R is fully equipped for manipulating character strings.

For each movie list, we will…

# For each movie list... For loops are frowned upon in R, let's use the classier # apply functions instead. Here I use the pblapply from the pbapply package. # It's equivalent to the regular lapply function, but it provides a neat # progress bar. Unlist to get a vector. urls.by.movie <- unlist(pblapply(urls.by.letter, FUN = function(URL)

list_of_films = [] # Go through each movie list page and gather all of the movie web page URLs for letter in letters: try:

…download the raw HTML content of the webpage,…

# Load raw HTML raw.html <- getURL(URL, curl = curl)

# Read the raw HTML from the web page page_text = urllib2.urlopen("http://www.moviebodycounts.com/movies-" + letter + ".htm").read()

…transform raw HTML into a more convenient format to work with,…

# Parse HTML content parsed.html <- htmlParse(raw.html)

# Convert the raw HTML into Markdown page_text = html2text(page_text).split("\n")

…find movie page entry, store the URL for later use and close the loop.

# Extract desired links from HTML content using XPath. # The desired links are all the URLs ("a/@href") directly following # ("/following::") the image which source file is called "graphic-movies.jpg" # ("//img[@src='graphic-movies.jpg']"). links <- as.vector(xpathSApply(parsed.html, "//img[@src='graphic-movies.jpg']/following::a/@href")) # Most links are relative URLs. Add root of the website to make them absolute. if (!is.null(links)) ix =grepl("http://www.moviebodycounts.com/", links) # Find relative URLs links[!ix] <- paste0("http://www.moviebodycounts.com/", links[!ix]) # Add root of website to make URLs absolute return(links) ), use.names =FALSE) # close the loop # One URL is actually just a symbolic link to another page. Let's get rid of it. ix <- which(grepl("movies-C.htm", urls.by.movie)) urls.by.movie <- urls.by.movie[-ix]

# Search through the web page for movie page entries for line in page_text: # We know it's a movie page entry when it has ".htm" in it, but not ".jpg", "contact.htm", and "movies.htm" # .jpg means it's a line with an image -- none of the movie entries have an image # contact.htm and movies.htm means it's a link to the Contact or Movies page -- not what we want # movies- means it's a redirect link to another page -- just skip over it if ".htm" in line and ".jpg" not in line and "contact.htm" not in line and "movies.htm" not in line and "movies-" not in line: #print line # The URL is in between parentheses (), so we can simply split the string on those # Some URLs are full URLs, e.g. www.moviebodycounts.com/movie_name.html, so splitting on the / gives us only the page name list_of_films.append(line.split("(")[-1].strip(")").split("/")[-1]) # If the movie list page doesn't exist, keep going except: print "\nerror with " + letter + "\n"

Now that we know where to find each movie, we can start the hard part of this challenge. We will go through each movie webpage and attempt to find its title, release year, count of on-screen deaths and link to its page on www.imdb.com. We will save all this information in a .csv file.

For each movie, we will…

# For each movie... # do.call(rbind, ...) to reorganize the results in a nice data frame data <- do.call(rbind, pblapply(urls.by.movie, FUN = function(URL)

# Now that we have every movie web page URL, go through each movie page and # extract the movie name, kill counts, etc. out_file = open("film-death-counts.csv", "wb") out_file.write("Film,Year,Kill_Count,IMDB_url\n") for film_page in list_of_films: try: # The information we're looking for on the page: film = "" kills = "" year = "" IMDB_url = "" # A flag indicating that we've found the film title on the page found_title = False

…download the raw HTML content of the webpage and transform raw HTML into a more convenient format to work with,…

# Load raw HTML raw.html <- getURL(URL, curl = curl) # Parse HTML content parsed.html <- htmlParse(raw.html)

# Read the page's raw HTML and convert it to Markdown (again) and go # through each line for line in html2text(urllib2.urlopen("http://www.moviebodycounts.com/" + film_page).read()).split("\n"):

…attempt to find movie title,…

# Find movie title # Title appears inside a XML/HTML node called "title" ("//title"). In this # node, it comes after "Movie Body Counts: ". I use gsub to get rid off "Movie # Body Counts: " and keep only the movie title. Film <- xpathSApply(parsed.html, "//title", xmlValue) Film <- gsub("Movie Body Counts: ", "", Film)

# If we haven't found the title yet, these markers tell us we've found the movie # title if not found_title and "!" not in line and "(" not in line and "[" not in line and line.strip() != "": film = line.replace(",", "").strip(":") found_title = True

…attempt to find movie year,…

# Find movie year # The year is usually a text inside ("/descendant::text()") a link node # ("//a") which source contains the string "charts-year" ("[contains(@href, # 'charts-year')]"). Year <-as.numeric(xpathSApply(parsed.html, "//a[contains(@href, 'charts-year')]/descendant::text()", xmlValue))

# The year is usually on a line with "charts-year" if "charts-year" in line: year = line.split("[")[1].split("]")[0]

…attempt to find link to movie on IMDB,…

# Find IMDB link # The IMDB link is inside a link node ("//a") which source contains "imdb" # ("/@href[contains(.,'imdb')]") IMDB_URL <- as.vector(xpathSApply(parsed.html, "//a/@href[contains(.,'imdb')]"))[1] # Note: We select the first element of the vector because for at least one of # the movies, this command returns two links.

# The IMDB url is on a line with "[imdb]" if "[imdb]" in line.lower(): IMDB_url = line.lower().split("[imdb](")[1].split(")")[0]

… and finally attempt to find the on-screen kill count. Here, Randy chose an approach that minimizes his coding effort, but that will potentially force him to make several manual corrections a posteriori. I chose to find a solution that works with minimal to no manual corrections, but that requires an extra coding effort (BTW, the following R code looks very long because I made detailed comments; it’s actually only 6 lines of code and some of them could be easily combined if it wasn’t for clarity). Whichever approach is best depends mostly on the size of the data you want to scrape and the time you have to do it.

# Find kill count. # Kill count is contained in the first non-empty text node # ("/following::text()[normalize-space()]") after the image which source file # is called "graphic-bc.jpg" ("//img[@src='graphic-bc.jpg']") Body_Count <- xpathSApply(parsed.html, "//img[@src='graphic-bc.jpg']/following::text()[normalize-space()]", xmlValue)[1] # Now we need to clean up the text node that we just extracted because there # are lots of inconsistencies in the way the kill counts are displayed across # all movie pages. For instance, counts are sometimes accompanied by text, not # always the same, and sometimes there is no text at all. Sometimes the total # count is split in two numbers (e.g., number of dead humans and number of # dead aliens). And sometimes the total count is displayed and accompanied by # a split count in parenthesis. First, let's remove everything that is # writtent in parenthesis or that is not a number. # Using gsub, remove everything in parenthesis and all non number characters Body_Count <- gsub("\\(.*?\\)", " ", Body_Count) Body_Count <- gsub("[^0-9]+", " ", Body_Count) # In case the total count has been split, we want to separate these numbers # from each other so that we can add them up later. Using strsplit, split the # character string at spaces Body_Count <- unlist(strsplit(Body_Count, " ")) # For now, we have extracted characters. Transform them into numbers. Body_Count <- as.numeric(Body_Count) # Sum up the numbers (in case they have been split into separate categories. Body_Count <- sum(Body_Count, na.rm = TRUE)

# The kill counts are usually on a line with "Film:" if "film:" in line.lower() or "kills:" in line.lower() or "count:" in line.lower(): kills = re.sub("[^0-9]", "", line.split(":")[1].split("(")[0])

Almost done! Now we just need to close the loop and write the data frame into a .csv file

# Return scraped data into a data frame form return(data.frame(IMDB_URL, Film, Year, Body_Count)) )) # close the loop # Save scraped data in a .csv file for future use write.csv(data, "movies-R.csv", row.names = FALSE)

out_file.write(film + "," + year + "," + kills + "," + IMDB_url + "\n") # If a movie page fails to open, print out the error and move on to the next movie except Exception as e: print film_page print e out_file.close()

And voilà! You should now have a .csv file somewhere on your computer containing all the information we just scraped from the website. Not too hard, right? Now, go give a try for real and let’s make the stats of www.MovieBodyCounts.com explode

Also, keep the .csv file, we will use it again next week to complete this challenge by scraping additional information from www.imdb.com.

3 – Source code

R and Python source codes are available here.

4 – Bonus for the braves

Today’s challenge was code and text heavy. No pretty pictures to please the eye. So, for all the brave people who made it to the end, here is a video of Randy explaining is scraping tactic, and a cat picture very relevant to today’s challenge

[theswarmlab-blog]

R vs Python - Round 1

var ifrm_advanced_iframe_2 = document.getElementById("advanced_iframe_2");jQuery("#advanced_iframe_2").css("visibility", "hidden");aiUseCookie();

[theswarmlab-blog]

Faculty position available at NJIT - Complex Biological Systems

The Department of Biological Sciences at the New Jersey Institute of Technology invites applications for a tenure-track, full time Assistant or Associate Professor position in the general area of complex biological systems. We seek candidates who will examine biological systems at any level, from cells to ecosystems, with an integrative view across levels. The candidate should use state-or-the-art technologies, including, but not limited to, molecular biology, cellular biology, genomics, proteomics, phenomics, transcriptomics, remote/optical sensing, and/or organismal tracking, to collect and analyze laboratory and/or field experimental data. Applicants must have a Ph.D. degree or equivalent, postdoctoral training, demonstrated research expertise, and a history of, or potential for, successful extramural funding. Candidates that link genetic, molecular, or cellular mechanisms to ecological outcomes are particularly encouraged to apply.

The successful candidate will be expected to develop a strong, externally funded and independent research program, and must demonstrate potential for teaching excellence at undergraduate and graduate levels. The candidate will have opportunities to collaborate with ecologists, evolutionary biologists, neuroscientists and cell biologists in the Department of Biological Sciences, as well as engineers, mathematicians, and computer scientists at NJIT and surrounding institutions.

Applicants should submit a single PDF document containing a cover letter, a CV, a brief statement of current and proposed research plans, and a teaching statement. The document should also include 5-7 keywords that best describe the applicant’s research interests, and a link to their Google Scholar page. All applications must be submitted on-line at njit.jobs, referencing posting 0601802 (direct link: http://njit.jobs/applicants/Central?quickFind=52682). Candidates should arrange for three letters of recommendation to be sent to [email protected]. Review of applications will begin December 15th, 2013, and continue until position is filled. Please address any questions to Jorge Golowasch at [email protected].

NJIT is an Equal Opportunity / Affirmative Action Employer. We welcome all applications regardless of race, color, religion, national origin, sex, age, disability or veteran status. Researchers from traditionally underrepresented groups are especially encouraged to apply.