Machine Learning - week 4 Assignment:
THE MOVIE DATASET.
TASK- IS to check which cluster perform best and to check and compare the statistics of each cluster( Ratings, Revenue, Votes , Budget, Average Vote count, Movie Title and Popularity)
#Check if rows contain any null values data_numeric.isnull().sum()
# -*- coding: utf-8 -*-
from pandas import Series, DataFrame import pandas as pd import numpy as np import matplotlib.pylab as plt from sklearn.cross_validation import train_test_split from sklearn import preprocessing from sklearn.cluster import KMeans
#
Read the movies metadata csv file
“”“ Data Management ”“” data = pd.read_csv(“movies_metadata.csv”)
#upper-case all DataFrame column names
data.columns = map(str.upper, data.columns)
#Only keep the numeric columns for our analysis. However, we’ll keep titles also #to interpret the results at the end of clustering. Note that this title column will #not be used in the analysis.
data.drop(data.index[19730],inplace=True) data.drop(data.index[29502],inplace=True) data.drop(data.index[35585],inplace=True)
data.columns
data_numeric = data[[‘BUDGET’,'POPULARITY’,'REVENUE’,'RUNTIME’,'VOTE_AVERAGE’,'VOTE_COUNT’,'TITLE’]]
data_numeric.head()
#Check if rows contain any null values
data_numeric.isnull().sum()
# Data Management #Drop all the rows with null values
data_clean = data_numeric.dropna()
# subset clustering variables cluster=data_clean cluster.describe()
#Let’s see the statistics for the votes counts for the movies data cluster['VOTE_COUNT’].describe()
#We see that a half of movies have been rated less than 10 times. For easier interpretability,
#let’s take only the movies that have more than 30 votes, i.e. top 26% of the movies.
cluster['VOTE_COUNT’].quantile(np.arange(.74,1,0.01))
cluster = cluster[cluster['VOTE_COUNT’]>30]
cluster.shape
cluster.columns
# standardize clustering variables to have mean=0 and sd=1 #Data Normalization
clustervar=cluster.copy() clustervar = clustervar.drop(columns='TITLE’) clustervar['BUDGET’]=preprocessing.scale(clustervar['BUDGET’].astype('float64’)) clustervar['POPULARITY’]=preprocessing.scale(clustervar['POPULARITY’].astype('float64’)) clustervar['REVENUE’]=preprocessing.scale(clustervar['REVENUE’].astype('float64’)) clustervar['RUNTIME’]=preprocessing.scale(clustervar['RUNTIME’].astype('float64’)) clustervar['VOTE_AVERAGE’]=preprocessing.scale(clustervar['VOTE_AVERAGE’].astype('float64’)) clustervar['VOTE_COUNT’]=preprocessing.scale(clustervar['VOTE_COUNT’].astype('float64’))
clustervar.head()
Apply K-Means Clustering
What k to choose?
Let’s fit cluster size 1 to 20 on our data and take a look at the corresponding score value.
from scipy.spatial.distance import cdist clusters=range(1,20) meandist=[]
for k in clusters: model=KMeans(n_clusters=k) model.fit(clustervar) clusassign=model.predict(clustervar) meandist.append(sum(np.min(cdist(clustervar, model.cluster_centers_, 'euclidean’), axis=1)) / clus_train.shape[0])
These MeanDist signify how far our observations are from the cluster center. A large positive or a large negative value would indicate that the cluster center is far from the observations.
Based on these MeanDist, we plot an Elbow curve to decide which cluster size is optimal. Note that we are dealing with tradeoff between cluster size(hence the computation required) and the relative accuracy.
“”“
Plot average distance from observations from the cluster centroid to use the Elbow Method to identify number of clusters to choose ”“”
plt.plot(clusters, meandist) plt.xlabel('Number of clusters’) plt.ylabel('Average distance’) plt.title('Selecting k with the Elbow Method’)
Our Elbow point is around cluster size of 5. We will use k=5 to further interpret our clustering result.
# Interpret 5 cluster solution
model3=KMeans(n_clusters=5) model3.fit(clustervar) clusassign=model3.predict(clustervar)
# plot clusters
from sklearn.decomposition import PCA pca_2 = PCA(2) plot_columns = pca_2.fit_transform(clustervar) plt.scatter(x=plot_columns[:,0], y=plot_columns[:,1], c=model3.labels_,) plt.xlabel('Canonical variable 1’) plt.ylabel('Canonical variable 2’) plt.title('Scatterplot of Canonical Variables for 3 Clusters’) plt.show()
As a result of clustering, we have the clustering label. Let’s put these labels back into the original numeric data frame.
len(model3.labels_)
cluster['cluster’] = model3.labels_
cluster.head()
Interpret clustering results
#Let’s see cluster sizes first.
import seaborn as sns plt.figure(figsize=(12,7)) axis = sns.barplot(x=np.arange(0,5,1),y=cluster.groupby(['cluster’]).count()['BUDGET’].values) x=axis.set_xlabel(“Cluster Number”) x=axis.set_ylabel(“Number of movies”)
We clearly see that one cluster is the largest and one cluster has the fewest number of movies.
Let’s look at the cluster statistics.
cluster.groupby(['cluster’]).mean()
We see that one cluster which is also the smallest, is the cluster of movies that received maximum number of votes(in terms of counts) and also have very high popularity and total runtime and net revenue. Let’s see some of the movies that belong to this cluster.
size_array = list(cluster.groupby(['cluster’]).count()['BUDGET’].values)
cluster[cluster['cluster’]==size_array.index(sorted(size_array)[0])].sample(5)
We see many big movie names in this cluster. So the results are intuitive.
Cluster that is the second smallest cluster in the results, has 2nd highest votes count and the most highly rated movies.
The runtime for these movies is on the higher end and popularity score is also good.
Let’s see some of the movie names from this cluster.
cluster[cluster['cluster’]==size_array.index(sorted(size_array)[1])].sample(5)
Lastly, let’s take a look at the least successful movies. This cluster represents the movies that received least number of votes and also has the smallest run-time, revenue and popularity score.
cluster[cluster['cluster’]==size_array.index(sorted(size_array)[-1])].sample(5)
As we can see this cluster also includes the movies for which our dataset has no information about the budget and revenue, hence there corresponding fields have 0 value in it. This pulls down the net revenue of the whole cluster. If we keep the cluster size slightly larger, we might get to see these movies clustered separately.
