Master Project

[masterproject]

Data Mining and Text Modeling Readme

Section One: Mining statistics from the server data I wrote five Python applications to extract useful data from these files: Application 1: Extracting black ticket data from the data pool

read_statistics_black_ticket.py

Application 2: Extracting red ticket data from the data pool

read_statistics_red_ticket.py

Application 3: Extracting subscription data from the data pool

read_statistics_subscription.py

Application 4: Extracting data about number of words from the data pool

read statistics_words.py

Application 5: Extracting data about number of page views from the data pool

read_statistics_page_views.py

These five applications read data files from the directory raw_data/book_statistics and save results to the directory cleaned_data Section Two: Text modeling and keyword extraction I wrote three Python applications to segment content and comments and extract keywords: Application 1: Content segmentation     content_segmentation.py Application 2: Keyword extraction from content     content_keywords.py Application 3: Keyword extraction from comment     comments_keywords.py These three applications uses Python Jieba library and input a customized dictionary from customized_dict.txt and a stop word list from stopwords.txt. These three applications read data files from the directory raw_data/book_comment and raw_data_book_content and save results to the directory topic_modeling_results

[masterproject]

Millionaire Web Novelist Interviews

I interviewed the millionaire writer Wuzui this morning. He answered all my questions and shared his experiences. His transitions from an boiler engineer in a thermal power plant to a millionaire web novelist is interesting and attractive. When he resigned from the power plant, Wuzui have been promoted as the vice director of a branch. He has a full time housewife who support him in this career. Because of his family's support, he has more time to write stories. He also gave advices to young college students about their career choices. Wuzui confirmed that the rumors on social networks about a manager and a group of editors are considering leaving Qidian, the first web novel start-up, is true. He said that this rumor has been confirmed. However, the PR person of Qidian said that this rumor was not correct. "Since it is a rumor, it will not affect the stock price of the company listed in NASDAQ."

Recalled the ethics class taught by Professor Richard Wald in 2011, I will give both sides the best shot in my investigative data-driven story.

[masterproject]

Comments about web novel publishing industry

Carmine Elvezio

Researcher of the User Interface and Augmented Reality Lab at Columbia University

Password for this video: masterproject

[masterproject]

Here is my updated targeted visualization

The millionaire writer Zhong Wang was hard to contact. I contacted both his company and the human resource department officer who works with him. Wang might suffer difficulties recently. He was involved in a lawsuit last year.

New start-up firms have been competing for well-known novelists on the web. The millionaire web novelist Wang was a contract writer for Qidian before joining a new startup web novel publishing firm in 2010, where he continued updating the story Yong Sheng (永生) that he had begun at Qidian. Qidian filed a copyright infringement lawsuit against Wang accusing Wang of contract infringement. Shanghai No.1 Intermediate People’s Court made the final judgment in 2012. According to the press release of the court, Shanghai Xuan Ting Entertainment Information Technology Co. Ltd, the Internet Operation Provider of Qidian under Shanda Interactive Entertainment (NASDAQ: SNDA), owned the serialized web novel Yong Sheng (永生). Wang paid 0.6 million Yuan (US$ 0.096 million) to the company as compensatory recovery.

As we can see from my data visualization, page-views of his work, the King of Wisdom, was decreased since July 2013. As the same time, Wang worked very hard to update his story. From the visualization, we can see that he constantly update this story. I need more information to explain this decreasing page-views

[masterproject]

Data Mining Using Python

Reading and Organization Page Views:

import os import glob import string import csv dataList = [] raw_data_path = 'raw_data/' cleaned_data_path = 'cleaned_data/' for infile in glob.glob( os.path.join(raw_data_path, '*.txt') ):     with open(infile, "r") as file:         for line in file:             #print line             collection = string.split(line.rstrip('\r\n'),',')             title = collection.pop(1) + "-" + collection.pop(1) + "-" + collection.pop(1)             collection.insert(0, title)             collection.pop(2)             collection.pop(2)             collection.pop(2)             collection.pop(2)             collection.pop(3)             collection.pop(3)             collection.pop(3)             collection.pop(3)             collection.pop(3)             collection.pop(3)             dataList.append(collection)         filename = cleaned_data_path + infile.replace("raw_data/", "")         with open(filename, 'wb') as f:             writer = csv.writer(f)             writer.writerows(dataList)         print dataList         dataList = []

Reading and Organizing Subscriptions:

import os import glob import string import csv dataList = [] raw_data_path = 'raw_data/book_statistics' cleaned_data_path = 'cleaned_data/book_subscription' for infile in glob.glob( os.path.join(raw_data_path, '*.txt') ):     with open(infile, "r") as file:         for line in file:             #print line             collection = string.split(line.rstrip('\r\n'),',')             title = collection.pop(1) + "-" + collection.pop(1) + "-" + collection.pop(1)             collection.insert(0, title)             collection.pop(1)             collection.pop(1)             collection.pop(1)             collection.pop(1)             collection.pop(1)             collection.pop(1)             collection.pop(1)             collection.pop(1)             collection.pop(2)             collection.pop(2)             collection.pop(2)             #print collection             dataList.append(collection)         filename = cleaned_data_path + infile.replace("raw_data/book_statistics", "")         with open(filename, 'wb') as f:             writer = csv.writer(f)             writer.writerows(dataList)         print dataList         dataList = []

Reading and Organizing Red Tickets:

import os import glob import string import csv dataList = [] raw_data_path = 'raw_data/book_statistics' cleaned_data_path = 'cleaned_data/book_red_ticket' for infile in glob.glob( os.path.join(raw_data_path, '*.txt') ):     with open(infile, "r") as file:         for line in file:             #print line             collection = string.split(line.rstrip('\r\n'),',')             title = collection.pop(1) + "-" + collection.pop(1) + "-" + collection.pop(1)             collection.insert(0, title)             collection.pop(1)             collection.pop(1)             collection.pop(1)             collection.pop(1)             collection.pop(1)             collection.pop(1)             collection.pop(1)             collection.pop(1)             collection.pop(1)             collection.pop(2)             collection.pop(2)             print collection             dataList.append(collection)         filename = cleaned_data_path + infile.replace("raw_data/book_statistics", "")         with open(filename, 'wb') as f:             writer = csv.writer(f)             writer.writerows(dataList)         print dataList         dataList = []

Single Book Mining:

import os import glob import string import csv dataList = [] with open("raw_data/book_statistics/175703.txt", "r") as file:     for line in file:         #print line         collection = string.split(line.rstrip('\r\n'),',')         title = collection.pop(1) + "-" + collection.pop(1) + "-" + collection.pop(1)         collection.insert(0, title)         dataList.append(collection)         with open("cleaned_data/175703.csv", 'wb') as f:             writer = csv.writer(f)             writer.writerows(dataList)

CSV Date and Month Conversion:

import os import glob import string import csv dataList = [] monthList = ["Jan", "Feb", "Mar", "Apr", "May", "Jun", "Jul", "Aug", "Sep", "Oct", "Nov", "Dec"] reader = csv.DictReader(open("75456_pageviews.csv")) for line in reader:     year = line["date"][0:4]     month = monthList[int(line["date"][5:7]) - 1]     temp = []     temp.append(line["book"])     temp.append(month + " " + year)     temp.append(str(line["clicks"]))     dataList.append(temp) with open("pageviews-month-year.csv", 'wb') as f:     writer = csv.writer(f)     writer.writerows(dataList) print dataList

Results of Conversions:

[masterproject]

Text Segmentation and Keyword Extraction using Jieba Library and Gemsim Library

Here is my python code for text segmentation:

#encoding=utf-8 import jieba import jieba.analyse import csv raw_data_path = 'raw_data/book_content/' file_name = ["37127", "37853", "38250", "39813", "41365", "44202", "45669", "48552", "69507", "75456", "81135", "89994", "135095", "158342"] #jieba.load_userdict("customized_dict.txt") stop_list = [] with open("stopwords.txt", "r") as stoplistfile:     for item in stoplistfile:         stop_list.append(item.rstrip('\r\n')) output_list = [] for name in file_name:     all_text = ""     multi_line_text = ""     temp_text = []     print "Book No. " + name + " keywords:"     temp_text.append(("Book" + name).encode('utf-8'))     with open(raw_data_path + name + ".txt", "r") as file:         for line in file:             if line != '\n':                 multi_line_text += line         templist = multi_line_text.split('\n')         for text in templist:             all_text += text                  for word in stop_list:             all_text = all_text.replace(word,'')         seg_list = jieba.cut(all_text,cut_all=False)         for item in seg_list:             temp_text.append(item.encode('utf-8'))         #print temp_text         output_list.append(temp_text)         with open("results/" + name + ".csv", 'wb') as onebook:             writer = csv.writer(onebook)             writer.writerow(temp_text) with open("results/content_segmentation.csv", 'wb') as f:     writer = csv.writer(f)     writer.writerows(output_list)

Here is the results:

  Code for Extracting Content Keywords:

#encoding=utf-8 import jieba import jieba.analyse import csv raw_data_path = 'raw_data/book_content/' file_name = ["37127", "37853", "38250", "39813", "41365", "44202", "45669", "48552", "69507", "75456", "81135", "89994", "135095", "158342"] stop_list = [] with open("stopwords.txt", "r") as stoplistfile:     for item in stoplistfile:         stop_list.append(item.rstrip('\r\n')) output_list = [] for name in file_name:     all_text = ""     multi_line_text = ""     temp_text = []     print "Book No. " + name + " keywords:"     temp_text.append(("Book" + name).encode('utf-8'))     with open(raw_data_path + name + ".txt", "r") as file:         for line in file:             if line != '\n':                 multi_line_text += line         templist = multi_line_text.split('\n')         for text in templist:             all_text += text                 for word in stop_list:             all_text = all_text.replace(word,'')         keyword_list = jieba.analyse.extract_tags(all_text, 20)         for item in keyword_list:             print item             temp_text.append(item.encode('utf-8'))         output_list.append(temp_text) with open("results/content_keywords.csv", 'wb') as f:     writer = csv.writer(f)     writer.writerows(output_list)

Results:

Book37127,身体,佛道,修行,苦痴,护法,苦痴师,女鬼,阴阳,阳身,尸体,玩意,体内,师父,生气,乌龙,教授,魔王,清明,土狗,眉头 Book37853,身体,老太爷,银子,灵气,尸体,修行,眉头,鬼童,攻击,迷山,鬼王,臭子,体内,父亲,老爷子,法术,糊涂,客气,气息,尸卫 Book38250,书记,身子,县长,转头,领导,省委,干部,市长,办公室,西山,市委,车子,漂亮,秘书长,挂断,俏脸,材料,办公桌,纪委,卧室 Book39813,身体,华夏,耶稣,龙王,燕京,养心,老爷子,皇帝,军师,保镖,答应,车子,公民,仇家,拒绝,黑色,关系,学生,解释,白残 Book41365,修罗,轩辕,身体,哥哥,玩家,攻击,魔法,生命,世界,华夏,轩辕剑,轮回,属性,力量,职业,全身,混沌,消失,速度,帮派 Book44202,淮东,崇州,奢家,兵马,江东,左军,淮东军,精锐,流民,兵力,东军,战船,骑兵,战事,淮西,青河,徐州,势力,河口,高丽 Book45669,灵力,修炼,符阵,神力,师兄,神识,妖术,门派,云海,魔兵,战部,法宝,剑意,弟子,实力,魔族,炼制,力量,剑诀,法诀 Book48552,神通,修炼,高手,弟子,秘境,力量,神族,身体,金仙,元气,浮屠,仙器,纪元,八部,法力,法宝,黄泉,炼化,境界,道器 Book69507,王安石,秦州,交趾,遵裕,京城,西夏,宰相,周南,邕州,西贼,木征,司马光,官员,功劳,宋军,进士,辽国,转运,消息,骑兵 Book75456,修士,妖兽,金丹,法宝,绿袍,灵石,神玄,真元,神识,元气,宗主,老祖,法器,道友,阴丽,气息,虚空,白色,法阵,妖丹 Book81135,法力,修炼,法术,贼头,祭炼,金刚,炼气,法器,玄黄,顾龟,太极图,魔族,龙诀,火鸦,法宝,真火,青城派,级数,弟子,太化 Book89994,叶邪,身体,玩家,力量,命运,攻击,迷失,生命,龙魂,世界,魔法,魔神,属性,司徒,哥哥,职业,试炼,伤害,鬼王,消失 Book135095,独孤,高手,真气,武功,剑气,剑法,宗师,姐姐,魔门,长生,境界,邪帝,原著,长剑,江湖,牧场,前世,玉真,闻言,气势 Book158342,球员,赛季,球队,塞罗,皇马,进球,进攻,冠军杯,场赛,防守,联赛,球迷,禁区,半场,皮球,出色,西甲,球场,前锋,主场

Code for Extracting Comments Keywords:

#encoding=utf-8 import jieba import jieba.analyse import csv jieba.load_userdict("customized_dict.txt") raw_data_path = 'raw_data/book_comment/' file_name = ["37127", "37853", "38250", "39813", "41365", "44202", "45669", "48552", "69507", "75456", "81135", "89994", "135095", "158342"] stop_list = [] with open("stopwords.txt", "r") as stoplistfile:     for item in stoplistfile:         stop_list.append(item.rstrip('\r\n')) output_list = [] for name in file_name:     all_text = ""     multi_line_text = ""     temp_text = []     print "Book No. " + name + " keywords:"     temp_text.append(("Book" + name).encode('utf-8'))     with open(raw_data_path + name + ".txt", "r") as file:         for line in file:             if line != '\n':                 multi_line_text += line         templist = multi_line_text.split('\n')         for text in templist:             all_text += text                  for word in stop_list:             all_text = all_text.replace(word,'')         keyword_list = jieba.analyse.extract_tags(all_text, 20)         for item in keyword_list:             print item             temp_text.append(item.encode('utf-8'))         output_list.append(temp_text) with open("results/comments_keywords.csv", 'wb') as f:     writer = csv.writer(f)     writer.writerows(output_list)

Results:

Book37127,加油,支持,红票,解灵,收藏,刷分,闲逛,灵异,圣魂,精彩,情节,宣传,分刷,潜力,投票,文笔,拜山,书挺,书写,结尾 Book37853,加油,支持,票票,收藏,刷分,猎灵师,红票,顶顶顶,宣传,投票,情节,文笔,闲逛,灵异,潜力,动力,送票,解灵,快乐,主角 Book38250,色戒,太棒,续加,官道,红票,书写,捧场,犒劳,精彩,加油,寂寞,支持,动力,潜力,推倒,宣传,官场,太慢,收藏,主角 Book39813,红票,书写,捧场,太棒,续加,加油,投票,支持,犒劳,精彩,书评,票票,顶顶顶,风骚,两票,保镖,快乐,书迷,快点,近卫军 Book41365,修罗,威武,火星,加油,顶顶顶,支持,刷分,红票,网游,灌灌,主角,刷刷,刷评,传说,书评,绿帽子,投票,点击,号群,票票 Book44202,枭臣,太棒,续加,书写,捧场,犒劳,红票,精彩,淮东,加油,奢家,支持,投票,官路,主角,动力,情节,东虏,书评,势力 Book45669,修真,书写,捧场,太棒,续加,红票,犒劳,世界,精彩,加油,支持,投票,黑票,创作,太慢,顶顶顶,快点,动力,主角,书评 Book48552,永生,太棒,续加,神机,捧场,犒劳,书写,精彩,红票,加油,支持,黄泉,主角,仙王,顶顶顶,神通,方清,神族,华都,造化 Book69507,书写,宰执,续加,太棒,捧场,犒劳,精彩,红票,支持,希越,加油,投票,主角,王安石,怪兽,历史,创作,收藏,穿越,哥斯拉 Book75456,太棒,续加,通路,捧场,书写,犒劳,精彩,红票,猥琐,支持,加油,金丹,投票,修士,神玄,主角,书评,妖兽,功法,流氓 Book81135,蛤蟆,太棒,续加,红票,捧场,犒劳,书写,精彩,加油,动力,烂尾,支持,法宝,潜力,仙侠,宇宙,造化,宣传,蜀山,赤城 Book89994,火星,邪龙,网游,捧场,书写,太棒,红票,续加,顶顶顶,书评,修罗,犒劳,精彩,加油,支持,尼玛,投票,命运,主角,晚安 Book135095,顶顶顶,独孤,书写,捧场,续加,主角,太棒,红票,加油,支持,犒劳,精彩,节操,投票,黄易,收藏,破碎,情节,太监,剧情 Book158342,书写,捧场,支持,投票,创作,加油,球员,球队,续加,太棒,红票,皇马,主角,赛季,冠军,欧冠,收藏,犒劳,郭怒,精彩

[masterproject]

Code for Bubble Chart Comments and Content Visualization

Data mined and analyzed using the Python Jieba Library and the Python Gensim Library

Developed using the Data-Driven Documents (D3) Library inspired by the Bubble Chart

var diameter = 1000, format = d3.format(",d"), color = d3.scale.category20c(); var bubble = d3.layout.pack()   .sort(null)   .size([diameter, diameter])   .padding(1.5); var comment_svg = d3.select("#comment_word_cloud").append("svg")   .attr("width", diameter)   .attr("height", diameter)   .attr("class", "bubble"); d3.json("json_data/comment_data.json", function(error, root) {   var node = comment_svg.selectAll(".node")     .data(bubble.nodes(classes(root))     .filter(function(d) { return !d.children; }))     .enter().append("g")     .attr("class", "node")     .attr("transform", function(d) { return "translate(" + d.x + "," + d.y + ")"; });   node.append("title")     .text(function(d) { return d.className + ": " + format(d.value); });   node.append("circle")     .attr("r", function(d) { return d.r; })     .style("fill", function(d) { return color(d.packageName); });   node.append("text")     .attr("dy", ".3em")     .style("text-anchor", "middle")     .text(function(d) { return d.className.substring(0, d.r / 3); }); }); // Returns a flattened hierarchy containing all leaf nodes under the root. function classes(root) {   var classes = [];   function recurse(name, node) {     if (node.children) node.children.forEach(function(child) { recurse(node.name, child); });     else classes.push({packageName: name, className: node.name, value: node.size});   }   recurse(null, root);   return {children: classes}; } d3.select(self.frameElement).style("height", diameter + "px");

[masterproject]

Code for D3 Line Chart Visualization

Data mined and analyzed using the Python Jieba Library and the Python Gensim Library

Visualization developed using the Data-Driven Documents (D3) Library inspired by the Line Graph with Dual-scaled Axes.

// Data array for pageview var data1 = [175482, 173921, 178700, 173445, 161303, 183206, 189261, 217394, 212578, 198895, 208623, 198026, 200070, 210005, 202120, 226972, 292047, 284975, 261037, 264295, 270422, 269353, 279135, 331320, 318608, 283991, 277259, 293945, 308135, 305437, 304406, 302415, 293567, 339569, 339286, 318603, 308628, 346517, 380255, 401014, 400551, 374231, 347277, 346659, 352628, 370844, 361397, 344931, 332978, 322979, 316816, 326260, 332066, 373403, 367229, 364258, 336124, 321106, 327089, 321979, 327933, 312738, 297174, 289470, 298910, 288113, 288649, 275576, 277637, 269326, 266297, 270346, 281770, 289829, 286486, 290191, 288483, 284481, 282943, 295527, 291992, 281244, 290836, 304520, 312638, 315248, 340015, 359397, 314658, 301604, 290045, 286285, 279677, 283368, 283455, 283472, 282046, 273739, 264889, 263804, 268360, 272038, 260891, 259681, 264792, 264437, 261575, 263431, 255635, 249191, 240464, 243134, 247868, 249436, 259109, 260240, 255172, 245218, 246684, 257279, 246445, 250681, 253160, 158083, 124375, 103424, 92013, 84288, 79570, 81353, 77501, 86423, 91367, 77134, 72733, 75086, 73187, 78561, 72893, 65001, 61389, 58644, 57300, 56425, 55216, 56025, 57242, 56695, 54606, 52382, 49065, 50394, 46373, 45043, 44081, 44282, 40008, 40695, 39078, 38132, 37556, 35148, 35020, 35007, 35611, 36426, 36385, 42478, 48364, 47425, 53957, 48190, 50320, 48604, 46921, 44851, 43966, 44070, 44526, 43346, 43620, 43827, 42943, 40981, 37430, 39619, 39029, 38970, 48079, 40831, 37650, 40742, 40700, 37518, 35734, 36837, 40780, 37878, 38511, 38254, 35875, 35839, 41012, 41748, 40820, 41776, 40940, 37594, 38706, 46713, 40631, 39482, 40146, 39345, 34579, 34940, 35261, 35055, 36278, 34577, 33946, 32180, 34504, 31709, 31277, 32966, 34396, 34261, 35065, 36156, 35215, 34973, 36506, 40976, 41071, 35887, 38473, 38858, 37412, 40535, 44409, 44536, 41461, 41800, 40481, 39448, 39021, 41847, 43265, 37960, 37131, 38756, 40067, 39809, 41746, 44520, 45984, 45759, 44644, 48365, 45358, 45164, 44856, 41710, 43218, 44381, 44425, 41489, 41245, 41952, 39868, 37437, 37874, 38719, 38276, 40039, 40005, 37065, 36563, 42630, 37110, 37776, 39769, 39497, 36519, 35525, 35092, 36555, 39983, 41987, 43808, 36193, 36513, 38174, 39410, 38374, 36556, 38514, 35384, 34289, 34999, 33570, 32052, 32420, 32501, 33185, 32231, 31748, 32109, 34841, 31627, 30443, 30402, 30155]; // Data array for revenue var data2 = [299442, 33180, 44164, 64320, 31188, 58728, 51170, 71110, 30904, 49890, 25060, 54592, 22534, 30910, 33200, 45380, 213470, 66480, 46698, 27956, 28510, 26698, 35240, 49826, 21032, 20684, 23058, 517844, 38820, 36968, 55738, 30792, 56606, 95368, 114406, 34798, 40270, 64806, 40984, 648874, 51596, 29918, 16188, 23722, 47734, 126518, 12404, 45352, 17304, 9216, 15720, 8050, 11972, 60074, 35944, 40022, 13448, 9750, 35652, 21612, 29616, 13816, 11736, 11358, 4496, 5008, 5320, 5164, 15472, 5052, 8838, 8098, 12758, 16916, 13324, 7406, 5320, 7150, 5096, 8826, 5830, 7562, 4886, 10584, 7030, 23480, 30198, 138520, 59914, 19928, 12696, 4886, 5396, 11786, 21128, 7054, 5818, 3986, 4764, 3366, 31038, 14782, 16628, 26604, 6150, 5606, 6096, 3298, 4062, 6750, 20108, 3952, 12722, 14540, 15840, 10362, 8848, 11416, 13842, 28736, 49166, 86948, 1565909, 900688, 557794, 339925, 664926, 473350, 689774, 402080, 420521, 638988, 392180, 431914, 269295, 261676, 270954, 339928, 275089, 253029, 247369, 234981, 232589, 244266, 244664, 233087, 224729, 217735, 168010, 146194, 152023, 153050, 142455, 168606, 140659, 143223, 239705, 128025, 121093, 128441, 127542, 118867, 115868, 119637, 128197, 123581, 129278, 132579, 125553, 110726, 111422, 127128, 121054, 119403, 112723, 111009, 111474, 119426, 113466, 114289, 322513, 115420, 106960, 105607, 112066, 108237, 113590, 106305, 109339, 137888, 103884, 107584, 106967, 111812, 100912, 119070, 148792, 104320, 131466, 108102, 99345, 108561, 109215, 97207, 92543, 87652, 94948, 109616, 109156, 133712, 148817, 135683, 147067, 130922, 129803, 123451, 137160, 135665, 140403, 134741, 138206, 143147, 148932, 155647, 152880, 158969, 451934, 164884, 170810, 160699, 151346, 167893, 155146, 154829, 151858, 165947, 116710, 187580, 156930, 153688, 159757, 166433, 158303, 146643, 142649, 155342, 146755, 141523, 174641, 151867, 142453, 146778, 134649, 141274, 153333, 168872, 175895, 199033, 190424, 218914, 165567, 155901, 144086, 136720, 174984, 171623, 144970, 146119, 145935, 157678, 128397, 148313, 147427, 136660, 138163, 183704, 137265, 143517, 111793, 163393, 114919, 147670, 111951, 125935, 136032, 114382, 123258, 123734, 140712, 120323, 147892, 121365, 123875, 107051, 127847, 100241, 93117, 97354, 77423, 90725, 82374, 89811, 72655, 91521, 87517, 84807, 90336, 80807, 86127, 101905, 87863, 84640, 73935, 77973]; var x = d3.scale.linear().domain([0, data1.length]).range([0, w]); var xlabel = d3.time.scale()     .domain([new Date(new Date("2012-03-01")), d3.time.day.offset(new Date("2013-01-08"), 1)])     .range([0, w]); var y1 = d3.scale.linear().domain([0, data1.max()]).range([h, 0]); var y2 = d3.scale.linear().domain([0, data2.max()]).range([h, 0]);   var line1 = d3.svg.line()     .x(function(d,i) {         console.log('Plotting X1 value for data point: ' + d + ' using index: ' + i + ' to be at: ' + x(i) + ' using our xScale.');         return x(i);     })     .y(function(d) {         console.log('Plotting Y1 value for data point: ' + d + ' to be at: ' + y1(d) + " using our y1Scale.");         return y1(d);     })     var line2 = d3.svg.line()     .x(function(d,i) {         console.log('Plotting X2 value for data point: ' + d + ' using index: ' + i + ' to be at: ' + x(i) + ' using our xScale.');         return x(i);     })     .y(function(d) {         console.log('Plotting Y2 value for data point: ' + d + ' to be at: ' + y2(d) + " using our y2Scale.");         return y2(d);     })     var revenue_pageview = d3.select("#revenue_pageview").append("svg:svg")       .attr("width", w + m[1] + m[3])       .attr("height", h + m[0] + m[2])       .append("svg:g")       .attr("transform", "translate(" + m[3] + "," + m[0] + ")"); var xAxis = d3.svg.axis().scale(xlabel).tickSize(-h).tickSubdivide(true); revenue_pageview.append("svg:g")       .attr("class", "x axis")       .attr("transform", "translate(0," + h + ")")       .call(xAxis); revenue_pageview.append("text")     .attr("class", "bottomlabel")     .attr("text-anchor", "end")     .attr("y", h + m[1]/2 +10)     .attr("x", w - 80)     .attr("dy", ".2em")     .text("Selected One Top Ranked Best Sell Fantasy Novel: King of Wisdom (奇幻玄幻: 圣王)"); revenue_pageview.append("text")     .attr("class", "bottomlabelauthor")     .attr("text-anchor", "end")     .attr("y", h + m[1]/2 + 30)     .attr("x", w - 60)     .attr("dy", ".2em")     .text("Wrote by millionaire author: Zhong Wang (王钟) Pen name: Meng Ru Shen Ji (梦入神机)") var yAxisLeft = d3.svg.axis().scale(y1).ticks(4).orient("left"); revenue_pageview.append("svg:g")       .attr("class", "y axis axisLeft")       .attr("transform", "translate(-15,0)")       .call(yAxisLeft);             revenue_pageview.append("text")     .attr("class", "scaleleft")     .attr("text-anchor", "end")     .attr("y", 1)     .attr("dy", ".01em")     .attr("transform", "rotate(-90)")     .text("Pageviews"); var yAxisRight = d3.svg.axis().scale(y2).ticks(6).orient("right"); revenue_pageview.append("svg:g")           .attr("class", "y axis axisRight")           .attr("transform", "translate(" + (w+15) + ",0)")           .call(yAxisRight);     revenue_pageview.append("text")     .attr("class", "scaleright")     .attr("text-anchor", "end")     .attr("y", w)     .attr("dy", ".01em")     .attr("transform", "rotate(-90)")     .text("Revenue"); revenue_pageview.append("svg:path").attr("d", line1(data1)).attr("class", "data1"); revenue_pageview.append("svg:path").attr("d", line2(data2)).attr("class", "data2");

[masterproject]

Targeted Three Line Visualizations about Millionaire Writers

After mining data for about one month, I found the following representation of data fit into my story:

These line visualizations using D3 library allow readers to learn the relationships between page views and revenues for millionaire writers, particularly Zhong Wang, pen name Meng Ru Shen Ji (梦入神机). This millionaire writers have rich life story and experiences in web novel.

I plan to make my rest visualization focus on novels wrote by Wang.

[masterproject]

A number of line chart visualizations applied to this master project

D3 line chart visualization with two books' statistics.

D3 line chart visualization with multiple books' statistics.

D3 short line chart visualization with multiple books' statistics.

Google line chart visualization tool for multiple books' statistics.

[masterproject]

Improved Line Chart for Revenue and Pageview Analysis

With improved colored labels and visual lines

[masterproject]

Novel Word and Page View Visualization

[masterproject]

Visualization of Comment Keywords (Multiple Story Bubble Chart)

Bubble Chart Visualization of Comment Keyword. Each color represents a story of the targeted web publishing platform.

Here is the source code of this visualization. I used one of the example of the D3 library.

The data file is in JSON format:

[masterproject]

Keywords Visualization using D3 Library

Jason Davies developed a word cloud visualization using D3 library. I modified his original version and feed my keyword results：

I created the following content and comment visualization:

The first two figures are keywords of story content. The last two figures are keywords of story comments

[masterproject]

Keyword Extraction of Comments

Using improved algorithm and libraries, I extracted keywords from comments:

Book No. 37127 keywords: 加油 支持 红票 解灵 收藏 刷分 闲逛 灵异 圣魂 精彩 情节 宣传 分刷 潜力 投票 文笔 拜山 书挺 书写 结尾 Book No. 37853 keywords: 加油 支持 票票 收藏 刷分 猎灵师 红票 顶顶顶 宣传 投票 情节 文笔 闲逛 灵异 潜力 动力 送票 解灵 快乐 主角 Book No. 38250 keywords: 色戒 太棒 续加 官道 红票 书写 捧场 犒劳 精彩 加油 寂寞 支持 动力 潜力 推倒 宣传 官场 太慢 收藏 主角 Book No. 39813 keywords: 红票 书写 捧场 太棒 续加 加油 投票 支持 犒劳 精彩 书评 票票 顶顶顶 风骚 两票 保镖 快乐 书迷 快点 近卫军 Book No. 41365 keywords: 修罗 威武 火星 加油 顶顶顶 支持 刷分 红票 网游 灌灌 主角 刷刷 刷评 传说 书评 绿帽子 投票 点击 号群 票票 Book No. 44202 keywords: 枭臣 太棒 续加 书写 捧场 犒劳 红票 精彩 淮东 加油 奢家 支持 投票 官路 主角 动力 情节 东虏 书评 势力 Book No. 45669 keywords: 修真 书写 捧场 太棒 续加 红票 犒劳 世界 精彩 加油 支持 投票 黑票 创作 太慢 顶顶顶 快点 动力 主角 书评 Book No. 48552 keywords: 永生 太棒 续加 神机 捧场 犒劳 书写 精彩 红票 加油 支持 黄泉 主角 仙王 顶顶顶 神通 方清 神族 华都 造化 Book No. 69507 keywords: 书写 宰执 续加 太棒 捧场 犒劳 精彩 红票 支持 希越 加油 投票 主角 王安石 怪兽 历史 创作 收藏 穿越 哥斯拉 Book No. 75456 keywords: 太棒 续加 通路 捧场 书写 犒劳 精彩 红票 猥琐 支持 加油 金丹 投票 修士 神玄 主角 书评 妖兽 功法 流氓 Book No. 81135 keywords: 蛤蟆 太棒 续加 红票 捧场 犒劳 书写 精彩 加油 动力 烂尾 支持 法宝 潜力 仙侠 宇宙 造化 宣传 蜀山 赤城 Book No. 89994 keywords: 火星 邪龙 网游 捧场 书写 太棒 红票 续加 顶顶顶 书评 修罗 犒劳 精彩 加油 支持 尼玛 投票 命运 主角 晚安 Book No. 135095 keywords: 顶顶顶 独孤 书写 捧场 续加 主角 太棒 红票 加油 支持 犒劳 精彩 节操 投票 黄易 收藏 破碎 情节 太监 剧情 Book No. 158342 keywords: 书写 捧场 支持 投票 创作 加油 球员 球队 续加 太棒 红票 皇马 主角 赛季 冠军 欧冠 收藏 犒劳 郭怒 精彩

[masterproject]

Improvement of Keyword Extraction

The previous keyword extraction results include a number of meaningless nouns. I added a stop word list to exclude these nouns. The improved keyword results is:

Book No. 37127 keywords: 身体 佛道 修行 苦痴 护法 苦痴师 女鬼 阴阳 阳身 尸体 玩意 体内 师父 生气 乌龙 教授 魔王 清明 土狗 眉头 Book No. 37853 keywords: 身体 老太爷 银子 灵气 尸体 修行 眉头 鬼童 攻击 迷山 鬼王 臭子 体内 父亲 老爷子 法术 糊涂 客气 气息 尸卫 Book No. 38250 keywords: 书记 身子 县长 转头 领导 省委 干部 市长 办公室 西山 市委 车子 漂亮 秘书长 挂断 俏脸 材料 办公桌 纪委 卧室 Book No. 39813 keywords: 身体 华夏 耶稣 龙王 燕京 养心 老爷子 皇帝 军师 保镖 答应 车子 公民 仇家 拒绝 黑色 关系 学生 解释 杰克逊 Book No. 41365 keywords: 修罗 轩辕 身体 哥哥 玩家 攻击 魔法 生命 世界 华夏 轩辕剑 轮回 属性 力量 职业 全身 混沌 消失 速度 帮派 Book No. 44202 keywords: 淮东 崇州 奢家 兵马 江东 左军 淮东军 精锐 流民 兵力 东军 战船 骑兵 战事 淮西 青河 徐州 势力 河口 高丽 Book No. 45669 keywords: 灵力 修炼 符阵 神力 师兄 神识 妖术 门派 云海 魔兵 战部 法宝 剑意 弟子 实力 魔族 炼制 力量 剑诀 法诀 Book No. 48552 keywords: 神通 修炼 高手 弟子 秘境 力量 神族 身体 金仙 元气 浮屠 仙器 纪元 八部 法力 法宝 黄泉 炼化 境界 道器 Book No. 69507 keywords: 王安石 秦州 交趾 遵裕 京城 西夏 宰相 周南 邕州 西贼 木征 司马光 官员 功劳 宋军 进士 辽国 转运 消息 吕公著 Book No. 75456 keywords: 修士 妖兽 金丹 法宝 绿袍 灵石 神玄 真元 神识 元气 宗主 老祖 法器 道友 阴丽 气息 虚空 白色 法阵 妖丹 Book No. 81135 keywords: 法力 修炼 法术 贼头 祭炼 金刚 炼气 法器 玄黄 顾龟 太极图 魔族 龙诀 火鸦 法宝 真火 青城派 级数 弟子 太化 Book No. 89994 keywords: 叶邪 身体 玩家 力量 命运 攻击 迷失 生命 龙魂 世界 魔法 魔神 属性 司徒 哥哥 职业 试炼 伤害 鬼王 消失 Book No. 135095 keywords: 独孤 高手 真气 武功 剑气 剑法 宗师 姐姐 魔门 长生 境界 邪帝 原著 长剑 江湖 牧场 前世 玉真 闻言 气势 Book No. 158342 keywords: 球员 赛季 球队 塞罗 皇马 进球 进攻 冠军杯 场赛 防守 联赛 球迷 禁区 半场 皮球 出色 西甲 球场 前锋 主场

As we can see, the new list of keywords give readers clues about the content of the story. I can feed this result to existing visualizations.

[masterproject]

Keyword Extraction for The Story Content

Using the Python Jieba library, I extract the 10 frequently appearing keywords from the story content. Below is my results:

dyn-209-2-236-101:Master Project puconghan$ python read_content_keywords.py Building Trie... loading model from cache loading model cost  1.34921598434 seconds. Trie has been built succesfully. Book No. 37127 keywords: 老鬼 胡琳琳 王勇 亮子 胡琳 张名 萧鬼 林阳 29 阳听 Book No. 37853 keywords: 苏扬 扎索 淘淘 唐菲菲 苏哥 身体 老太爷 银子 灵气 露出 Book No. 38250 keywords: 思宇 地道 书记 笑笑 倩影 瑶瑶 李青 柳媚 燕妮 微微 Book No. 39813 keywords: 秦洛 倾城 九九 闻牧 医生 身体 华夏 耶稣 龙王 千重 Book No. 41365 keywords: 修罗 风瑶 轩辕 身体 哥哥 玩家 宝宝 攻击 技能 魔法 Book No. 44202 keywords: 林缚 淮东 江宁 崇州 奢家 兵马 顾悟 江东 左军 董原 Book No. 45669 keywords: 左莫 修者 灵力 修炼 晶石 韦胜 符阵 神力 公孙 师兄 Book No. 48552 keywords: 方寒 神通 羽化 居然 修炼 高手 弟子 秘境 力量 神族 Book No. 69507 keywords: 韩冈 赵顼 王安石 王韶 王舜臣 吕惠卿 玉昆 王厚 秦州 交趾 Book No. 75456 keywords: 魏索 修士 妖兽 金丹 灵珑 法宝 韩薇 绿袍 威能 灵石 Book No. 81135 keywords: 陈七 法力 修炼 法术 祭炼 贼头 金刚 炼气 法器 玄黄 Book No. 89994 keywords: 叶邪 身体 苏菲 玩家 力量 命运 果果 攻击 迷失 左破军 Book No. 135095 keywords: 独孤 寇仲 徐子陵 尚秀 婠婠 宋玉 商秀 高手 真气 晴雪 Book No. 158342 keywords: 马洛卡 球员 赛季 球队 塞罗 皇马 进攻 进球 埃托奥 冠军杯

As we can see, most of the keywords are the name of the main characters. These number of frequently appearing keywords do not contain meaningful stories about the platform or the story. I will try to apply the gensim library to extract keyword with low TF-IDF score (frequently appear in one chapter but not frequently appear in all chapters).

The source file saves all chapters into one txt file. I have to find a way to parse these chapters and save as separate files for the gensim Topic Modeling analyses.