Musings Nondescript

[laymanstrange]

Data Retrieval

1. Get list of indices/ tickers for which data retrieval will work Q. Is there a way to automate this as well?

2. Pull data for all the above from yahoo fin and store into a database Q. Which database? 3. Incorporate basic filters (to be defined)

3. Update recent data daily in the database for the above in the database

[laymanstrange]

Rationality & Death

As once said a wise man of our age, Mr. Joey Tribbiani, “when you are dead, you are gone, you are worm’s food”. Aren’t we so busy with our community, iphones and ipads that we take them for truth and certainty. Apologetically, death is the only truth. There is a fair chance that one might be born or one might not be - it’s all probability. But once born, there is no other certainty than death. We may chose to become a scientist or a doctor, an airline pilot or a policeman and even the president of a country, but what is not afforded to us is our choice over death. 

Interestingly, this sets us apart in terms of what we do here, how we do it and why we do it. I strongly feel that once we see things from the vantage point of death as being the real truth and nothing else, everything is vanity. 

Surprisingly enough, humans are supposed to be rational in decision making, aren’t we! It’s safe to say that I am not straying away too much from the common wisdom when I say that rationality is one thing which sets us apart from other animals. All other species might be concerned unswervingly about food and survival, we however in numerous ways have mastered the art of managing those, haven’t we! Ergo, the only thing left to measure the human progress is to measure our collective rationality, having conquered most other aspects species concern themselves with. Ironically, neither have we been successful in conquering basic needs for our masses with several millions going hungry everyday nor have we mastered being rational - just watch the news for 10 mins. 

On observation though, death seems to serve me as a common denominator for the way we ought to move towards rationality. The way we think about it, the way we feel about it, the need to embrace it and see everything other than rational as vanity is what might make us humans. As a matter of thought, these achievements are exercise in futility - given that we are not even able to display the innate behaviour which we are born with - rationality.

[laymanstrange]

Trend Prediction

Create a fashion corpus. Fashion corpus would be roughly created using Wikipedia articles, RSS feeds from fashion magazines which will require scraping in some form, etc. The whole superset would have to be cleaned up in a big way, possibly using NLTK python libraries (Natual Language Processing) to remove stopwords, repetitions, and more. 

List of sources:

1. Wikipedia - explore metadata to pinpoint/ download fashion specific content. Apparently Wiki has structured downloadable data available at https://en.wikipedia.org/wiki/Wikipedia:Database_download but the question to answer is how the 13 gigs of data would be sorted and the relevant data associated with fashion would be downloaded. It’s a somewhat stupid approach to download all 13 gigs and then use stopwords to find fashion content. Wondering on the approach. Shoutout to #Wikipedia though for an amazingly rich content they are making freely available. 

Interestingly, Wikipedia itself provides an API wrapper in python for querying Wiki database. However, that didn’t seem useful. Here is what we are at: using Special Export downloads articles but not inside subcategories and using PetScan gives a list of all articles inside categories and its subcategories depending on “depth” input. The workaround is to get a list of all articles inside a category re: Fashion using PetScan, then download all articles using Special Export, following which we would be working with Gensim probably to do a document analysis and discard articles which are in terms of tolerance level remote to fashion.   

So, summarily after some trials and lot of errors, I have come to the conclusion that the best way to get the content from Wiki is to use the Wikipedia API calls which doesn’t take markups into consideration and renders the output in a plain and simple readable English format. The only issue remains is the number of API calls and rate limits associated with it. Since there is no strict limit mentioned by Wikipedia folks, I am using time.sleep with a uniform distribution of seconds of code sleep. For now it seems there is no other choice especially given that wikipedia api wrapper doesn’t come with good documentation touching on the API calls rate limitation, as opposed to Tweepy documentations for Twitter API call. For now however the API calls are working fine and the expectation is to download all articles associated with Fashion (used Index of fashion related articles here rather than the Categories as described earlier).

Snippet of code: (mostly brute and might be a genuine programmer’s nightmare)

import wikipedia import os import time from requests import exceptions from random import uniform input_file = open('article_list.txt', 'r') output_file = open('article_content.txt', 'a') for line in input_file:    # time.sleep(uniform(1, 3))    try:        article = wikipedia.page(line)        print article.title        output_file.write(article.title.encode('utf-8'))        output_file.write('\n')        time.sleep(uniform(2, 4))        output_file.write(article.content.encode('utf-8'))        output_file.write('\n')    except:        exceptions.ConnectionError        try:            time.sleep(uniform(3, 5))            article = wikipedia.page(line)            print article.title            output_file.write(article.title.encode('utf-8'))            output_file.write('\n')            time.sleep(uniform(1, 2))            output_file.write(article.content.encode('utf-8'))            output_file.write('\n')        except:            exceptions.ConnectionError            time.sleep(uniform(4, 6))            article = wikipedia.page(line)            print article.title            output_file.write(article.title.encode('utf-8'))            output_file.write('\n')            time.sleep(uniform(2, 3))            output_file.write(article.content.encode('utf-8'))            output_file.write('\n')            continue        continue    # output_file.write(article.title)    # output_file.write(wikipedia.summary(line,sentences=1)    # output_file.write(article.content) output_file.close() print line os.system('say "your program has finished"')

And yes, everything on the community edition of PyCharm 

Error catching: 1. HTTP Connection errors - owing to rate limitations on API

Error catching: 2. DisambiguationError - owing to multiple meanings of one article title

Have been bypassing both of these for now. And till now we have a collection of 700 articles and ongoing. 

Update on 4th Dec: Corpus of around 4k documents created out of wikipedia. Data pre-processing using nltk toolkit and bit of regex. 

LDA - gensim package (ldamodel.py tweaked)

Source Code

import glob import nltk import chardet from collections import defaultdict import logging from lda_model import LdaModel import gensim import pyLDAvis.gensim import warnings warnings.filterwarnings('ignore') # basic logging comments logging.basicConfig(format='%(asctime)s : %(levelname)s : %(message)s',

level=logging.INFO) # setting folder for file retrieval folder = '/Users/abhinavprakash/PycharmProjects/Data/new_texts/' # detecting encoding def encoding_detection(rawtext):    return chardet.detect(rawtext).values()[1] # POS tagging and removal of other parts of speech. Making sm def pos_tagging(tokens):    return [item[0].lower() for item in nltk.pos_tag(tokens)

if item[1] in ('FW', 'NN', 'NNS', 'NNP', 'NNPS')] raw_list = [] for filename in glob.glob(folder+'*'):    coding = encoding_detection(open(filename).read())    raw_list.append(nltk.word_tokenize(open(filename).read()

.decode(coding))) # raw_list = [nltk.word_tokenize(open(filename).read()

.decode('utf-8')) for filename in glob.glob(folder+'*')] print raw_list[0:10] raw_list_POS = [pos_tagging(row) for row in raw_list] print raw_list_POS[0:10] total_length = len(raw_list_POS) frequency = defaultdict(float) for texts in raw_list_POS:    for text in set(texts):        frequency[text] += 1 print frequency tokens = [[token for token in text if 1.0 < frequency[token]

< 0.4*total_length] for text in raw_list_POS] tokens = [[token for token in text if token not in           ('=', '==', '===', '%', ']', '[', '-', 'c. ',

'>', '<', 'ii', 'iii' '====', 'women', 'designer',

'men', 'century','woman', 'man', 'centuries', 'mm',

'product', 'products', 'i', 'g.', "i'll", "you're",

"it's", "we've",'"i', 'ms.', 'mr.', 'm43', 'id',

'c/o', 'd.getelementbyid', 'window.__stp', "'https",

'//widgets.rewardstyle.com/js/shopthepost.js',            'window.__stp.init', 'd.location', "'object",

'd.createelement', '/^http', 'shopthepost-script',            'javascript', "that's", "don't", '|', 'x', 'k', '=====',

"i'm", "i've", "'shopthepost-script")]           for text in tokens] print 'Tokens below:' print tokens[0:10] # creating and saving BoW representation dictionary = gensim.corpora.Dictionary(tokens) dictionary.save('token_dict_BoW.dict') print 'Dictionary saved' corpus = [dictionary.doc2bow(text) for text in tokens] gensim.corpora.MmCorpus.serialize('token_BoW.mm', corpus) print 'Corpus saved' # loading BoW representation id2word = gensim.corpora.Dictionary.load('token_dict_BoW.dict') mm = gensim.corpora.MmCorpus('token_BoW.mm') # running lda lda1 = LdaModel(corpus=mm, iterations=5000,

id2word=id2word, num_topics=50, update_every=1, chunksize=500, passes=14) print 'LDA run' # lda.print_topics(20) data1 = pyLDAvis.gensim.prepare(lda1, corpus, dictionary) pyLDAvis.show(data1)

Results (first cut, needs optimization - don’t kill me)