Natural Language Processing
After a hectic couple of weeks I may have been neglecting this project - that is no more and I am right back on top of it with more interesting progress:
So after the shambles that was grabbing all the data and adding it into a .csv file I now have a well structured data set and with only 100 entries it should be alright to start testing natural language parsing. The small data set is mainly to help with quick testing as I go to ensure that it is as accurate as possible.
With the columns as follows (ID of recipe, recipe name, number of ingredients, list of ingredients, rating for recipe, number of raters) this is the output in .csv format at the moment:
Looking through the initial numbers in the raters column I noticed I was collecting the incorrect numbers - originally I had a lot of 11′s in that column. This was due to the numbers I was pulling being number of 1*, 2* etc... I changed this and it now works correctly - being able to look at the numbers in detail and double check them really helped.
Originally I decided on a word-by-word method to split the ingredients from the measurements and other ingredients. This method would split the ingredients into different words and then count the number of them throughout the recipes to find the most common. Once this was finished I was then planning on cross-referencing the x-most ingredients used with a dictionary list of food types. This was after removing any odd characters from the mix.
I actually got most of this working until I realised I wasn’t going to likely find a list of all foods readily available and this still wouldn’t solve my issue of things that are likely the same sort of food but are slightly differently spelled or branded.
Here’s what the output looked like (I likely should have removed numbers also from the mix - an afterthought):
I realised this probably wasn’t the best way to go ahead with word processing so after a lot of research I decided to instead split the words, keep them in their own recipe array of ingredients and then use something called synset - this is a part of the Natural Language Tool Kit (NLTK). I did take some things that I learnt from the previous method, such as removing numbers and any funky characters. This was done via regex to remove everything except for the lower and upper case characters. I then made everything lower case to ensure easy matching later on in the process.
One of the main reasons I chose to use python was for its word processing library NLTK that allows me to split up ingredients and compare them. I went on to download this and install all of the packages that looked relevant to ensure that it would run smoothly. One package that was suggested to me by another person working on the same sort of project as me was the wordnet package.
So I got this sort of working - I’m currently using just 3 recipes in order to test thoroughly as I go through. I split the ingredients into words and then check them against the word “cheese” at the moment, and to be honest it sort of worked... Until I hit this error:
The word stemming is an issue due to a dictionary/thesaurus usually only holding the present tense or a version of the word. So then I got to thinking... I have used word stemmers before at my last job where we did sentiment analysis on in game chat. SNOWBALL! Guess what - it gets better, I had already downloaded it via NLTK earlier (go past me).
The it occurred to me I had been kinda silly because stemmed words may be great but they turn “cheese” into “chees” which isn’t in the dictionary... The stemming will definitely come in use later when grouping ingredients but for the moment I need to find a new way to fix the issue with finding lemmas. Now to move all that code into a new file so I don’t have to rewrite it again...
I got this working so instead of crashing it will just add the ingredient to a list of other ingredients that can’t be cross-referenced. Looking through these most don’t seem that important (as seen below):
Given this is only a small sample I can’t fully say that this is okay, and there are things such as peppers that would be important. Most of these items seem to be irrelevant or cooking preparation guidelines/measurements - such as chopped or cups. Luckily I don’t actually need these as a part of my project, if I were to look more into this in the future (with a much larger timeframe) I would consider adding preparation and the measurements of ingredients to the algorithm. Then again if I had all the time in the world I would also be looking at the actual comments on the recipes in order to gain a much broader idea of sentiment.
One way I can think of to possibly run these items through the similarity process again via a stemmed version but this would cause problems with the peppers as it would clash with pepper (both very different things, thank you so much English language!!).
I left it as it was and used a 5000 word sample with the previously created count words method to see how many of each word it struggled to place.
There are a lot of words in there I really should use and definitely need to have within my project - a scrape through these is definitely necessary in order to pick out raisin instead of raisins and walnut instead of walnuts... Weighing up the costs of a few ingredients: I think this may be the only way to get the vast majority of these recipes correct.
If I run through these after stemming them this is the result set of those that couldn’t be cross-referenced:
This really did clean it up - but unfortunately stemming doesn’t work that well, cherries has been stemmed to cherri - which obviously isn’t a word and should be cherry. Now a way to sort this out is to possibly use lemmas, instead of using a stemmer I need to use a lemma (say that 3 times fast..). I’ve never actually done this - but hey lets give it a go:
It worked first time - I am officially in love with this library. Instead of just running through these I’ll add the lemmatization to the original run through as it just makes everything singular and awesome.
Thats my progress for today - thanks for reading, I will continue this on Monday as tomorrow I am going to pamper my Mother for Mothers Day :)
