shields of dreams @shieldsofdreams - Tumblr Blog

A (Really Brief) Intro to Natural Language Processing

Hacker School: Week 10, Day 2

It's been far too long since the last post. I promised my check-in group — our daily time for talking about what we're doing here — every day last week that I would write a blog post. I didn't. But here it is, in all its glory.

What is Natural Language Processing?

My brief sojourn into working with words — specifically, the Academy Award speeches — sparked an interest in all of the possibilities of combining programming and language. Everything I read and heard about this idea led me back to natural language processing. So, as is my custom, I went straight to good old Wikipedia.

"Natural language processing (NLP) is a field of computer science, artificial intelligence, and linguistics concerned with the interactions between computers and human (natural) languages."

So NLP combines a few things — CS, AI, linguistics — but ultimately concerns how computers work with natural languages (aka our human languages). Don't worry, it doesn't take a background in CS, AI, or linguistics to get the basics.

Why Do I Care?

I care because NLP has a huge number of potential applications — many of which are challenging, fascinating, and (eventually) world-changing (if computers could actually understand anything — anything — we ask them). But more immediately, I care because NLP is actually tangible and accessible, even for beginners. Much of what I've learned in programming is conceptual, or mathematical, or involves a large number of moving pieces. NLP (at its most basic) is taking organized text — a thing I already understand and can see — and manipulating and analyzing it in a way that I could talk about with anyone who has read or written text.

For this (again, really brief) intro, I'm going to use a Python library called Natural Language Toolkit (NLTK). I'll try to assume as little knowledge as possible, but a basic understanding of how to write Python and interact with the terminal will definitely be helpful. I'm essentially following along with the start of this book (or this version if you like Python 3). Just the first section of the Preface gave me a much better understanding of why NLP is important and uniquely challenging. It's written by the creators of NLTK, and gives a much more thorough and understandable intro to using the NLTK library than I ever could. I'll use two of their intro examples to show what you could do right away.

Getting Set Up

I'll assume you already have Python and some kind of interactive terminal (if you have a Mac, both should already be there). If you're completely new to Python, but are interested in learning about NLP in Python (or in general), the book I referenced before does a much better job than I ever could easing a newcomer into both Python and NLP/NLTK.

In fact, I'll just point you to this section for getting set up with NLTK (just those few paragraphs up to 'Searching Text.') If you have some experience with pip and/or virtual environments, it can make installing NLTK easier (it's what I used).

Once you have your terminal open, here's what you'll type:

$ python

This should open the Python interactive interpreter (or REPL). You can type in Python code and the results of the code will appear immediately. I find it's a great way to try out my own code, or try using a new library, package, or API, to actually see the results of particular code. I also just love typing some code in and getting immediate feedback. If you have NLTK installed, try (any text following ">>>" is Python code you can enter):

>>> from nltk.book import *

This'll import a bunch of books! NLTK has a number of text sources — books, news stories, chat logs (anonymized, don't worry) — to analyze/use in your code. You should see the name of each print as they load. Should look something like this:

Now you can use them!

What Can I Even Do?

Try this:

text1 — the full text of Moby Dick — is what you just loaded from NLTK above, and is now accessible/usable by Python. More specifically, it is an object of type 'Text' — this is specific to NLTK and allows you to perform a bunch of cool operations on it. In fact, try this:

>>> dir(text1)

It should look like this:

dir() is a really helpful Python command — it effectively shows all of the attributes associated with the object you put in the parentheses (important to note that everything in Python is an object, so you can use dir() with everything — even dir() itself). Put more simply, it's everything you can do or access for a given object. It might seem overwhelming, but for now I'll ignore everything with trailing and/or leading underscores (these are for the most part standard Python operators) and focus on the other stuff, specific to Text objects in NLTK (like 'collocations', 'plot', and the one we're about to use).

Searching Text

Promised we would, so let's search the text!

>>> text1.concordance('whale')

concordance is one of the actions (aka methods) of Text objects (it's just to. So you have a text object — text1 — call a method on it — .concordance() — and put some argument in the parentheses — 'whale'. The argument is a string you want to search for in the text. The result of this method is what you see above — all of the places in the text with the word 'whale' along with its context (all of the words surrounding it). It's limited to displaying the top 25 (of 1226 — unsurprisingly, lots of whale talk in Moby Dick). This is kind of amazing! It's probably one of the simplest features of NLTK, but I can imagine a ton of applications — not the least of which is it just looks cool: you could make a web app just to do this and show the context for any word in some text!

Sidenote: 'Concordance’ comes from publishing, and refers to exactly what we did above (list words in a text with their contexts). Its Wikipedia entry highlights what makes NLTK, and computing in general, so amazing: ‘Because of the time, difficulty, and expense involved in creating a concordance in the pre-computer era, only works of special importance, such as the Vedas, Bible, Qur'an or the works of Shakespeare or classical Latin and Greek authors, had concordances prepared for them.’ Now we can do it with one command! That’s just the best.

Finding Similar Words

Let's try another cool method — similar:

>>> text1.similar('whale')

According to the NLTK book, similar finds 'other words [that] appear in a similar range of contexts' as the given word. So 'ship' is found in more contexts with 'whale' than any other word — because of this I can reasonably conclude that the two words are similar in Moby Dick. Already I can see how I'd apply this elsewhere — e.g., I could find words similar to a particular topic, which could form the basis for more complex sentiment analysis. Or, again, I could just make a fun web app that allows you to type in a word and get similar words in a body of text.

This obviously barely scratches the surface of natural language processing and the Natural Language Toolkit, but hopefully gets you a little interested. I'd love to write more about some of the other features/possibilities, and maybe even walk through a project I'll create with it (though I have to do it first). I'm sure I'll have more to say soon.

Resources:

NLTK book: The better-written basis for this post. Go try it out!

NLTK: The home for the Natural Language Toolkit. Has some simple/cool examples and additional resources.

Thank The Academy

Hacker School: Week 7, Day 4

Big gap between writing.

I've made pretty steady progress on two projects - one the web framework I've been writing, and one a web app (try it out!). I'm not really finished either - one of the major problems I've noticed in programming is the eternal question I have, "is it finished?" - but the web app is simpler to understand, more fun, and super relevant for the Oscars! (or irrelevant if you don't like the Oscars).

Last week, I found this, a database for Academy Award acceptance speeches. I decided I had to use it. So I first got all of the data together - using Python to gather the data (I'll leave out the details, but they involve regular expression matching, parsing HTML, and, depressingly, copying and pasting - the site is not the best). And then I built a web app - one with questionable styling, but which does two different, real things!!! - on top of the data! And you can use it!

First, it randomly generates an Oscar thank you speech (like above) based on all speeches ever given. It does this through a concept called a Markov chain - the basic idea is that you take a body of text (here, all speeches) and break it down into sequences of words of a specific number (here, 3 words - but that's arbitrary) called an n-gram. So a 3-word n-gram could be "Thank you to" or "cast and crew" or "Tommy Lee Jones." An algorithm (that another Hacker Schooler wrote - thanks again!!) then collects every unique n-gram in all of the text, and matches/maps each one to a list of all of the words that follow it. So "Thank you to" would be mapped to "the", "God", "my", and anything else that followed that n-gram in the text. Once that's done, you can generate a seemingly random block of text - except it's only random in that it randomly chooses the words based on that previous mapping. So "Thank you to" might be followed by "the", "God", or "my" - but never by something it wasn't mapped to. It's much more complex than that, and I'm glossing over specifics, but see it for yourself - it'll "randomly" generate a thank you speech for you, one that looks almost real because of the Markov chain being used, and you'll almost definitely never see the same one twice.

Second, I did a bit of analysis on the length of the speeches. There are two interactive visualizations - one for the length of every single speech since 1966 (data before that is a bit spotty) and one for the average per year. You can see (above) that, on average, speeches are definitely getting longer, at least by number of words - especially when you include Honorary speeches (those are getting exactly as long as you think - aka REALLY LONG - calm down, D.A. Pennebaker).

You can also see how long individual speeches are, and filter for particular categories. The one above is Best Picture - if you hover over any of the bars you'll get all of the associated info - for Titanic. (Shock, they went super long too.) You can even click on "Speech (click to show)" at the bottom to actually see the speech, or be taken directly to the Oscar website entry for that speech (if you're tryna watch that D.A. Pennebaker speech - hint: it's so long they don't even have the full video).

Overall, this has been a really fun project - but also somewhat frustrating/against what I know is best for my learning. I spent too much time on styling, on data cleaning, on interactivity, and ultimately feel like I could've learned more by diving into one particular thing. What's more, I know this hasn't been great in that I worked alone on it - Hacker School is especially good as I am surrounded by other programmers, and they're all willing to work together on really cool projects. I somehow keep finding myself drifting back to working on my own. Ultimately, it makes for projects I want to work on - usually movie-related, clearly - but also leads to me getting lost in my own head, going down rabbit holes, and not focusing on what would be best for learning. I'd like to make a concerted effort in these last few weeks to work with others on projects - even if they're not exactly what I want to work on.

Until then, happy Oscars!!!! (if you don't celebrate, enjoy the weekend, I suppose)

Fundamentals

(Hacker School: Week 5, Day 5. Fair warning, this one's pretty technical.)

Here I am, it's the fifth day of the fifth week of Hacker School. I haven't written a post in a week - out of laziness, maybe; out of a desire to code instead of write, oh most definitely1. That's been the value of this week - writing code on my own (as opposed to from tutorials or books).

Damn Hard

A friend sent me a great post the other day that encapsulates much of what has been so strange, and newly frustrating, about learning how to program (mostly) on my own. It's worth a read, especially if you've been learning on your own - you'll just nod your head a lot - but the author's point is that learning to code on your own is hard (damn hard). It's probably not unique in that respect (I can't speak to other disciplines); but, the author speaks well to the particular difficulties I've encountered so far. In the beginning, my learning was certainly characterized by an initial rush of understanding - I know how to write Python! - and a subsequent drop of confidence - I have no idea what to do with Python! It's an empty feeling - I know something, but I don't know what to do with that knowledge. And it began to erode my confidence.

So in my time off between work and Hacker School, I learned how to build apps/sites in Django and in Meteor - big web frameworks that aren't exactly beginner-friendly. I shot for the moon, and missed, but landed in like, George Clooney territory, not among the stars.2 That is, I got something to work! And had no idea how it worked.

Is that useful - to get something done, have something to show, but not know how it happened? Hacker School doesn't think so - and I don't either. I think it's much more important to understand something - especially something as complex as a web app - from the ground up.3

The Fundamentals

And that's what I've been focusing on here - learning why and how something works. But it's time-consuming, often confusing, and there's not always something visible to show for it. So this week has been especially interesting in being mostly spent on two very different projects.

The first project is a small web framework, written in Python. I've been working alongside a fellow Hacker Schooler, Paul, and we've been building similar but separate frameworks. We spent much of the week digging as deeply as possible into how to build a web application in Python from scratch. And we've learned a lot - about how the internet works, how and why simple web frameworks (like Bottle and Flask) do what they do, and how to handle seemingly simple things like HTTP requests. It's fascinating, fun, and often frustrating to build something from scratch - it's helped me to appreciate the people who have spent their time making sure 99.5% of us never have to think about things like HTTP requests or URLs. But at the same time, here's my visible progress thus far:

But is visible progress even important? Would I learn more if I actually built something - something that does more than allows me to type in my name and favorite movie and shows a lame, code-esque representation of exactly what I typed in?

Overcoming 'Fear'5

My second project came to me late Wednesday afternoon. I've been wanting to use the Rotten Tomatoes API - Rotten Tomatoes is a movie review aggregation site, so naturally I wanted to use it - but had been legitimately scared off because I couldn't conceptualize how it would work. Most simply, I was scared of the word (acronym) API. Programmers around me have used 'API' to describe so many seemingly unrelated things that it became almost impossible for me to wrap my head around what it meant concretely. But in keeping with the spirit of Hacker School, I was determined to confront that fear. Here's a great passage from HS's manual (linked above):

Another obstacle we try to remove is fear. We think this is one of the most pernicious impediments to education. In most of the world, but especially school and work, people are afraid of looking stupid. This fear frequently keeps us from asking important questions like "how does that work?" or even just "why?" Worse, it keeps us from saying "I don't understand." That means many of us muddle on with a half-baked or entirely incorrect understanding of core concepts. This is particularly bad with programming, because these misunderstandings compound, and over time become harder and more embarrassing to admit to and address.

So I pressed forward to find out how it worked. And as it turned out, there was nothing to be scared of - it entirely depends on the API, but the one given by Rotten Tomatoes is easy to use if you have some understanding of URLs and working with JSON data (if you don't, that's ok too! It takes some time to learn but is at least pretty intuitive). The basic concept is you, as a developer, get a key from them - this tells them you're not a robot, essentially - and then you can access a certain amount of data in a predefined format (JSON) using predefined queries/parameters - like a movie you want to search for, or a specific data set like upcoming movies (in the form of URLs). And then you can do what you want with that data. So here's what I did!

I made a command line interface! It's not done yet - you can't actually access all of those things (well, you can't access any of them yet, only I can) - but I've made good progress. It's less cool-looking than a fully-functional web app - but also a lot easier to make, and thus a lot less frustrating. It was an easy win for me, got me motivated about building something on my own, and was fun to do - all a great contrast to something more esoteric like building a web framework.

More importantly, let's search for a movie - 'batman'!!!

I don't care about The Dark Knight6 - let's get more info on Batman & Robin!

So 'more info' just means small snippets of reviews for now - but it's something. And hey, you can easily see who hated and loved Batman & Robin! (Answer: everyone hated it except the Times, inexplicably. And 'wounded yeti' could stand to be part of the lexicon over, like, selfie.)

In conclusion: it's a valuable struggle to build something from scratch - I for one have learned a lot so far from the experience of building a web framework. At the same time, I'd argue that it's extremely important when learning any skill - but especially something as broad and potentially overwhelming as programming - to work on small, doable, motivating projects applying some small portion of what's been learned so far. These allow for a mental breather - I don't have to learn a whole new concept - and help to actually solidify learning in a meaningful, lasting way. It was rejuvenating to build something quickly, easily, and with showable results - and in color! What a week.

1. If by some unfortunate accident you haven't watched Arrested Development do yourself a favor and stop reading this right now, come back in 25 hours when you're done, thanks.

2. I really wanted to clumsily work that overly inspirational quote in there somehow.

3. Bold move, plugging your own blog in your blog.

4. No, I didn't watch The Social Network and get really inspired to go and learn programming. The Zuck of that movie is lonely and sad and Citizen-Kane-esque and I don't want to emulate it. It's just a great movie by a great director who can't stop making great movies. (And I just like programming.)

5. Not this Fear.

6. Lie.

From the Ground Up

(Hacker School: Week 4, Day 5)

Like I said at the beginning of the week, it's really hard to write something every day. It's been most difficult this week, when I've felt like I've made steady progress, but have very little to tangibly show for it.

Though it's more difficult to put into words or show an example, this week has been especially good for two things: deliberate practice and understanding a topic from the ground up.

Deliberate practice is the idea - introduced by a group of researchers ~20 years ago, and since popularized in a bunch of books (Outliers, mainly, but also in Make It Stick) - that those who become experts in a particular area do so less because of innate ability and more due to their specifically-structured practice over a long period of time (that's the 10,000 hour rule). Generally, it takes the form of picking a specific part of a topic a bit beyond one's reach - but building upon previous experience - practicing that part, getting targeted feedback on one's performance, and repeating the process, over and over. It's a step-by-step process - each thing one practices builds upon the last and strengthens overall skill in the area (so the theory goes). Though there's some dissent with the idea that it explains expert performance, I still think it can be applied advantageously to my own learning. I'm not an expert in programming by any means - not even close, and may never be - but reflecting upon what I've learned, looking ahead and structuring my next step to be a bit harder than the last, and practicing until I've fully grasped that topic, has been more valuable to my progress than anything so far. It just wasn't until reading Make It Stick (and at least one blog post) that I put a name to it.

That directly relates to the other part - understanding something from the ground up. Though I like knowing the big picture - i.e., can be a global learner (as Mel Chua highlighted to us in our first week here) - day-to-day I'm usually much more concerned with learning something step by step - i.e., sequentially. I get easily frustrated if I'm going through a tutorial or book or class and realize there are 25 things that the author or professor assumes I already know. This single thing - the assumption of knowledge, and the overwhelming amount of assumed knowledge - has been my biggest frustration in learning programming and computer science. With a few exceptions, nearly everyone with information about a particular topic (usually, authors of books and tutorials) assumes you come in knowing a, b, and c (and usually d-z and 1-10). (Granted, much of the learning resources are free - and anyway, every single author can't be expected to hold your hand through their topic.)

My lack of knowledge often comes from the obvious place - not having a computer science degree - but it doesn't come from a lack of curiosity. I want to learn these things! It just takes so much time! If I want to build an app, do I really need to learn how the language works, how databases work, what routing is, what HTTP requests are, how to authenticate users, how to keep it secure, how to interact with the front end, how to deploy it to Heroku or Amazon so everyone can actually see it? Hacker School would - I think, rightly - argue that yes, it is more important that I understand fundamentally how each piece works before building something. Do I need to know every single thing in minute detail? Definitely not. But I will be a better programmer - and ultimately a better learner - if I take the time, and yes, the frustration, to learn the fundamentals and build from there.

So to bring it back to deliberate practice, this week I structured my time around starting from the ground up in developing a web app in Python. Doing this ensured I stepped from one level to the next, more or less sequentially - it's also much easier to grasp what I don't know when starting at the very bottom. I learned how to build an app without a framework (it's using something called WSGI - likely a topic for a future post). Then I started to learn why frameworks exist, and why each small piece was added. This was a slow but ultimately rewarding process - by the end of the week I felt much more comfortable using something like Flask - one of the Python web frameworks I've mentioned before - to build an app, than I did before. And I did it by identifying things I didn't yet know, practicing them, and then moving onto the next thing.

I still feel like I didn't do much this week, but I suspect I'll be grateful for the foundation I've been building. Hopefully I can remember everything I had to learn - the other most difficult thing about learning is knowing what to learn next, and the best resource to do it. I'd love a trusted source for sequential learning of particular programming topics or concepts. (There are great ones for CS topics - Coursera has a bunch of great (FREE!!!!) courses from top schools - I'm currently moving between three different Intro courses - the other major frustration is how difficult CS can be to start understanding.)

Some Things I Learned in Week 3 of Hacker School

(Hacker School: Week 4, Day 1)

It's unexpectedly difficult to write a blog post every single day. So, while I'll try to, it's naive of me to think I can do that before/after spending all day learning.

It's hard enough to focus in a directed environment like college or a job; it's that much harder to do it in a completely undirected environment. It takes a lot of willpower to direct my attention toward only a few things in a given day or week and shift my attention away from a million other interesting things.

Because of that effort, I know I'll gradually improve at focusing in this environment, which will only be more helpful in a more directed environment.1

Programming is a complex and involved discipline. When I started learning over a year ago, I definitely did not have enough respect for how much I'd need to learn and how difficult it is. Learning a language is only a tiny piece (and likely the most straightforward) of becoming a programmer, and I know I still have a while to go.

Recursion is an inherently confusing concept, and even when I thought I understood it (at many points in the last year), I realized I really didn't. Last week brought me a little closer.

Hacker School has presentations every Thursday where they encourage anyone and everyone to present whatever it is they're working on or interested in, in any state of completion. It really feels great, and motivating, to present something to a group of people who aren't primed to judge you or your project. It's also much easier for someone like me who is usually nervous to show anyone any of my work - because I'd be afraid they'd think it was simple, or bad, or uninteresting. That's a silly way for me to think - if anyone would be so judgmental about my creating something, however simple, bad, or uninteresting, they're probably not worth my time anyway.

When programming (but I think when doing anything), while it feels good and productive to build something - to display visible progress - it feels equally good, if not better, to go back and refine it, to fix any issues and shore up the foundation, to make it smaller, less complex, and ultimately better grounding on which to build more. I did this with the project I presented - I refactored the code2, made indentation and variable names consistent, and added comments to clarify anything that's confusing about my code. It sounds boring; it felt great.

Reading about not-directly-relevant things often helps me to understand more about what I'm focusing most of my energy on. I talked a little in other posts about Make It Stick - which I'd definitely recommend, especially if you're interested in learning itself - it helped me to structure my time and energy here in a way that engenders better, more lasting learning. And it helped me to be more comfortable with getting frustrated, and feeling lost and incompetent. After I finished that, I started Quiet - a similarly structured book (drawing on equal parts empirical research, anecdotes, and history to make a larger point about the topic) - and it's already helped me to view my introversion and desire to work on my own or spend time thinking about things in a more positive light. Neither have a single thing to say about programming (yet) but both have already supported my time here in a very positive way.

And now I begin my work for the week - which will be completely destroyed by the STORM OF THE CENTURY.

1 This is related to what that book, Make it Stick, had to say about learning - it's only going to stick when it's effortful. I think that's generalizable to other things - like focusing. If it's hard now, and I struggle through it, it'll be easier later, especially in a more structured environment.

2 Pretty much exactly that - refined it, made it smaller, simpler, easier to understand what's going on.

Slow Progress

(Hacker School: Week 3, Day 3)

This is a quick post about today and yesterday - not on any particular topic, and more on my own sluggish and sometimes frustrating progress. This one's definitely HS-specific.

I spent much of yesterday following a fairly involved tutorial in Flask - but only got through the first five pieces. As has often been true for me lately (and seems to be true of any programming/CS topic), learning something really means learning 25 things - and I never know upfront how much I don't know.

The rest of the day, and some of today, I started a (very free!!) online course (aka MOOC) in Algorithms from Stanford. I find that the concepts are complex, but I can wrap my mind around them - but struggle with translating them into code. I'm lucky that I have the time and energy to struggle through problems - and hopefully in that way making them stick (tm) - but I find that process a bit disheartening sometimes. It's a similar struggle, between spending a ton of time understanding every detail and being comfortable moving on from something if it isn't important.

Finally, today I picked up my own project that I'd been working on prior to starting here. I essentially took all of the available data from the IMDb Top 250 - a publicly-voted list of the best films of all time (HIGHLY subjective - and often biased) - since it began in 1996. I'll leave out the specifics of how I did that for now, but I think that could be a later post. I then translated it into an interactive visualization - one where you can easily digest how the public's feelings (through a pretty big and untrustworthy, but best available, proxy) on particular movies have changed over time. I was able to port all of the data I have from Excel/csv to a real database, and got it to display the visualization - but it's over 800,000 rows of data, so it's extremely slow. Hoping to come up with a way to speed it up and make it usable, because I think it'll be pretty cool.

The best parts of the past two days have been working with others - though I've often approached it less positively than I should, because I have so much I want to work on by myself. When I can come to terms with the idea that I'll never finish everything I want, and that I always learn more from working with others than on my own, I'll be a bit more comfortable. But I sat in two really interesting sessions - one by Mary, one of the facilitators here at HS, about the inner workings of Git (it helped in thinking about the seemingly impossible-to-penetrate way that Git actually works); and one in which a group of us met to talk about the Algorithms course so far. Both were enlightening and made me feel less frustrated and stuck in my own progress; I hope to keep working in those kinds of sessions.

More tomorrow.

Some Resources for Flask

(Hacker School: Week 3, Day 1)

Wasn't technically at HS yesterday, but I wanted to give a little more detail on how I learned Flask last week. This is really only interesting if you want to work with Flask, so I guess you've been warned.

Here are my major takeaways from learning Flask:

It's a web framework written in and for Python.1

It's relatively easy to learn - but that's relative to larger frameworks like Django (for Python) or Rails (for Ruby). If you're truly new to web development, there's still a good amount of assumed knowledge/understanding.

It's easy to learn because it does only a few, specific things, and those things don't take too long or too much pre-requisite knowledge to understand.

Here are some great resources for getting started, though they do require a minimal amount of assumed knowledge of intermediate topics in Python, the underlying concepts of web applications (routing, templating, HTTP requests, etc.), and working with databases:

Official Flask documentation: Great if you have some experience in other frameworks/have developed web apps on your own before.

Quickstart (also part of the official Flask docs): Perfect if you just want to get something working, even if you're just starting out. Does assume you've installed it already, which is outlined by them here.

Tutorial (again, official): A little more advanced - they take you step-by-step through building a web app using a database and some templates. More assumed knowledge here, so a bit more confusing if you're just starting out.

The Flask Mega-Tutorial: Written by Miguel Grinberg, who wrote the O'Reilly book on Flask2, a more in-depth tutorial that spends more time getting you oriented, even if you're just getting started. If you have some more time and patience to get into the details and possibilities of using Flask, this is a thorough but straightforward intro.

Jinja2 documentation: One of the two3 pre-installed libraries for Flask is called Jinja2. It allows you to use templating to build dynamic HTML templates that will render using data derived from databases and manipulated using Python. Once you have an understanding of how it works, it's powerful - but until then it can be a little confusing. Here's a simple example.

Using SQLite3 with Flask (again, part of the offish docs): assumes that you know basic SQL and how to work with databases. If you're not familiar, it's worth struggling through this part on your own - and especially trying to understand what each piece of code does. Hopefully I'll write a more detailed post soon that walks through that; it's just a broad and potentially complex topic.

My Flask Tutorial: A former HSer (bev-a-tron on GitHub) wrote what looks to be (and recommended by multiple HSers as) an excellent intro to Flask for people who don't have all of the knowledge the above tutorials tend to assume. I haven't worked through it yet but even the intro makes it clear that the tutorial is truly a tutorial - for people with no web development experience (just some Python experience).

I'm sure there are other good resources - I'll try to add them here if I encounter them - but these are great for just getting started in Flask. A lot of the concepts introduced in Flask also translate easily to other web frameworks - and it's worth knowing what the framework actually does for you, in case you can't rely on it for some reason (e.g., it doesn't do what you want it to).

1. Obvious, yes. But if you don't know what that means, you probably won't be interested in the rest of this. But if you are interested, and you don't know what that means, you just have to learn a bit more - and that's ok too! Or you might just want to struggle through learning this - it'll make it stick, so to speak.

2. O'Reilly is a terrific - and pretty much the definitive - resource for books on programming. I've read a few, and they tend to be easy to read, step-by-step, and comprehensive. (Also, many are free online!)

3. There's another - Werkzeug - that takes care of WSGI/HTTP stuff for Flask. I haven't learned much of it yet - I just know that it allows you not to deal with the lower-level details necessary for building a web app in Python. They also have a tutorial - I haven't worked through it but it seems detailed and actually kinda fun.

Some Things I Learned in Week 2 of Hacker School

(Hacker School: Week 2, Day 4)

Out of laziness/inability to put together a coherent and thoughtful summation of my week, I'll stick to the random list format. (h/t randomlists.com)

What a web framework does. Using Flask as a proxy for understanding the larger point of web frameworks was a good choice - I'd recommend it to anyone looking to start learning web development. It takes a bit of effort to connect the underlying concepts to what Flask does specifically, but it is a good starting point. I'd like to elaborate more in a (more coherent and thoughtful) post on what exactly it does, how to use it to make something simple, and what's difficult about using it for someone unfamiliar with programming or web frameworks.

How to work with a database in the simplest possible way. I had a decent grasp of how to write SQL queries - basically, what you write to get, add, or update data in a database - but had very little understanding of how to set one up, make it work with a web app, or even what one really is. Again, a later, better post.

What actually happens when you visit a website on your browser.1

It's not a good idea to install whatever you want at any time. This is probably good advice in general - but especially for a programmer. In Python, it's considered bad practice to install all but very few packages globally2. Instead, you should create - and consistently use - virtual environments. This is something that always gave me some confusion/fear - the name "virtual environment" in and of itself is not super helpful in understanding what it does - but they basically allow you to install sets of packages specific to a particular project, without having them conflict across projects. Again, a whole blog post.

User authentication is still confusing; I'll return to it later.3

It's a good idea to stick to the specifications (aka specs, a good fancy tech word) you set out for yourself when building a program - no matter how small. You can return to and refine those specs at any time, but without a map, you can easily get lost. Don't be me when coming out of a subway station - orient yourself before just walking in the direction that seems kinda-sorta-yeah-it-must-be-north, and then walk south for like ten minutes, especially in an unfamiliar setting.4

Regular expressions! They are scary and potentially confusing and I'd avoided them for a while. But they are also super cool - and there's a great step-by-step, interactive tutorial for them here.5

How to learn, and how to reflect on how I learn. Already plugged this book, but Make It Stick is really insightful on this topic - though I'm surprised by how common-sensical the author's arguments seem. For example - it's a good idea to weave (or their word, interleave) different topics together over the course of learning a larger topic - rather than the typical chunks of information (like a math textbook, where you learn one topic completely, then move onto the next and forget about the last one). It makes so much sense! I definitely don't learn this way, so I'd like to refine my learning habits to be more in line with what they recommend during my time here.

I'm sure I learned a million other things, but these have stuck with me this week. Looking forward to learning more, and more deeply, next week.6

1 You type in a URL - that's a request - the website's server sends back data that's been formatted in HTML - that's the response - and the browser renders it/makes it look like a website. Or you don't - that's your prerogative.

2 Packages - basically programs written for use with Python (and other programming languages; in some other languages they're called something different). Installing globally - installing in your main/root directory so that you can access those programs at any time with any program you write.

3 Is this a cheating one? Kind of, but I learned that it is still confusing, so that's something!

4 And especially especially outside of the confines of the Manhattan-above-Union-Square grid (or your city's/town's equivalent).

5 If you don't know what a regular expression is - it's essentially a way to search for patterns in text, used in computer science but also in formal language theory (not my words). For example, you want to find all words that have three a's - no more, no less - you can use the regular expression a{3} (I think, would be embarrassing if I have that wrong).

6 ...on Serial.

Making It Stick

(Hacker School: Week 2, Day 3)1

If Tuesday was frustrating in the lack of traditional or measurable productivity, yesterday was gratifying in actually completing defined steps of a project. But the question is, did I actually learn anything?

After my post yesterday, a few gracious Hacker Schoolers (HSers, for brevity's sake) wholeheartedly recommended the book Make It Stick as a way to think about how I learn, and potentially more effective ways of learning than the traditionally understood techniques. I was convinced2 and got the eBook3 and started reading. In the first few pages4, the authors summarize their primary arguments - I'll highlight two (especially relevant) quotes here:

"Rereading and massed practice give rise to feelings of fluency that are taken to be signs of mastery, but for true mastery or durability these strategies are largely a waste of time."

"Learning is deeper and more durable when it's effortful. Learning that's easy is like writing in sand, here today and gone tomorrow."

The first quote effectively captures my first few days here at HS. If you've read my first few posts5, you might be thinking to yourself, "hm, that sure sounds like the authors of this book are describing Jim's first few days at HS as a quote unquote waste of time." And it had the potential to be a waste of time. But last week, I also pair programmed on a small Python project, and in doing that had a - wait for it - effortful experience putting the concepts I'd read about and practiced into action in an unfamiliar situation (another technique the authors advocate). So I learned!

This week, I spent much of the first few days reading the documentation of Flask - a Python web framework that allows you to build small apps - and working through some Flask tutorials. And I felt like I didn't really learn anything. I'd read a lot - and practiced a lot - but it felt like nothing truly stuck. But yesterday, I put what I'd learned into action in an unfamiliar situation - a web app that tracks my to-do list on a daily basis. It's a simple concept, but it seriously solidified my learning - not only did I have to understand the specifics of Flask, but also had to generalize that learning to the concepts that underlie any web framework, and any web application.

And I actually finished some defined steps in building it! So, yesterday was doubly successful in that I had something to show for it - a small app that I can use (and am actually using) - and in building that app, I deepened my learning in a way that's difficult to impossible when learning by rote, or working through tutorials. Hope to continue that progress today.6

1 I wrote an entire blog post, did not save a draft, and lost it in the Great Google Chrome Crash of 2015 (almost as bad as the Twig of '93). Note to future self - save your work. Note to Tumblr - automatic saving, or at least a way to save a draft without closing out of the post, would be nice! (I assume there's a reason it's not there, but man would it have helped here.)

2 I'm easily swayed.

3 In the process, I lost my library eBook of Infinite Jest - a book I borrowed 3+ months ago and now own the hardcopy of - because I turned on the WiFi. Yes, if you want, you can scam the library into never returning eBooks by just turning the WiFi off. Yes, if you do this, you should probably feel a little bad. But I bought it! It's just so heavy to bring on the subway!a

4 I'm a slow reader.

5 Or are me in the future, looking back at how naive I was in the past.

6 Though I've already spent like two hours writing this blog post twice.

a Consider this my plug for Infinite Jest. I'll try to never talk about it again - even mentioning it can sound super pretentious.

How Do I Learn?

(Hacker School: Week 2, Day 2)

In my first week here at HS, I felt like I learned a lot. I finished one textbook, read a lot of another one, and by good old osmosis I knew everything there was to know about Python. I pair programmed - a technique HS advocates as a way to learn alongside others, regardless of experience level - and had a valuable if challenging experience. I even got to talk to a ton of different people to understand their projects, learning goals, and interests.

Now we're in week 2, and that feeling of progress has slowed. It's easy to feel progress as I'm reading - every page I read and understand os a success. As I'm putting what I've learned into action, though, it's easy for me to get frustrated, unfocused, and discouraged - especially when the implementation of what I learned isn't as straightforward as I'd like.

For example - to get a bit technical - on Monday I read all about building an application in Flask. I went through the tutorial - a step-by-step guide to building a small app - and understood each piece as I followed along. Yesterday, I went to put what I'd learned into action - I wanted to build a page that tracks my to-do list.1 But when I went to start over and build a new app, I got frustrated - I didn't remember certain steps, new and unfamiliar steps had to be introduced - and eventually got completely discouraged. Worse (in my mind), I know that building this kind of app (where you have a single page, connect to a database, and read and write from the database through the web framework) is about as easy as it gets in web development - a lot of tutorials and help guides try to emphasize how "simple" it is to build this kind of thing. And yet I couldn't finish it in a day? "You must not be cut out for this, Jim" - me to myself at some sad point yesterday.

So I took a step back. Why am I frustrated and discouraged? It could be a lot of things, but I think it's mostly due to the way I like to learn. Traditional learning environments - grade school, high school, college, work - never gave me this kind of frustration. And why? Because my learning was structured by someone else. Here is a textbook - read this chapter - do this assignment - here's what you did right and wrong (or, for work - here's what you need to know - do it - good/bad job - try again). If I did something wrong, or didn't understand something, there is a clear way to better understand it (read it again, do the exercises again, talk to the teacher, get help from a classmate; ask a coworker, talk to the manager, let someone else do it). When I got a good grade, or a good review, that was always, to me, an affirmation that I understood something - and when I got a bad grade/review it was a signal to go back and try again, with some clear, directed feedback. This kind of step-wise learning - learn one thing, ensure understanding, then learn the next, until those successively more difficult pieces build up to the overall course2 - was, for me, at least easy to understand. I knew where I stood at any time, I knew how much more there was to learn, I knew what was and was not necessary to understand.

HS is pointedly non-traditional. Their about page says it all - "We don't have grades, degrees, classes, or a fixed curriculum." For me, it's much easier to learn with grades, degrees, classes, and a fixed curriculum (or defined roles, set expectations/goals, regular reviews); I like having successively harder steps, with an external reward (grades, degrees, promotions, etc.) for getting to that next step. But I want to actively try to adapt to this new learning style - or, at the very least, mesh it with the way I learn. HS advocates this environment for a lot of reasons - but I think the main one is that extrinsic reward systems don't actually encourage real, lasting learning or engagement with what one is learning. It just makes it a lot harder to start, especially for someone as used to and motivated by the traditional extrinsic reward system as me. So in the coming days, I'd like to get less frustrated and discouraged - and remember that just because I'm not given a grade, or am unable to finish a project, doesn't mean I haven't learned something.3

1 The world needs it.

2 Mel Chua, one of HS's many residents, gave a great talk last week on learning styles - specifically in learning programming, but I think it can be generalized. This kind of learning (step-by-step) leans toward what she called Sequential learning - as opposed to Global learning, a style used to describe people who like to be immersed in the big picture. I am aggressively Sequential (it's a spectrum, but man oh man do I fall on that Sequential side).

3 Some things I HAVE learned in the last two days upon some self-reflection: how HTTP really works, why web frameworks exist, why web frameworks maybe shouldn't exist, how to authenticate users, what hashing is and how to implement it, what WSGI is, how sockets work, what a call stack is, what an ORM is, how to decide when to move on from a project, how to write footnotes.

#hackerschool #learning #programming

A Primer (for me) on Web Frameworks

(Hacker School: Week 2, Day 1)

Yesterday - day 1 of week 2 of HS1 - I learned about web frameworks. Like a lot of other things in programming - object-orientedness, version control, package managers, variables, the Internet, Y2K - "web framework" is a term I've heard a lot, but to which I'd attached little meaning. "Yeah, Django's a web framework, it like makes websites more dynamic and scalable and has these views and is the best" is probably a quote I said to myself at some point in the (not too distant) past, with little understanding of what Django actually does (and if you think Django is this, this, or this, or none of those, that's cool too).

So, yesterday, I set out to change that. And thanks to this terrific, simple, insightful post by Jeff Knupp - I think I did OK. (Much of this will be summarizing his points - and so are mostly just notes for myself to digest them.)

(I'll try to be as clear as I can to people who know all of this and people who don't in most posts - I think it helps me to communicate what these complex concepts are in a concise, easy-to-understand fashion. Most people aren't programmers.)

It really starts with "How does the Internet work?" Let's assume we're just talking about me trying to go to amazon.com on my laptop through Google Chrome.2 My browser - Chrome (or Firefox, or Internet Explorer, or whatever) - is the client. Most simply, anytime I go to a website, the browser/client is requesting data from the website's server. So I type in www.amazon.com and hit Enter - that's the request. The server then responds with data. That data is always, when it gets back to the browser/client, in the form of HTML (and CSS and JavaScript, but that's not important right now). The browser/client then renders that HTML (and CSS and JS) - takes all of that data and turns it into the page (amazon.com) you're used to seeing. Anytime you click on a link, or go to a new site, it goes through that process again.

So if the process is that simple - a browser/client sends a request, as a URL/web address, to the server, the server sends back some HTML, and the browser/client renders the HTML - why would it need to be complicated by a bunch of stuff in the middle3? Well some4 would argue you don't need to5 - but whether or not it's a good idea to use web frameworks, it's good to understand what they are and do, how to use them effectively if at all, and how you'd do what they do without them.

As simply as possible, web frameworks are there to allow you as a programmer to write as little code as possible, primarily for web-specific tasks that are done over and over again. For example, every page on Amazon has the same exact (or extremely similar) header toolbar (where you search for stuff or pick a category). They could write all of the HTML for that bar and copy it for every single page they have - millions, probably - or they could use some kind of template for that header, and attach the stuff under it (all the product info) automatically. Amazon may not use a standard, freely-available web framework, but many frameworks allow you to do similar templating.

Yesterday, I learned Flask6. Flask is one of those web frameworks7 - this one's Python-based. It's a good one to learn, though, because it does as little as possible and it's easy to follow how the creators actually went about writing the code to make it work8. It has the aforementioned templating (implemented using a library/package/whatever-you-want-to-call-it called Jinja2), and takes care of all of the magic that happens when the client requests something from the server9. It also helps with URL routing - using the same example, if you type in anything after amazon.com (like amazon.com/books), the "/books" part is used to determine which data to send back to you. It can get more complex and interesting but that, in a nutshell, is URL routing.

So what can I actually do with it? Yesterday, I got it to show me some pages in HTML, and even connect to a database. But the project I was attempting takes a ton of data, so I'd like to step back today and do something simpler - create a very small but usable web app where I can track my to-do list. Once I'm convinced I know the ins and outs of using Flask - and what it's actually doing for me - I'll move onto something bigger.

Hope that was helpful, future Jim.

1 I like giving some context, for when I inevitably come back here to re-read all of my thoughts like a weirdo.

2 I'm leaving a ton of possibilities out - and also a lot of the evolution of the Internet - but this is the most straightforward example.

3 Web frameworks could be pretty effectively summarized as "a bunch of stuff in the middle."

4 Including last night's HS speaker - r0ml (his preferred moniker)

5 If I was a weaker man, his talk would've effectively invalidated all of my learning for the day.

6 Probably an overstatement. I learned the basics and have an OK grasp of what's going on.

7 Other examples (that I know anything about): Django, Rails, Bottle, Sinatra. Here's a Wikipedia list of all frameworks - there are a lot.

8 Also it has great, straightforward, step-by-step documentation - not always assumed for any kind of software/package/</sup>etc.

9 It involves HTTP - basically the standard set of rules for making those requests - and WSGI - Python's standard set of rules for using Python to interact with the data - both of which are blog posts on their own.

Some Things I Learned in Week 1 at Hacker School

Here's a random assortment of things I learned this week at Hacker School (HS from here on out). You can probably apply some of these to yourself as well (just substitute 'I' for 'you'), but I'm not gonna make that kind of assumption about you. (No, Vice, I'm NOT too old to make small talk, and it's rude to assume that about me. You don't know me.)

I'm not as unknowledgeable as I think, even about something as new to me as programming.

I know more about some things, and less about other things, than other people. 100% of the time. (Example: I knew how to take the NYC subway. Many others did not, but they did know a lot more about programming. We learned from each other!)

Corollary - Because of this, it's okay to ask questions. (Like, what's a corollary? It's "a statement that follows readily from the previous statement." Was it necessary that I use this to make a point? Probably not.)

I am not as smart as I think (or thought, before I started). There is so much more to learn - specifically about programming, but that can probably be generalized too.

Other people are by far the best source of learning. And they are willing to help - everyone feels like they don't know enough about something and has been in that position. (Could be a generalization, but I think it's a fair one.)

Teaching someone else about something - big or small - is just as terrific for learning. It helped me to actually talk out loud about concepts I'd learned, and pinpoint what I didn't understand.

"Programming" can mean a lot of things. When I started learning, I thought about programming in relation to Codecademy courses, my favorite websites/apps, or specific programming languages. Turns out it's so much broader than that - with Python (the language I'm focusing on at the moment) you can build hugely popular apps, create the system underlying the best-known visual effects studio out there, forecast the weather, or just do some math. I'm barely scratching the surface, and that's one language, of dozens, all of which are implementing complex and often non-intuitive core computer science concepts. It's a hard and extremely broad discipline - but because of that is much more fascinating than I'd expected.

Related - learning a programming language is a lot easier for me than learning to be a programmer. It's kind of like learning Spanish (or any language) - I learned all of the words and syntax, I know how to string them together to make basic sentences, and I can find a library in Spain. But I could not hold a single conversation with another person in Spanish, or live in Spain, or write a book in Spanish. (The analogy doesn't fully hold, but it's close enough. There's a lot more to programming than the language - the language is just a means to solve problems.)

Reading about something in a book felt like I was just absorbing knowledge - it's a powerful feeling. I spent much of the first two days reading textbooks and typing out code. But it was so much easier than trying to solve a problem using what I'd learned. And it's doubly easier than working with another person on that problem. But it was vastly more valuable for me to work with other people - usually of very different levels of understanding (and usually more experienced than me). And like I said, learning from and teaching other people was a great source of learning for me.

I learned so much from trying to build my own simple version of Git, and working with others on it, than anything else this week. So much so that it could be one or several blog posts. A small project - even/especially one that isn't shiny or fully-functional - can be such an incredible way to learn how you work, what you need to improve, and how much really goes into building real programs. (If you're unfamiliar with Git - and even if you aren't, it's still a confusing tool - it's basically a way to track changes in your code over time. There's a lot more to it, which is why it's so hard to build, but that's the general idea.)

Related - and this is definitely something HS emphasizes - it's so much more important to understand the underlying technology than how to implement it. There are a lot of web frameworks - essentially, a collection of code that makes developing complex websites easier - out there. They are often intricate/seemingly impossible to understand. I tried to learn and use one (Django) on my own, before I started at HS - it would've been such a better use of my time to understand what it's actually doing than try to first make something with it. Had I known all of what it actually did, I would've used something simpler (like Flask, which I am learning about now.) This is also a general life thing I've learned about myself - I'm much more comfortable understanding something from the ground up than accepting something works and not knowing what to do when it doesn't.

These two simple blog posts have been amazing just for processing what I'm doing and learning. Even if I'm the only person who ever reads them, it's a worthwhile endeavor.

It's really hard for me to organize my thoughts. I could use some help there. (e.g., Is it lazy that I just threw it in a list, and didn't formulate thought-out paragraphs about it? Maybe. Just wait a few months, and I'll be the next Buzzfeed/Vice/Upworthy/Dose.)

Also if you're new to programming, HS has a great manual - ostensibly for people going to HS, but I think it can be applied to anyone at any level interested in programming.

Previously, On

(I'm not gonna finish that title, I like the beginning but "Previously, On My Life" sounded just the worst. This'll do, pig.)

This is mostly just a catch-up for me, to remember what I've actually been doing for the last few weeks, months, years. If that's something you want to read about, cool, thanks! If not, there are so many things to read in the world, go read those!!

Sometime in the last eight years, I decided I didn't love finance, my college major. Then I graduated with a degree in finance, and got a job at a financial technology startup, and still didn't love finance. (This is not a sob story - a person not loving their job or career path is the like bottom 0.1% of important problems - but hopefully it's mildly interesting at worst.) I worked there for 3 1/2 years (from graduation until October 2014) - honestly, much longer than I ever would've expected. The time there was stressful, draining, and led to several periods of burnout, obvious to coworkers and friends and family - but it was genuinely valuable. Every person I worked with was more talented, driven, and intelligent than I was; every day I learned something new about how to (or not to) make decisions in a company, small or big; I gained a respect if not a love for finance and investing, and a much better understanding of those who choose to pursue it; most importantly/relevantly for my own life, I (very) gradually understood what was important to me, both in a career and in my life.

What's important to me in my career is that I'm interested in and engaged by the problems I'm trying to solve, and that it's not affecting my own life or personality in a negative way. I considered the idea - for a while, like periodically between grade school and year 3 of that job, so maybe 10 years - of being a part of the movie industry. What part? Who cares! It's the movies! It's all fun! At about the aforementioned year 3 I realized I do care, and it's probably not all fun - I didn't want to spend years in a non-creative role with the distant/impossible possibility that someday I'd be a director. I love movies, more than most things, but I knew I'd have to throw away most of those other things to pursue that dream. So somewhere in year 3, I moved on.

And that's about when I started doing a few Codecademy courses on a whim. My job required a lot of Excel - if you're familiar, I'll pour one out for you; if not, skip ahead, or you've already stopped reading - and only when I started these simple courses did I realize that I was (very) basically programming already. The same skills - logical thinking and problem solving, working with large datasets, understanding how to write formulas - translate pretty well to actual programming. So I kept doing Codecademy. And started reading textbooks. And gradually started writing small programs. Eventually I made my own website (and the companion blog, both a serious impetus for learning programming). Somewhere in the middle there, I knew I could be, and had to be, a programmer. I liked learning about it, I liked writing code, I liked that I could use it to do pretty much anything. (Important thing to take away from that sentence - I liked it. That's all I needed.) The question went from What Do I Want to Do? to How Do I Do It?

So in October, I left the job. From then until mid-November, I studied on my own - and struggled through making that website. Then I started a Web Development course at General Assembly. I have nothing but great things to say about GA - the people working there were wonderful, my classmates were engaged and diverse and genuinely kind, and the program is perfect if you're just getting started in programming. But I realized early on in the course that I wasn't just getting started; I'd learned a lot on my own already, and it probably wasn't the best use of my time to learn it all again, in a different language (I'd been learning Python, they focus on Ruby). At that point I decided to drop the course and pursue learning on my own (GA was extremely gracious at this and every other point too - they talked to me and helped me through that difficult decision, and completely understood my difficulty - again, just the best people).

Days later, I finished my application to Hacker School - something I probably should've just done in the first place, but I was glad I did. (A lot of this blog will probably focus on my experience here, but if you're interested, it's worth reading everything on their site. They're very open about their goal - to provide a space for people to self-direct their learning about programming alongside each other and together, while enforcing minimal structure - it's definitely an adjustment from class-style learning.) I (miraculously) got in on the first try, and just started this past Monday. And so far it's been a challenging, but stimulating, environment for learning. Consistent with my past experiences, everyone here is just the nicest - there are a lot of nice people in this world! It's one of those crazy surprises about being an adult! - and help to preserve an environment where people can learn what they want, without the regular external pressure (quizzes, tests, grades, jobs, money, etc.) that tend to drive learning. Per their site - "We value intrinsic motivation and self-direction, and believe people learn best when they're free to explore their passion and interests." It might sound high-minded, or touchy-feely, or unrealistic - those all might be true. But they seriously strive to make that happen, and in the first week, that's what's happened for me.

I'll try to keep this regularly updated - ideally every day - to give an idea of my experience at Hacker School. I'll also go into technical stuff - like the actual things I'm learning - so if you're not interested, probably don't read those, because if you're reading things you're clearly not interested in and then getting mad later, that's kind of a silly thing to do. (Also a thing I've done - why would I read all the way through a Buzzfeed quiz just to be angry about it?? Life is short!) That's more so I'm sure I can clearly express what I'm learning. So what I'm saying is, this blog is really just for me - but hope you enjoy it!

Trending Blogs

Recently Viewed Blogs

shields of dreams