Kun.io App Studio Blog

[missourihome]

Semi-Raw Cheater's Hot Fudge

Heat oil or butter in a pan, just enough so the peanut butter will melt. Remove pan from heat. Add some peanut butter first to help cool down the oil and melt the peanut butter. Then, add molasses, honey, and cocoa powder. Mix it up and enjoy!

Ratio: 1 tsp oil to butter (I prefer coconut butter for this.) 2 tbsp peanut butter 1/2 tsp blackstrap molasses 1 tsp honey 2 tsp cocoa powder

[missourihome]

Thought I’d see what kind of spin the Italians put on macaroni and cheese. Came across this recipe: Conchiglie con Fonduta or “shell pasta with cheesy cream sauce.”

This is a tasty hearty sauce. I like the lack of flour as a thickener, comparing it to Mornay Sauce.

Not something to eat...

[mongodb]

Some of the best MongoDB tools come from the Open Source community. Projects like the Node.js Driver, Mongoose and Meteor have become the backbone of many MongoDB apps and have helped support the developer community all over the world. We want to see more of what the community has built.

...

[wprl]

Review of "Evolutionary Computing"

This is an excellent, thought provoking book. Working through the problems will give you great insight and challenge you into not only a better understudying of Evolutionary Computing and related fields, but Computational Intelligence in general. It will make you a better developer by showing you new and interesting approaches to optimization and coding in general.

[wprl]

Baucis around the World

Baucis has users spanning the globe. From San Francisco to New York, London to Tokyo. Build your next REST API with Baucis!

While you're waiting on v0.16.0 and the new streaming features, why not check out what's being said about Baucis around the world?

Servizi REST con NodeJS + Express + Mongoose + MongoDB

Marionette.js(Backbone.js)のチュートリアル with yeoman その１(準備からサーバー側実装まで)

Einführung in den MEAN Stack: Teil 10

Getting Started Full Stack, Node.js Backbone.js MongoDB

angularJS의 $resource를 이용한 REST service 호출 예제

[wprl]

On JavaScript Style #1: Use One-Line If Statements

One-line if statements have an undeserved reputation of being unreadable.

Two common arguments against them are 1) It's easy to forget to add braces when expanding the if statement, and 2) everything will be crunched and unreadable for lack of space.

To 1) I say, bah, it's not hard to remember, not any harder than adding braces after a function. 2) is easily avoidable when using a minimum level of forethought.

So, one-line ifs aren't bad, but are they good? Yes, they greatly reduce the number of lines of code needed for error handling, contract programming, and other common uses of ifs.

function (request, response, next) { if (!request.isAuthenticated()) return response.send(401); doSomethingAsync(function (error, result) { if (error) return next(error); if (!result) return next(new Error('Not found.')); if (!result.authorized) return response.send(403); if (result.conflict) return response.send(409); // Logic response.send('Good job.'); }); }

This way it's easy to arrange functions into sections of error handling, contracts, and then logic. And, it's easy to read and navigate visually.

Using with else is also pleasing on the eye, and easy to read:

if (ok) doSomething(); else doSomethingElse();

However, mixing one-liners with braces is weird:

if (ok) doSomething(); else { doA(); doB(); }

Do this instead:

if (ok) { doSomething(); } else { doA(); doB(); }

One line statements should not be very long. They should not be longer than one line of text. They should not wrap. They should not run off the margin (one or two characters is OK).

I end lines at 80 characters but then I like big fonts and short lines. A lot of people use 120 characters or more. The biggest issue is consistency. If you have a consistent maximum number of characters in a column, it will be easier to read a document, whether it's JavaScript, HTML, or prose.

In paper printing, margin width and font size can be adjusted based on the size of the page and other stylistic considerations. So too, in code.

Again, do not have crazy long one-line statements in your code. People will hate you!

If the line's too long, use braces.

Next in this series: On JavaScript Style #2: One-Line Functions

[wprl]

Links of the Month: February 2014

This is a list of links I found interesting in the past 30 days or thereabouts. If you're a nerd, you might find them interesting as well.

RFC 3986

It was interesting to note the use of ; and , in the path. I'd never seen that before, and it's a method that could be used for resource/API versioning and I haven't seen it discussed. Coincidentally, I was working with the Yahoo Fantasy Sports API this month and noticed they use this for some feature or other.

http://www3.nd.edu/~jvanderk/sysone/

I'd read this article before, but stumbled across it again while looking for old Mac resources. It's a nostalgic look back at Mac System 1.0.

(I set up an old iMac G3 to use as an SSH terminal on my desk. I now have 4 screens on my desk, 3 machines running them.)

Here are some cool old mac games you can still purchase. I will be! I love retrogaming.

Realmz

http://www.robotroom.com/StormImpact.html

http://www.spiderwebsoftware.com/exile/macexile.html

Here's a bunch of legacy Mac software Apple provides for download. System 7.5.5 here I come!

http://www.info.apple.com/support/oldersoftwarelist.html

This Japanese SNES RPG looks cool. I like the graphics. I hear there is a translation available...

https://www.youtube.com/watch?v=cHWtcq8xpF8 (Ryukihei Dan Danzarubu)

I run a lot of apps on my laptop and often have many tabs open. I realized it would not be that hard or expensive to double my RAM, and that's what I'm going to do soon!

http://support.apple.com/kb/HT1270?viewlocale=en_US&locale=en_US#link1

http://www.newegg.com/Product/Product.aspx?gclid=CJqR_ofnx7wCFRRo7Aodzl4Afg&Item=N82E16820148604&nm_mc=KNC-GoogleAdwords&cm_mmc=KNC-GoogleAdwords--pla--Mac+Memory-_-N82E16820148604&ef_id=UdYnvgAABMbibVGe:20140213000504:s

[wprl]

On JavaScript Style #0

Coding, and beautiful code are something I have been thinking about for 23 years. Over the years, I have read many opinions on this or that aspect of coding style, considered the opinions of many friends and colleagues, and have experimented with and revised my own style thousands of times.

Style is something interesting in the otherwise quantifiable profession of software development. It is not wholly subjective, though parts of it are. And yet the objective qualities are just that — qualitative rather than quantitative.

I'm not sure I am a practitioner of anything as formal as Literary Programming, created by the esteemed Dr. Knuth. However, I have been greatly inspired by that approach to coding and documentation. Code is text, and a great deal can be learnt from poetry and prose writing when considering organization, readability, brevity, and other aspects of good style.

Code quality touches every stage of software development. Good organization and readability are key to the health of a software project. If code is not easy to find or to revise, code quality suffers in the classic snowball pattern. Sadly, this expensive malady affects too many projects.

We can design software systems in code or pseudo-code. We can approach coding iteratively, much like the draft and revision process of creative writing. Code, and therefore text, is the very essence of software.

I hope that I will be able to share a deeper appreciation for style and lessons from prose and poetry as applied to coding, JavaScript, and Node.js.

Next in this series: On JavaScript Style #1: Use One-Line If Statements

[wprl]

Semver Versioning Below v1.0.0

I haven't seen this covered much anywhere, apologies if I missed some salient piece of information floating around on the web. It doesn't seem much touched upon in the available semver literature.

I like to think of v0.0.0 as the source of all projects, the blank slate, the primal origin of code. As 0 isn't a release, but the state of there being no releases, anything goes: code that doesn't compile, scanned images of penned ravings on cocktail napkins, etc.

v0.0.1 is the first attempt at a functional release; it is just a stepping stone on the path to a functional release. It may compile, it may not. It was a good stopping point while you began to bring form to your idea, reconsidering and revising your plan.

v0.1.0 is the first functional, compiling release. Things get less interesting from here on. Increase the patch number (v0.1.1) for bug fixes, refactors, and redesigns. Increase the minor version for each new feature or breaking release (v0.2.0, v0.3.0).

When everything's solidified into what will be v1.0.0, follow semver procedure as normal.

What are your thoughts? How do you approach pre-v1.0.0 version numbers?

Cheers,

William

[wprl]

ClamAV sure eats a lot of memory! Even with the additional swap ClamAV is still broken, presumably from memory exhaustion.

I was about to uninstall it... but then I thought that I had better go ahead and see if resizing the VM works. Yep. Setting RAM to 1 GB (+swapfile) seems to have done the trick. Who knew anti-virus scanning could be so resource intensive for only a few users?

Follow up to a previous post.

[wprl]

Additional Steps to Set up Spam Blocking (Postfix/Ubuntu)

This is a follow-up to a previous post: Installing a Spam Filter for Postfix on Ubuntu.

It's funny. I tend to expect code to speak for itself, but I write about things that are less transparent to me, as a way to learn, share, and record observations for later use...

I was getting ready for bed last night, when I received notification of an undeliverable email. It was an email I had sent to a client Friday. I had been looking forward to a fresh start to the week, with all the chaos of the blizzard and smaller emergencies that sprang up in the new year behind me. Life had other plans.

I poked around the server logs and realized Amavis had crashed and this is what was blocking incoming AND outgoing mail!

Here's a brief account of how I diagnosed and solved the issue.

The first thing I noticed in the log (/var/log/mail.log) was that ClamAV was giving errors. Seemed like a reasonable place to start looking for a problem, though I wasn't sure it was related to the issue with Amavis.

sudo less /var/log/clamav/clamav.log

The clamav log was surpsigingly empty.

sudo less /var/log/clamav/freshclam.log

Received signal: wake up

ClamAV update process started at Sun Jan 12 09:35:14 2014

WARNING: Your ClamAV installation is OUTDATED!

WARNING: Local version: 97 Recommended version: 98

DON'T PANIC! Read http://www.clamav.net/support/faq

sudo aptitude update

I followed the link. Checked in apt, and there was no update for the package yet. Oh well.

I decided to focus on resending the bounced message now. I had restarted Amavis and everything seemed to be moving along again. I checked what mail was queued:

mailq | less

Quite a few bounced and undelivered messages...

man postsuper

May regret this but...

sudo postsuper -r ALL

Eeep 107 messages requeued. A lot of it looked like incoming spam, so hopefully that will just be filtered by Amavis... the sense of dread grows as I reopen Mail.

OK didn't blast 100 spams out into the world or anything. Just flooded my email with spam and a surprising number of relevant messages that had been queued... hope I didn't miss too much...

Now, I'm looking for the really important bounced message to my client:

man postcat sudo postcat -q 0123456789

I couldn't find it. Didn't show up in my sent messages in Mail. Apparently requing all the messages had worked, though it never did appear in my sent mail folder in Mail. All the other dequeued messages reappeared in the sent mail folder. I only realized it had worked after receiving a reply from the client.

While looking through the mailq output, I noticed mail to the root user was not able to be delivered. I added an alias so that messages to root would be sent to me instead.

sudo nano /etc/aliases

I added root: william. Then I restarted the various mail servers.

sudo /etc/init.d/postfix restart sudo /etc/init.d/spamassassin restart sudo /etc/init.d/clamav-daemon restart sudo /etc/init.d/amavis restart

A new message in mail.log!

warning: database /etc/aliases.db is older than source file /etc/aliases

I noticed these messages were related to the delivery of messages to root:

dovecot: lda(root): Error: user root: Initialization failed: Namespace '': stat(/root/Maildir) failed: Permission denied (euid=---(nobody) egid=---(no group) missing +x perm: /root, dir owned by 0:0 mode=0700)

dovecot: lda(root): Fatal: Invalid user settings. Refer to server log for more information.

dovecot: lda(root): Error: chdir(/root) failed: Permission denied

Looking through various sources of information, I discovered that I needed to run this command to rebuild the alias index:

newaliases

OK, two issues were solved, but I had another Amavis crash at the beginning of the afternoon. Time for some more snooping.

I rechecked the ClamAV log and there was now some output in it (/etc/log/clamav/clamav.log).

It didn't have enough memory! I decided to enable swap to fix it, by following the directions from DigitalOcean found here:

https://www.digitalocean.com/community/articles/how-to-add-swap-on-ubuntu-12-04

I followed those instructions, then restarted the server:

sudo shutdown -r now

Everything in /var/log/mail.log looks good, no errors now. Hopefully the Amavis issue was due to low RAM as well, and will be solved by the addition of the swap file.

I'll be keeping an eye on the server to monitor health. I'm also looking for a good way to get this information in my DevOps dashboard.

Hopefully that's the end of that!

[wprl]

Installing a Spam Filter for Postfix on Ubuntu

Readers of the blog will recall that I built the Kun.io mail server using open source software such as Ubuntu and Postfix. Well, I had an interesting couple months living without a spam filter. Finally the volume of spam got bad enough (around Christmas) that I decided it was time to prioritize installing a spam filter on my mail server.

Installation went very smoothly, and all I had to do was follow these additional steps listed on the Ubuntu Community Help Wiki.

By following these instructions I installed Amavis which is basically a daemon that runs email middleware. It hosts SpamAssassin and ClamAV, which filter spam and scan for viruses, respectively.

As I said, worked like a charm. The only additional step I had to take was to quiet ClamAV's alteration of the email subject line when it couldn't parse an email. To do this, I added the following line to /etc/amavis/conf.d/50-user:

$undecipherable_subject_tag = undef;

[wprl]

PATCH, Idempotence, & the Importance of Standards

PATCH is only a proposed standard (RFC5789), granted a widely used one. I think fundamentally it addresses a need that doesn't exist, because there's absolutely nothing in the HTTP spec (RFC2616) that states you can't use PUT for full or partial updates (or both or neither).

A lot of room is left to interpretation for the server implementation, and rightly so.

Other areas are explicitly defined. It's hard to understand how a reading of the spec can lead one to think of POST as an update. I covered that a bit in the last post.

A common statement is that arguments about REST are overly theoretical, and perhaps some are, but often there are subtle advantages to rigorously following the specification.

One example of the advantage of using PUT as the spec outlines is this: PUT is idempotent, so a browser (or proxy) knows it is fine to retry a failed request. POST is not defined to be idempotent, and therefore it would be inappropriate to retry the connection.

Another advantage is that as client logic continues to (finally) detach from the server, and is moved to the browser, relying on standard semantics means clients can be more generalized, and just know that if a resource is to be updated, a PUT to that resource's URL should be used. The server defines much of how to interpret the PUT request's body, headers, etc.

Clients are all the time moving closer and closer to being implemented solely in the browser using technologies such as HTML5, RequireJS, Backbone.js, and static hosting — no client code hiding in the back end. (I should mention Bower too. It incredibly tames the tangle that is client-side package management.)

The interpretation of RFC2616 as mapping directly to CRUD is a beautiful thing. Not only can most, if not all, of the "database" (REST API) queries be moved to the client, but standard HTTP technologies like caching proxies can be used.

Another advantage to using standards is lower development time, and therefore lower project cost. For example, the infamous cache-busting query parameter is used to workaround caching of GET requests. Sure, the cache-buster works but you have added complexity when a standard solution was already available, a much more powerful and efficient alternative was right under the developer's nose: the cache control headers such as Last-Modified and ETag, already a part of HTTP.

On a final note, things like supporting deleting properties of a resource vs. just being able to set a property to null, or overwriting the object without knowing its current state are implementation details, and I would think this is why the spec largely leaves such considerations to the server implementation.

For example, in a JSON API, you could set a property to e.g. { $del: true } to delete it, or if the semantic meaning of having the attribute, but storing a value of undefined or null instead of deleting the attribute is really meaningful, you could use other means to specify this.

A key question there is do these distinctions have semantic meaning in the context of your API, and if they do, is there a more straightforward way of representing this semantic?

Similarly, you could always send a full update, even if partial updates are supported... there's nothing stopping you.

[wprl]

Use PUT for Partial Updates in your REST API

Or, why Baucis uses PUT for updates

There is a myth that I see resurface fairly often when discussing HTTP methods. It is the myth that PUT should not be used for a partial update, and indeed, the POST vs. PUT vs. PATCH debate is riddled with misconceptions.

One can safely map the four main HTTP methods 1:1 with CRUD, though it's not the only possible correct interpretation of the specification (RFC 2616).

Idempotence

I believe much of the confusion around the issue stems from the fact the PUT should be an "idempotent" operation.

In math, idempotence has the meaning of always yielding the same value, for example multiplying or dividing by one will always yield the original value. My best guess is that this leads people to think that idempotency in HTTP has to do with the resulting "value" or "server state" after an idempotent request such as PUT.

However, this is not the definition of idempotence in the context of HTTP. The definition is given in section 9.1.2 of RFC 2616, the definition of HTTP/1.1, and of RESTful APIs over HTTP:

Methods can also have the property of "idempotence" in that (aside from error or expiration issues) the side-effects of N > 0 identical requests is the same as for a single request. The methods GET, HEAD, PUT and DELETE share this property.

Notice that server state is not mentioned. Only server side-effects are considered. Side-effects are not equivalent to server state.

PUTing Partial Updates

For an example, let's consider a RESTful JSON API, such as those created with Baucis.

PUTting a partial update should always have the side-effect of updating the fields listed in the partial update. It would therefore be idempotent in the context of HTTP because the same side-effect (updating the partial field list) always occurs given identical PUT requests.

Furthermore, section 9.6 of RFC 2616 goes on to say that the spec "does not define how a PUT method affects the state of the origin server." (This is the only time server state is even mentioned in RFC 2616.)

Don't use POST for updates!

Using POST instead of PUT for partial updates goes against the spec. Some systems use this approach. I'd have to say that updates are semantically much different from adding a comment to a thread, annotating a document, or adding a file to a directory which is what POST has been designed for.

Section 9.5 of the RFC defines POST:

The posted entity is subordinate to [the resource at the POSTed URI] in the same way that a file is subordinate to a directory containing it, a news article is subordinate to a newsgroup to which it is posted, or a record is subordinate to a database.

This doesn't sound like an update, partial or full, to me.

A note on PATCH

PATCH seems extraneous, but maybe some people find a use for it. It exists outside of RFC 2616, so I tend to ignore it. I can't fathom why you would need it in addition to POST and PUT.

Conclusion

I hope I have shown that PUT does map very well to the U in CRUD (update). There are many possible correct interpretations of RFC 2616, and mapping to CRUD is one of them. This isn't to say an API must support partial updates through PUT, it's certainly not a requirement of REST. But, what is certain is that using PUT for partial updates adheres to the HTTP specification, and is therefore RESTful.

[wprl]

Are Node.js Workers a Good Thing?

Using workers adds unnecessary complexity to otherwise stateless Node.js instances for little or no benefit when building an API server, and most other back-ends.

Without workers, you only have to worry about scaling across instances. You load balance equally across all instances.

With workers, you have to worry about scaling within multi-core instances AND scaling across instances. You load balance twice, first to the cluster, and then to a worker. You introduce state to a system that could be stateless, and therefore simpler (and therefore less expensive).

Why reinvent the wheel when it comes to load balancing? When there are much more mature and powerful solutions available, it wastes resources duplicating, then maintaining load balancing logic via custom code that uses workers.

[wprl]

Code Obfuscation is Worthless

The purpose and reasoning behind client-side code obfuscation is flawed.

In practice, any client software is inherently less secure because it is much more available to the end user for disassembly, reverse engineering, and other inspection and tinkering.

This is a good thing for client software, some would even say it is a fundamental right of every user to disassemble and inspect code running on his or her own machine.

In any case, obfuscation tries to secure the wrong part of the system. Security is the job of the server and network, and energy securing the system should be focused in these areas. All secure logic should ultimately be performed on the server, and additionally in the client if necessary for end user experience (like form validation). All secure data should be deleted from the client as soon as possible, or kept out of it entirely if possible. Secure resources should be prevented from being sent to the client at all.

There should be no code in the client that when analyzed can yield security flaws. That is a fundamental design flaw. You must only have non-valuable code in the client. Anything else is negligence.

For intellectual property, your average user is a windmill, and the obfuscaters are like many Don Quixotes tilting at imaginary giants. Why would you really care if an end-user downloads an image from your site or peeks at the code? The images should not be hi-res originals, for example, and the client code should contain no secrets.

As much as the intellectual property cabals want you to believe that casual piracy by users hurts content owners, there is more evidence that it boosts sales. Too many people buy into the myth that piracy hurts them, and only to their detriment.

The real "enemy" in the case of most of the web are the mass pirates, people who will steal your images, music, etc. and mass produce them for sale. Of course, this isn't applicable to most sites, and obfuscation and other UI hacks will not stop it.

In short, if a piece of a system must be secured, that should happen on the server level, and never on the client.

[wprl]

Why My Open Source Stack Is Right for Your REST API

If a developer says they use a custom stack, it should raise flags. Especially if they only have a few years professional experience. Or little experience with long-term maintenance and modification of existing software systems.

My stack leverages the power of open source JavaScript, both in the client and server, including Baucis for rapid, yet scalable development, and a few proprietary packages to provide the "secret sauce."

The master excels by knowing what to leave out, seeking the core of the problem, until it is naked in its simplicity, with no unnecessary layers to obscure its pure function.

App frameworks have their place. MeteorJS, Sails, etc. are all a great way to rapidly develop an app, and a great platform for less-experienced or even inexperienced developers to build an MVP or prototype quickly, and will carry them some distance into production, depending on the complexity of the app and supporting server architecture.

But, if you are in a position to both quickly build an MVP and also avoid rearchitecting your app after the initial development phase, why not do it?

Let's talk soon!

William Riley-Land