TheWebCafes

As a self-learned and self-help web developer, I appreciate knowledge sharing by various developers in the Internet. I always never fail to find anything that I would like to know from Google. Sometimes, I get a perfectly fit solution for what I want. Sometimes, I didn't, but at least will gain the basic knowledge to get started. I started TheWebCafes as a place for me to document my knowledge and share it out to the community. I hope it will be like a Cafe, where we could have some discussion here!

[thewebcafes]

Dumping and Importing MySQL database with emoji in UTF-8 safe way

The incorrect way:

mysqldump -uroot -p database > database.sql mysql -uroot -p database < database.sql

The correct way:

mysqldump -uroot -p --default-character-set=utf8mb4 database -r database.sql mysql -uroot -p --default-character-set=utf8mb4 database mysql> SET names 'utf8mb4' mysql> SOURCE database.sql

[thewebcafes]

Tip: General Purpose Proxy with SSH Tunnel on Mac OS X

There is an easy way to setup a general purpose proxy if you have SSH access to a linux server. The main idea is to use SOCKS proxy to tunnel all your traffics through the linux server via SSH connection. The following is the steps if you are using Mac OS.

Open 'Terminal' app, and type the following command and enter. Then leave the Terminal open.

ssh -D 8080 -C -N <username>@<ip address>

Then open System Preference - Network - Advanced... - Proxies. Then checked the SOCKS Proxy. Then enter the settings as below: SOCKS proxy server: 127.0.0.1 Port: 8080

[thewebcafes]

Automatically Start Apache and MySQL on Mac OSX El Capitan with Launch Daemon

In general, to automatcially start a process during startup on Mac OSX El Capitan, we need to create a plist file under /Library/LaunchDaemons

Launch Terminal app

Type the following, and enter your password to gain root access:

sudo su

Create the plist file to launch Apache:

cd /Library/LaunchDaemons nano info.mamp.start.apache.plist

Paste the following content into the file, edit according to your need. Note that this example use the control script of Bitnami MAMP stack.

<?xml version="1.0" encoding="UTF-8"?> <!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd"> <plist version="1.0"> <dict> <key>Label</key><string>info.mamp.start.apache</string> <key>ProgramArguments</key> <array> <string>/bin/sh</string> <string>/Applications/mampstack-5.6.12-0/ctlscript.sh</string> <string>start</string> <string>apache</string> </array> <key>RunAtLoad</key><true/> </dict> </plist>

Press Ctrl + X to exit, save when asked.

Create the plist file to launch MySQL:

nano info.mamp.start.mysql.plist

Paste the following content into the file, edit according to your need. Note that the control script of Bitnami MAMP stack doesn't work when I tried, so I use the mysqld binary directly

<?xml version="1.0" encoding="UTF-8"?> <!DOCTYPE plist PUBLIC "-//Apple Computer//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd"> <plist version="1.0"> <dict> <key>Label</key><string>info.mamp.start.mysql</string> <key>ProcessType</key><string>Interactive</string> <key>Disabled</key><false/> <key>RunAtLoad</key><true/> <key>KeepAlive</key><true/> <key>SessionCreate</key><true/> <key>LaunchOnlyOnce</key><false/> <key>UserName</key><string>_mysql</string> <key>GroupName</key><string>_mysql</string> <key>ExitTimeOut</key><integer>600</integer> <key>Program</key><string>/Applications/mampstack-5.6.12-0/mysql/bin/mysqld</string> <key>ProgramArguments</key> <array> <string>/Applications/mampstack-5.6.12-0/mysql/bin/mysqld</string> <!--string>--defaults-file=/Applications/mampstack-5.6.12-0/mysql/my.cnf</string--> <string>--port=3306</string> <string>--socket=/Applications/mampstack-5.6.12-0/mysql/tmp/mysql.sock</string> <string>--datadir=/Applications/mampstack-5.6.12-0/mysql/data</string> <string>--log-error=/Applications/mampstack-5.6.12-0/mysql/data/mysqld.log</string> <string>--pid-file=/Applications/mampstack-5.6.12-0/mysql/data/mysqld.pid</string> <string>--lower-case-table-names=1</string> <string>--user=_mysql</string> <string>--basedir=/Applications/mampstack-5.6.12-0/mysql</string> <string>--plugin-dir=/Applications/mampstack-5.6.12-0/mysql/lib/plugin</string> </array> <key>WorkingDirectory</key><string>/Applications/mampstack-5.6.12-0/mysql</string> </dict> </plist>

Press Ctrl + X to exit, save when asked.

Reference: http://dev.mysql.com/doc/refman/5.7/en/osx-installation-launchd.html

[thewebcafes]

Linux: Kill all processes matching a string

ps -ef | grep "some string" | grep -v grep | awk '{print $2}' | xargs kill

Explanation

ps -ef List all processes in full format grep "some string" Filter to show only processes with "some string" grep -v grep Exclude process with the word "grep". Otherwise, this command will show itself in the output list awk '{print $2}' Extract and output column 2 only, which is the process ID xargs kill Run kill for each line of the output. Essentially it will run kill <pid>

[thewebcafes]

post-receive Git Hooks for Auto-Deployment Based on Branch

#!/bin/bash # Reference for Git hooks: http://stackoverflow.com/questions/7351551/writing-a-git-post-receive-hook-to-deal-with-a-specific-branch # Reference for bash script: http://www.thegeekstuff.com/2010/06/bash-if-statement-examples/ # Get arguments from stdin (i.e. use 'read <arg1> <arg2> <arg3>') while read oldrev newrev refname do # Parse 'refname' to get the branch name branch=$(git rev-parse --symbolic --abbrev-ref $refname) if [ "master" == "$branch" ] then # If pushed branch is master, deploy to production GIT_WORK_TREE=/var/www/html else # Otherwise, should deploy to test GIT_WORK_TREE=/var/www/test fi # Force checkout the pushed branch to work tree export GIT_WORK_TREE git checkout -f $branch done

[thewebcafes]

Bulk Renaming of Files in Linux Shell

Example:

for f in store-*.png; do mv $f ${f//store-/}; done

What It Does:

Basically the script loop through all files in the current directory in the form of store-*.png and remove the ‘store-’ from the file name.  For example ‘store-1.png’ will be renamed to ‘1.png’ and etc.

How It Works:

The for loop:

for f in store-*.png; ... ; done

basically loop through all the files in the form of store-*.png and keep the actual file name in the variable $f on each loop.

The command in between:

mv $f ${f//store-/}

rename the file from the original name (variable $f) to a new name using variable manipulation in the following form:

${VARIABLE//search/replace}

In this example, we replace ‘store-’ with empty string ‘’.

[thewebcafes]

Android: Image Picker / Chooser and Upload To Server

Objective: To allow user to choose image from photo, gallery, 3rd party apps (e.g. Dropbox) or take photo and upload it to server.

Step 1: Create and start a chooser intent

private static final int FILECHOOSER_RESULTCODE = 2888; private Uri mCapturedImageURI = null; public void ChoosePhoto(View v) { try { // Create folder at external storage // This allow us to use higher quality image from camera File imageStorageDir = new File( Environment.getExternalStoragePublicDirectory( Environment.DIRECTORY_PICTURES) , "ImageUploader"); if (!imageStorageDir.exists()) { imageStorageDir.mkdirs(); } // Create camera captured image file path and name File file = new File( imageStorageDir + File.separator + "IMG_" + String.valueOf(System.currentTimeMillis()) + ".jpg"); mCapturedImageURI = Uri.fromFile(file); // Camera capture image intent final Intent captureIntent = new Intent( android.provider.MediaStore.ACTION_IMAGE_CAPTURE); captureIntent.putExtra(MediaStore.EXTRA_OUTPUT, mCapturedImageURI); // Intent to get image from other app Intent i = new Intent(Intent.ACTION_GET_CONTENT); i.addCategory(Intent.CATEGORY_OPENABLE); i.setType("image/*"); // Create a chooser intent based on get image intent Intent chooserIntent = Intent.createChooser(i, "Image Chooser"); // Add camera intent to the chooser chooserIntent.putExtra(Intent.EXTRA_INITIAL_INTENTS , new Parcelable[] { captureIntent }); // Start the activity startActivityForResult(chooserIntent, FILECHOOSER_RESULTCODE); } catch(Exception e) { Toast.makeText(getBaseContext(), "Exception:"+e, Toast.LENGTH_LONG).show(); } }

Step 2: Capture Selected Uri and Convert to File Path

@Override protected void onActivityResult(int requestCode, int resultCode, Intent intent) { if(requestCode==FILECHOOSER_RESULTCODE) { Uri result=null; try { if (resultCode != RESULT_OK) { result = null; } else { // retrieve from the private variable if the intent is null result = intent == null ? mCapturedImageURI : intent.getData(); String picturePath = null; if (result.toString().startsWith("file://")) { // File from 3rd-party app (etc. Dropbox, Camera) picturePath = result.getPath(); } else { // Media content from Photo, Gallery String[] proj = { MediaStore.Images.Media.DATA }; Cursor cursor = getContentResolver().query(result, proj, null, null, null); int columnIndex = cursor.getColumnIndexOrThrow(MediaStore.Images.Media.DATA); cursor.moveToFirst(); picturePath = cursor.getString(columnIndex); cursor.close(); } // Call the ImageUploader (implemented in Step 3) new ImageUploader().execute(picturePath); } } catch(Exception e) { Toast.makeText(getApplicationContext(), "activity :"+e, Toast.LENGTH_LONG).show(); } } }

Step 3: Upload Image to Server Using AsyncTask

Note: This code rely on Apache HTTP Client library.  It was tested with httpcore-4.3.4.jar, httpmime-4.3.4.jar and httpclient 4.3.4.jar.  Latest version of binary can be downloaded from: http://hc.apache.org/downloads.cgi

private class ImageUploader extends AsyncTask<String, Void, String> { @Override protected String doInBackground(String... params) { // TODO Auto-generated method stub String result = ""; // Client-side HTTP transport library HttpClient httpClient = new DefaultHttpClient(); // using POST method try { HttpPost httpPostRequest = new HttpPost(new URI("http://example.com/upload")); // creating a file body consisting of the file that we want to // send to the server FileBody bin = new FileBody(new File(params[0])); /** * An HTTP entity is the majority of an HTTP request or * response, consisting of some of the headers and the body, if * present. It seems to be the entire request or response * without the request or status line (although only certain * header fields are considered part of the entity). * * */ MultipartEntityBuilder multiPartEntityBuilder = MultipartEntityBuilder.create(); multiPartEntityBuilder.addPart("photo", bin); httpPostRequest.setEntity(multiPartEntityBuilder.build()); // Execute POST request to the given URL HttpResponse httpResponse = null; httpResponse = httpClient.execute(httpPostRequest); // receive response as inputStream InputStream inputStream = null; inputStream = httpResponse.getEntity().getContent(); if (inputStream != null) result = convertInputStreamToString(inputStream); else result = "Did not work!"; return result; } catch (Exception e) { return e.getMessage(); } // return result; } @Override protected void onPreExecute() { // TODO Auto-generated method stub super.onPreExecute(); //dialog.setMessage("Uploading..."); //dialog.show(); } @Override protected void onPostExecute(String result) { // TODO Auto-generated method stub super.onPostExecute(result); dialog.hide(); Toast.makeText(getApplicationContext(), result, Toast.LENGTH_LONG).show(); //mFrag.setText(result); } } private static String convertInputStreamToString(InputStream inputStream) throws IOException { BufferedReader bufferedReader = new BufferedReader( new InputStreamReader(inputStream)); String line = ""; String result = ""; while ((line = bufferedReader.readLine()) != null) result += line; inputStream.close(); return result; }

[thewebcafes]

Decreasing Root Disk Size of Amazon EC2 Instance

Stop the instance you would like to decrease the root disk size.

Create a snapshot of its EBS volume.

Use the snapshot to create a new EBS volume from that - lets call this "source" volume. (Work with this copy instead of original EBS volume in case something goes wrong).

Create a new EBS volume of your desired size - lets call this "target" volume.

Launch a new instance (use the cheapest option as we only need this to do disk resizing and copying).

Attach the "source" volume (e.g. as /dev/sdf) and "target" volume (e.g. as /dev/sdg) to the new instance.

Login to the terminal of the new instance.

Check the "source" file system:

e2fsck -f /dev/sdf

Maximally shrink the "source" volume:

resize2fs -M -p /dev/sdf

This will output the number of blocks

Choose a chunk size (e.g 16MB), calculate the number of chunks to copy: blocks * 4 / (chuck_size_in_mb * 1024) - round up to integer

Copy the data:

dd if=/dev/sdf ibs=16M of=/dev/sdg obs=16M count=<count>

Resize the "target" volume:

resize2fs -p /dev/sdg

Check the file system:

e2fsck -f /dev/sdg

Detach the EBS volume of the original instance

Detach the "target" volume and attach it to the original instance

Start the original instance and wait (it may need more time for initializing).

If everything OK, clean up if needed (e.g. delete the snapshot, delete the "source" volume, delete the original volume and terminate the instance used for copying). Otherwise, repeat.

[thewebcafes]

SPF Record for Dummies: Improving Mail Delivery

SPF (Sender Policy Framework) record in short, is a TXT record that you could (and MUST!) add to your DNS records to tell the outside world which hosts (based on IP address) is permitted to send mail under your domain name.  It is currently the industrial standard for email verification to prevent email spoofing.

A typical SPF record looks like this:

yourdomain.com. IN TXT "v=spf1 a mx -all"

This tell the receiving Mail Transport Agent (MTA) to accept email from IP addresses that matches the A/AAAA records (the 'a' rule) and the MX records (the 'mx' rule) of 'yourdomain.com' and reject email from all other sources (-all).

In general, a SPF record that the following form:

v=spf1 <q-1><rule-1> <q-2><rule-2> <q3-><rule-3> ...

Where <q-n> is one of the qualifies below: + (or omitted) means pass (MTA should accept, mark as 'pass') ? means neutral (MTA should accept, mark as 'neutral') ~ means soft fail (MTA should accept but mark accordingly) - means fail (MTA should reject) And <rule-n> is one of the mechanism below: all - match any IP address a - match A/AAAA records ip4 - match a given IPv4 address range ip6 - match a given IPv6 address range mx - match the IP addresses resolved from MX record ptr - match the PTR record in DNS exists - match as long as the given domain name resolves to any address include - Include policies from another SPF record

SPF record is assess rule-by-rule from left to right.  So, you should notice that the 'all' rule is usually at the end to act as the default action. For example:

"v=spf1 a mx +all" is no different from "v=spf1 +all" and means anyone is allow to send email using yourdomain.com (which is a bad idea).

Details of Mechanism

a - match A/AAAA record

yourdomain.com. IN TXT "v=spf1 a a:example.com"

a:example.com will match the A/AAAA record of example.com.  If the domain name is omitted (i.e. just 'a'), it means matching A/AAAA record of yourdomain.com

mx - match MX record

yourdomain.com. IN TXT "v=spf1 mx mx:example.com"

mx:example.com will match the MX record of example.com.  If the domain name is omitted (i.e. just 'mx'), it means matching MX record of yourdomain.com

ip4 - match a given IPv4 address range

yourdomain.com. IN TXT "v=spf1 ip4:123.456.789.12 ip4:12.345.678.0/8"

ip4:123.456.789.12 match a single IP address as stated, the second rule match an IP address range 12.345.678.* (note the '/8' at the end).

ip6 - match a given IPv6 address range

Similar to ip4 but matches IPv6 address instead.

include - include policies from another SPF record

yourdomain.com. IN TXT "v=spf1 include:_spf.google.com"

This include the SPF record of _spf.google.com, which is common if you use GMail of Google Apps as your email service.

ptr and exists are less used.  We will skip the details here.

Here is a good tool to check your SPF record

http://mxtoolbox.com/spf.aspx

Why SPF (for the curious one)?

The reason for implementation of SPF is that SMTP allows any computer to send email claiming from any source address just by setting the Return-Path and From headers.  With SPF, a receiving mail server could compare the domain name specified in the Return-Path with SPF record of the domain to check if a host (based IP address) is allow to send email on behalf of the domain name.

For example, if you use a server (IP address: 123.456.789.012) to send out an email to a GMail account, claiming from [email protected].  GMail will check the SPF record of yourdomain.com to see if 123.456.789.012 is permitted to send email on behalf of yourdomain.com.  If it is not permitted, GMail will drop the message.  Many mail servers nowadays are very strict on SPF record in a sense that if SPF record is missing, the message will be dropped.  Thus, it is important to setup your SPF record correctly.

[thewebcafes]

Tip: Kill a Running PHP Cron Job

If you started a long-running PHP script as a cron job and wish to terminate it for whatever reason.  Here is what you can do.

Step 1: Run the following command to find the Process ID

ps -ef | grep php

"ps -ef" is basically to show all (-e) processes in full details (-f) and the output is pipe (with '|') through "grep php" to find all processes with the word "php".  From the output, you should be able to identify the process and its ID.

Step 2: Kill it with the following command

kill -9 <pid>

[thewebcafes]

Exim: Speed-Up Mail Delivery by Disabling SpamAssassin Scanning for Outgoing Mail using ACL

Our exim mail server was experiencing slow newsletters delivery today (0.5s per mail instead of the usual 0.05s).  Upon troubleshooting, I noticed that it was caused by SpamAssassin.  It appeared that our server was set to scan all outgoing emails that are less than 100Kb in size.  And this time round, our newsletter happened to be less than 100Kb and SpamAssassin is working very hard to scan all of the outgoing newsletters.  Thus, it was decided that we should disable SpamAssassin scanning for outgoing newsletters in the future.  Here is how it can be done:

Step 1: Run the following shell command to find your exim configuration file:

exim -bP configure_file

Step 2: Open the configuration file and look for the line:

begin acl

This line marks the beginning of Access Control List (ACL). After this line, you could see a number of rules in the following form:

accept <abc> = <xyz> warning <abc> = <xyz> deny <abc> = <xyz>

With these rules, the ACL section actually controls Exim's behaviour in handling incoming and outgoing mails.  More details on ACL can be found here.

If Exim is compiled with SpamAssassin extension, you should see this condition at some part of the rules:

spam = <username>

This is where Exim asks the message to be scanned by SpamAssassin.

Step 3: Add rules to accept outgoing mail without scanning

The solution is then to add rules that accept outgoing mail without any "spam" condition.  There are many ways to do it.  For us, our outgoing newsletter are non-SMTP mails, so we added the following rules under the acl_not_smtp part.

acl_not_smtp: accept sender_domains = mailer.ourdomain.com accept sender_domains = ourdomain.com

Note: If you are sending mails via SMTP, it is better to use rules that based on 'hosts' instead of 'senders' and 'sender_domains' as the latter can be forged easily.

[thewebcafes]

HowTo: Check Your Mail Server (MTA) in Linux

Your Linux server may be configured to run different Mail Transfer Agent (MTA) such as Sendmail, Postfix, Exim, QMail and etc.  To check out which one is running on your server, use this command:

alternatives --config mta alternatives --display mta 

If the alternatives command did not work, check what program is listening to port 25 as follow:

netstat -alnp | grep :25

[thewebcafes]

HowTo: Check Your Linux Distribution (Distro)

Linux comes with different distribution (commonly known as distro) such as RedHat, Ubuntu, CentOS, Fedora and etc.  When there is time you need to know what distro your server is running on, just use this command:

cat /etc/*-release

[thewebcafes]

Getting Uploaded Filename and Mimetype in Google App Engine

The GAE documentation gives a good example of getting the content of an uploaded file using self.request.get('file'):

http://code.google.com/appengine/docs/python/images/usingimages.html#Uploading

However, it doesn't give a clue on how could we get the filename or the mimetype.  It takes me a while to search on Google and finally found the solution...

You can get the parameters from:

self.request.POST['file'].filename self.request.POST['file'].type 

[thewebcafes]

Email Validation in Google App Engine - Python

Email validation from user input has always been a problem without perfect solution. The most common practice is usually to match the input string against a regular expression and/or MX records.  Unfortunately, many of the regular expressions proposed are overly strict and reject some perfectly valid email addresses.  And MX records can fail occasionally as well.  This will end up giving terrible signup experience to your potential users.

Google App Engine SDK has choose to don't bother about email validation at all by providing the google.appengine.api.mail.is_email_validate() function that only check for non-empty string instead of validating the email. Unfortunately, we don't hope for overly loose validation that permit input like 'asdfghjkl' either.

The best approach is probably using the regular expression provide by Django (django.core.validators.email_re) for light validation that reject obvious non-email input while doesn't provide false positives:

from django.core.validators import email_re if email_re.match(input):   // Email is valid else:   // Email is invalid 

[thewebcafes]

images.get_serving_url() bugs in Google App Engine Development Server

If you are testing blobstore features related to images for your Google App Engine app locally in your development server. You may notice that images.get_serving_url() did not work correctly in development server.

First issue:

key = 'YOUR-BLOB-KEY' blob_info = blobstore.BlobInfo.get(key) print images.get_serving_url(blob_info.key())

This code will output error as get_serving_url expect a string rather than key object in development server.  The following code will work on both development and production server:

key = 'YOUR-BLOB-KEY' print images.get_serving_url(key) 

Second issue:

The above code will run but the URL returned will not work on development server. It will return HTTP/500 Server Error if you visited the URL.

I notice it always return URL end with "%3D%3D".  If you "URL-decode" it by replacing "%3D%3D" to "==", it somehow will work.  So, you can do this:

print images.get_serving_url(key).replace("%3D%3D","==")

The URL returned will work on both servers.  I did not use regular expression for URL decode as this seems to be enough for the moment and it is faster.

[thewebcafes]

Iterating through dictionary in GAE-Django

Iterating through dictionary in GAE-Django doesn't seems to be as intuitive.

Instead of:

{% for key,value in dictionary %}   {{ key|escape }}: {{ item|escape }} {% endfor %}

You should do:

{% for item in dictionary.items %}   {{ item.0|escape }}: {{ item.1|escape }} {% endfor %} 

Also, note that dictionary do not have any order.