Idle Headed Scientist @idle-headed-scientist - Tumblr Blog

Check SSL Certificate Dates with OpenSSL

echo | openssl s_client -connect site:port 2>/dev/null | openssl x509 -noout -dates

inotify Using Processes

If you are using pyinotify, you might be familiar with the error "No space left on device (ENOSPC)". What you have to do when this happens, they say, is to increase the maximum amount of watches on your system.

You can see the max number of watches with

sysctl -n fs.inotify.max_user_watches

And set it to a new value with

sysctl -n -w fs.inotify.max_user_watches=16384

But wait, there's more. inotify using processes have a link to "anon_inode:inotify" among their file descriptors in /proc. So, you can use the following command to identify them:

find /proc/*/fd/* -type l -lname 'anon_inode:inotify'

To have more information (thanks to good folks here):

ps -p $(find /proc/*/fd/* -type l -lname 'anon_inode:inotify' -print 2> /dev/null | sed -e 's/^\/proc\/$[0-9]*$\/.*/\1/')

#linux #proc #inotify

Profiling Memory in Python

The Python module memory_profiler is a nice tool that gives line-by-line memory cost of Python programs. Just add a decorator on top of the function to profile. See:

@profile def import_many(): import numpy import cv2 import theano if __name__ == '__main__': import_many()

Save this as mem_test.py, and run it. A line-by-line breakdown is output:

$ python -m memory_profiler mem_test.py Filename: mem_test.py Line # Mem usage Increment Line Contents ================================================ 3 16.824 MiB 0.000 MiB @profile 4 def import_many(): 5 90.883 MiB 74.059 MiB import numpy 6 102.859 MiB 11.977 MiB import cv2 7 126.270 MiB 23.410 MiB import theano

So, importing numpy consumes more than 90 megabytes. Nice.

I installed memory_profiler like this:

$ sudo pip install psutil $ sudo pip install -U memory_profiler

#python #memory #profiling

Histogram of Values in Linux

Let's say you have a bunch of numbers in a file called data.txt. Let's say you want to get the number of occurances of each value. What you do is:

$ cat data.txt | sort -n | uniq -c 1 10 1 11 1 13 3 15 1 18 2 19 6 21 1 23 1 24 1 27 5 29 7 30 21 31

#linux #bash #data

Disable buffering in STDOUT

If you have an executable that writes to standard output, chances are, the code might not have controlled output buffering. If that is the case, once you redirect the output to another executable via pipe, or to a file, the output will be buffered and will not be written immediately. A more detailed write up about the problem is given here.

There is a - relatively - quick solution to this problem you can try if all else (like stdbuf or unbuffer) fails:

# echo '__attribute__((constructor))void f(){setvbuf(stdout,NULL,_IOLBF,0);}' \ | gcc -s -include stdio.h -x c - -fPIC -shared -o "line_buffer.so" # mv line_buffer.so $HOME/lib/line_buffer.so # LD_PRELOAD="$HOME/lib/line_buffer.so" executable

First we build a library that will set the stdout buffer to be line buffered, instead of fully buffered. Then we move the library to somewhere tidy. Then we use the LD_PRELOAD trick to override the default properties of stdout for the executable.

Taken from here. An explanation for the LD_PRELOAD trick can be found [here](http://stackoverflow.com/questions/426230/what-is-the-ld-preload-trick

#linux stdout buffering

Testing SSL Connections with OpenSSL

The openssl executable is found on almost any Linux. If you have SSL performance problems, it is a good tool for testing:

openssl s_time -connect remote.host:3338

This command will try to connect to the given host on the given port, and produce timing information.

#ssl #linux

URL Encoding for JBoss and mod_jk

If an encoded URL is decoded in the wrong character encoding, you obtain gibberish. So, you need to be careful when decoding the request parameters you get from browsers. Your best bet is to decode according to the character encoding of the HTTP request.

If you use Java, the standard HTTP request object automatically decodes the request parameters, using the character encoding set in the request object. That's fine, unless the character encoding of the request is not correct when this decoding is done, in which case you obtain gibberish.

Character encoding of the request is determined by the HTTP request itself, but can be modified by the server. So, you need to configure your server correctly to not obtain gibberish.

For example, if you use mod_jk to handle SSL on your front end, and you then forward to JBoss, you need to add the following line to your mod_jk configuration:

JkOptions +ForwardURICompatUnparsed

Also, you should enable UTF-8 on all your connectors in JBoss server.xml (under deploy/jboss-web.deployer). Don't forget the AJP connector!

#apache #mod_jk #JBoss #character encodings in html #utf8

Apache mod-rewrite by default re-encodes the complete URL. This is a problem. Because, see:

https://invidyo.com/vms/mobile/camerainfo;jsessionid=0FE062BB1E32D7DC3075D6A0338AB4DD.node1?name=%C4%9F%C5%9F%C3%A7%C4%B1&action=update&serial=801F026EFE4E

is converted to

https://invidyo.com/vms/mobile/camerainfo%3bjsessionid=0FE062BB1E32D7DC3075D6A0338AB4DD.node1?name=%25C4%259F%25C5%259F%25C3%25A7%25C4%25B1&action=update&serial=801F026EFE4E

which is definitely not the desired thing.

Add the NE flag to you RewriteRule to prevent this. For example,

RewriteRule (.*) https://%{HTTP_HOST}%{REQUEST_URI} [NE]

#apache #rewriterule

JSESSIONID in the request

If you are using Tomcat, and you want to append the session ID to the request, you can add it to the end of the request after a semi colon.

https://example.com/listStuff;jsessionid=1BCE434F4F9952E3E7712096AAA678EA.node1?id=12

Unix Time in MySQL

Use FROM_UNIXTIME() and UNIX_TIMESTAMP() functions to convert between MySQL dates and Unix epoch time. But beware: these functions work on seconds instead of milliseconds.

#mysql #unix

Bash Variable Test

To check if a variable is null, use -z:

$ [ -z "$MY_VAR" ]; echo $? 0 $ MY_VAR="a" $ [ -z "$MY_VAR" ]; echo $? 1 $ MY_VAR= $ [ -z "$MY_VAR" ]; echo $? 0

To check if a variable is not null, use -n:

$ [ -n "$MY_VAR2" ]; echo $? 1 $ MY_VAR2="a" $ [ -n "$MY_VAR2" ]; echo $? 0 $ MY_VAR2= $ [ -n "$MY_VAR2" ]; echo $? 1

Sed Tricks

Now, the linux utility sed is awfully useful, if I may say so. However, it is really hard to keep in mind the tricks of the language. I'll list below the few different uses of sed I've encountered so far.

Replace all occurances of a regular expression in a file,

$ sed 's/regexp/replacement/g' filename > output

If you want to edit the original file, then use the -i switch.

$ sed -i 's/regexp/replacement/g' filename

If you want to print a particular line in a file,

$ sed -n '10p' filename

prints the tenth line of the file.

If you want to change a particular line in a file,

$ sed '10p c changedline' filename

You can also strip away unwanted text. For example,

$ sed -n 's/$[0-9]\{1,\}$ .*/\1/p'

will only print the numbers followed by at least one blank. Those lines that do not have a number followed by a blank are omitted (note the -n switch). Here, "\1" makes the first matching group in the regular espression to be printed.

#linux #sed

Make a hotcopy backup of SVN

Use svnadmin.

$ svnadmin hotcopy [repository-path] [target-directory]

For example:

$ svnadmin hotcopy /usr/local/svn-repositories/ ./svn.backup.130513

#linux #svn

Add new repo to SVN

This process is a bit strange. You have your files in a directory on your local computer. First you import the directory to SVN.

$ svn import [project-path] [new-repo-address] -m [log-message]

Now your project is in SVN, but your local files are not a working copy. You need to checkout a working copy to your computer. However, it is perilious to checkout to your project directory, because svn will complain about the files there. So, it is better to checkout to an empty folder.

$ svn checkout [new-repo-address]

An example would help clarify things. Below, the contents of the directory ~/myProject is first imported to SVN, then it is backed up just in case. Finally the contents are deleted and a working copy checked out.

~$ cd myProject ~/myProject$ svn import . https://192.168.1.100/repos/myProject -m "Initial import." ~/myProject$ zip ~/myProject.zip ./* ~/myProject$ rm -rf . ~/myProject$ svn checkout https://192.168.1.100/repos/myProject .

#linux #svn

Linux date timestamp

Format the date thusly:

$ date +"%y%m%d-%H%M%S" 130509-154140

#linux #date

Find Target of a Symbolic Link

Use readlink -e to follow a chain of links and find the final target:

readlink -e path.to.link

Also handy for finding the actual executable for commands:

$ readlink -e `which keytool` /usr/lib/jvm/java-6-openjdk-amd64/jre/bin/keytool

REST with cURL

cUSL is versatile. You can specify HTTP parameters and commands with -d and -X options.

For example, if you want to PUT some data you can do this:

curl -d "data1=value1&data2=value2" -X PUT http://www.example.com/action

#linux #curl #REST

Trending Blogs

Recently Viewed Blogs

Idle Headed Scientist