Notes to Myself

Talat UYARER's notes to himself taken in the Wonderland World

[talatuyarer]

How to use a self signed certificate repo with Gradle

Sometimes we need to use repos by self signed certs. Even Though you import self sign cert on your jvm. Gradle ignore to use keystore. The issues is mostly likely “Exception in thread “main” javax.net.ssl.SSLHandshakeException” or packages that can’t be downloaded because of an invalid certificate.

To be able to use self signed cert repo you should follow these steps:

Export certificates of Your Artifactory Open your artifactory in browser and export its certificate (in order to get certificate from chrome browser open console, move to security tab and click view certificates, after that you will be able to export by dragging certificate icon to desktop).

Import self signed cert to cacerts keystore

keytool -import -trustcacerts -alias mvn -file <self-signed-cert-file> -keystore $JAVA_HOME/jre/lib/security/cacerts -storepass changeit

Add JVM params on gradle.properties file On the project’s root directory create or edit the gradle.properties file and add the line below. org.gradle.jvmargs=-Djavax.net.ssl.keyStore="$JAVA_HOME/jre/lib/security/cacerts" -Djavax.net.ssl.keyStorePassword=changeit

Now you are ready to use self signed maven repo. 

[talatuyarer]

gRPC Performance Optimization

Streaming is more efficient than sending a separate RPC for each chunk of big data, but less efficient than a single RPC with all chunks in a repeated field. The overhead of streaming can be minimized by using a batched message type.

[talatuyarer]

29 yaşındayım. 5 yıldır bir firmada arge şefi olarak çalışıyordum. Malum durumdan ötürü firma battı malesef. 8 aydır işsizim. Yazılım öğrenmek ve yazılımcı olmak için çok mu geç? Sadece fikrinizi almak istedim...

Merhaba, Bence hic birsey icin gec degil. Eger yazilim isinden zevk alacaginizi dusunuyorsaniz. Kesinlikle deneyim derim. Akliniza takilan baska sorulariniz varsa cevaplamaktan memnuniyet duyarim.

[talatuyarer]

How to reset Bash`s Cache

Today I learned Bash has a cache for recent running command. When I've run iPython previously in a shell session, that command was cached. Thats why even I run in virtualenv it still run my local machine’s ipython. hash command save my life. hash command in Linux system is the built-in command of bash which is used to maintain a hash table of recently executed programs. It remembers and shows the program locations. It will give the full pathname of each command name.

From your bash shell, run: 

hash -r This command will reset bash’s cache. it refered to the version of iPython that I installed in my virtual environment.

[talatuyarer]

Life saving Bash Settings

I saw this settings on a tweet flood. I did not want to lose them. I copied all of content in here. Thanks @kadirsert 

If you are continuously running shell commands on more than one Linux terminal, probably you want all of the shell (mostly bash) prompts to remember any command from any terminal.

With the following environmental variables to in the .bashrc file, you can do it so

Ignoring duplicate entries:  export HISTCONTROL=ignoredups:erasedups

large history size: export HISTSIZE=102400

This is for a big history file: export HISTFILESIZE=100000

Appending commands to history file: shopt -s histappend

Last but not least, this is for saving and reloading the history after each command is run: export PROMPT_COMMAND="history -a; history -c; history -r; $PROMPT_COMMAND"

Source: https://twitter.com/kadirsert/status/1092694397441978369

[talatuyarer]

How to fix PostgreSQL error "duplicate key violates unique constraint"

If you get this message when trying to insert data into a PostgreSQL database:

ERROR: duplicate key violates unique constraint

That likely means that the primary key sequence in the table you're working with has somehow become out of sync, likely because of a mass import process (or something along those lines). Call it a "bug by design", but it seems that you have to manually reset the a primary key index after restoring from a dump file. At any rate, to see if your values are out of sync, run these two commands:

SELECT MAX(the_primary_key) FROM the_table; SELECT nextval('the_primary_key_sequence');

If the first value is higher than the second value, your sequence is out of sync. Back up your PG database (just in case), then run thisL

SELECT setval('the_primary_key_sequence', (SELECT MAX(the_primary_key) FROM the_table)+1);

That will set the sequence to the next available value that's higher than any existing primary key in the sequence.

Reference: https://hcmc.uvic.ca/blogs/index.php?blog=22&p=8105&more=1&c=1&tb=1&pb=1

[talatuyarer]

How Spark SQL evaluate our Expressions

There are different evaluation paths in Spark SQL:

Interpreted. Expressions are evaluated using an eval(...) method. Plans are evaluated using iterators (volcano model). 

Expression Codegenerated. This means that all expressions are evaluated using a code generated function. Plans are evaluated using iterators.

Wholestafe Codegenerated. All expressions and most plans are evaluated using code generation.

For more information: https://issues.apache.org/jira/browse/SPARK-17728

[talatuyarer]

Google Developers Web Updates contains the latest news from the Chrome, looking at new features on the open web and in Chrome DevTools.

It is amazing :) 

[talatuyarer]

Java happens before rules for Concurrency

The rules for happens-before are:

Program order rule. Each action in a thread happens-before every action in that thread that comes later in the program order. Monitor lock rule. An unlock on a monitor lock happens-before every subsequent lock on that same monitor lock. Volatile variable rule. A write to a volatile field happens-before every subsequent read of that same field. Thread start rule. A call to Thread.start on a thread happens-before every action in the started thread. Thread termination rule. Any action in a thread happens-before any other thread detects that thread has terminated, either by successfully return from Thread.join or by Thread.isAlive returning false. Interruption rule. A thread calling interrupt on another thread happens-before the interrupted thread detects the interrupt (either by having InterruptedException thrown, or invoking isInterrupted or interrupted). Finalizer rule. The end of a constructor for an object happens-before the start of the finalizer for that object.

[talatuyarer]

HDFS hflush vs hsync

hflush:  This API flushes all outstanding data (i.e. the current unfinished packet) from the client into the OS buffers on all DataNode replicas.

hsync: This API flushes the data to the DataNodes, like hflush(), but should also force the data to underlying physical storage via fsync (or equivalent). Note that only the current block is flushed to the disk device.

[1] https://github.com/apache/hadoop/blob/trunk/hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/DFSOutputStream.java

[talatuyarer]

How to really persist your file in Java

Use FileChannel.force(boolean) or FileDescriptor.sync() to force data to be persistent on disk. Either of them can work. FileChannel.force use FileDispacther.force[1] and it calls fdatasync or fsync in Java 8. 

When you use OutputStream.flush, it does not guarantee the data to be written to disk, just flush it to OS. Better to use FileOutputStream.getChannel().force(true) or FileOutputStream.getFD().sync() to guarantee the persistency, performance might not be good.

Special Thanks to Yongkun. He wrote very good blog post. [2]

[1] http://hg.openjdk.java.net/jdk8/jdk8/jdk/file/687fd7c7986d/src/solaris/native/sun/nio/ch/FileDispatcherImpl.c#l141

[2] http://yongkunphd.blogspot.com/2013/12/how-fsync-works-in-java.html

[talatuyarer]

Bloom Filters

Good tutorial for Bloom Filter understanding: http://billmill.org/bloomfilter-tutorial/

Bloom filters use case is following:

You have very large data sets that typically don't fit in memory and you want to check your element it contains or not contains. Obviously It works very well for not contains detection.

if the bloom filter gives a hit: the item is probably inside if the bloom filter gives a miss: the item is certainly not inside

How can I use in Java. Guava Provide a library for Bloom Filter:

https://github.com/google/guava/blob/master/guava/src/com/google/common/hash/BloomFilter.java

m denotes the number of bits in the Bloom filter (bitSize) 

n denotes the number of elements inserted into the Bloom filter (maxKeys)

k represents the number of hash functions used (nbHash) 

e represents the desired false

positive rate for the bloom (err) If we fix the error rate (e) and know the number of entries, then the optimal bloom size 

m = -(nln(err) / (ln(2)^2) ~= nln(err) / ln(0.6185)

The probability of false positives is minimized when k = m/n ln(2).

[talatuyarer]

Disable auto-opening nautilus window after mount your phone

On the Terminal,

gsettings set org.gnome.desktop.media-handling automount-open false

[talatuyarer]

Why does Java's hashCode() in String use 31 as a multiplier?

The value 31 was chosen because it is an odd prime. If it were even and the multiplication overflowed, information would be lost, as multiplication by 2 is equivalent to shifting. The advantage of using a prime is less clear, but it is traditional. A nice property of 31 is that the multiplication can be replaced by a shift and a subtraction for better performance: 31 * i == (i << 5) - i. Modern VMs do this sort of optimization automatically.

(from Chapter 3, Item 9: Always override hashcode when you override equals, page 48, Joshua Bloch's Effective Java)

[talatuyarer]

How to build Hadoop Native Library with Snappy Compression Support

Snappy is a compression library that can be utilized by the native code. It is currently an optional component, meaning that Hadoop can be built with or without this dependency.

Download and compile snappy codecs. or you can install from your distro repo. I installed libsnappy and libsnappy-dev packages from Ubuntu repo. If everything is fine you can use -Drequire.snappy to fail the build if libsnappy.so is not found. If this option is not specified and the snappy library is missing,silently build a version of libhadoop.so that cannot make use of snappy. After than You just need to enter below command:

mvn clean package -Pdist,native -DskipTests -Dtar -Drequire.snappy

If you build snappy and It is located different place you can use this parameters

-Dsnappy.prefix to specify a nonstandard location for the libsnappy header files and library files. You do not need this option if you have installed snappy using a package manager.

-Dsnappy.lib to specify a nonstandard location for the libsnappy library files. Similarly to snappy.prefix, you do not need this option if you have installed snappy using a package manager.

-Dbundle.snappy to copy the contents of the snappy.lib directory into the final tar file. This option requires that -Dsnappy.lib is also given, and it ignores the -Dsnappy.prefix option.

After compiling finished you can find your native libraries 

<source_folder>/hadoop-dist/target/hadoop-2.5.2/lib/native/

Good luck

[talatuyarer]

How to connect HBase using Apache Phoenix from Pentaho Kettle

In Our Office Mustafa needs to connect HBase from Pentaho Kettle. We find a solution for the problem. I want to share who need this.

Download suitable  Apache Phoenix version for you from the website: http://phoenix.apache.org/download.html

Copy two files from source directory to: PENTAHO_INSTALL_PATH/lib/ phoenix-core-4.3.1.jar phoenix-4.3.1-client.jar

Create a new project in Pentaho: File -> New -> Transformation

From left pane select **Design -> Input -> Table Input **and drag it to your transformation

Double click to your table input and give a name to your step

Click new next to Connection select box to create a new database connection

Give your connection a name (Ex: Phoenix) Connection Type: Generic Database Access: Native (JDBC) Custom Connection URL: Your ZooKeeper Hosts (Ex: jdbc:phoenix:localhost:2181:/hbase) Custom Driver Class Name: org.apache.phoenix.jdbc.PhoenixDriver And then click Ok to close database connection settings popup

Thanks to Mustafa Artuc

[talatuyarer]

Bottom Line Tuning Tips for G1GC

When I read HBase User Maillist. I come cross Bryan Beaudreault’s experiences[1] about When they using G1 garage colllector with HBase in Hubspot. I want to note those. You can see below:

- If an allocation is larger than a 50% of the G1 region size, it is a humongous allocation which is more expensive to clean up.  You should want to avoid this.

- The default region size is only a few mb, so any big batch puts or scans can easily be considered humongous.  If you don't set Xms, it will be even smaller.

- Make sure you are setting Xms to the same value as Xmx.  This is used by the G1 to calculate default region sizes.

- Enable -XX:+PrintAdaptiveSizePolicy, which will print out information you can use for debugging humongous allocations.  Any time an allocation is considered humongous, it will print the size of the allocation.

- Using the output of the above, determine your optimal region size. Region sizes must be a power of 2, and you should generally target around 2000 regions.  So a compromise is sometimes needed, as you don't want to be *too* far below this number.

- Use -XX:G1HeapRegionSize=xM to set the region size.  Use a power of 2.

[1] http://apache-hbase.679495.n3.nabble.com/How-to-know-the-root-reason-to-cause-RegionServer-OOM-tp4071357p4071402.html