Bigdatablogs

Venkaiah's technical blogs on Big Data and Datawarehousing

[bigdatablogs]

Using JDBC Connection pool for Netezza backed applications

Using Connection pool is normal for Java web applications or enterprise applications. In case of a web application whose reporting is backed by Netezza, is it a good idea to use Connection Pool? definitely, but the connection pool size should be choosen rationally. As per Netezza docs or forums an appliance support 42 concurrent connections. So as per my experience the application should hold the connection pool which is reasonably half of this size, so that the other connections would be reserved for data loading etc. In our case we used an open source connection pool c3p0 and it was very effective to serve the purpose. After introducing the connection pool in our application the processing time for a job generating reports was cut down by 30%.

[bigdatablogs]

Is C/C++ suitable as backend for web application development

Software industry has been progressing with rapid web application frameworks like Groovy or Grails, Ruby on Rails , Scala etc. Still there are companies which are developing C/C++ backend frameworks as part web application.C/C++ always has been the best choice for embedded systems where memory is a constraint. Is it suitable for web application development?. It would be as long the the framework has the support for the below features

1. Generate the output in different formats like JSON and XML

2. Support to send emails

3. Should be able to generate CSVs and PDFs

4. Database layer to handle DB transactions

5. Transaction handling etc

But is it worth building a C/C++ framework with all these features, keeping aside lot of pre-built frameworks in other languages. It all depends on the company's arichtecture team efficiency. 

[bigdatablogs]

WHERE WOULD YOU MOST LIKE TO VISIT ON YOUR PLANET?

London

[bigdatablogs]

Faster Approach for MySQL Dump file restore

Restoration of huge databases like 30Gigs will take long time restore, mostly because of default settings.In general mysql default settings for innodb of innodb_buffer_pool_size, innodb_additional_mem_pool_size, innodb_log_file_size, innodb_log_buffer_size would be in 8MB,1MB,5MB and 1MB As these buffer sizes are way too small and also every single transaction is committed to disk with default which causes a huge amount of IO for every transaction.   running below queries confirm these  mysql -uflixtrix -p -e "select @@innodb_buffer_pool_size, @innodb_additional_mem_pool_size, @@innodb_log_file_size, @@innodb_log_buffer_size, @@innodb_flush_log_at_trx_commit"   mysql -uflixtrix -p -e "select @@innodb_doublewrite"  would return 1   The restoration time can be reduced by increasing the innodb engine settings as per below steps,   1. sudo su 2. vi /etc/mysql/conf.d/innodb.cnf 3. Paste the following into innodb.cnf:

[mysqld]

innodb_buffer_pool_size = 1024M

innodb_log_file_size = 1024M

innodb_log_buffer_size = 8M

innodb_flush_log_at_trx_commit = 0

innodb_doublewrite = 0

  now restart mysql   4. service mysql restart  and    run mysql -uflixtrix -p -e "select @@innodb_buffer_pool_size, @@innodb_additional_mem_pool_size, @@innodb_log_file_size,  @@innodb_log_buffer_size, @@innodb_flush_log_at_trx_commit"  if these are set to the above values then every thing is fine, otherwise in most cases ( mysql ver 5.5) refuses to start ... (mysql ver 5.1) seems to start but without innodb support you can verify this using  mysql -u flixtrix -plocal -e "show variables" | grep innodb will have result like have_innodb NO ignore_builtin_innodb OFF move ahead to step 4 4.'sudo su' to root again   and run cd /var/lib/mysql 5.rm ibdata1 ib_logfile0 ib_logfile1   now restart mysql  6.service mysql restart   7.Run the command mysql -u flixtrix -plocal -e "show variables" | grep innodb  you should see mysql have_innodb YES and the other variable set at step 3.   With these changes the datbase dump should get restored in mins (with default mysql settings it was taking 18hrs+ in my case), this can reduced by further varying the innodb settings. PS: please note that these results are on a PC with dual core procy and 4gig ram