TK's blog @taekyung - Tumblr Blog

MySQL gap lock, next key lock by example

MySQL’s gap lock or next key lock is a specific mechanism to guarantee it’s isolation level.

Gap lock is set on before or after values of the accessed record. Next key lock is the accessed row lock + gap lock. Therefore they have similar features.

Gap lock is activated in the following conditions.

Isolation level is Repeatable read (default value)

Storage engine is innodb

The query is executed in…

View On WordPress

#lock #mysql

The difference of Isolation level - Read Committed vs Repeatable Read

The difference of Isolation level – Read Committed vs Repeatable Read

MySQL’s default isolation level is “Repeatable Read“. But Oracle’s default isolation level is “Read Committed“. In this post, I will show you the difference by example.

Test environment : MySQL 5.7

Test table

create table lock_test( col1 varchar(10), col2 varchar(20), primary key(col1)); insert into lock_test values('1', 'A'); insert into lock_test values('2', 'B'); insert into lock_test…

View On WordPress

#isolation level #mysql #transaction

Kafka tip - keep the order of delivery

Kafka tip – keep the order of delivery

Kafka guarantees the order of delivery within a partition. That is, the order is not kept among partitions in a topic. There are some tips to keep the order of delivery in a topic.

1. One partition for a Topic

This is what Kafka basically supports.

<Pros>

Simple solution

<Cons>

A partition is applied to only one consumer, which has limitation in scale out

2. Custom partitioner

Partition number…

View On WordPress

#Kafka

Spring Batch tip - Partitioning

Spring Batch tip – Partitioning

Basically, Spring Batch Job runs in single thread. To increase the throughput, we need to parallelize the job by partitioning. The core architecture of Job partitioning is as follows.

Partition Step : The wrapper of common Step with a partitioner and grid size (or partition size)

Single Step : Common Step with a Reader, a Processor and a Writer

Partitioner : The core part of partitioning.…

View On WordPress

#spring batch #Spring framework

Spring Batch - Read once, Write multi example

Spring Batch – Read once, Write multi example

Sometimes, we need to read once, write multi ETL with Spring Batch. Spring Batch supports it with CompositeItemWriter (org.springframework.batch.item.support.CompositeItemWriter).

Sample table schema

CREATE TABLE "TB_SOURCE" ( "COL1" VARCHAR2(10 BYTE), "COL2" VARCHAR2(20 BYTE), "COL3" VARCHAR2(20 BYTE) ); CREATE TABLE "TB_TARGET1" ( "NEW_COL1" VARCHAR2(10 BYTE), "NEW_COL2" VARCHAR2(20…

View On WordPress

#spring batch #Spring framework

More on Spring batch Reader

Spring Batch Job is composed of Reader, Processor and Writer. This post shows some more details on Reader.

Reader is the implementation of org.springframework.batch.item.Reader. I recommend to use the following implementations.

AmqpItemReader (org.springframework.batch.item.amqp.AmqpItemReader) : to interact with queue such as RabbitMQ

FlatFileItemReader(org.springframework.batch.item.file.FlatFi…

View On WordPress

#spring batch

Simple ETL with Spring Batch

Spring Batch is literally a batch framework based on Spring Framework. I usually use it to develop a simple ETL(Extraction, Transaformation and Loading) program.

In this post, I’ll show you how to write a simple ETL program. (This sample is tested on Spring Batch 3.0.10)

Prerequisites

Database (MySQL or Oracle)

Spring batch context database

Spring batch libraries (spring-batch-core, spring…

View On WordPress

#spring batch #Spring framework

A tip on using user transaction on Mybatis

When using transation on Mybatis, there are 3 ways.

Container managed transaction – JEE engine manages transaction

User managed transaction – transaction is controlled by user source code

Spring managed transaction – transaction is controlled by Spring Transaction policy

I usually use Spring managed or container managed transaction. But sometimes, I need to control transaction in source code. For…

View On WordPress

#Mybatis #transaction

Some useful commands to manage Kafka

This post shows some useful commands to manage Kafka. (based on version 2.12)

Detailed information is on Kafka official documentation

Notice that some commands use zookeeper ip/port and others use kafka ip/port

Topic management commands List all topics

${KAFKA_HOME}/bin/kafka-topics.sh –list –zookeeper zookeeper_ip:port

Create a topic

${KAFKA_HOME}/bin/kafka-topics.sh –create –zookeeper zookeeper…

View On WordPress

#Kafka #management

At which point does Kafka consumer start to read

At which point does Kafka consumer start to read? It depends on the following scenarios.

Basic concept

Kafka can have several topics and each topic can have several partitions

Each partition keeps offset for each message

Offset starts from 0 and increases by 1 when a new message is published

Kafka server keeps the next offset to read for each consumer (or consumer group)

Kafka doesn’t have any…

View On WordPress

#Kafka

Managing Kafka log

Kafka log is not an informative file but repository for incoming queue message. Some queue software deletes queue message when it is acked by consumer. But Kafka keeps log regardless of consumer’s ack.

But it is anyhow a file, so it has storage limitation. Now I’m showing how to manage Kafka log. (The following contents are tested on version 2.12.)

Test environment

For test, I created a topic…

View On WordPress

#Kafka

Building a robust Zookeeper client : connection management

To support high available Zookeeper service, we need to configure Zookeeper ensemble. But this is not enough. we need to understand how client manages connection to Zookeeper server.

For test, I set up a Zookeeper ensemble. (port : 2181, 2182, 2183) After then, write a test client with 2 important parameters – 1) connect string 2) session timeout

Case 1) connect string is targeting one zookeeper…

View On WordPress

#Ensemble #Zookeeper

Mockito thenReturn() vs thenAnswer()

Mockito is a very powerful test library for Java. In most cases, setting some mock action with thenReturn() is enough. But in some rare cases, we need to use thenAnswer().

thenReturn() example

First, let’s review the basic use of thenReturn()

public class TestTargetClass { public String getResultWithParam(int param) { return null; } public String getResultWithoutParam() { return null; } }

View On WordPress

#Java #mockito

How Kafka consumer works?

Kafka clients are composed of producer and consumer. Understanding consumer is very important for overall architecture.

A Topic is composed of several partitions (the number is defined when creating the Topic). And Kafka consumer is composed of group and client. (There could be multi groups and multi clients in a group)

Each group and client is identified by group id and client id. (which can be…

View On WordPress

#cluster #Kafka

How Kafka Topic failover works?

After Kafka cluster has been configured, we need to create a Topic which enables failover and data replication.

The conclusion in advance is that if a Topic’s replication factor is more than 2,

Kafka supports automatic leader failover

Data rebalance is supported only in manual operation

Test environment

Kafka 2.12

3 Kakfa brokers (Id : 0, 1, 2) on a PC

To create a Topic

${KAFKA_HOME}/bin/kafka-…

View On WordPress

#failover #kaf

Building Kafka cluster

Building Kafka cluster is crucial for the production system.

Kafka cluster gives the following advantages.

Support for failover in case of a node down

Queue replication

Support for consumer scale out (standalone Kafka also supports consumer scale out)

Step 1 – Build Zookeeper ensemble

Kafka depends on Zookeeper for it’s configuration management. Therefore, Zookeeper needs to run in cluster. (Ref…

View On WordPress

#cluster #Kafka

Trending Blogs

Recently Viewed Blogs

TK's blog