hamblr

about.me/hamburgerkid

[hamburgerkid-blog]

That's all because of R.

Through life in R, I met elephants. Through the elephants, I met brilliant community. Through the community, I met great people. Through the people, I have learned sophisticated software engineering. Through the engineering, I found the way to go. And now I've decided to step out of R and get ahead on the way. Thank you so much for giving me a lot of opportunities. Definitely I'm not here now without it. That's all thanks to R. Goodbye.

[hamburgerkid-blog]

Indexing Paused in Xcode 7

When I wrote a deep? structured list in a class, my Xocde 7 never finished indexing of my Swift project, and nothing can build subsequently.

The list looks like below.

Before

let list = [

    [1: [11, 12, 13]],

    [2: [21, 22, 23, 24, 25]]

]

So I gave up to keep the sturcure and splitted the list to 2 different lists. I know it’s worse as an implementation but needed to carry on my project with Xcode 7.

After

let list1 = [1, 2]

let list2 = [

    [11, 12, 13],

    [ 21, 22, 23, 24, 25]

]

Anyway I like Xcode in usual, to date.

[hamburgerkid-blog]

B

I have learned B in B-boy stands for Break and Bs in AirBnB stand for Bed and Breakfast. I see. By the way I have bought Blowing by B'z at that time. :p

[hamburgerkid-blog]

Why unit test people?

several days ago I got a chance to rethink what the major positive effects unit test brings to us.

and I reached the following conclusion.

it can bring unix philosophy “dotadiw” to the code.

and the above can make the code readable for developer.

it can be a sample code for developer.

it can streamline bug tracking in the long term.

the 1st one seems the biggest point for me.

these pages (POSTD, Developers.IO) helped me, thanks.

[hamburgerkid-blog]

Jenkins behind proxy

filling in the http proxy conf in the advanced plugin management page is required, even if the process is running with JVM options for proxying. xo

[hamburgerkid-blog]

鈴木さんにも分かるネットの未来

before read this essay, I had assumed suzuki-san means anonymous japanese people, but actually that points mr.suzuki from studio-jiburi.

anyway, his broad insight to the future and present and past of the internet is very instructive for me. 

in addition to his point of view as a leading person of japan web sphere, his technical one helps his words convincing much more.

I must learn it, incorporate my tech knowledge into my daily petty thoughts.

[hamburgerkid-blog]

NGINX listen backlog

if you increase the value of net.core.somaxconn of Linux kernel to over 512, do not forget to increase the value of the backlog in the listen directive too. xo

[hamburgerkid-blog]

CROSS 2015

hi, long time no see.

I went to CROSS 2015 on last thursday for the first time. it had several technical talks but also emotional ones.

let me pick up my highlights here. (I will translate into english sooner or later. :p)

1. 恋愛至上主義

挿入至上主義

sexの回数を、射精の回数でカウントしているよね

sexless解消の極意

キスを改まってしてみて下さい キスを大事にしてみて下さい

sexの極意

too hard もっとやさしく too fast もっとゆっくり too much いろいろ試しすぎ too heavy いれなくてもいいじゃん

2. 銀河と小宇宙

- 導入時の障壁

新技術を取り入れるにあたって、大きなチームをどう教育するか 現在利用中の技術にかかりきりで、手が回らない 導入時の障壁はない、が、運用、第2導入(全社展開的な?)時に障壁がある 利用者と非利用者の間で知識の極端な分断 ノウハウの共有

3. エンジニアの幸せな未来

働きやすさと働きがい 不幸にならない 会社は驚くほど簡単につぶれる e.g. 雪印, ダイエー 転職できる技能を持ち続ける 上手くさぼる 他人の働きがいをどれだけ考えられるか 自分に合った大きさの組織で働く 勉強して下さい、特に基礎技術、数学、などなど

4. 全文検索エンジン

comparison between solr, groonga, elasticsearch and cloudsearch.

- es

Q: 多機能をひとつのクラスタでまかなうのか? A: 目的により分かれていることが多い Q: kibanaで無茶なクエリされた時の対応は考えられているか? A: kibana側を改善していっている Q: kibanaのバックエンドを柔軟にする予定や要望はないか? A: 基本esのためにkibanaがある Q: 人気の秘訣は? A: kibanaの見栄えが良い、クラスタ構築の敷居の低さ

- solr

Q: 最適なjvmオプションを見つけるために工夫していることは A: gc-logをzabbixで監視、slave一台に適用してみる Q: 辞書が辛いという印象があるがどう対応しているか A: (didn't catch) Q: ユーザーの満足度をどのように計測しているか A: 複数の指標を組み合わせている Q: jsonでクエリーすることの楽さ辛さ A: jsonだとdiff見やすい Q: 他に比べて性能は A: 使い方ひとつ

- cloudsearch

Q: cloudsearchにsolrの辞書を流用できるのか A: kuromoji互換、synonymも使い回せる

- groonga

Q: (didn't catch) Q: 他に比べてgroongaの分散機能の良さは A: mysqlの仕組みに乗っかれる

- sum up

Q: 全文検索クラスターのオンプレとhostedの流れ A: cookpadがawsでcloudsearchではなくsolrを使うのは、ビジネスのコアなので、自由度を重視した A: chatworkは自由度より運用の負荷を優先した

at last, I would like to say thank you to all the persons who involved with this event. I enjoyed it  and will go one next year too. 

[hamburgerkid-blog]

AWS Summit Tokyo 2014 Day 2

Let me open my note of the day 2 sessions as well.

- keynote skipped

- gov system with AWS everyone uses AWS in US gov.

- inside of game infra with AWS for PSN/PS4 PSN managed by sony entertainment network (not by SCE) set subnet mask to 25 bit 3rd number signify the service. ease to see the service from the ip address. use EIP rather than NAT to make maintenance easy. use chef repository to store EC2 instance info by the instance id to automate.

- big data and open data in cloud key1. cloud x crowd = open innovation key2. platform key3. code for X key4. education wcould : w stands for workplace lab.cloud BODIC.org so call, BODIC is a platform for bid/open data, crowd(code for x) and edu. Kaggle : crowd sourcing for data science

- kinesis official fluentd plugin for kinesis provided at github. MQTT adapter also contributed. hori-san : before kinesis, manges internal monitoring (billing) system team -- issue in the monitoring scale interval ops cost -- needs from user real time elastic keep everything reuse > very similar to the issue. -- tips hava to handle duplication by user itself. need to match with customer demand (use case).

- dynamo db in a nutshell, that is a near perfect KVS

- game platform with cloud - JAWS skipped

I really enjoyed the event as usual on both 2 days. Thank you!

[hamburgerkid-blog]

AWS Summit Tokyo 2014 Day 1

Let me open my note of the day 1 sessions.

1 keynote - CTO Dr.WV we are really customer first company. - monex securities create mission critical product by themselves (in house) with using AWS Prof.Kitsuregawa share intermediate date rather than only result one by using cloud. 2. AWS accelerates edu tech easy to stop using (shut off) can make people think big. cost only use no longer possible to use. easy to stop can mean easy to start.

3. movie ad by Yako-san and 2 guys. migrate hadoop to redshift reduce cost by 50%. consulted by Flydata. - key to use redshift at least TB size short interval minute to hour. structured data.

4. targeting ad delivery system by CA ad tech studio delivery logs in dynamo delivery flags in elastic cache analytics/reporting with redshift

5. CS frontier along with cloud every field can be powered 'by IT' big data = evidence leads innovation. formerly with intuition but now can be based on the evidence. magnetic disk has increase the limit of size but very hard to improve its i/o speed. flush also meets similar situation. and store all data in flush seems hard at this moment. formerly computer tends to focus on compute but nowadays archive and store are more. 'automatic reorganisation' implemented in storage array controller by his lab. 'collaborative green storage' sleep part of storage set until in demand and 80% electricity saved. - pros of using cloud for CS easy inventory flexibility of HW spec - cons consistency reproducibility black box of physical settings, e.g. iops guarantee (but basically good for users.) - challenge de-virtualization on virtualisation performance isolation

6. amazon app store 7. panel talk skipped

Thank you!

[hamburgerkid-blog]

JAWS DAYS 2014

2 weeks ago I attended the event. belatedly let me write down my feedback memo here. It took relatively long time to recap my thought on that event but yesterday that finally figure out in my head.

A large number of attendees are there and the immutable infrastructure track was always packed in particular. I also wanted to catch that track but too crowded space to get into it so that basically I went to iJAWS track. luckily mosts of JAWS users weren’t interested in English talk. :p

- Immutable Infrastructure

Unfortunately still I’m not quite familiar with this field. but even I can feel that cloud native architecture will fit with this new movement. it will bring us no more care of idempotent and disposable infra and so on. I believe this big wave will strongly change our near future infra design and operation style, like plastic bottle for portable drink.

- English

I could ask question in some of the sessions, and in iJAWS track I could do it at every ones. I suppose my questions could hit relatively good point. that experience encourages me to learn English from now on too.

- Consideration

I needed about 2 weeks to sum up this but finally figure out it.

Of course JAWS stands for Japan Amazon Web Service and that event is intended to AWS user and every attendee loves AWS. but like the immutable infrastructure track, high profile session are not just focus on AWS but furthermore on fundamental topic through AWS. and I think the other interesting session provided by the SA also has this tendency.

Every time I learn this kind of thought, the key point still comes from very basic/fundamental computer science knowledge naturally, when I join AWS related event.

You can learn a lot of meaningful things though AWS, even if you don’t have interest in AWS. I strongly suggest that you come and join it next time!

Thanks,

[hamburgerkid-blog]

ElasticSearch Study Session #3

Last friday, I tried to attend the ElasticSearch (ES) study session.

It was already 3rd session but I joined the series for the first time. Let me write down my feedback here.

Geohashing with Elasticsearch

By Mr.Florian from Elasticsearch Inc,

At that time, the main topic is its Geohash implementation. Basically there are roughly 4 ways to query the location to search engine.

1. Bounding box

2. Geo distance

3. Polygon

4. Geohash

In these ways, Geohash is more familiar with data storing in search engine. Because Geohash can be stored in search engine as its term easily. 

Geohash is composed of multiple layer and each layer consists of 32 cells based on Base32 encoding. So every geo informatin convert into bit code and search engine can store it as string type data easily.

From the others

ES is very good to run on AWS. elasicearch-aws-plugin is really useful to manage ES on AWS. If you don't use VPC, you can separate ES clusters by its tag or its security group settings.

And also node.js is nice to use ES because of its REST API.

yamabiko, a data replication library from MySQL to ES created by @yoshi_ken san, looks great!

Sense, a chrome extension to access ES, is helpful to check ES data.

Its schema design is very important because...

Index : its design can heavily affect the query performance, scalability and maintenance-ability.

Routing : can query faster but requires considerable index design in advance.

Mapping : hard to change the definition after launching your service because it requires reindexing.

Inner object or Nested or Parent/Child (P/C) data : naturally everything has trade-off. But in particular the P/C needs to define when you create index.

Anyway, ES seems still young compare to Solr, and will grow up more and more rapidly. But most of attendees seems to prefer it to Solr due to its friendly REST API. I think this simple clear motivation in user (only in attendees?) can help ES direction to head for the future.

Thank you.

[hamburgerkid-blog]

Cassandra Summit Japan 2014

Last Friday, I attended the event. Let me share my feedback here.

Cassandra community in japan

In the opening talk, Tomita-san mentioned the tough situation of the community.

Current status of cassandra community in japan is tough. Few japanese docs on web. Even the small amount of public docs are tend to out of date. Poor relationship between the developers, unfortunately. But now the core community member has been preparing the consortium to solve the above issues.

Cassandra 2.0 and 2.1

Mr.Jonathan Ellis explained some of the new features in the 2.x. the following points seem significant in particular, I think.

1. Consistency level 'Serial'

we can see the details more in their blog post.

2. On-heap/Off-heap (Memory optimisation)

Utilize memory space outside of the heap to access the metadata more quickly than from disk.

3. Hyperloglog (HLL)

Use the algorithm to improve Bloom filter quality in case of inefficient SSTable size for compaction. 

Paxos

Every Datastax guys emphasized its low performance again and again. They implement that consensus algorithm to add the new consistency level by themselves without any external product such as Zookeeper but also noted the trade off to the performance.

100 nodes cluster admin operation in CyberAgent

@oranie-san from CA made presentation about their experience through the operation with the version 1.1 cluster.

His presentation was really straightforward and practical. everyone seemed to learn the difficulty and the strength of Cassandra from this.

Anyway, I have never known where the famous phrase comes from. apparently that is from CA.

'鬼の哭くシステム'

But unfortunately and naturally any Datastax guys seemed not to see the joke at that moment.

At last, I just would like to say thank you to all the host members! I really enjoyed it. Thanks!

[hamburgerkid-blog]

MySQL Cluster Casual Talks

On last Thursday, I attended the event. It was first time for me to attend the event related MySQL. The atmosphere was casual and relax but the contents were interesting and helpful.

Let me share my feedback here.

By the way, at this time the details mainly refer to the version 7.3.

Full ACID properties support

No SPOF, good for DR because it can be composed of nodes across data centers and so on, they mentioned and emphasized some good points of it.

But just for me the full ACID support can be the key feature as a distributed product. for MySQL guys it seems natural thing but I think  MySQL Cluster can stress the ACID support more if compared to other distributed system.

Strict limitation of the scale out

An expert said, MySQL Cluster can have 48 nodes as the SQL node and also 256 as the data node at the most.

I was surprised with the strict number. as far as I know all the other distributed system doesn't have such a concrete number of limitation. I'd like to check the reason of it later.

Notes from experience with MySQL Cluster for 4 years

Another expert gave presentation based on his experience. It was very informative for me. That kind of topics will never be written in the white paper. 

If GCP (Global Check Point) stops, your system severely damaged. Adjust the threshold for it.

Data node rolling restart is enable, but if you do it without service out, you must consider a plan for coincident errors during the restart in advance.

Hard to locate bottleneck. It seems to me that any distributed system requires monitoring system to be maintained.

Take care of data life cycle. For instance, if 'DELETE' data to make spare memory space without restating the data node, the spare will reuse only for the same table.

NDB has online backup function, but recovery data with it will be very slow. 

QA

An attendee asked some questions. One of them is very interesting for me.

Q. How can we design data schema for NDB storage engine ? Or can we reuse data schema for InnoDB to it ?

A. There’s only one way to find out. :p

Not a silver bullet

Anyway, there is no silver bullet. Just demonstrate it and go for yourself!

My impression

Before decide to go with MySQL Cluster, do consider MySQL.

At the last, I'd like to thank all the community member involved and GMO guys. I had good time in the event. Thanks so much !

[hamburgerkid-blog]

AWS Summit Tokyo 2013

I attended the event on last Wednesday and Thursday.

This is my brief impression of that.

For 2 days, the event hall was really packed with audience. even a SA acted as a receptionist to clear up stack of the crowd on the 2nd day. AWS seems to be attracted high attention. Several guest speakers also mentioned that they could see the event hall filled up with audience for the first time.

I took my time at the Tech track mostly and had about 10 sessions. But I would like to talk several impressive points only here.

Core Competence

In each key note, the executive said that AWS can help you to focus on your core competence. It seems to me that if your business strength requires your own infra, you should make use of it mainly and also must keep it same service level to AWS to survive in your market. because your competitor can choose AWS or other cloud. How about your company?

GameDay

They seem to hold the game day regularly. The game can mean practice in this context. On that day, they intentionally break their own system partially and test the availability and check the recovery plan. Like fire fighter, practice for emergency is very important.

Paxos

A famous consensus algorithm. Anyway all engineers involved with distributed computing must know it. I began to read the paper again. Let me learn more and more. Thank you for the nice notice.

SA

I think SA is their crucial front man but their skill level seems to start scattering a little. I hope they keep the brilliant status, even if the member increased.

NetApp girl

At the last, a NeApp girl overwrote everything I learned for the 2 days. Too hot!

@tagomoris photo!! twitter.com/muga_nishizawa…

— Muga Nishizawa (@muga_nishizawa)

June 6, 2013

Through this event, I think again that I have to learn basic technology skill more. whether cloud or on-premises, I believe everything consists of 0 and 1.

Thank you very much for the interesting event, all AWS guys.

[hamburgerkid-blog]

Deleting Archive To Amazon Glacier Vault Empty Without Archive ID.

I have used Arq to back up my private photo and movie. It’s really useful to back up data into S3/Glacier or so on.

To lower cost, I have chosen Glacier as the storage. But in particular case, Arq has a major problem in handling Glacier.

If User remove a directory from the backup source set on Arq GUI, all backuped data in AWS deleted excluding Glacier vault.

To delete the vault manually on AWS management web console, firstly every user must delete all archives in it and make it empty. But in this case there is no archive ID. Arq completely forget it.

For such a silly situation?, AWS has provided API to get the archive ID as well as other services.

So I created a Python script to empty the vault and publish it to my Github repository. It handles the API through boto framework.

If you also have trouble with abandoned archive. Let's try it!

[hamburgerkid-blog]

Executing Bash Function In Parallel.

We can use xargs command for the parallel execusion. For instance, see below.

#!/bin/bash

source `dirname $0`/external.sh

function func_a () {echo "hello ${1}. I'm $ME."}

function func_b () {func_a $1}

export -f func_a func_b

export ME=$ME # Variable ME is from external.sh.

for name in 'andy' 'bob' 'charlie'

do

    xargs -L1 -n1 bash -c "func_b ${name}" &

done

exit 0

This sample seems too small to parallel processing. but you can utilize this approach in your heavy workload.

By the way, GNU Parallel can be your alternative.