tglstory

var _gaq = _gaq || []; _gaq.push(['_setAccount', 'UA-17377104-4']); _gaq.push(['_trackPageview']); (function() { var ga = document.createElement('script'); ga.type = 'text/javascript'; ga.async = true; ga.src = ('https:' == document.location.protocol ? 'https://ssl' : 'http://www') + '.google-analytics.com/ga.js'; var s = document.getElementsByTagName('script')[0]; s.parentNode.insertBefore(ga, s); })(); dog lover, data person

[tglstory]

Why you shouldn’t care about quality

Ok, this is a clickbaity title and of course you should care about quality. In this post I want to highlight how one of my projects failed (specifically failed in terms of I failed to write tests) miserably, how things are different now, and what I ultimately learned from the experience.

The Promising Start

Some time ago, I was in charge of writing a software application that:

reads some data from an external source

apply business logic to transform the data

write the transformed data into a data warehouse

Sounds like a pretty standard ETL job. But there was one twist: the application also had to read deltas (user inputs) from the data warehouse and re-apply the business logic and output it again in near real-time.

This core feature necessitated some kind of streaming application, so I decided to use the relatively new (at the time) Kafka stack of Kafka / Kafka Connect / Kafka Streams, now more commonly known as the Confluent Platform. Coincidentally, the platform team supporting the application developers decided to onboard to a Kubernetes stack called Openshift. It felt like some exciting times were ahead, so many cool new technology to work with, so much value to deliver to the stakeholders.

Since everybody was onboarding a new platform, there was a lot of head first learning involved. I wrote multiple posts to the Kafka Google group, had to pull frequent all-nighters, worked weekends, and did some other really unhealthy things I will never consider doing again. But it was definitely worth it, because I more or less finished the application in a fairly ridiculous amount of time. 

Until everything fell apart.

Things Fall Apart

Turns out, business requirements change. After I finished my first implementation, I was trying to improve its quality by post-writing tests and refactoring it because it was so hastily written. Then came the news: I needed to make calculation changes. No big deal I thought, I just need to change the component that deals with the calculation. The change caused unexpected errors in downstream calculations, and as you can probably imagine I started panicking because I couldn't untangle the mess that this project had become. In my head, there was the initial implementation, the half-refactored version, and the calculation fix all munged together and I couldn't think straight. Productivity went down, and I started becoming visibly unhealthy. After I finally made all the pieces work together again, there was another required change that came out of nowhere. And another followed. And another followed. I never found the time to write tests for the entire project.

Hasty development on a tight deadline was not the sole contributor to not writing tests. 

Our Openshift stack did not work well with Kafka Streams applications especially when they were stateful. 

Our jenkins environment was never fully supported by anyone because of the lack of time and resources. 

At one point I had to fork off to a custom version of Kafka because there was a bug on the Kafka side. 

Of course the largest blame for not writing tests is on the engineer, but I believe the engineer being okay with not writing tests in the first place is largely a cultural problem.

How Things Improved

I eventually rotated out of the team because I could not take the stress anymore. The project eventually got rewritten multiple times depending on who was working on the project. Even though I was working on different projects, the engineering environment did not really change for a while. Then came in a few waves of great positive influence.

Cycling of personnel: A lot of oldies who had a lot of tribal knowledge had moved onto different companies or teams, which was great in forcing the tribal knowledge to become documented. Then, some great people came in to backfill those positions and this was a very important part of reinvigorating a stale culture.

Standardizing on development methodology: For a very long while our team members were arguing against each other with the topic of monorepo vs. polyrepo. We eventually decided to go full polyrepo fashion (in compliance with the 12 factor app) as much as other monorepo people wanted to continue using the monorepo. Having this lengthy argument conclude was a great relief to everyone, as topics like these become a massive time sink when left unattended. 

Relying on hosted tools: We decided to use Gitlab to host our code and to run our CI pipelines, and converged onto a hosted instance of Kubernetes. We started using Databricks as our compute environment and scheduled everything with Airflow. Having standard tools for everyone meant that developers had confidence that they will get the required troubleshooting, and their problems will be solved. 

This set of improvements had one effect that surfaced itself after a while: everybody had more time. When engineers have more time, they care about the code they write and they review the code that others write. This leads to a culture in which people care about quality in general. I believe this is the ideal way for an engineer to learn about quality and apply it to their daily work. As I stated in my previous post, if you want to go above and beyond a glorified Googler, you have to slow your work down and actively try to learn new things. And the ideal way to achieve this is with the help of your engineering organization's culture.

I really like this tweet as it perfectly sums up the lesson I learned from my failed project. Yeah, the project failed. But that's okay, that failure doesn't define who I am as a software engineer. I continue to work as hard as I can to foster an engineering culture that helps me, and everyone around me succeed.

[tglstory]

How to improve yourself as a programmer

I was thinking about my work and my hobbies and thought of something strange. For all my hobbies and other interests, repeating the act was a good way to improve on it. However, for programming / coding / software engineering, Googling seems like such a better way to improve than being inside a well and writing code constantly.

If I want to be a better breakdancer, I have to stretch, strengthen my joints, lose some weight, and most importantly practices the moves I want to do. If I want to learn how to dance on a longboard, I need to ride more with both feet, practice carving, and practice dancing. If I want to draw better, I need to sit down in front of a sketchbook and really put in the work to become a better drawer.

However, when I reflected back on my recent engineering work, I realized that I can’t really apply the same principles to it (I was working on this). Specifically, I wanted to write tests for my method that sends a request to a URL and download the contents of the response into a file on S3. If I did not have the option to Google, I would have either abandoned the project altogether after spending a lot of time trying to find what the correct combination of testing framework and dependencies are or continued on writing untested code. The way I improved my code was through Googling, and not writing more code (without Googling).

When I shared this thought with my wife, she said that it's similar to my sister in law's situation. She writes for a documentary series on mountains, and said that most of her work time is occupied with searching for how other people describe mountain in written text. My wife was right (as she always is); it was similar. I continued thinking about the difference between programming and writing.

A writer reads a lot to eventually produce a writing that will be read by other people and in the future be another source of reference. This applies to other academic work as well; academics constantly research what other people have done in order to come up with a unique piece of knowledge, with the ultimate goal of sharing it with the world to make the world richer in collective knowledge.

When we Google, it's 90% because there is a problem we cannot solve by ourselves. And when we find the answer, we celebrate and when we encounter a similar problem in the future, we re-use the solution. So in that sense, programming and academic research is similar.

Unfortunately, industrial software engineering doesn't make your code public. The majority of the work that I've done over my professional career is now the property of the companies I've worked for. I learned a lot through Googling and produced a lot of code, but none of that can be shared the same way as academic results because they are not open source and because products are typically what gets shared with the world, not the code that builds the product.

The technology industry of late is also focused on "crunching" and producing high-quality or multi-featured products in a short amount of time. When engineers are rushed, they react to situations instead of thinking about the code they produce. I need to finish this module by tonight. When I reactively write code without thinking, I will write code that fails. When a failure occurs, I will Google for the solution. I find the solution, but instead of internalizing it and making it part of my knowledge, I have already moved on to the next line of code. When this kind of development pattern becomes a habit, it's hard to remember what the problem was, what the solution was, and what the overall situation of the codebase was at the time of the problem because of a simple reason: there is no thought behind any of it. The code is purely driven by adrenalin.

Occasionally there will be times when you Google things that are not big problems, but things that are easy to forget. For example, how do you cast an int to String in Python? If you know, good for you, but it's not the end of the world if you don't know this because this is just one Google away. Relying on the safety net of Googling as a mentality is definitely something we can improve upon. Why not spend one day of your week reading through how types work in Python? Why not spend a day experimenting with this curiosity and internalize what could be trivially Googled? This kind of activity can be compared to sitting down with a sketchbook and a pencil and drawing to improve.

Through this thought exercise I learned that in order to improve as a programmer, I need to 1) contribute back to the community where I learned from, 2) adapt the speed of my work so I can put thoughts behind the code, and 3) intentionally spend time reading code and learning about things. I hope to put these points into practice and write more about the experience in the future.

[tglstory]

mulholland

[tglstory]

flowers

[tglstory]

want more love

[tglstory]

skies

[tglstory]

20190102

It is my little brother’s 7th birthday.

I still remember the day I discovered your kidney stone. Every day I thank the powers that be that I found out at the earliest moment. When they had to open you up it felt like my heart was opened up.

I still remember the phone call telling me that you have a genetic disorder. I still remember the drive to UC Davis on that rainy, rainy day. I still remember how much you barked in the halls of the hospital. I still remember picking up your little body after you so bravely recovered through the second surgery.

Every day since that day I know you’re living your second life. When I see you asleep curled up next to me I know that you are a tiny tragedy waiting to happen. But I want to give you the happiest life to the extent I can give it. Even though it means that one day you will give me the saddest moment of my life.

Happy birthday my little dog.

[tglstory]

20181225

Today is Christmas.

Christmas seems to lose its special magic year by year, but I don’t necessarily feel worse every year. We had a brief conversation about life in 2018 today morning over tea and some keks and I realized this year has been so kind and generous to us. I’m very grateful for all the improvements I’ve made as a person and we’ve made as a family.

Dog training has been a monumental success in December as well, though sometimes it doesn’t go as planned. Dogs need socialization and positive experiences that don’t lead to barking and over-excitement. This means dogs have to learn from other dogs and other dog owners that it’s okay to meet other walking dogs. If my dog was in a place where he or she could help other dogs, I would totally do it in a heartbeat because I understand how frustrating it can be to have a hyper dog when you’re walking them. Unfortunately not all dog walkers share the same sentiment and it really burns and hurts my feelings when they act like assholes towards us. I mean, I’m sorry I’m trying to improve my dog’s manners?

Rejections and things not going as planned always hurts. Maybe 2019 will be a good year to strengthen my internal muscles.

[tglstory]

20180107

Fuck. forgot to write my diary. :(

The culmination of small successes is working well for me but the one deterrent is the pain in my left hand. I should visit physical therapy again. Guitar is getting better (getting calluses again) but the pain is preventing the development of grip strength. what a shame...

pilates has wrecked me for the last two days. Today we ventured off into little tokyo and I realized how lucky i am to be able to spend time with my wonderful wife. 

[tglstory]

20180105

별 탈 없는 일주일이었다. 얼마만에 이렇게 평화로운 마음을 가지고 일주일을 보냈는지 모르겠다. 여러가지 이유가 있겠지만 좀 꼽아보자면 회의의 부재, 독서, 컴퓨터 줄이기, 기술적 문제 해결 등이 있겠다. 앞으로 이런 주들만 계속되면 좋겠지만 별로 그럴것같지는 않다... ^_ㅜ 남이 어떻게 영향을 주는 것 말고 내가 조절하는 요소들은 최대한 긍정적이고 열정적으로 할 수 있게 해야겠다.

요즘 리크루팅 오퍼가 오는 회사들은 흥미롭다. goop.com, 기억해 두겠어.

새롭게 시도해보는 삶의 자세와 행동들이 효과가 있는것 같아서 좋다. 물론 발전의 양은 매일 고정되지 않아 변동이 있겠지만 항상 목적보다는 시스템을 염두에 두고 내일 하루도 열심히 살아보자. 좋은 주말이면 좋겠다.

[tglstory]

20180104

확실히 게임을 하면 할 일들을 잊게 되고 그 일들은 잊고 나면 다시 생각할 때 귀찮아진다..... 게임을 더 줄여야겠다. 부지런히 씻고 책을 읽고 하루를 마무리해야지.

나는 남을 설득하는 일에 항상 자신이 있었는데 딱히 설득하지 않아도 되는 당연한 것들을 설득해야 하는 날들이 반복되다보니 남과 대화하는것 자체가 귀찮아져서 설득은 더욱 귀찮아졌다. 근데 성공을 위해서는 어쩔 수 없이 내 생각을 효과적으로 전달하는 법이 필요하고, 남을 설득하지 못해 결국 제일 답답한건 내 자신이기 때문에 조금만 더 그쪽으로 용을 써봐야 겠다.

뭔가를 내 마음대로 하려면 두배로 일해야 한다는 말이 피부로 와닿는다. 잠을 좀만 더 줄여보려고 노력해야겠다. 그럴려면 좀 더 일찍 자야겠지.

가사를 트위터에 좀 적고싶은데 너무 오그라든다. 으으.

[tglstory]

20180103

생각보다 매끄러운 하루였다. 회의가 없다는 사실 하나만으로도 업무시간의 질과 하루의 행복도 자체가 달라진다. 회의 자체가 싫다기 보다 회의를 하는 구성원들이 회의에 대한 큰 준비와 생각을 안해오니 수많은 사람들이 모여 시간만 낭비하는 것이다. 이것에 대한 해결책도 한번쯤 생각해 봐야 하나...

노트는 적는것도 아직 습관이 안됐고 보는것도 습관이 안됐다. 이것도 하나의 시스템으로 생각해봐야겠다.

오늘은 오랜만에 낮잠을 잤다. 집에 있을때 생각없이 컴퓨터앞에 앉아있는 시간을 줄이니 자연스레 몸이 원하고 필요한것들을 하게 된다. 오늘은 결국 다시 컴퓨터 앞에 앉긴 했지만 내일은 시간을 더 줄여야겠다. 

기타를 오랫동안 안 쳤지만 딱히 아쉬운게 없다. 한번도 제대로 앉아서 의도를 가지고 연습을 해본적이 없으니 이제 새롭게 시작하는것이라고 생각하고 다시 달려보자.

[tglstory]

20180102

나이는 서른살까지 먹고 멈춘다고 나도 모르게 생각했다. 새 해가 시작한지 하루 되는날 올해도 한살을 먹는다는 생각에 갑자기 가슴이 먹먹해졌다. 바쁘게 움직이고 많은걸 하더라도 한살 한살 흘러가는 나이 앞에선 그냥 아무것도 안하는 어린이 같은데. 

하고싶은 불평 불만들에게 입밖으로 나오지 말라고 했더니 갈 곳을 못찾아 마음을 이곳 저곳 헤집고 다닌다. 아무리 평정심을 가지고 조용히 말하려고 노력해도 어느새 미간이 찌뿌려진 내 표정을 느낀다.

처음으로 메트로놈을 가지고 기타를 치면서, 오늘따라 유난히 배에 힘을 주고 했던 운동을 하면서 모든 행동에 의도를 가져야 한다는 것을 배워본다. 오늘의 이 고통이 내일 뭘 더 잘하게 해달라는 생각보다는 내 의도와 내 몸을 좀 더 맞춰주길 바래. 그러기 위해선 애초에 내가 의도하는게 뭔지부터 생각해 봐야겠다.

숨을 내쉴때 내것이 아닌 것들을 내쉬고 내 안에는 내것만 두라고 하는데 언제부턴가 내 안에 뭐가 있는지를 모르겠고 이 느낌이 싫다. 알맹이가 가득차있던, 생각에 이글이글 불타던 내 자신이 언제부터 이렇게 희미해졌는지 모르겠다.

새소년 노래가 너무 좋다. 요즘은 데뷔앨범 없이도 활동을 할수 있는게 밴드 입장에선 좋겠지? 근데 노래를 많이 듣고 싶은 입장에선 그게 답답한것 같다. 아무리 좋아도 아껴듣지 않으면 금방 질리니까. 

내일은 더 음악을 많이 들어야겠다.

[tglstory]

Baby’s first steps on a Kafka Streams app - Part 1

A few months ago I started working on a new project at work. Its business logic is basically obtaining data from a source, apply some calculations, and write them out to a database. Sounds simple enough. Although the source data is provided at a specified time in a batch fashion, there will be a future requirement of an hourly feed so it seemed like a fairly reasonable idea to try and work with this in a streaming computation model. Given the rise of Confluent Platform, our engineering organization thought we should give Kafka Streams a try- this was an adventure because we were traditionally running spark batch jobs and had limited usage of Kafka itself.

I was inclined to use Kafka Streams for the following reasons:

I never used Kafka Streams before so I was curious

After using Azkaban after Luigi, I had a strong sentiment that tasks should be organized in an event-triggered fashion

I personally did not enjoy using the existing tech stack and wanted to use something completely new

I think I feel comfortable enough with the framework itself to give a meaningful insight, and I want to provide a retrospective for myself about the above three points.

Takeaway: Don’t underestimate the effort of doing something you haven’t done before.

All the business requirements were laid out in a user story, and all of them were fairly simple stories because they were broken down quite a lot. Read in this data source, convert it into avro objects. Ok cool. Seems simple enough. Based on my previous experience with MapReduce and Thrift, it’s a fairly trivial task. I’m sure there’s some things I need to learn about Kafka Connect but it should still be a trivial task. <- This is what I thought, but experience has proven otherwise.

Avro doesn’t have a native notion of an optional field, so you need to search for the best way to have optional fields. Now, why does the schema registry keep denying my messages? Ah, my schema change is non-backward compatible, that took a while to google. Hmm, Avro doesn’t have a concept of datetime? It’s actually an int/long epoch time? Uhh, and I’m supposed to use this Kafka Connect library to convert from java Dates to Avro dates? And I have to use java.util.Dates? Oh crap, and java.util.Date isn’t threadsafe?

The above questions were all answered and the correct implementation took a few weeks to write, because there were numerous unexpected differences from my experience to a new one. In planning for tasks where you haven’t worked with the tech stack, you should aggressively overestimate and consider all unturned stones to be landmines.

Takeaway: Read the fucking documentations

I had a left join that was basically combining two avro objects into one. And since joins are always key-based in the Kafka Streams DSL, this join just has to work. However, I noticed that only some keys are producing the right values and some aren’t. What the heck? Source topic A and B are easily verifiable, I literally see the data there, but why isn’t the join triggering?

Turns out, joins require co-partitioning of the data and that is outlined in great detail in the documentations. Kafka Streams actually is one of the best documented projects I’ve ever used, and I would suggest it as the #1 resource that every Kafka Streams user should know from start to end. 

Lurking around the Github Kafka project and the Google group for Confluent platform taught me a lot about the platform as well. I feel fortunate that I got to work with frameworks that have a great community.

In the next article I will write about why I think KS is pretty cool.

[tglstory]

Around July/August of 2015, Caltrain experienced another one of its frequent failures. I don’t recall the specific reason for the stoppage that day. 

At the time I was still getting used to the whole commuting from San Jose to San Francisco thing. You know, the whole spending a large chunk of your day in a train that had spotty network connections and uncomfortable seats. Wanting to just get home to your wife and dogs. Only to end up in a station that was a few miles away from home. It fucking sucked, but it got me home around the time I want to be home and I could forget about the commute until next morning.

But when the train stops working, that system fails. The part where I forget that this fucking commute is temporary and I can spend the better part of the day at home. Some people just hit up a bar and get some drinks while they’re waiting for the train to get back to normal, but I wasn’t one of those people. I was just so antsy and so fucking... desperate to get back home. I couldn’t stand one more minute in San Francisco.

As much as I value the enjoyment of my own life I also value my wife’s. Coincidentally me being at home for dinnertime is a necessity for the enjoyment of both of our lives. But I didn’t want her to drive all the way to San Francisco; so I suggested that we meet somewhere in the reasonable middle. I would bike to 22nd St. or Bayshore. We got off the phone and I started biking.

When you bike from the 4th & King station to either one of those stations, you go through these neighborhoods that you would never go to on an ordinary day. Mission Bay, Dogpatch. Portola Place and Vistacion Valley. Biking through that neighborhood when the sun sets is a very surreal experience. All these warehouses with broken windows, construction sites that hasn’t been touched for months, people passed out in muni stops. Juxtaposed to one of the most beautiful sunsets I’ve ever seen, what’s left of the sun casting over the ocean. I’m sweating through this neighborhood, trying to navigate through this neighborhood that I’ve never been through before. At the end my wife is waiting for me at the station, and we end our day together. But this experience of biking through that neighborhood is exclusively mine, as we have no reason to bike that path together. 

I occasionally thought I would try that path again with my wife. It gave me this intangible feeling that I’ve never experienced before. Desperation mixed with this hint of adrenalin and awe. Equal portion of greed and guilt. It left a big mark in my heart but I’ve never had a dream about this experience, it must not be something traumatic. That’s nice I guess. We never got to try this bike ride, and I rode through the same path about two or three times and never after that.

At the time I thought my agony was something that I couldn’t leave. I loved living in the South Bay and so did my wife. I didn’t want to move to San Francisco and so did my wife. Other neighborhoods, I’ve heard a lot of bad things about them; there are a lot of complaints about San Jose but hey, we’ve lived here for a while and it’s not fucking intolerable, right? Well, after a year and half, there came a point where I simply could not bear the burden of this commute anymore.

So I tried this thing called disregarding what other people say and experiencing things for myself. We went to all these neighborhoods and places that other people complained about and judged for ourselves. In the end we ended up in a place that was much more easier to commute to but was also not far from where we wanted to be. Later on, where we moved to became a place where we wanted to be. It all worked out well.

And now I think about this experience and it tells me something. Life isn’t really fair, that’s a fact; but that doesn’t mean you should always take things without asking for better. You always have to ask or fight for what you want. You might not get everything that you want, but you might; and it’s better than getting nothing that you want. Go places and ask for things you want. If you’re feeling miserable try everything to escape from that hellhole. When a suboptimal yet appealing offer comes to you be confident enough to say no to get something optimal. Make life something that’s worth living for yourself.

It’s June 2017, almost two years passed since that day. A lot of new things came into my life and I’ve left a lot of things behind. Some things you don’t even realize they’re there, but you can’t separate them from you. I wish the me of today was there next to the me that day. Now that I think about it, I’d say it’s a great fortune that I’ve found a me who can be of help to the past me. Here’s to me. The one person who can completely understand me and completely love me.

[dfincher]

La La Land (2016) dir. Damien Chazelle

[sanguis-potestas-est]