Coffee,coffee,coffee,coffee.. water

[coffeecoffeewater-blog]

Local Jenkins in docker using GCP code repositories

Today I decided to setup my own jenkins CI server that runs on my desktop, using docker seems like a good idea. I also needed it to have access to GCP source repositories.

Here's what I did:

Create a volume and run a vanilla jenkins docker container which mounts it as it's home directory like so:

docker volume create --name jenkins_home --driver local docker run -d -p 8080:8080 -p 50000:50000 --name myjenkins \ -v jenkins_home:/var/jenkins_home \ jenkins

Create a service account just for jenkins and add read only source repository access to the role. This will give you a .json file with a private_key and associated project id.

Copy this into the jenkins_home docker volume

docker cp certs.json myjenkins:certs.json

You will need this to get a tty within the docker container:

docker exec -t -i myjenkins bash

Within the docker container:

Install the google-cloud-sdk in userspace into the jenkins_home volume, /var/jenkins_home/google-cloud-sdk.

curl https://sdk.cloud.google.com | bash # and follow instructions (there's a headless version of this too)

Activate the service account

gcloud auth activate-service-account --key-file certs.json

Tell git to use the gcloud git credential helper

git config --global credential.helper gcloud.sh

Within the jenkins UI, add gcloud tools to the path. There's an environment variables section, set name: PATH and value: $PATH:/var/jenkins_home/google-cloud-sdk/bin.

And now we are done! You can now create jenkins pipelines that check out GCP source repositories using their respective https git address. eg:

https://source.developers.google.com/p/my-project-name/r/my-code-repo

You don't have to do this again if you recreate the container, as all the changes were within the volume. So you can delete the container and recreate it with the same volume and all will be well.

The process seems a bit more manual than I'd like, but rather than automate it more I decided to document it.

Extra bonus work I should do: use google cloud storage for the volume driver instead of a local volume. This seems like it should be easy.

[coffeecoffeewater-blog]

Hosting python packages privately on s3.

Today I set out to answer the question, what's the best way for me to host my own python packages?

I am using AWS. I want them to be accessible without authentication from within my VPC and closed to the public.

I don't really want to manage and monitor any servers for this so I looked into hosting them on s3. Which led me to this article, suggesting pip2pi.

Experimenting, I made a requirements.txt containing:

psycopg2==2.6.1

Then ran:

virtualenv-3.4 env source env/bin/activate pip install pip2pi pip2pi packages --no-use-wheel -r requirements.txt

This creates a directory containing packages as tgz files.

(env)$ find packages packages packages/psycopg2-2.6.1.tar.gz packages/simple packages/simple/index.html packages/simple/psycopg2 packages/simple/psycopg2/index.html packages/simple/psycopg2/psycopg2-2.6.1.tar.gz

The deeper files are symlinks.

From this we could run our own mini pypi by just serving these files. Eg:

python3 -m http.server

Then

pip install -r requirements.txt \ --index-url=http://127.0.0.1:8000/packages/simple/

There's also an --extra-index-url option if you only want to host your internal packages in their own repo.

So if we want to host these in s3, we could aws s3 sync packages s3://bucket/packages, then enable website hosting on that bucket. But I do not want my packages to be available to the world, only to my ec2 instances within my VPC.

I was led to this stackoverflow answer, which led me to this post about new amazon VPC endpoints for s3. This lets me create a endpoint that allows access to any s3 resource. What this means is that s3 traffic will go via this endpoint instead of via an internet gateway. This also means you can now add security policies on your s3 buckets to limit access to be from that VPC only.

So as directed, I created an VPC endpoint, and added the following security policy to my bucket (change the bucket name and vpc id).

{ "Version": "2012-10-17", "Id": "Policy1234567890123", "Statement": [ { "Sid": "Stmt1234567890123", "Effect": "Allow", "Principal": "*", "Action": "s3:*", "Resource": "arn:aws:s3:::mybucket/*", "Condition": { "StringEquals": { "aws:sourceVpc": "vpc-12345678" } } } ] }

I then turned on static website hosting and voilà. I can use it to install python packages. This is accessibly without any authentication from within my VPC, but when I try from my laptop I get 403 forbidden. \o/

My current thinking is my build scripts will just internally cache the project dependencies so I'm not dependent on pypi. I could also use this to deploy my projects themselves as modules, like so:

aws s3 sync s3://bucket/python/packages packages pip2pi packages --no-use-wheel /path/to/project aws s3 sync packages s3://bucket/python/packages

This adds my project as a package, adds any new dependencies and updates index.html.

My only concern is there's a race condition if two projects build at the same time. The index.html might end up having only one of projects, but I think this would only matter when the project is brand new. The index.html only points to the module names, not the individual versions.

[coffeecoffeewater-blog]

what goes around comes around

RC has overlapping batches. That means that half the people leave just as you’re getting to know them. Then a whole bunch of new people arrive.

Both of these groups are nice people. Excellent people in fact. Lovely smart people I am glad I met.

But let’s be honest... The old batch thought they owned the place and the new batch are impostors.

At RC the space belongs to us and it also belongs to no-one. This is weird and excellent.

RC is full of contradictions like, isn’t it grand to have all this free time to do anything you want, but why do I feel like I haven’t got time to get anything done?

It feels like a home, but also it doesn’t need me.

The contradiction I like the most is how the nature of RC builds confidence and enthusiasm, but it also also reminds me I don’t matter in the grand scheme of things. Good things will happen without me.

It’s humbling to be a part of something you love that feels bigger than you.

[coffeecoffeewater-blog]

Pairing

So pairing is great. I know this after 7 months of it, but it’s always good to be reminded.

Last week I paired on a react app, then a bit of golang distributed systems. Good times.

Now I’m pairing on deriving concept maps by implementing AddIntent in haskell. So much shared braining.

It's awesome.

Also cocktails are great.

[coffeecoffeewater-blog]

i make no sense

[coffeecoffeewater-blog]

Overwhelming reasons

This has been a weird week.

I stayed home all of wednesday due to an overwhelming sense of reasons.

I managed to drag myself into RC thursday for the presentations and the goodbye party for batch spring 2.

Sometimes things just knock me sideways. I lose my balance and find it difficult to even know how to think. Brain chemistry that has gotten out of hand, or unforeseen events, or foreseen events I've constructed and walked right into.

Being stuck in a state of speechless, gasping, choking, can be oddly enjoyable. It's like staying awake to avoid the next day. Or fasting.

"It's a trap! The reasons will still be here when you come out! They'll just be older." I've spent a lot of time in there.

I owe people gratitude today. Thank you for asking how I am and if I'm okay. It helped me breath. It all helped.

Then I could nod, I'm good, I'm fine, I'm okay, I'm not. That's something I can work with.

After that the party was good. <3

[coffeecoffeewater-blog]

Spark on EMR

Today I found out that Amazon web services finally supports Spark on Elastic Map Reduce properly. With a drop down and everything!

Elastic map reduce being the thing that will spin up a hadoop cluster on demand. For big data processing bla bla.

Spark being that cool thing that totally replaces map reduce and is more flexible and just better at pretty much everything. What you really get is Spark on YARN (Yet Another Resource Negotiator) with the hadoop distributed filesystem (HDFS) and s3 (amazon's simple storage service) support built in. This is goodness.

Here's how to start a spark cluster on the command line:

aws emr create-cluster \ --ec2-attributes KeyName=INSERT_YOUR_KEY_NAME_HERE \ --applications Name=SPARK \ --instance-groups InstanceCount=1,Name=Master,InstanceGroupType=MASTER,InstanceType=m3.xlarge InstanceCount=3,InstanceGroupType=CORE,InstanceType=m3.xlarge \ --no-auto-terminate \ --ami-version 3.8.0

If successful it will return the cluster id.

This will launch a cluster with 1 master and 3 workers in the region you have configured the aws command line tool to use.

If you don't have the aws command line tool, you can install it using python's package manager, pip.

sudo pip install awscli aws configure

Once your cluster is running:

aws emr ssh --cluster-id CLUSTER_ID --key-pair-file PATH_TO_PEM_FILE

Then

$ pyspark .. ... ....lots of info logs... .. >>> stuff = sc.textFile("s3://somebucket/somestuff.txt") >>> stuff.first()

You can also use ipython, but you will need to set python 2.7 as the default

sudo alternatives --set python /usr/bin/python2.7 IPYTHON=1 pyspark

[coffeecoffeewater-blog]

Recurse week 1

So ends week one at the recurse center and my first week in NY. Summer batch 1 2015.

The intro day was great. I especially liked the very thoughtful setting of context around what we expect to the space to feel like. Inclusive. I wish I had a copy of the speech.

Exhibit A:

This week I have been trying to relearn haskell. https://github.com/tims/learnmeahaskell

I implemented labelling an argumentation framework with grounded semantics. Next week I’ll either try preferred. Or try and connect it up to IO. Eventually I’d like it if you could add new nodes and only reprocess the parts that change.

I’ve also been building a blog tool in react and making it pseudo-flux, just ‘cause. I’d like to have a stab at implementing a pseudo-Relay thingi. https://speakerdeck.com/laneyk/mutations-in-relay

It sounds pretty neat.

So far I’ve had some pretty terrible coffee and few good ones. I’m digging the tasty vietnamese foods.

I have also concluded that flexbox is cool.