Karen McCulloch @k-a-z-mcc95 - Tumblr Blog

Speed up Django Querysets with RawSQL

I’m writing this because the documentation makes it seem pretty scary, but it has saved me a couple of times.

Today, I wanted to filter a queryset based on average of a value from a related model. The average had to be grouped by something too. In SQL that would be something like...

``` SELECT something.*, AVG(relatedthing.att) AS avg_att FROM something JOIN relatedthing ON relatedthing.s = something.id HAVING avg_att > 3 GROUP BY something.thing ```

But, I'm determined to use the Django ORM. So, I started out with

``` queryset = SomeThings.objects.prefetch_related( "relatedthings_set" ).Values( "thing" ).annotate( avg_att=Avg("relatedthings__attribute) ).filter( avg_att__gte=3 ) ```

But, SomeThings is a large set of objects, several thousand objects, and RelatedThings is also a large dataset. There are like a hundred RelatedThings for every SomeThing. So this took a long time to return and I could see Django was doing lots of strange queries in the console...

I then remembered about the RawSQL annotation.

``` queryset = SomeThings.objects.annotate( avg_att=RawSQL( """ SELECT AVG(relatedthing.att) FROM relatedthing WHERE relatedthing.s = something.id """, params=[] ) ).filter( avg_att__gte=3 ) ```

And this ran much faster. It created a query like, but probably uglier,

``` SELECT somethings.id, somethings.x, somethings.s ( SELECT AVG(relatedthing.att) FROM relatedthing WHERE relatedthing.s = something.id ) AS avg_att FROM somethings HAVING avg_aat >= 3 ```

#django #sql

Zeit is awesome! So easy to use :D

allanimemangaquotes

requested by a-be-ba-blom FB | TWITTER | QUOTURES LIST

codingflumps

The programmers wisest friend.

k-a-z-mcc95

ahh rubber-ducking.

docker + heroku

Assumptions:

You have a heroku account

You have downloaded the CLI so you can push to heroku on your commandline

You have already become aquainted with docker.

To ship a docker container to heroku

Install the container registry plugin and login to it.

From the directory containing the docker file, create the heroku app.

build the image and push to heroku container registry

check it out in your browser

heroku login heroku plugins:install heroku-container-registry heroku container:login

heroku create

heroku container:push web

heroku open

TBH I have problems with the heroku open command - it tries to open winebrowser for no reason what so ever. Best open it manually in a browser

python django + docker

GOOD GOD WAS THAT DIFFICULT.

Running a django server in a container is more convoluted that it should be.

Create the app

Create the Dockerfile

Build the image

Run the container

Open your browser: localhost:8080

django-admin startproject <project name>

1 FROM python:2-onbuild 2 ENV PYTHONUNBUFFERED 1 3 4 EXPOSE 8080 5 6 RUN mkdir -p /usr/src/app 7 COPY . /usr/src/app 8 WORKDIR /usr/src/app 9 10 RUN apt-get update 11 RUN apt-get -yqq install tree 12 13 RUN pip install -r requirements.txt 14 15 RUN tree 16 CMD python codingTest/manage.py runserver 0.0.0.0:8080 17 18

docker build -t <image name > .

docker run -t -p 127.0.0.1:8080:8080 <image name >

checkoutmycode

Gonna learn vim!

dash

99 little bugs in the code 99 little bugs in the code Take one down, patch it around 117 little bugs in the code

Alex Shchepetilnikov (@irqed)

Developers/Programmers can relate…

(via dash)

Vim:folds

Folds are handy: They let you choose what you want to see within a file plus make you look extra cool.

Open:

all: zR or zO

current: zo

Close:

all: zM or zC

current: zc

manual: zf i.e. v{motion}zf or zfa} will create a fold from the current position to end of the block.

Toggle:

all: zA

current: za

For more: http://vim.wikia.com/wiki/Folding

vim: splitting

splitting allows us to open many files at once in vim.

Start vim with many files

vim -O <file1> <file2> <file3> [vertically]

vim -o <file1> <file2> <file3> [horizontally]

Open a new file

CTRL + W, S :e <file> [horizontal]

CTRL+ W, v :e <file> [vertical]

:sp <file> [horizontal]

:vsp <file> or :vs <file> [vertical]

#vim

Docker Stuff

Create image

docker build -t <username>/<image_name> .

Create container for image - random port number

docker run -P <username>/<image_name>

docker ps

find the image and look for its host and port

Create a container for an image - specific port

docker run -p <new port>:<exposed port> <username>/<image_name>

Where the exposed port is mentioned in the docker file.

localhost:<new port>

Push it to docker hub

docker push <username>/<image-name>

Copy image from docker hub to bluemix

cf ic cpi <username>/<image-name> registry.ng.bluemix.net/<my_namespace>/<new_image_name>:<tag>

scikit-image and docker

Using scikit-image with docker is proving difficult.

I’ve tried copying over relevant parts of the mobo/sklearn dockerfile into my Dockerfile, https://hub.docker.com/r/mubo/sklearn/~/dockerfile/

Eventually, I got it running with

FROM node:argon MAINTAINER Karen McCulloch EXPOSE 4000 EXPOSE 5432 # copy the application folder inside the container RUN mkdir -p /usr/src/app COPY . /usr/src/app WORKDIR /usr/src/app # install system-wide deps for python RUN apt-get update RUN apt-get -yqq install python python-pip RUN apt-get -yqq install pkg-config RUN apt-get -yqq install python-tk python-imaging RUN apt-get install -y libopenblas-dev liblapack-dev git-core build-essential gfortran libfreetype6-dev libjpeg-dev libhdf5-dev liblzo2-dev libbz2-dev RUN curl https://bootstrap.pypa.io/get-pip.py | python RUN apt-get -yqq install python-dev RUN pip install cython && \ pip install scipy && \ pip install matplotlib && \ pip install six && \ pip install numpy && \ pip install scikit-image RUN pip install image \ pillow # start up the node app. RUN npm install CMD npm start

I've just started using Docker, if you have any suggestions: how to make it tidier, work better,... Just send me a message on tumblr. I would really appricate it.