#syscalls

[an-incomplete-heap]

An uncomplete list of C syscalls I used today

getw pread pwrite open close

does this have a purpose? absolutely not. but then again, it’s not for you 

[inercia]

The underlying technology enabling the Windows Subsystem for Linux

WSL executes unmodified Linux ELF64 binaries by emulating a Linux kernel interface on top of the Windows NT kernel. One of the kernel interfaces that it exposes are system calls (syscalls).

[cleberz]

Asael, syscalls e eventos

 Os eventos, no Asael, são um tipo de simulacro do que acontece nos sistemas Unix (e qualquer outro sistema operacional) chamado "system calls" (ou, simplesmente, syscalls).

 E como funciona uma syscall?

 Digamos que estou escreendo um programa e preciso abrir um arquivo. Eu não quero saber detalhes sobre o acesso ao arquivo em disco, qual sistema de arquivos está sendo usado, se os bytes são big ou little endian na arquitetura, enfim, nada. Eu não quero saber de nada, apenas que preciso abrir o arquivo tal e quero lê-lo. Eu passo o caminho para o sistema operacional e espero que ele me dê um file descriptor pronto para uso.

 Detalhes de implementação de como liberar um file descriptor? Ignoro completamente. E a interface para abrir arquivos é a mesma em qualquer sistema "POSIX compliant". Tanto faz se meu programa rodará no Linux, no FreeBSD ou no Mac OS X. A interface, em si, é a mesma.

 E assim deveria ser nos nossos projetos Django. Se eu quero gerar um formulário, eu não deveria me importar com qual módulo e qual classe gera um formulário para mim. Minha app deveria apenas chamar o sistema/framework e dizer: "por favor, preciso de um formulário para editar este meu determinado modelo".

 Nada mais!

 E é isso o que o Asael te ajuda a fazer.

 Em resumo bem rápido:

@events.listener('NOME': u'DESCRIÇÃO')

def meu_listener(args):

 pass

Para chamar a função de qualquer outro lugar:

x = events.trigger('NOME')

 Mais detalhes no próximo post.

[nelhagedebugsshit]

Monitoring EventMachine using /proc

So we run a bunch of EventMachine at Stripe. I personally hate EventMachine, but it's what we've got, and it's probably the best answer if you really want async I/O in ruby.

One question you inevitably find yourself asking the question: How close is my EventMachine worker process to capacity? How many more requests/second can this worker handle?

This is, frustratingly, not a super straightforward question. Because of the asynchronous nature of EM, you might have multiple requests logically in-flight at a given moment, but all asynchronously waiting on I/O, and so the process is quite able to handle more. You can look at CPU usage, but because Ruby is much less good than, say, node.js at being religious about async I/O, our workers do also do a fair amount of synchronous I/O -- so a worker might be burning very little CPU, but spending all its time inside synchronous I/O, and thus not have any spare capacity.

Thinking about it more, we decided that the metric we wanted to measure was, "what fraction of the time is this worker spending waiting inside the EventMachine main dispatch loop?" -- that is, essentially, exactly "what fraction of the time is this worker available to handle additional requests?"

Unfortunately EM doesn't expose any easy hooks (that we found, at least) to measure this. We considered doing some truly awful monkeypatching, but rejected that idea as too risky and grody (although we later learned that Conrad Irwin, developer of a bunch of nice ruby software, had written this monkey-patch already and packaged it -- depending on your tastes for other peoples' monkey-patches, this may be a better idea).

But thinking more, we realized that we don't need any hooks into EM at all: The question we are trying to answer is phraseable in terms of a simple question about the Linux system-call layer: "What fraction of the time is my process spending in the main eventmachine select() system call?"

If we can recognize that externally, we can answer this entirely from outside the process! So, what does an EM process sleeping in its event loop look like? Let's answer this two ways:

First, let's go to the source. We trace the Run() method in em.c, and (knowing that we happen to be using the epoll backend) find a _RunEpollOnce method, the meat of which is simple:

FD_ZERO(&fdreads); FD_SET(epfd, &fdreads); if ((ret = rb_thread_select(epfd + 1, &fdreads, NULL, NULL, &tv)) < 1) { if (ret == -1) { assert(errno != EINVAL); assert(errno != EBADF); } return true; }

So assuming that rb_thread_select is a simple wrapper around select, we expect to see a process in a

select(N+1, [N], NULL, NULL, &timeout)

call, where N refers to an epoll fd.

We can verify this via strace, of course. We run

[nelhage@anarchique:~]$ ruby -reventmachine -e 'EM.epoll; EM.run {}'

and then:

[nelhage@anarchique]$ strace -p 30413 Process 30413 attached select(8, [7], NULL, NULL, {0, 83342}) = 0 (Timeout) select(8, [7], NULL, NULL, {0, 90000}) = 0 (Timeout) select(8, [7], NULL, NULL, {0, 90000}) = 0 (Timeout) select(8, [7], NULL, NULL, {0, 90000}) = 0 (Timeout) select(8, [7], NULL, NULL, {0, 90000}) = 0 (Timeout) select(8, [7], NULL, NULL, {0, 90000}^C Process 30413 detached <detached ...> [nelhage@anarchique]$ ls -l /proc/30413/fd/7 lrwx------ 1 nelhage nelhage 64 Mar 27 23:06 /proc/30413/fd/7 -> anon_inode:[eventpoll]

Ok, great. But how do we do this programmatically? We could use strace, but that seems heavyweight and overkill. We could write something ourselves using ptrace (I've done it before!), but custom C code twiddling around my production services with arcane kernel interfaces is not my idea of fun. (Ok, well, it is kinda my idea of fun, but that doesn't make it a good plan...).

But we really don't need very much information ... just the current system call ... and it turns out that /proc has this for us!

[nelhage@anarchique]$ cat /proc/30413/syscall 23 0x8 0x19a0240 0x0 0x0 0x7fffac3e0910 0x3f 0x7fffac3e0880 0x7f938ef7ea53

So the first number is the syscall -- on amd64, 23 is select.

The next values are the arguments. 0x8 is the "max FD +1" number. Next are the (read,write,except) fd_sets. We can't look inside them trivially, but we can verify that only read is set, and that the timeout (0x7fffac3e0910) is non-NULL. Everything else is garbage, because select only has 5 arguments.

We know from this that the nfds argument is 8, which means that the epoll fd should be 7, and we can confirm that using /proc/$pid/fd as shown above. So putting it all together we get this ruby function. We can run that inside a monitor process that discovers all our workers (at Stripe, we do this via the einhorn command socket) and periodically polls each of them, sampling to determine if they're in the EM main loop. We dump this into graphite, and, sampled across time, it gives us pretty good visibility into when it's time to spin up more worker processes!

Note I've only verified this with EM around 1.0.0 on ruby 1.8 and 1.9, and it depends by construction on using epoll on Linux amd64. It's probably tweakable to other environments.