Mar 2, 2023
The Standards Council of Canada has approved a standard which would eliminate new data silos and copies when organizations build new apps.
seen from United States
seen from United Kingdom
seen from Netherlands
seen from Netherlands
seen from China
seen from China
seen from Singapore
seen from United States
seen from Türkiye
seen from China
seen from Greece
seen from Russia
seen from Russia
seen from United States
seen from Japan
seen from Canada
seen from China
seen from Argentina
seen from Australia
seen from United Kingdom
The Standards Council of Canada has approved a standard which would eliminate new data silos and copies when organizations build new apps.
Splice is a Linux system call that can send data from one file descriptor (ie, a socket) to another one with zero-copy. This is of vital importance for some applications like proxies, or for some general cases like receiving data that is going to be immediately saved to disk. It can save many copies and many CPU cycles…
Why does top report that Cassandra is using a lot more memory than the Java heap max?
Cassandra uses mmap to do zero-copy reads. That is, we use the operating system's virtual memory system to map the sstable data files into the Cassandra process' address space. This will "use" virtual memory; i.e. address space, and will be reported by tools like top accordingly, but on 64 bit systems virtual address space is effectively unlimited so you should not worry about that.
What matters from the perspective of "memory use" in the sense as it is normally meant, is the amount of data allocated on brk() or mmap'd /dev/zero, which represent real memory used. The key issue is that for a mmap'd file, there is never a need to retain the data resident in physical memory. Thus, whatever you do keep resident in physical memory is essentially just there as a cache, in the same way as normal I/O will cause the kernel page cache to retain data that you read/write.
The difference between normal I/O and mmap() is that in the mmap() case the memory is actually mapped to the process, thus affecting the virtual size as reported by top. The main argument for using mmap() instead of standard I/O is the fact that reading entails just touching memory - in the case of the memory being resident, you just read it - you don't even take a page fault (so no overhead in entering the kernel and doing a semi-context switch). This is covered in more detail here.
https://wiki.apache.org/cassandra/FAQ#mmap