Ione v1.2 was released a little over a month ago and I forgot to announce it here. It now supports making TLS connections and starting TLS servers.
There's also some additions to the futures API and the timer scheduler is no longer unusable.
TLS
Starting a TLS server or opening a TLS connection is now almost as easy as starting any server or opening any connection. The only thing you need to do extra is to create an OpenSSL::SSL::SSLContext:
ssl_context = OpenSSL::SSL::SSLContext.new ssl_context.verify_mode = OpenSSL::SSL::VERIFY_PEER ssl_context.cert_store = OpenSSL::X509::Store.new ssl_context.cert_store.set_default_paths f = reactor.connect('google.com', 443, ssl: ssl_context) f.value # => an open TLS connection
Futures
Futures got some optimizations and new features. You can now compose operations with #then, which works like a mix between #map and #flat_map. It can be very convenient in situations where you may want to perform an additional asynchronous operation depending on the result of the previous operation. Before you had to use #flat_map and return a pre-resolved future in the case where you did not.
For example, maybe you're loading something from a database, and when it's not what you wanted, for example an empty result, you load something else, as a fallback. It might look like this:
f = load_the_thing ff = f.flat_map do |result| if result.empty? load_fallback_thing else Future.resolved(result) end end
You need to use #flat_map because otherwise there's no way to chain another asynchronous call – but it also means you must return a future even when you don't want to chain. With #then you're not required to:
f = load_the_thing ff = f.then do |result| if result.empty? load_fallback_thing else result end end
The futures API also got two other additions: Future.traverse and Future.reduce. The former is a combination of Enumerable#map and Future.all, and encapsulates a pattern I use all of the time: mapping an array of queries or IDs to an array of futures that loads something from a databased for each query or ID, and then using Future.all to create a single future of all of the result – Future.traverse does that elegantly in one go:
f = Future.traverse(ids) do |id| load_thing(id) end f.value # => [thing, thing, thing]
Finally, Future.reduce takes a list of futures and as their values become available it reduces them down to a single value, like Enumerable#reduce.
Timers
In previous versions of Ione the timer scheduler was extemely naïve. It was basically just an array of timers, in the order that they were scheduled. For each tick the reactor would scan through the array to find any expired timers and trigger their callbacks. It would also not remove the timers, just mark them as triggered, so the next tick would have to scan through the same array again. When a new timer was scheduled the array was compacted and all triggered timers were removed.
There were two reasons for this implementation: the first was that I wanted to avoid locking in the reactor thread, which was possible using this metod, and the second was that I just didn't consider timers a very important feature and didn't have time to make something better.
Not surprisingly timers were basically useless, and when people started using timeouts in cql-rb and the DataStax driver it got ugly. I had not planned to do anything more than TLS in v1.2, but when I realized how bad the situation was I read up on priority queues, implemented a heap data structure and replaced the scheduler. This delayed v1.2, but it was really worth it.
After one and a half years of development I'm handing over the responsibility of developing and maintaining the Ruby driver for Cassandra to Datastax.
They've worked tirelessly for the last few months to create a new Ruby driver to match the API of the Java, Python and other official drivers, using cql-rb as a foundation.
The new driver has a few new cool features that I never had the time to add to cql-rb, including configurable strategies for which node to send a request to, for request retries and for reconnections.
Besides a blog post announcing the release of the first beta, there's lots of new documentation of the new API and the new features.
cql-rb v2.0 has finally been released. This is the first release to support all the new features in Cassandra 2.0 like result paging, mixed batches, SASL authentication.
Read the full release notes on v2.0
The next step is to add LZ4 compression support – it's already completed, but the lz4-ruby gem didn't release a compatible version in time to include it in v2.0, and there will be support for connection strategies, routing strategies, and hopefully support for encrypted connections, as well as support for the new features that will be introduced in Cassandra 2.1.
This release is forwards compatible with all versions of Cassandra that support v2 of the CQL binary protocol, which includes Cassandra 2.1 and most likely all releases in the coming year or two.
It's been way too long in the making, but v2.0 of cql-rb is very nearly here. I released the first release candidate yesterday, and the final version should be released this week.
The reason it has taken so long is twofold: first I wasn't sure enough that I'd found all the bugs, but didn't have the time to take it through the process I usually use before releasing a version, and after I had come that far I wanted to be sure about the public API.
What I do to satisfy myself that a new version is completely stable and reasonably bug free is a blog post I've been meaning to write for a while, but suffice to say that I don't take it lightly. v2.0 is a big release, there's a ton of new code and feature support, and lots of things have had to changed to accommodate supporting two protocol versions at the same time. The code that does the initial connection and peer discovery was also heavily refactored, it's by far the most complex part of the driver, but now it's at least possible to understand it again. Being confident that you've weeded out most of the bugs after such a big change requires more than just some integration tests.
Since bumping the version to a new major means the possibility of making backwards incompatible changes I also wanted to make sure I took full advantage of that and removed things I didn't want to support, dropped deprecated features, and changed things I wasn't too happy about. In the end I decided on keeping the API backwards compatible with v1.2, and just remove the deprecated features from the documentation. Some pieces that may have accidentally been in the public API documentation have also been marked as private.
Once v2.0 is released I will start working on new features, and I should probably get going on supporting Cassandra 2.1 features like user defined types.
Ione can now be used to create network servers, in addition to clients. The v1.1 release is the big first step towards making Ione the general purpose reactive toolkit for reactive programming in Ruby that I want it to be.
There is a simple message of the day example server that shows the basics of starting a server, but I've also been working on a higher level abstraction for writing RPC clients and servers – still a work in progress, but check out the example to see how it works.
cql-rb v2.0.0.pre1 has just been released, it's the first version with an external dependency.
Before I tell you more I want to give you some background about why cql-rb has until now been completely standalone: When I started cql-rb I was hoping to use some existing gems for things like futures, IO, byte buffers, UUIDs, etc., but I didn't find any that fit.
For example, there are many UUID gems for Ruby, but all that I looked at had issues mostly around performance and most of them were overengineered for something so simple. I've been asked a few times why I didn't just fix the problems of one of the existing gems and made the world a little better, but considering that the Uuid and TimeUuid classes in cql-rb, including the TimeUuid generator is only around 150 lines of code, it feels like it would be much more trouble than it would be worth depending on something else.
In contrast, the case with byte buffers seems it seems like everyone is happily using strings and arrays for buffers. With a bit of help from a friend I build a pretty performant byte buffer implementation that is orders of magnitude faster than appending and slicing from a string. Using strings and arrays as buffers and queues in Ruby is not a good idea, but something that most people don't need to worry about, which means that those that do don't know that they should worry about it.
When it comes to concurrency and IO I had an idea that I might try to use Celluliod and Celluloid::IO as a core and build the driver around that. In the end I decided against it, mostly because it would be such a heavy dependency and I wouldn't use much of it. In hindsight I'm very happy about this decision. I've played around a bit with Celluloid since then and either I don't understand how to test it properly, or the performance is just not very good. From what I've understood it uses fibers in its message dispatch, and that could explain things.
One of the benchmarks that ship with Celluloid shows it making 8K async message sends per second, within the same process. Compare that to a simple client/server setup that uses cql-rb's IO reactor and does over 50K async RPC calls between two processes, over TCP. That includes encoding frames and everything. 1
This brings us to cql-rb's new little sister, Ione. I have used cql-rb to build network clients for other things and it feels a bit strange depending on a Cassandra driver when the application doesn't talk to Cassandra. From the start I've tried to keep the parts of cql-rb that aren't strictly related to the CQL protocol as generic as possible. The UUID implementation is usable by itself, futures and byte buffers are completely generic, and the IO reactor doesn't assume anything about who it's talking to. As proof of that, and to keep myself straight, the integration tests have included a full-featured Redis client.
With v2.0 of cql-rb around the corner it felt like the right moment to extract the reactive core, including the futures implementation, the IO reactor and the byte buffer. These are the building blocks of network clients and reactive applications, and I've already got code that makes it possible to run servers too (a feature I had built for an application which didn't fit at all to merge back into cql-rb).
The UUID implementation remains in cql-rb, it doesn't fit in Ione, and I don't think the Ruby world need another UUID library, whatever I think about the existing ones.
So, does the world really need another IO reactor and futures implementation? I don't know. I felt like I needed one, and I didn't find the right one for cql-rb. Now what I build to solve that problem is available as a standalone project, and hopefully it will be useful to someone else but me.
About Celluloid, don't read too much into what I'm saying here, I think it's a very interesting project with many successful users. All I'm saying is that it looks to me like it wouldn't have worked for this use case, and not with the kind of performance I need from a Cassandra driver. Making something tailor made for a purpose will always make it better for that purpose than a general tool. ↩︎
It's been long in the making, but now you can get full Cassandra 2.0 support with cql-rb. Including batches, paging, SASL authentication and bound variables in non-prepared statements.
cql-rb, the Cassandra driver for Ruby, just reached v1.2.0 and is now feature complete with respect to the v1 of the CQL binary protocol.
It's been about a year since I set out to build a new, high performance, Ruby driver for Cassandra. v1.0 was released eight months ago and was about delivering the bare necessities with great performance. v1.1 followed a few months later and had everything you could expect from a driver for a distributed database. v1.2 is a tiny release in comparison to those two, but wraps up the feature set with support for request tracing and compression.
The the goal posts always move, as it were. Cassandra 2.0 came out in September with lots of new features. The next release of cql-rb, besides bug fixes and performance improvements, will also be 2.0. We'll have to see if Cassandra 2.1 comes out before, but the idea is not to have synchronized version numbers, I just think that it will be a significant number of changes and new features and a major release seems appropriate.
The plan is to do a release with support for all of the features of v2 of the CQL binary protocol, including batching, bound variables for non-prepared statements, result set paging, and support for SASL authentication. The new version also opens up some great new optimization possibilities. Keep a look out for the prereleases, but if you can't wait you can peek at the protocol_v2 branch, which already has some of the features mentioned.
Video of my talk on building a CQL driver from Cassandra Summit Europe 2013. You can find the slides here, and the videos for the other talks from the conference can be found on YouTube.
Sorry about the lack of light, I turned down the lighting in the room to make sure my slides would be readable -- it looks like it was pitch dark, but it's mostly the camera focusing on the brightest source of light in the room.
Last night at the Cassandra Summit Europe I was named a Cassandra MVP Rebel Elite along with a few other people who have "gone above and beyond to share their knowledge and expertise to help the Cassandra community". I'm really happy to have received this recognition from the community. Thanks everyone!
I was at Cassandra Summit Europe and talked about my experience building a CQL driver. Here are my slides:
I also took the opportunity to release the first release candidate of cql-rb v1.1, which has automatic reconnection, peer discovery, better performance, especially for prepared statements, and many more improvements. It's been a long time coming, but it's among the hardest code I've ever had to write.
For the last months I've been working on a driver for Cassandra's new native protocol, and today I released version 1.0.
Before 1.2 all Cassandra drivers used Thrift, but in 1.2 a new binary protocol was introduced. The Ruby Thrift libraries have never worked well, especially not in JRuby, and because of that using Cassandra with your Ruby application has never been a great experience. Hopefully that will change now.
cql-rb is pure Ruby, works with both MRI and JRuby and has no other dependencies. It has support for most of the features of CQL3 and Cassandra 1.2, including prepared statements and authentication. It's built around a non-blocking IO reactor to make full use of the binary protocol's support for request pipelining, and maximize performance.
For version 1.0 the focus has been to support the features needed for most applications, to be stable enough for production and to provide good performance. Now when 1.0 is out I will try to implement some of the more advanced features that exist in Datastax Java driver, for example automatic peer discovery, reconnection on node failures, support for tracing requests and compression.
Finally, here's a snipplet to demonstrate basic usage, there are many more examples in the README, and in the API docs.
require 'cql' client = Cql::Client.connect client.use('system') result = client.execute('SELECT * FROM peers') result.each do |row| p row end
germansingergirl
pixelskylines
origamiaround
gummygunk
dededos
izzy806