#4store

[semantic-web-links]

[apassant-blog]

A simple PHP library for 4Store

I was recently playing with 4store, the new RDF-store engine by Steve Harris / Garlik, after having used 3store for a few years in a previous project. As I don't want to use the HTTP server right now, but need to manage data input / query in PHP, I wrote a tiny lib that you can get here. It provides methods to import and delete graphs, as well as running SPARQL queries, and eventually outputs results with the requested content-type (XML, JSON or text). Then, adding graphs and querying data can be simply done as follows:

$s = new FStore('demo'); $s->import('http://rdfs.org/sioc/ns'); $s->query("select ?s where { <http://rdfs.org/sioc/ns#Item> ?s ?o }"); $s->delete('http://rdfs.org/sioc/ns');

[steveharris-blog]

Want to store hundreds of megatriples on lowend hardware?

The answer is, buy a SSD. Might seem crazy to spend that much on disks for a lowend machine, but economically, it makes sense. More details below.

This post specifically talks about 4store, but probably applies to other RDF stores too.

When I was designing the index used in the current version of 4store, I had in mind that solid state disks were probably coming along, so the IO optimisation is a bit different to how you would normally tune DB indexes, to take more advantage of the low seek and read times, but relatively expensive writes in flash media. However, I didn't predict the huge internal block sizes in flash storage, so until this weekend I wasn't sure how it would actually perform. This might mean that 4store takes more advantage from SSD hardware than other RDF stores, but I wouldn't want to predict that until someone compares them.

Well, at least on the SSD I tried, a Corsair Force F120, the performance is excellent. On a 3 year old, lowend desktop machine with 2x 2GHz cores, and 2GB of RAM I managed to import almost 280MT in under 4 hours. The machine wasn't idle during that time, it was doing it's normal job as a file server, but the SSD wasn't used for anything but 4store indexes.

I've not done any formal query performance testing, but anecdotally it seemed to be compeletly acceptable.

It's dangerous to extrapolate out from the behaviour of a low-end machine, but I'd expect a more sensibly specced server to be able to go to similar ratios of storage to main memory, hopefully making gigatriples in single nodes a pracical proposition.

The data I used was from the US data.gov datasets - it was just the first thing that came to hand, and I strongly believe in testing with real not, not synthetic data. I used the follwing files: data-1058.nt, data-10.nt, data-57.nt, data-793.nt, data-795.nt, data-805.nt.

The average performance over the whole import came to just over 20kT/s, and the speed over the last few million triples was about 17kT/s, but it wasn't decreasing very rapidly.

The workingset size for this data is about 30GB, and normally we'd recommend a server with at least 16GB of RAM to hold that amount of data, and probably 24GB would be safer. A Dell R210 with 4GB of RAM, and a 120GB SSD (you will need to add that yourself) will set you back about £700, whereas one with 16GB of RAM, and a second fast HD will set you back more like £1000, draw significantly more power, and probably have a lower usable capacity. So, economically SSDs look very sensible, on the face of it.

The unkowns at this point are:

Will the performance scale usefully to the class of servers you'd actually justify having in a 2011 datacentre.

What the lifespan of the SSDs will be - I'm using a consumer one, "enterprise" SSDs have a longer lifespan, but are substantialy more expensive. I'd expect a consumer SSD to last long enough to become obsolete with a realsonable daily write load (few tens of MT/day), based on what I've seen so far.

I've included an abbreviated log of the import below, and the full log is available elsewhere. The figures in []s are the size of the import, and the average import speed for the last 5 million triples of that import:

$ 4s-import flashtest -v data-* Reading <file:///storage/rdf/data-1058.nt> [ 20449300 triples, 87242 triples/s ] Reading <file:///storage/rdf/data-10.nt> [ 4719314 triples, 94577 triples/s ] Reading <file:///storage/rdf/data-57.nt> [ 26946509 triples, 100992 triples/s ] Reading <file:///storage/rdf/data-793.nt> [ 96307678 triples, 28270 triples/s ] Reading <file:///storage/rdf/data-795.nt> [ 98193090 triples, 15586 triples/s ] Reading <file:///storage/rdf/data-805.nt> [ 32505405 triples, 16346 triples/s ] Imported 279121296 triples, average 20098 triples/s $ 4s-size flashtest seg quads (s) quads (sr) models resources 0 139533763 +0 7 24076141 1 139587533 +0 6 24077738 TOTAL 279121296 +0 7 48153879