Tiny Bits of Code

This is a place to share code that I'm writing for various projects. It's mostly meant to keep me focused, but by putting it here it just might help someone out (maybe you?). Other things might slip in over time, we'll just see how it goes.

[tinybitsofcode]

Walkthrough: Installing Jena and Fuseki

PREREQUISITES

An apache based web server already pushing content to the web that you have administrative access to.

Simple background knowledge of Linux\Unix such as how to navigate around a directory tree.

Ruby installed on the web server.

SETUP

tar -zxvf apache-jena-2.11.2.tar.gz  tar -xvzf jena-fuseki-1.0.2-distribution.tar.gz

[Note that you may need to change the file names to account for version changes and the like.] [Note that Fuseki will require Ruby to be installed or you won’t be able to use the special scripts for managing data with it, like “s-put” for loading although we are not going to use it here.] You’re going to need to set some environment variables and modify the PATH to help things work so you might as well do it as soon as they are unpacked:

export JENA_HOME=/your_path/apache-jena-2.11.2 export JENA=/your_path/apache-jena-2.11.2/ export FUSEKI=/your_path/jena-fuseki-1.0.2/ export JENAROOT=/your_path/apache-jena-2.11.2 PATH=$JENAROOT/bin:$PATH PATH=$FUSEKI:$PATH

EDIT THE FILES DIRECTLY [Note that you will need to change the directory locations to represent the locations on your system.  Basically any time you see /your_path/ you are going to need to substitute the path to your home directory or whatever directory you are choosing to work from. You might not need all the exports.  The one’s here were cobbled together from different sources and it should not hurt you to have them all.] The export commands allow certain features of Jena to know where the program files are no matter where you run them from on the system. Updating the path will give you access to the scripts in the apache-jena-2.11.2/bin/ directory without having to either be in that directory or type the full path to it. If you’re going to be using a triplestore then chances are you have a lot of data to load.  Let’s assume that this is the case and use the OCLC 1 million most highly held books as an example. You can get this file at http://purl.oclc.org/dataset/WorldCat/datadumps/WorldCatMostHighlyHeld-2012-05-15.nt.gz .  When you have it put it on the same computer as Jena/Fuseki and in some directory that you can find/navigate to. You’ll need to make a directory for the triplestore to be held in and this can be done as follows (again, make changes to represent the file structure that you want).  This following set of commands will create the directory structure TStoreData/WorldCat/ off your home directory (If you’re not sure where that is then use the print working directory command “pwd” after you have entered “cd ~”).

cd ~mkdir TStoreDatamkdir TStoreData/WorldCat

Unfortunately there are some small problems that Jena will have with the WorldCat file.  In short, there are unicode escape sequences that get read as angle brackets and as far as Jena is concerned these angle brackets should not be there and so it throws an error.  I’m not sure why they are there so we’re just going to get rid of them at this point so that we can get some data loaded.  To do this we run the following two commands from the directory where you put the original download:

sed 's_\\\u003E__g' < WorldCatMostHighlyHeld-2012-05-15.nt > WorldCatMostHighlyHeld-2012-05-15_angleFix1.nt

sed 's_\\\u003C__g' < WorldCatMostHighlyHeld-2012-05-15_angleFix1.nt > WorldCatMostHighlyHeld-2012-05-15_angleFix2.nt

sed is the “stream editor for filtering and transforming text”.  It is going to run through the WorldCatMostHighlyHeld-2012-05-15.nt file looking to match and replace based on what is inside the single quotes and then

LOAD DATA

Now you’re ready to load some data.  There are multiple ways to do this but we’re going the use the fastest.  Execute the following command:

tdbloader2 --loc=/your_path/TStoreData/WorldCat/ /data2/home/jes6/CorporaAndData/WorldCat/WorldCatMostHighlyHeld-2012-05-15_angleFix2.nt

What you are doing here is telling the program tdbloader2 to go get the World Cat file we downloaded and then fixed, WorldCatMostHighlyHeld-2012-05-15_angleFix2.nt, from the directory it is in (on my machine it is /your_path/CorporaAndData/WorldCat/) and put the persistent triple store it will build in /your_path/TStoreData/WorldCat/.  This method will create a directory for the triplestore if it doesn’t already exist so we didn’t need to make the directory earlier but it’s good to know how.  It is also good to remember that if you enter the wrong directory then you’ll get data somewhere you didn’t expect. This method won’t load data into a directory that already has data in it so if you need to clean out a directory you could do something like:

rm -rf TStoreData/WorldCat/*

Executing this would delete all the contents of the TStoreData/WorldCat/ folder, including other folders. If you have a really big file it might take quite some time to load.  This will be a pain if you have to sever your connection or have intermittent wifi access.  To get around this issue the same tbdloader2 command but with “nohup” (short for “no hangups”) appended to the front (This is a great trick to use with any long process from the terminal).

nohup tdbloader2 --loc=/your_path/TStoreData/WorldCat/ /data2/home/jes6/CorporaAndData/WorldCat/WorldCatMostHighlyHeld-2012-05-15_angleFix2.nt

You might also find it useful to use the same terminal window for other things while the tbdloader2 process is working away.  The easiest way to do this is to append an ‘&’ at the end of the command.  So, our tdbloader2 command would become:

tdbloader2 --loc=/your_path/TStoreData/WorldCat/ /data2/home/jes6/CorporaAndData/WorldCat/WorldCatMostHighlyHeld-2012-05-15_angleFix2.nt &

Of course, we don’t always remember to background something up front so it is helpful to know how to do it after the fact too.  If we had run one of the tbdloader2 commands and forgotten to add the ‘&’ we could add it after the fact by pressing “ctrl" + “z” together to suspend the process in the current window and then entering the command bg to send the last suspended process (in this case the one we just suspended) into the background and continue it.  If you want to nohup and background then use this process, execute the nohup version, suspend it, then background it. We could then use the jobs command to check that it is running.  This command will also give us the job numbers allowing us to do things like:

bg %3 to send job 3 into the background

fg %4 to bring job 4 into the foreground

kill %7 to kill job 7

kill -9 %7 to kill job 7 after asking in to shutdown nicely with the previous kill command and it didn’t listen (or is taking too long) and now we really want it to stop no matter what.

OPEN AND RUN THE SERVER From the fuseki directory issue the following command to start the server:

./fuseki-server --loc=/your_path/TStoreData/WorldCat/ --port=3030 /WorldCat

--loc=FILE is the location of the triplestore to load.  In this case we are telling it to use the WorldCat data that we loaded above.  There are more ways to load data an you can see them by looking at http://jena.apache.org/documentation/serving_data/index.html#running-a-fuseki-server .  --port=NUM is the port that you’d like to provide access to the triplestore through.  The default port that fuseki will use is 3030 so if you are ok using that port then you can just leave this option out.  The last option in the line is extension to the base url that you would like it to use.  Here we’re using “/WorldCat” since it is relatively short and a good description of what will be there. Note that the server is very likely something that you will want to run in the background, just like tdbloader.  You can also issue the nohup command with it if you would like it to persist after you disconnect.  See above for descriptions and examples for both of these. (In the long term you'll want to set it up as a service and have it start automatically with the server, but that is another process). While the server may start if you have a firewall it might not be possible to access it if the firewall does not allow access on the port you are requesting access through.  We get through this by issuing this command:

sudo iptables --insert INPUT 8 -m state --state NEW -m tcp -p tcp --dport 3030 -j ACCEPT

and simply changing the port number as needed.  You will need administrator (superuser) access to do this so make sure you know the password and/or that your account has these privileges. With the server started and the port open you can check your data and your server by visiting http://cwrc-apps-03.srv.ualberta.ca:3030/ (substituting your base URL for cwrc-app-03.srv.ualberta.ca and the appropriate port number).  This will give you a simple dashboard that you can use to interact with fuseki and the data that it represents.  Choose the “Control Panel” link and then the WorldCat dataset from the drop down menu.  You will end up on a page that will allow you to enter SPARQL commands.  Let’s ask the World Cat dataset  to show us all the predicates that are used in it and to list each of these only once.  Type: SELECT DISTINCT ?y WHERE {?x ?y ?z} into the to SPARQL Query box and press “Get Results” (note that the capitalization does not matter, it is just a convention).  Depending on the speed of your server after a short while you should see the following as output, which is a list of all the predicates used in the WorldCat material that you loaded. ------------------------------------------------------------ | y                                                        | ============================================================ | <http://www.w3.org/1999/02/22-rdf-syntax-ns#type>        | | <http://purl.org/dc/terms/title>                         | | <http://purl.org/dc/terms/description>                   | | <http://purl.org/dc/terms/created>                       | | <http://rdfs.org/ns/void#dataDump>                       | | <http://purl.org/dc/terms/publisher>                     | | <http://vocab.org/waiver/terms/norms>                    | | <http://creativecommons.org/ns#morePermissions>          | | <http://creativecommons.org/ns#attributionName>          | | <http://purl.org/dc/terms/license>                       | | <http://creativecommons.org/ns#attributionURL>           | | <http://rdfs.org/ns/void#exampleResource>                | | <http://www.w3.org/2002/07/owl#sameAs>                   | | <http://rdfs.org/ns/void#uriSpace>                       | | <http://rdfs.org/ns/void#vocabulary>                     | | <http://rdfs.org/ns/void#subset>                         | | <http://rdfs.org/ns/void#openSearchDescription>          | | <http://creativecommons.org/ns#useGuidelines>            | | <http://www.w3.org/2000/01/rdf-schema#label>             | | <http://xmlns.com/foaf/0.1/homepage>                     | | <http://xmlns.com/foaf/0.1/page>                         | | <http://schema.org/name>                                 | | <http://www.loc.gov/mads/rdf/v1#isIdentifiedByAuthority> | | <http://schema.org/contributor>                          | | <http://schema.org/about>                                | | <http://schema.org/author>                               | | <http://purl.org/library/hasManifestation>               | | <http://schema.org/inLanguage>                           | | <http://purl.org/library/holdingsCount>                  | | <http://schema.org/copyrightYear>                        | | <http://purl.org/library/oclcnum>                        | | <http://purl.org/library/placeOfPublication>             | | <http://schema.org/description>                          | | <http://schema.org/publisher>                            | | <http://www.w3.org/1999/02/22-rdf-syntax-ns#value>       | | <http://www.w3.org/2004/02/skos/core#inScheme>           | | <http://schema.org/isbn>                                 | | <http://purl.org/library/hasCarrier>                     | | <http://schema.org/bookFormat>                           | | <http://schema.org/datePublished>                        | | <http://schema.org/url>                                  | | <http://schema.org/genre>                                | | <http://schema.org/numberOfPages>                        | | <http://schema.org/bookEdition>                          | | <http://schema.org/isFamilyFriendly>                     | | <http://schema.org/contentRating>                        | | <http://schema.org/illustrator>                          | | <http://schema.org/reviews>                              | | <http://schema.org/awards>                               | | <http://schema.org/itemReviewed>                         | | <http://schema.org/reviewBody>                           | | <http://schema.org/image>                                | | <http://purl.org/dc/terms/isPartOf>                      | ------------------------------------------------------------

Congratulations!  You have a working triplestore! Interested in accessing the triplestore through a program?  Here’s as example script showing a simple way to do it with Python.  Of course, you will need to change the endpointURL to point at the location of your store.

#Simple script to send a SPARQL query to the CWRC SPARQL endpoint on our Jena triplestore and output the result #Modified from the example on p. 209 of Learning SPARQL, O'Reilly, Bob DuCharme import urllib2 endpointURL = "http://cwrc-apps-03.srv.ualberta.ca:3030/WorldCat/" query = """ SELECT DISTINCT ?y WHERE {?x ?y ?z} """ escapedQuery = urllib2.quote(query) print escapedQuery requestURL = endpointURL + "query?query=" + escapedQuery print requestURL print urllib2.quote(endpointURL + "query?query=" + query) request = urllib2.Request(requestURL) result = urllib2.urlopen(request) print result.read()

[tinybitsofcode]

Getting GraphViz 2.34 to Work in Protégé 4.3 with OS X Mavericks

Protégé is a really powerful tool for building and editing ontologies for use on the semantic web.  Much of its usefulness is lost though when it is not possible to see graph representations of just what it is you've built and for months this is exactly where I've been stuck.  It was clear from poking around the web that I wasn't pointing OWLViz at the right directory even though I was able to find (what I thought was) the DOT file.

Solution?  Somewhere along the way to Protégé 4.3, OS X Mavericks, GraphViz 2.34, and pointing to /usr/local/bin/dot (found buried deep in the comments of the GraphViz download page) in the OWLViz panel within the preferences window and it all works beautifully.

Now to make some ontologies.

[tinybitsofcode]

4.9 Days to Load 8GB of OCLC Data into Sesame 2.7.7

Jeff Antoniuk and I have been working to get the triples for the 1 million most commonly held books in libraries as reported by OCLC in a Sesame (2.7.7) triple store. With 8GB of data this took longer than expected and required an intervention in the parameters of the store.

The problem was we kept receiving the following error when trying to load the data into a native store with all the vanilla options:

Exception in thread "main" java.lang.OutOfMemoryError: Java heap space at java.lang.AbstractStringBuilder.<init>

The trick was to modify console.sh to use:

JAVA_OPT=-Xmx12000m

Once this was done the entire process took 425486808 milliseconds (or 4.9 days) on a server with 12GB RAM, 4 cores, and a 512GB disk.

I'm worried now about how long it will take to load in all the RDF data from VIAF.

[tinybitsofcode]

Installing a Sesame Triple-Store and Loading in Some Data

Two things are assumed upfront:

You have a unix based system to install Sesame on and that you are generally familiar with navigating around that system.

You have a set of properly formatted RDF triples to load into the triple-store.

To begin, navigate to the directory where you would like to install Sesame. Get the Sesame code:

$ wget http://sourceforge.net/projects/sesame/files/Sesame%202/2.6.10/openrdf-sesame-2.6.10-sdk.tar.gz/download

Unpack the tarball:

$ tar -zxvf openrdf-sesame-2.6.10-sdk.tar.gz

Run the console:

$ cd openrdf-sesame-2.6.10/ $ bash bin/console.sh

Create a repository (The command below creates a "native" repository, which means that the triples are stored on disk.  This is important for persistence but can hurt performance.  With only 40k triples at this point performance is not going to be an issue for me.  The ID and Title can be pretty much anything you want but avoid spaces in the ID.  At this point we'll ignore indexing and just hit return when asked to contribute that.):

> create native. Please specify values for the following variables: Repository ID [native]: Orlando40k Repository title [Native store]: Repository for first 40k triples from Orlando Triple indexes [spoc,posc]: Repository created

Open Repository (use whatever you chose for ID when creating the repository):

> open Orlando40k.

Load Repository (file must be in the openrdf-sesame-2.6.10 directory since this is where we opened the console and all file input/output will be relative to this directory.  If you put the file somewhere else or opened the console from another directory then will need to modify the path to the file):

> load orlandoPoetessRDF1-Clean.rdf.

We'll get to playing with data in the triple-store later.  For now we'll just close the repository (This is important since not doing so can lead to problems in the future as various changes that you thought you made might not actually get committed):

> close.

And then exit the console:

> exit.

[tinybitsofcode]

A lovely little line from Donna Haraway’s “A Cyborg Manifesto”

#Lit

[tinybitsofcode]

Scraping RDF Namespaces

I'm involved with a project that needs to scrape the namespaces used for RDF on the web.  This will be done with Scrapy (http://scrapy.org/), a python-based web crawler. 

from scrapy.contrib.spiders import CrawlSpider, Rule from scrapy.selector import HtmlXPathSelector from scrapy.selector import XmlXPathSelector from scrapy.contrib.linkextractors.sgml import SgmlLinkExtractor from tutorial.items import DmozItem class DmozSpider(CrawlSpider):     name = "dmoz"     start_urls = [                   "http://datahub.io/dataset"                   ]     rules = [              Rule(SgmlLinkExtractor(allow=('.+',),deny=('(.+?\.zip)|(.+?\.gz)',)), callback='parse_item'),              Rule(SgmlLinkExtractor(allow=('http://datahub.io/dataset\?.+'),deny=('(.+?\.zip)|(.+?\.gz)',)), follow=True),              #Rule(SgmlLinkExtractor(deny=('(.+?\.zip)|(.+?\.gz)',)), follow=True),              ]          def parse_item(self, response):         xxs = XmlXPathSelector(response)         xxs.register_namespace("rdf", "http://www.w3.org/1999/02/22-rdf-syntax-ns#")         items = []         namespaces = iter(xxs.select('/rdf:RDF').re(r'xmlns:([\w]+?)="(.+?)"'))         for word in namespaces:             item = DmozItem()             item['short']=word             item['link']=next(namespaces,None)             item['url']=response.url             items.append(item)         #if stats.get_value('pages_crawled') > 500:         #    raise CloseSpider('crawl limit reached')         return items

items.py

from scrapy.item import Item, Field class DmozItem(Item):     short = Field()     link = Field()     url = Field()

settings.py

BOT_NAME = 'tutorial' SPIDER_MODULES = ['tutorial.spiders'] NEWSPIDER_MODULE = 'tutorial.spiders' CONCURRENT_REQUESTS = 100 #LOG_LEVEL = 'INFO' COOKIES_ENABLED = False RETRY_ENABLED = False #DOWNLOAD_TIMEOUT = 30 DEPTH_LIMIT = 10 ROBOTSTXT_OBEY = False

To run it from the command line: scrapy crawl dmoz -o items.csv -t csv

[tinybitsofcode]

Loading the Hume text into an NLTK corpus for analysis

This proved to be much more of a challenge than Natural Language Processing with Python implied that it would be.  The problem is that NLPP uses the import command for almost all of its examples and the PlaintextCorporaReader for the others.  This implies that they are the same thing (or at least it did to me): they are not. 

Treating them as if they are the same thing means that the categories() method is unavailable and thus makes trying to recreate the examples in chapter 6.1 with your own corpora a problem because doing so will throw this error: 'PlaintextCorpusReader' object has no attribute 'categories'.

To summarize, I think that I finally found a decent explanation worth sharing with the rest of the world at http://bit.ly/mxE9Cr .  It is apparently a chapter out of Python Text Processing with NLTK 2.0 Cookbook.  It might just be worth getting a hard copy of this.

So, I'll be starting with CategorizedPlaintextCorpusReader and the cookbook chapter when I go at this next.

[tinybitsofcode]

WHAT IS YOUR FAVORITE INANIMATE OBJECT?

String.

[tinybitsofcode]

Preparing to do some text analysis on Hume

The intent is to test out text classification from the Python NLTK on an interesting problem in Hume's Dialogues Concerning Natural Religion. Philo is a skeptic up until the last chapter (Part 12) where he relinquishes his position. There is debate in the community as to the source of this conversion and the thought is a little classification of speakers might provide another affordance on the issue. The plan is to take the dialogs up to the last chapter and use each speaker as training for the software. Following this the speakers in the last chapter will be presented and we'll see how the software categorizes them. In short, we are asking, "Is Philo still Philo?"

Original File From: http://digihum.mcgill.ca/~sgs/temp/hume-dialogues.html

Tag Conversion

Might be necessary to grab things based on XML tags rather than HTML span class tags. So, what was once a <span class="PHILO> tag should be converted to a <PHILO> tag or else it will get removed in HTML stripping. This was done with Text Wrangler's find and replace and some regex:

Search for: <span class="([A-Z]+)">([\s\S\n\r]+?)</span>

Replace with: <\1>(\2)</\1>

Dividing training material from hypothesis material

This was just done with simple cutting and pasting from one document into another and could be done with any word processing tool. These were saved as .txt files since the HTML will ultimately just be stripped away. This is also the time that all the header information was stripped.

Strip the html and leave the speaker tags

This is done on each file separately. It is done after breaking the files up since the HTML is helpful in breaking the files up, but not helpful for our question. This approach takes advantage of the assumption the the first character in plain HTML is always lower case while the first character in the speaker tags we are using is upper case (as is the whole name in this case). Again, a find and replace in Text Wrangler with a little regex makes easy work of this:

Search for: <[a-z].*?>|</[a-z].*?>

Replace with: nothing (just leave the field blank)

Extracting speakers as separate files

This was done with a little python script. It assumes that the content for each speaker is captured between matching tags while anything outside these tags is to be discarded. It creates a file for each speaker that it finds in the document and names than file speakerName.txt. This process turned up a miss tag (one instance of <Philo> that should have been <PHILO> and was creating a separate file. This was corrected and run again. The script was as follows with basic adjustments for the name of each file.

import codecs readFile="hume12.txt" fileDesignator="12" with codecs.open(readFile, encoding='utf-8', mode='r') as inputFile:     speaking=0     speakingCharacters=[]     spoken=""     readingName=0     nameCharacters=[]     name=""     names={}     for line in inputFile:         for character in line:             if character=="<":                 if speaking==1:                     speaking=0                     names[name].write(spoken.join(speakingCharacters))                     speakingCharacters=[]                     nameCharacters=[]                     name=""                 else:                     readingName=1             elif character==">":                 if readingName==1:                     readingName=0                     speaking=1                     name = name.join(nameCharacters)                     if name not in names:                         filename = name + fileDesignator + ".txt"                         names[name]=codecs.open(filename, encoding='utf-8', mode='w')             else:                 if readingName==1:                     nameCharacters.append(character)                 elif speaking==1:                     speakingCharacters.append(character)                          for item in names:         names[item].close()

This produced the following files

Next Steps: Training the Python NLTK with the training files to see how it classifies the testing files. This will draw extensively from the O'Reilly book Natural Language Processing with Python, Chapter 6.