#twxplorer

[dividebysomething]

Some tricks for handling non-ascii text when analyzing tweets

For Twxplorer, we needed to clean up "terms" in tweets so as to not clutter the counts with emoji and other noise.  

In the first implementation, we simply forced unicode strings to ascii using this python code:

s = s.encode('ascii', 'replace').replace('?', '').lower()

This broke down when we wanted to support non-English languages, which often use non-ascii characters such as ñ, ç and é. Here's a solution which seems to work, at least for the languages we support. I think it would break with CJKV languages.

import unicodedata s = filter(lambda x: unicodedata.category(x)[0] != 'C',s.lower())

[eisema]

via @tweetsmania

[buzzfork]

Northeastern Lab Unveils twXplorer Search Engine

First came Topsy. Now, Northeastern University's Knight Lab has released a new Twitter search engine, twXplorer, with the goal of bringing Google-like search quality to Twitter data.

twXplorer works by analyzing the results of a Twitter search and returns a list of tweets as well as patterns in those tweets: the terms, hashtags and links that show up most often in your search.  Even better, each of those terms, hashtags and links is hyperlinked, so you can click on it to filter just those tweets from your original search. These features provide helpful trend context on top of Twitter search, a feature that hasn't been available previously outside of advanced Topsy searches.

Although the user interface and feature-set is fairly minimal, there are already quite a few things to like about twXplorer, including:

Related hashtag and link ranking data from search results

Snapshot save feature

List rankings that provide contextual data to your existing Twitter lists

For researchers, data analysts, journalists and digital marketers active in the Twitter ecosystem twXplorer is well worth a look. Want to see twXplorer in action? Have a look for yourself.