This minor stunt is nothing comparable to Danny but hey, its great fun! If you want to see the king of trial click here. If you have something special to you which makes fun or is difficult (etc) please comment or add a video reply.
Sadly Sony don’t want me to promote ‘Electric Feel’ from MGMT on youtube. In my opinion this is a fair usage (only a 42 second snippet) of the song in my video … Watch the video via blip tv.
In jetwick we only want to show relevant tweets for a search. No noise, no spam.
So first problem is solvable when we the user sorts by retweets, filters out by a specific criteria of his choice or when he refines its search: adds more specific terms.
But how can we get rid of spam at twitter? First, what is spam at twitter?
Several years ago Paul Graham gave a nice definition of email spam: ‘unsolicited and automated’. With this definition we can identify 4 situations for twitter spam:
With the following algorithm we can try to solve point 2 and 3.
I applied this algorithm on the data of jetwick and grabbed the twitter users with a lot of spammy tweets (the number in brackets) for the last week:
careerfan (968) -> bot teamnapalm (587) -> spam endy_pink (481) -> spam manypro (312) -> bot i_want_napalm (294) -> spam gutlazaro (216) -> spam (canceled from twitter) appstoreadam (210) -> bot+spam livralivro (207) -> bot sigaajesus (195) -> spammy thakiddunncase (195) -> ups, no spam lauberte_ (167) -> spam 2dvdlsnorjeuvou (158) -> spam josialemrossi (152) -> spam (canceled from twitter) malucomunic (139) -> spam (Was this account hacked!!?? Because no one of the followers is a spammer!)
The idea is simple, but the results looks promising. There could be a lot of use cases. E.g. twitter clients like hootsuite could add tweet quality to its available filters … the user specific klout score is not useful, because even less popular tweeters can create great tweets 🙂
Let me know what you think!
… Anhänger!
After reading about countdown traffic lights I created my music-player version of a traffic light:
Or should we interpret the yellow light differently?
In the previous blog post I tried to introduce node.js and its nice features. Today I will introduce my little search engine prototype called jsii (javascript inverted index).
jsii provides an in-memory inverted index within approx. 1000 lines of JavaScript. Some more lines are necessary to set up a server via node.js, so that the index is queryable via http and returns Solr compatible json or xml. The sources are available @github:
git clone git@github.com:karussell/jsii.git
Try it out here: http://pannous.info:8124/select?q=google e.g. filter queries works like id:xy or queries with sorting works like &sort=id asc. The paramters start and rows can be used for paging. For those who come too late e.g. my server crashed or sth. ;-), here is an image of the xml response:
Solr XML Response
The solr compatible xml response format makes it possible to use jsii from applications that are using SolrJ. For example I tried it for Jetwick and the basic search worked – just specify the xml reponse parser:
solrj.setParser(new XMLResponseParser());
The first thing I needed was a BitSet analogon in JavaScript to perform term look-ups fast and combine them via AND bit-operation. Therefor I took the classes and tests from a GWT patch and made them working for my jasmine specs.
While trying to understand the basics of a scoring function I stumbled over the lucene docs and this thread which mentions ‘Section 6 of a book‘ for a good reference on that subject.
My understanding of the basics is now the following:
With jsii you can grab docs from a solr index or feed it via the javascript api. jsii is very basic and simple, but it seems to work reasonable fast. I get fair response times of under 50ms with ~20k tweets although I didn’t invest time to improve performance. There are some bugs and yes, jsii is a memory hog, but besides this it is amazing what can be done with a ‘script’ language. BTW: at the moment jsii is a 100% real time search engine because it does not support transactions or warming up 😉
Hints
“Everybody (incl. me) is laughing at you JavaScript” – Me (before the year ~2003)
“It’s time to laugh back!” – JavaScript
Wouldn’t it be amazing to make all your programming tasks in one programming language? No need to switch? No need to relearn?
I dreamed that dream with Java. Sadly the Java-plugin is not the future (not user friendly, not search engine friendly, …) and ordinary web development can be done with pure Java solutions such as Vaadin/GWT or Wicket, but at some point you will need to know JavaScript.
With node.js – although this is not the first server side js solution – it is possible to do all your tasks in JavaScript! Plus a bit css and html knowledge, of course. But can we save files or querying databases with pure javascript? node.js has the goal to provide an implementation for such a server side API and designed its API to be non-blocking. Another interesting feature are web sockets: node.js makes it possible to directly communicate from server to client (and back) with pure JavaScript. So you can send js- or json-snippets back and forth. Check out one amazing example of this. Behind the scenes of node.js the V8 acts as virtual javascript machine and makes it all amazingly fast.
To better understand things I need to code them. Now, that I wanted to get a better understanding of an inverted index – I had to code it. But implementing it in Java would have been boring, because there already is the near-perfect Lucene. So I choose JavaScript: I wanted to get a better understand of this language and I wanted to try node.js.
In my upcoming blog post I will write about JSii – an inverted index implementation in javascript (apache 2 licensed). The index is not limited to node.js – you can use it in the browser. But more interestingly it can be queried via http and even SolrJ – in that case node.js hosts the inverted index.
But why is a search engine in JavaScript useful? First of all, it is a prototype and I learned a lot. Second, you can check it out and learn something more and think about other possibilites/ideas.
Third, I can imagine of the following scenario – reducing the need of “server-side” architectures. You can call it “ad-hoc peer to peer networks over HTTP”. Imagine a user which visits your website is willing to give you for some minutes or seconds a bit of its browser-RAM and CPU. Then you can push some minor parts of your data (in our case a search index) to the users’ browser and include it into your network. This architecture is extremly difficult to manage. You will have to avoid that users think your site is compromised by malware. But this network will work better/faster/more reliable/… the more users visits your site!
A similar concept is used in electric power transmission. Centralized power stations produces the predictable ‘source of energy’. In the future decentralized power stations such as wind turbines, solar plants, etc will pop up for hours or even minutes and contribute its energy to the world wide energy network.
We will see what the future will bring for the IT sector. But there is no doubt: the future of JavaScript is bright.
I always looked for a simple way to visualize our log data e.g. from solr. At that time I had in mind a combination of gnuplot and some shellscripts but this session from the lucene revolution changed my idea. (Look here for all videos from lucene revolution.)
I thought: “hey thats it! Just put the logs into solr!” So I coded something which simply reads the log files and named it Sogger. Without sharding, without message queues, … but it should work on real systems without any changes to your system (but probably to sogger).
I hope Sogger doesn’t suck, but it does not come with any warranty, so use it with care! And: It is only a proof of concept – nothing comparable to the guys from loggly.com
To get your logs sogged:
hg clone http://timefinder.hg.sourceforge.net/hgroot/timefinder/sogger sogger-code
svn co -r 1023329 https://svn.apache.org/repos/asf/lucene/dev/trunk solr-code
Sogger doesn’t necessarily need the trunk version but I didn’t tested it for others yet
cd sogger-code/ java -jar dist/Sogger.jar url=http://localhost:8983/solr logFile=data/solr.2010-10-25.log.gz
Now you should see something like this
Sogger has several advantages over simple “grep-ing” or scripting with your solr logs:
to do’s:
Es gibt Sprüche die finden Gegner und Befürworter toll:
In the next days I will release a minor changed UI of jetwick.com. It is nice that for this only css+html changes were necessary (I am using wicket btw). A nice consequence of this is that a lot “float: left;” and “width: xy;” are now needless.
This will be three the column style:
Before it was a mix with a lot white areas:
Let us start with the fun tweets. Ok, this week a lot Java bashing tweets, but I like them!
maven 3 is out. It now lets you download the internet even faster than before.
The world needs to stop hyping “html5” as though it’s markup alone that builds rich web apps. It makes JavaScript angry.
Wolfenstein 3D… in 1K of JavaScript: The JS1K conference wrapped up recently. One of the winners that jumped out a… http://bit.ly/bzSm0A
I laughed out loud at this “classice IE6 effect” ported to HTML5: http://mrdoob.com/lab/javascript/effects/ie6/ via http://waxy.org/links
“JavaScript is the only language that people feel they dont need to learn before they start using it.” – Crockford
Little known fact: JavaScript also has an isNaaN() function for when you aren’t sure if you’re working with Indian food
I have seen an app with SQL code in the *views*, looked like a java coder was given a php book and told to make a rails app.
Matz on #ruby speed: Build your website in Ruby until you have more traffic than Twitter, then use your riches to hire Java programmers.
tweeted before, but worth revisiting. java: where this seems normal. http://t.co/w8T0HBy
OH: “Java is just a DSL for turning XML into core dumps.”
got a good laugh out of buzz today… http://twitpic.com/2vs9ie #python #java
Two Java programmers walk into the CS lounge… http://bit.ly/dgYAVj
(PS: I don’t get it so please enlight me in the comments)
judging Clojure/Lisp by its parens is like judging Java by its classpath
And last but not least some intersting infos:
Tomorrow I’ll release 7 advisories that affect Java VM Applets
“JavaScript – It’s a Real Language!” by @clubajax #tech #javascript http://bit.ly/bzLMcD
New release! javaewah A compressed alternative to the Java BitSet class http://bit.ly/bhITOQ , github : http://bit.ly/dDne3T
ever want to preload all the java classes your application ultimately depends on at runtime? automatically? http://gist.github.com/612643
Of course this list isn’t complete! So, watch out for more fun and infos at twitter and contact me or comment if you want to add it here or for the next week.
Navigatone 
