Shortest Code for a Simple Calculator on Android

String RESULT;
String input = "(1+3)/4 * 2 - 7";
...
webSettings.setJavaScriptEnabled(true);
...
webView.addJavascriptInterface(new JavaScriptInterface() {
   public void returnResult(String o) {
       RESULT = o;
   }}, "JavaCallback"));
webView.loadUrl("javascript:window.JavaCallback"
   + ".returnResult("+input+")");
// now RESULT is -5

Is there a shorter one? BTW: this is only a sketch not sure if I miss a bracket somewhere …

jsii – full text search in 1K LOC of JavaScript!

In the previous blog post I tried to introduce node.js and its nice features. Today I will introduce my little search engine prototype called jsii (javascript inverted index).

jsii provides an in-memory inverted index within approx. 1000 lines of JavaScript. Some more lines are necessary to set up a server via node.js, so that the index is queryable via http and returns Solr compatible json or xml. The sources are available @github:

git clone git@github.com:karussell/jsii.git

Try it out here: http://pannous.info:8124/select?q=google e.g. filter queries works like id:xy or queries with sorting works like &sort=id asc. The paramters start and rows can be used for paging. For those who come too late e.g. my server crashed or sth. ;-), here is an image of the xml response:

jsii-response

Solr XML Response

The solr compatible xml response format makes it possible to use jsii from applications that are using SolrJ. For example I tried it for Jetwick and the basic search worked – just specify the xml reponse parser:

solrj.setParser(new XMLResponseParser());

His-story

The first thing I needed was a BitSet analogon in JavaScript to perform term look-ups fast and combine them via AND bit-operation. Therefor I took the classes and tests from a GWT patch and made them working for my jasmine specs.

While trying to understand the basics of a scoring function I stumbled over the lucene docs and this thread which mentions ‘Section 6 of a book‘ for a good reference on that subject.

My understanding of the basics is now the following:

  • The term frequency (tf) is to weight documents differently. E.g. document1 contains ‘java’ 10 times but doc2 has it 20 times. So doc2 is more important for a query ‘java’. If you index tweets you should do tf = min(tf, 3). Otherwise you will often get tweets ala ‘java java java java java java…’ instead of important once. So for tweets a higher entropy is also relevant
  • The inverted document frequency (idf) gives certain terms a higher (or lower) weight. So, if a term occurs in all documents the term frequency should be low to make that term of a query not so important compared to other terms where less documents were found

With jsii you can grab docs from a solr index or feed it via the javascript api. jsii is very basic and simple, but it seems to work reasonable fast. I get fair response times of under 50ms with ~20k tweets although I didn’t invest time to improve performance. There are some bugs and yes, jsii is a memory hog, but besides this it is amazing what can be done with a ‘script’ language. BTW: at the moment jsii is a 100% real time search engine because it does not support transactions or warming up 😉

Hints

Full text search in 100% JavaScript – The future of JavaScript is bright.

Everybody (incl. me) is laughing at you JavaScript” – Me (before the year ~2003)

It’s time to laugh back!” – JavaScript

See the next post about the implementation – jsii

Wouldn’t it be amazing to make all your programming tasks in one programming language? No need to switch? No need to relearn?

I dreamed that dream with Java. Sadly the Java-plugin is not the future (not user friendly, not search engine friendly, …) and ordinary web development can be done with pure Java solutions such as Vaadin/GWT or Wicket, but at some point you will need to know JavaScript.

With node.js – although this is not the first server side js solution – it is possible to do all your tasks in JavaScript! Plus a bit css and html knowledge, of course. But can we save files or querying databases with pure javascript? node.js has the goal to provide an implementation for such a server side API and designed its API to be non-blocking. Another interesting feature are web sockets: node.js makes it possible to directly communicate from server to client (and back) with pure JavaScript. So you can send js- or json-snippets back and forth. Check out one amazing example of this. Behind the scenes of node.js the V8 acts as virtual javascript machine and makes it all amazingly fast.

What has this all to do with full text search?

To better understand things I need to code them. Now, that I wanted to get a better understanding of an inverted index – I had to code it. But implementing it in Java would have been boring, because there already is the near-perfect Lucene. So I choose JavaScript: I wanted to get a better understand of this language and I wanted to try node.js.

In my upcoming blog post I will write about JSii – an inverted index implementation in javascript (apache 2 licensed). The index is not limited to node.js – you can use it in the browser. But more interestingly it can be queried via http and even SolrJ – in that case node.js hosts the inverted index.

But why is a search engine in JavaScript useful? First of all, it is a prototype and I learned a lot. Second, you can check it out and learn something more and think about other possibilites/ideas.

Third, I can imagine of the following scenario – reducing the need of “server-side” architectures. You can call it “ad-hoc peer to peer networks over HTTP”. Imagine a user which visits your website is willing to give you for some minutes or seconds a bit of its browser-RAM and CPU. Then you can push some minor parts of your data (in our case a search index) to the users’ browser and include it into your network. This architecture is extremly difficult to manage. You will have to avoid that users think your site is compromised by malware. But this network will work better/faster/more reliable/… the more users visits your site!

A similar concept is used in electric power transmission. Centralized power stations produces the predictable ‘source of energy’. In the future decentralized power stations such as wind turbines, solar plants, etc will pop up for hours or even minutes and contribute its energy to the world wide energy network.

We will see what the future will bring for the IT sector. But there is no doubt: the future of JavaScript is bright.