When I first discovered Elasticsearch, in late 2010, I immediately considered it an elaborate hoax.
What else can you think when you discover a software project which comes sprinkled with so many buzzwords: a RESTful HTTP interface, working with schema-free documents, talking in JSON through a rich DSL, horizontally scalable and distributed by design, cloud-ready and cloud-friendly? And that software project is a search engine? A hoax, clearly.
But then I downloaded the source code, had a look a it… and it seemed quite legit. I became suspicious – it seemed like a little bit too much work for a prank. So I installed said project, and, to my astonishment, it did seem to work the way it was advertised on the tin.
At that time, I was working as a lead developer for a social media monitoring application, which was using CouchDB and CouchDB-Lucene as core technologies, so, as you can easily imagine, my neural paths had just exploded like fireworks. Working with schema-free JSON-based documents over HTTP and using full text queries and facets was something I was used to, only this time, it looked way, way better.
Being a Ruby shop, we naturally needed a Ruby library for talking to Elasticsearch, and not being satisfied with the offering, I set out to write a Ruby gem for Elasticsearch. In spite of working hard to make it usable outside of our pretty specific use case, I never imagined it will get nearly nine hundreds of GitHub watchers and be featured on Railscasts in the process. Or that I will get to solve more then three hundred issues and pull requests, curating other people’s code and making sure the library does not crumble under the weight of different styles, approaches and opinions.
Today marks a special day on this journey. As a member of the Elasticsearch.com team, I will be able to put more of my time and energy into Elasticsearch and the surrounding ecosystem — the Ruby library, provisioning and monitoring tools, research and documentation. My specific goal is to make it even more easy for Rubyists to work with full text search in their applications, to understand it better, and to make most of it.
After all, search is the primary means for keeping up with the vast amounts of data; every day, from Google to Mac OS X Spotlight, we rely on search to find information. In fact, the future of our civilization depends on how well we understand the data about ourselves, be it the global Twitter chatter, air pollution metrics, DNA sequences or evidence gathered by the Curiosity rover on Mars. I believe that full text search, and Elasticsearch in particular, will be an important part of this future.