I presented a webinar with 451 Research, where I talked about DBaaS and the disruption that’s occurring with Elasticsearch. As we get further into 2018, I think the content stands as a pretty good review of how far Elasticsearch has come, and can serve as a basis for where I think the technology is going. There’s no way I could start this blog post without first offering major kudos to both the Elasticsearch community and Elastic for building an amazing ecosystem of open source software that solves real problems.
A New Database
There is no shortage of stories or analyses about the NoSQL movement and how it’s driving a lot of expansion and change in the database space. A trend that I find even more interesting is the increase in both database technologies and database types over the past couple of decades. I used DB-Engines (because it’s awesome) and a few other online resources to map out a timeline of database technologies and what has been introduced each decade, which you can see below.
There are two major trends :
- The number of databases introduced every year has soared in the last two decades
- The number of different categories or types of databases has also escalated.
The first of these two trends is not terribly surprising. There’s been many evolutionary changes in computing with the internet and open source software over the past 20 or so years, so you would expect a similar growth in databases. Sure enough, the total number of databases (and open source databases) has continued to grow as people develop new technologies to solve a completely new set of challenges.
This brings me to the second trend. As new challenges have risen with data, smart technologies have been created to fulfill that need. Internet services and big data have driven growth in technologies like Hadoop and NoSQL. Now we’re starting to see technologies like IoT driving growth in streaming and time-series databases.
Out of this massively growing mess of databases a few have outperformed and become truly industry-changing. One of those is Elasticsearch. The increase in popularity of Elasticsearch over the past few years is staggering and can best be summed up by another few charts from DB-Engines:
In the first image, you can see that Elasticsearch has risen from relative obscurity in 2013 to surpassing it’s closest rival. Now you can see that it is jumping into the top 3 or 4 non-relational data stores. If you compare that against other technologies that had roughly the same popularity in 2013 (the second image), you will notice how much Elasticsearch has surpassed them.
How did Elasticsearch outperform competitors and other rising data stores in this time period?
Elasticsearch Strikes Back
I attribute this rapid adoption to a few main areas:
- Ease of Use
- The Elasticsearch Community
- Compatibility
- Broad Use Cases
These areas all build on each other. The great focus that the product and its creators have placed on making it easy to bootstrap and use has led to a quickly growing community of people and products. This active community has created a massive number of clients, plugins, and additional technologies to ensure that Elasticsearch is compatible with everything. The fact that Elasticsearch is compatible with so many technologies has allowed the use cases for Elasticsearch to rapidly expand.
If you look at the timeline of the Elasticsearch community’s launches and acquisitions over the past few years, you see a core focus on search, then expansion into logging with Logstash, then ad=ded visualization with Elasticsearch Kibana, expansion into analytics with aggregations, then into monitoring with Beats. That is a huge amount of use case expansion in a small period of time!
All of these features have been getting more established and more robust over time. Meanwhile Elastic.co, as a company, has really focused on beefing up a lot of these capabilities with their acquisitions of Found, Packetbeat, Prelert, Opbeat, and most recently Swifttype. All of these are doubling down and moving up the stack in the key use case areas mentioned above.
Return of the Elastic Stack
All of this expansion has created a couple of interesting adoption models for Elasticsearch:
Main datastore
“I want to build an app/feature, so I choose Elasticsearch as the datastore.”
This first model is pretty tried and true, and reflects how database adoption usually goes.
Complementary technology
“I want to add visualization to SQL/MongoDB/etc, add indexing to Hadoop, add processing/storage to Kafka, etc.”
This model covers a fairly untapped market. I am sure there have been many companies selling enhancements to databases, but none come to mind that can handle so many disparate uses with the same technology.
Additive technology
“I’m already shoving my logs into Elasticsearch, why not add metrics, monitoring, etc.”
This final, additive technology path, is in my opinion the most disruptive thing Elasticsearch does. Elasticsearch can get in the door as either an add-on, or small part of the app. From there, it grows exponentially as people get comfortable with it and see how flexible it is.
This is where Elasticsearch challenges many other players in the market. I use the term disruptive specifically because I think the technology very closely matches the disruptive technology model from “The Innovator’s Dilemma”. We see Elasticsearch meeting the needs of many basic uses and quickly working up the stack to add capabilities that nip at the heels of bigger players in the monitoring, log analysis, analytics, and enterprise search markets. Not only is Elasticsearch doing this in one use case, it’s doing it in multiple areas.
As you can tell, I’m very excited by what I see coming out of Elastic and the Elasticsearch community. I think it is truly a technology that will continue to grow at an amazing rate and I would love to work with you to make that happen! Contact us and we can talk about it!
In the meantime, you can try ObjectRocket for Elasticsearch with Kibana free for 30 days.
