Elastic Stack in Data Engineering — Case Study

One of my clients is a company, that provides a platform for the content analysis. This platform is used by knowledge workers of the company to analyze social messages and to detect trends in data such as social sentiment.

Not only the title is important, but also the content itself. So we integrate Data Mining module of Elastic stack that allows extracting patterns from data. With this approach, users could customize analysis via creating filters and using it on the data.

The main idea of this article is to show how you can go from analysis specification to final result with Elastic stack.

Let’s start!

Photo by Myriam Jessier on Unsplash

Note: in this article, we use Excel files to capture data for the sake of simplicity. We will use a real data set in the future.

1. Analysis specification

Let’s start with analysis specification, where we map what we are looking for in the data.

The example file looks like the following:

As you can see, we have a set of information about our objectives, the audience, how often the content was published and links to the source page to allow interacting with the content.

Now we need to transform this file into Elasticsearch efficient data structure. So let’s create…