Introduction To Elastic Search, Kibana and X-Pack

6 min readDec 11, 2017

- Elasticsearch is a search engine. It is powered by Lucene, an open-source full-text search library.
It provides a distributed full-text search engine with an HTTP web interface and schema-free JSON documents.

- Kibana is the data visualization plugin for elasticsearch.

- X-Pack is an Elastic Stack extension that bundles security, alerting, monitoring, reporting, and graph capabilities into one easy-to-install package.

Installation :

Required:

Elasticsearch requires at least Java 8

$ java -version
java version “1.8.0_92”
Java(TM) SE Runtime Environment (build 1.8.0_92-b14)
Java HotSpot(TM) 64-Bit Server VM (build 25.92-b14, mixed mode)

Steps:

1) Download elastic search and kibana from https://www.elastic.co/start

* Following example is for Mac

$ ls
elasticsearch-5.5.0.tar.gz kibana-5.5.0-darwin-x86_64.tar.gz

2) Extract ElasticSearch and verify files and folders

$ tar zxf elasticsearch-5.5.0.tar.gz
$ ls
elasticsearch-5.5.0 elasticsearch-5.5.0.tar.gz kibana-5.5.0-darwin-x86_64.tar.gz
$ cd elasticsearch-5.5.0
$ ls
LICENSE.txt NOTICE.txt README.textile bin config lib modules plugins

3) Run elasticsearch by typing command bin/elasticsearch from the parent folder.

$ bin/elasticsearch
[2017–08–19T02:41:02,805][INFO ][o.e.n.Node ] [] initializing …
….
[2017–08–19T02:41:08,001][INFO ][o.e.n.Node ] [CkWGb2R] starting …
[2017–08–19T02:41:08,233][INFO ][o.e.t.TransportService ] [CkWGb2R] publish_address {127.0.0.1:9300}, bound_addresses {[fe80::1]:9300}, {[::1]:9300}, {127.0.0.1:9300}
[2017–08–19T02:41:11,509][INFO ][o.e.n.Node ] [CkWGb2R] started

* Elastic is running on the node CkWGb2R which is created by default. Node name can be changed in config/elasticsearch.yml (node.name) before staring the elasticsearch.

* Its running on 127.0.0.1(localhost) port 9200

4) Open browser and load http://localhost:9200. You can see the following information.

{
“name” : “CkWGb2R”,
“cluster_name” : “elasticsearch”,
“cluster_uuid” : “OxfxTDEJSju3VUlX8GFpqw”,
“version” : {
“number” : “5.5.0”,
“build_hash” : “260387d”,
“build_date” : “2017–06–30T23:16:05.735Z”,
“build_snapshot” : false,
“lucene_version” : “6.6.0”
},
“tagline” : “You Know, for Search”
}

Cluster :

- A cluster is a collection of one or more nodes (servers) that together holds entire data and provides federated indexing and search capabilities across all nodes.

- A cluster is identified by a unique name which by default is “elasticsearch”.

Node:

- A node is a single server that is part of the cluster, stores your data, and participates in the cluster’s indexing and search capabilities.

- A node is identified by a name which by default is a random UUID that is assigned to the node at startup. One can define any node name if do not want the default.

5) Extract Kibana

$ tar zxf kibana-5.5.0-darwin-x86_64.tar.gz
$ ls
elasticsearch-5.5.0 elasticsearch-5.5.0.tar.gz kibana-5.5.0-darwin-x86_64 kibana-5.5.0-darwin-x86_64.tar.gz

6) Run Kibana in another terminal by typing command bin/kibana from the parent folder.

$ bin/kibana
log [21:21:53.232] [info][status][plugin:kibana@5.5.0] Status changed from uninitialized to green — Ready
….
log [21:21:53.907] [info][listening] Server running at http://localhost:5601
log [21:21:53.909] [info][status][ui settings] Status changed from uninitialized to yellow — Elasticsearch plugin is yellow
….
log [21:21:59.907] [info][status][ui settings] Status changed from yellow to green — Ready

7) Access http://localhost:5601 in browser

Kibana Dev Tools

- Go to http://localhost:5601, kibana UI -> Dev Tools -> Console.

- Left panel to write query and execution.

- Right panel will show the result.

Type “GET /” and run.

This will fetch the current node and cluster information that we have already seen.

Terminology

Index:

- An index is a collection of documents having more or less similar characteristics. For ex, index for product data, index for customer data etc.

- And index is identified by a lowercase name which is used to refer to the index when performing indexing, search, update, and delete operations against the documents in it.

Type:

- A type is a logical category/partition of your index whose semantics is completely up to you.

- A type is defined for documents that have a set of common fields.

- For example, let’s assume you run a e-commerce platform and store all your data in a single index. In this index, you may define a type for user data, another type for product data, and yet another type for feedback data.

Document:

- A document is a basic unit of information that can be indexed.

- For example, you can have a document for a single customer, another document for a single product, and yet another for a single order.

- This document is expressed in JSON

Create Document

Kibana UI -> Dev Tools -> Console

Type the following and RUN.

POST /my-index/my-type/1
{
“body” : “test”
}

* This will create a document of id 1 of type ‘my-type’ under index ‘my-index’. In order to index a document, we must tell Elasticsearch which type in the index it should go to.

* Elasticsearch does not require you to explicitly create an index first before you can index documents into it. In the previous example, Elasticsearch automatically created the ‘my-index’ index

* Elasticsearch has fast search responses because, instead of searching the text directly, it searches an index like retrieving pages in a book related to a keyword by scanning the index at the back of a book.

Get Document

GET /my-index/my-type/1

Search (API)

GET my-index/_search
{
“query”: {
“match”: {
“body”: “test”
}
}
}

Output

{
“took”: 24,
“timed_out”: false,
“_shards”: {
“total”: 5,
“successful”: 5,
“failed”: 0
},
“hits”: {
“total”: 1,
“max_score”: 0.2876821,
“hits”: [
{
“_index”: “my-index”,
“_type”: “my-type”,
“_id”: “1”,
“_score”: 0.2876821,
“_source”: {
“body”: “test”
}
}
]
}
}

Delete Document

DELETE /my-index/my-type/1

Output:

{
“found”: true,
“_index”: “my-index”,
“_type”: “my-type”,
“_id”: “1”,
“_version”: 2,
“result”: “deleted”,
“_shards”: {
“total”: 2,
“successful”: 1,
“failed”: 0
}
}

GET /my-index/my-type/1

{
“_index”: “my-index”,
“_type”: “my-type”,
“_id”: “1”,
“found”: false
}

Install X-Pack

* Install at Elastic Side

- stop elastic search by Ctrl+C

- Install x-pack plugin

$ bin/elasticsearch-plugin install x-pack

-> Downloading x-pack from elastic

….

-> Installed x-pack

-Start elastic again

$ bin/elasticsearch
….
[2017–08–19T03:22:28,975][INFO ][o.e.p.PluginsService ] [CkWGb2R] loaded plugin [x-pack]
….
[2017–08–19T03:22:33,927][INFO ][o.e.n.Node ] [CkWGb2R] starting …
….
[2017–08–19T03:22:37,479][INFO ][o.e.n.Node ] [CkWGb2R] started
….

* Install x-pack UI at kibana side

- stop kibana by Ctrl+C

- Install x-pack plugin

$ bin/kibana-plugin install x-pack
Attempting to transfer from x-pack
Attempting to transfer from https://artifacts.elastic.co/downloads/kibana-plugins/x-pack/x-pack-5.5.0.zip
Transferring 119276235 bytes………………..
Transfer complete
Retrieving metadata from plugin archive
Extracting plugin archive
Extraction complete
Optimizing and caching browser bundles…
Plugin installation complete

- Restart kibana

$ bin/kibana
log [22:01:43.618] [info][listening] Server running at http://localhost:5601
log [22:01:43.620] [info][status][ui settings] Status changed from uninitialized to green — Ready

* Go to http://localhost:5601

You can find the login screen. This comes because of x-pack which gives role based permissions and authentication.