Elasticsearch built on Apache Lucene is a free and open-source search and analytics engine for all types of data which includes structured and unstructured data. It is known for its performance, speed, scalability, and its distributed nature. Elasticsearch can also be used as a database, but like every other database, Elasticsearch has its strengths which include retrieving large datasets near real-time without affecting performance amongst others.
Advantages of Elasticsearch
- It supports REST-API: Unlike many other databases which require a database client to perform operations, Elasticsearch provides the ability to make a request via HTTP verbs. This provides the flexibility to quickly view and perform operations via HTTP clients (such as your internet browser, terminal, etc ) which has become available and widely spread.
- It supports sorting, filtering, and pagination out of the box.
- Elasticsearch provides a lot of search features which includes
- Fuzzy search – Fuzzy search allows you to identify or match items that are related to the search item. For example: if the Elasticsearch database contains a list of all animals which includes a cat, dog, antelope, rabbit, etc. Using the search item “at” will return the following results cat, antelope, rabbit. The example used is an oversimplified example of its powers
- Autocompletion
- Full-text search – Performs query operation against the Elasticsearch database searching for text fields that contain the searched word
- Elasticsearch is fast – This is because of its inverted index and distributed architecture, which makes it suitable to retrieve data from millions of documents in seconds
- DSL (Domain Specific Language) Queries. The DSL feature in Elasticsearch provides the power to perform complex data retrieval from the database
- Elasticsearch supports many languages out of the box such as Javascript, Php, Java, Python, etc
- Elasticsearch is scalable
Installing Elasticsearch
Now that we have understood Elasticsearch and its advantages let delve into installing and setting up a mini-project to see how to use Elasticsearch. You can install the Elasticsearch engine using docker (the fastest) or manually into the host machine. In this tutorial I will be installing Elasticsearch via docker; before we start the installation via docker, if you want to use other installation approaches, you can follow the installation links below for the different operating system
Installation on Docker
- Ensure docker is already installed on the host machine. Click the link to install the docker desktop (Windows or Mac) on your machine, which automatically installs required docker services on your machine
- Run the command docker pull docker.elastic.co/elasticsearch/elasticsearch:7.13.2 in your terminal. It fetches the docker image into the local docker repository in your host machine
- Then create a new Elasticsearch container by running the command docker run -p 9200:9200 -p 9300:9300 -d –name “elastic-search-7.13.2” -e “discovery.type=single-node” docker.elastic.co/elasticsearch/elasticsearch:7.13.2
- Side notes on the docker run command
- the -p flag instructs the docker to bind the container’s port (the second port) to the host port (the first port). This means that if you don’t want Elasticsearch to be accessible at port 9200 just change the first port to a desirable port as seen here – docker run -p 7777:9200 -p 9300:9300 -d –name “elastic-search-7.13.2” -e “discovery.type=single-node” docker.elastic.co/elasticsearch/elasticsearch:7.13.2. Ensure the port is not used by any other service
- The -d flag makes the container run in the background
- The –name flag allows you to give the container an easily identifiable name, without the name tag, docker generates a random name for the container.
- The -e flag is to set environment variables for your container
Hurray we now have the Elasticsearch engine up and running in our machine. Next we will be setting up our application in Nodejs
Setting up Elasticsearch in Nodejs
- Ensure Node is installed in your machine, NPM is automatically installed once Node has been installed
- Create a folder for this tutorial in any directory on your host machine
- Open a new terminal and move into the directory folder
- Run npm init -y this creates a new package.json file with default values
- Create a new file called index.js this will contain application entry point
- Open the tutorial folder in your favourite code editor
- There are two major ways to perform operations on your Elasticsearch database in Nodejs; It is either using an HTTP client such as the axios package or the dedicated Elasticsearch client for Nodejs. We will be using the axios package for this tutorial. Understanding how to use the REST api for Elasticsearch will provide you with more flexibility
- Run the command npm install axios express to install the required packages. The express package is to take advantage of its routing and server functionalities
Yipee, setup completed it is time to code
Coding time!!!
Express Boilerplate
Open the index.js file and add the code below
const express = require('express');
const app = express();
app.post('/create-index', async (req, res)=>{
res.json('Index created')
})
app.post('/data', async (req, res)=>{
res.json('Data inserted')
})
app.get('/data', async (req, res)=>{
res.json(`Searching for ${req.query.search}`)
})
app.delete('/data/:id', async (req, res)=>{
res.json(`Deleted data ${req.params.id}`)
})
app.listen(3000, () => {
console.log('App listening on port 3000!');
});The above code is just a boiler plate for our Crud operation, we also used the opportunity to setup a CRUD server. Now in your terminal run the command node index.js; A server is created and you will be able to access any of the endpoint.
Creating Index and Mappings
Next, we will be configuring our Elasticsearch database by creating an index and its mappings, index is similar to a table in a relational database and mapping to a relational database schema. In the SQL world before you can start using your database you will need to create tables and schemas, while Elasticsearch is flexible with schemas, that is, it is not important to create schemas(mappings) before you begin inserting data; it is advisable to create a mapping to improve Elasticsearch database performance
Update the index.js file to include the below code
const express = require("express");
const app = express();
const axios = require("axios");
const esUrl = "http://localhost:9200/";
app.use(express.urlencoded({ extended: true }));
app.use(express.json());
app.post("/create-index", async (req, res) => {
try {
const checkIndexExist = () =>
new Promise((resolve) => {
axios
.get(`${esUrl}${req.body.index}`)
.then((_) => {
resolve(true);
})
.catch(() => {
resolve(false);
});
});
const ifIndexExist = await checkIndexExist();
if (!ifIndexExist) {
const esResponse = await axios.put(`${esUrl}${req.body.index}`, {
mappings: {
properties: {
name: {
type: "text",
},
email: {
type: "text",
fields: {
raw: {
type: "keyword",
},
},
},
country: {
type: "text",
},
age: {
type: "integer",
},
company: {
type: "text",
},
jobRole: {
type: "text",
},
description: {
type: "text",
},
createdAt: {
type: "date",
fields: {
raw: {
type: "keyword",
},
},
},
},
},
});
res.json(esResponse.data);
} else {
res.json("Index exist already");
}
} catch (error) {
res.status(500).json(error);
}
});
In the updated code above, we created a new variable called esUrl to hold the Elasticsearch database Url, the axios package was also initialized to enable access to the database. The express middlewares app.use(express.urlencoded({ extended: true })), app.use(express.json()) was also included to enable us include post data in our post request.
Now restart your nodejs server (this is important so that the server can get the new changes made on the index.js file). Open your Postman app and make a post request to http://localhost:3000 with index included in data body as seen below
If you try to create an index that exists already, a message will be returned displaying a message that Index exist already as seen below
In the create-index route, before making a put request to create the index, we first checked that the index existed by making a GET request, if the index existed it will return a 200 but if the index does not exist the endpoint will return a 404 as seen here. It is important you check if an index existed before creating it, so you don’t encounter errors.
In the PUT request for creating index, the mapping objects contains the expected fields that will be inserted in Elasticsearch database, at the top level is properties which contains the actual schema for the index, for more information on the accepted keys in mapping see here. Next the properties object contains the fields and field types of the data we will be inserting into the database. The types indicated for each field matters to Elasticsearch, using the wrong type can affect the performance (positively or negatively) of your queries. For more information on the different field types see here.
More importantly you can see email and createdAt contains fields key, this is to inform elastic search that the fields will make use of multiple fields type, more information can be found here. Now that we have understood the create-index endpoint, it is time to insert some data.
Inserting data into the Elasticsearch Database
For the sake of this tutorial, we will be inserting data from a JSON file. You can download the JSON file here. Then update the post data section of the code
app.post("/data", async (req, res) => {
try {
const sampleData = require("./sample.json");
const sampleDataLength = sampleData.length;
for (let idx = 0; idx < sampleDataLength; idx++) {
await axios.post(`${esUrl}${req.body.index}/_doc`, sampleData[idx]);
}
res.json("Bulk data inserted");
} catch (error) {
res.status(500).json(error);
}
});
Ensure sample.json is in the same directory as the index.js file. Then go to your postman and make a post request to the data endpoint as seen below. Voila, we have successfully inserted the bulk data.
Retrieving Data from Elasticsearch
Elasticsearch has a plethora of ways to perform simple and complex queries in retrieving data from the database, and it is impossible to cover them in this tutorial, but for a start, we will cover how to perform simple data retrieval from the database. The code below shows how to retrieve data in descending order of CreatedAt, and how to query from single and multiple fields in Elasticsearch.
app.get("/data/:index", async (req, res) => {
try {
let response;
const test = req.query.test;
switch (test) {
case "sorting":
response = await axios.post(`${esUrl}${req.params.index}/_search`, {
sort: {
createdAt: "desc",
},
});
break;
case "matching":
response = await axios.post(`${esUrl}${req.params.index}/_search`, {
query: {
match: {
country: "samoa",
},
},
});
break;
case "multi-matching":
response = await axios.post(`${esUrl}${req.params.index}/_search`, {
query: {
bool: {
must: [
{
match: {
name: "Anastacio Stamm",
},
},
{
match: {
country: "Samoa",
},
},
],
},
},
});
break;
default:
response = await axios.get(`${esUrl}${req.params.index}/_search`);
break;
}
res.json(response.data);
} catch (error) {
res.json(error);
}
});In your postman make a GET request to http://localhost:3000/data/users, this will retrieve the first 10 results of all users in the Elasticsearch Database.
Deleting data from the Elasticsearch Index
Deleting data is quite easy, all that is needed is to get the document id required to be deleted and then make a delete request to the Elasticsearch delete doc api as shown in the code below
app.delete("/data/:index/:id", async (req, res) => {
try {
const response = await axios.delete(
`${esUrl}${req.params.index}/_doc/${req.params.id}`
);
res.json(response.data);
} catch (error) {
res.json(error);
}
res.json(`Deleted data ${req.params.id}`);
});Voila! We have learnt about Elasticsearch, its advantages and how to make CRUD operations in Elasticsearch using Nodejs and Axios. It is important to note you can use any HTTP client of your choice to make request to Elasticsearch APIs or use the Elasticsearch Javascript Client.
To wrap it up, here are some of the use cases of Elasticsearch
- Website search
- Application Search
- Log Analytics
- Security Analytics
- Business Analytics
- Full-text search, amongst many other use cases
I hope you enjoyed this tutorial, comment, like and share. Cheers
