Posted inBlog

Creating a Microservice Using Node.js

No Comments
Node Js

JavaScript consistently holds a top position in programming language rankings. There are two primary environments for JavaScript: the browser and Node.js. The latter is extensively used for developing web services.

I began learning JavaScript when I started my I.T. career about eight years ago, and a few years back, I read the excellent book “JavaScript: The Definitive Guide” by David Flanagan.

This book answered many of my questions and introduced me to some intriguing JavaScript techniques that I now use in my daily work. More importantly, it led me to start developing web services with Node.js, having previously used JavaScript only for frontend development. I highly recommend this book to both novice and seasoned JavaScript developers.

In this post, I will demonstrate how to build a microservice using Node.js. Drawing from my web development experience, I aim to create a scalable, robust, reliable, and high-performance solution using the popular Node.js stack.

Introduction

In this article, I will demonstrate how to build a microservice using a Task Management web service as an example. The service will provide the following API:

  • Create a task with a name and description
  • Retrieve a task by its identifier
  • Update the status, name, or description of a task

This simple API will illustrate the power of Node.js in building web applications, showcasing a development process that is both fast and straightforward.

Some application requirements include:

  • Tasks should be created with the status ‘new’
  • Available status transitions: from ‘new’ to ‘active’, from ‘new’ to ‘cancelled’, from ‘active’ to ‘completed’, from ‘active’ to ‘cancelled’
  • Avoid race conditions (more details to follow)

The main non-functional requirements are:

  • Scalability — the microservice should handle an increasing number of requests
  • Elasticity — the microservice should accommodate traffic spikes
  • Performance — the microservice should respond quickly to enhance user experience
  • Resilience — the microservice should be fault-tolerant and capable of recovery to maintain correct functionality
  • Monitoring — the microservice should provide mechanisms to monitor its health
  • Observability — the microservice should generate log streams and metrics for maintenance purposes
  • Testability — the microservice should be easy to test
  • Statelessness — the microservice should not store client context; instead, state should be stored in a database
  • Deployability — the microservice should be easy to deploy and update

All of these requirements can be met when developing a web application with Node.js. Let’s discuss how to achieve these objectives in the following steps.

Stack

When starting to build a web service from scratch, choosing the right tech stack is crucial. Naturally, the first consideration is the programming language. For this microservice, I’ll be using Node.js. Here are some advantages of using Node.js for developing web services:

  • JavaScript is already the predominant language for frontend development, so using it for backend development allows the same developers to create full-stack applications.
  • The JavaScript community is vast, providing answers to almost any development question. Additionally, numerous libraries are developed and maintained by the community, offering various third-party solutions with unique features.
  • Node.js interprets JavaScript using Google’s V8 engine, which efficiently compiles JavaScript into machine code.

The benefits are extensive, but the choice of Node.js is just the beginning.

Database

I need to maintain data between web requests. Developing scalable stateful web services can be complex, so it’s advisable to keep your web application stateless and store the state in an external database instead.

For this service, I will use the popular document-oriented database MongoDB:

MongoDB is a NoSQL database that offers several advantages over SQL databases:

  • Schemaless — A MongoDB collection (similar to an SQL table) can contain documents with varying schemas. There’s no need to define a structure before storing documents in the collection.
  • Scalability — MongoDB is designed to scale out across multiple servers.
  • Performance — MongoDB is optimized for read-heavy workloads and can handle large volumes of data.

MongoDB is a popular choice when developing Node.js web services.

Web Framework

A web framework is essential for building web applications, as it manages many routine tasks necessary for developing web services, such as routing, security, and binding.

There are numerous options for Node.js web frameworks, with Express being the most popular:

The primary advantage of Express is its simplicity and minimal code requirement to start the web server. Here is a ‘Hello, World!’ example written with Express:

const express = require('express')
const app = express()
const port = 3000;

app.get('/', (req, res) => {
  res.send('Hello World!')
})
app.listen(port, () => {
  console.log(`Example app listening on port ${port}`)
})

Also, express has a huge community. You can find different libraries that extend your server functionality.

Validation

Validation is a crucial aspect of web applications since it’s impossible to predict how users will interact with your API. Malicious users could potentially disrupt your application with invalid input.

To ensure the parameters provided in web requests (such as those in the path or body) are correct, I will use Joi.

It’s a powerful library used to validate different models. There is an example of validating one of the requests I’ll develop later:

const createTask = {
  body: Joi.object().keys({
    name: Joi.string().required(),
    description: Joi.string().optional(),
  }),
};

It validates that the object has a nested object body with two strings: mandatory name and optional description.

But that’s not all. Providing some dangerous MongoDB injection that can drop collection is still possible. To sanitize web requests, I will use the express-mongo-sanitize package:

Configuration

You need to make your application configurable so the same build artifact can be run in different environments by changing the configuration. The standard approach is to provide configuration via environment variables.

You probably don’t want to manually set environment variables on your local machine before the application starts. The popular Node.js solution for this issue is implemented in the library dotenv:

This library loads content from a file named .env (filename can be changed) and sets content from this file to environment variables.

Static Analysis

With JavaScript applications, you can easily maintain consistent code style and adhere to uniform rules by using the ESLint library.

ESLint enhances code quality and can detect bugs and security vulnerabilities. By incorporating ESLint checks into your continuous integration (CI) process, you can ensure your team follows the established coding standards. I will implement this later.

Testing

Writing automated tests for your application is a good practice to ensure it functions correctly even after changes. There are various types of tests, including unit, integration, load, and end-to-end (E2E). To guarantee your application’s quality and performance, it’s essential to have a comprehensive suite of tests.

One of the most popular libraries for testing JavaScript applications is Jest. With Jest, I will implement unit and integration tests. These tests will help me improve and refactor code in later stages, ensuring the application remains functional.

Logging

An application log stream allows you to ‘remotely debug’ your web service. By covering your code with logs, you can easily trace the code execution path and understand the request logic under different circumstances.

The most popular package for collecting JavaScript logs is Winston. Winston is a simple yet powerful logging library that enables you to collect logs using various transports (console, file, etc.). You can also customize the log format (e.g., simple text, JSON).

Metrics

Metrics enable you to monitor the health of your application. They provide insights such as the number of incoming requests, average request execution time, and the number of 5XX responses. With metrics, you can set up monitors to alert you via email or notifications if something goes wrong.

For my application, I will use Prometheus middleware to collect standard web application metrics through express-prom-bundle:

More about Prometheus in the next section.

Monitoring Stack

Log stream and metrics should be collected in some database so later that can be used to monitor them or visualize.

I will use the next stack to collect and visualize logs and metrics:

  • Prometheus — open-source monitoring alerting toolkit that uses a pull model to collect metrics;
  • Promtail — an agent that contains and ships logs;
  • Loki — log aggregation system;
  • Grafana — observability system.

Local Infrastructure

To create the local stack needed to develop and test the application locally, I’m going to use Docker:

With Docker, you can start a local environment similar to what will be used in staging and production environments. You don’t need to install many tools on your local machine. Instead, you can execute several commands to start the stack you need.

With Docker Compose, you can define all the infrastructure with a single compose.yml file and start it with a single command:

docker compose up -d

Continuous Integration

To be sure commits do not break anything, you need continuous integration (CI).

For this purpose, I will use GitHub Actions:

There is a free tier for your GitHub account so you can run simple builds to check your application code.

Develop Application

I’ve started with application structure and found a great repository that helped me to follow suitable project styles:

I’ve borrowed some code from this repository (e.g., validation middleware), so I recommend for you to check it.

Later, I’ll share a link to the repository, but before that, I want to show some interesting application parts.

I’ve used mongoose to integrate applications with MongoDB. First, I’ve defined the schema of the model:

const mongoose = require('mongoose');
const { Schema } = mongoose;
const TaskSchema = new Schema(
  {
    name: {
      type: String,
      required: true,
    },
    description: {
      type: String,
      required: false,
    },
    status: {
      type: String,
      enum: ['new', 'active', 'completed', 'cancelled'],
      default: 'new',
    },
    createdAt: {
      type: Date,
      default: Date.now,
    },
    updatedAt: Date,
  },
  { optimisticConcurrency: true },
);
module.exports = mongoose.model('task', TaskSchema);

This object can be used to validate the model and perform different MongoDB operations from your code. This is an example of a task update:

async function updateTaskById(id, { name, description, status }) {
  if (!name && !description && !status) {
    return { error: 'at least one update required', code: AT_LEAST_ONE_UPDATE_REQUIRED_CODE };
  }

if (status && !(status in availableUpdates)) {
    return { error: 'invalid status', code: INVALID_STATUS_CODE };
  }
  for (let retry = 0; retry < 3; retry += 1) {
    // eslint-disable-next-line no-await-in-loop
    const task = await Task.findById(id);
    if (!task) {
      return { error: 'task not found', code: INVALID_STATUS_TRANSITION_CODE };
    }
    if (status) {
      const allowedStatuses = availableUpdates[task.status];
      if (!allowedStatuses.includes(status)) {
        return {
          error: `cannot update from '${task.status}' to '${status}'`,
          code: TASK_NOT_FOUND_CODE,
        };
      }
    }
    task.status = status ?? task.status;
    task.name = name ?? task.name;
    task.description = description ?? task.description;
    task.updatedAt = Date.now();
    try {
      // eslint-disable-next-line no-await-in-loop
      await task.save();
    } catch (error) {
      logger.warn('error during save', { error });
      if (error.name === 'VersionError') {
        // eslint-disable-next-line no-continue
        continue;
      }
    }
    return task;
  }
  return { error: 'concurrency error', code: CONCURRENCY_ERROR_CODE };
}

The most exciting part is saving the model after the update. I’m using an optimistic lock to fight against the race condition problem.

Imagine in two concurrent requests, you’re trying to complete and cancel the same task. Race conditions might occur when they both get a task with the status ‘active’ and save the model. The first task status might be changed to ‘completed’ and then to ‘canceled’ (or vice versa). This is wrong because the transition ‘completed’-’canceled’ and ‘canceled’-’completed’ is prohibited.

Mongoose has the solution for this issue implemented with optimistic locks. Optimistic lock is a strategy used in databases to handle concurrent requests. Each document has an additional version property. When the transaction tries to save/update the model, it checks the version. If the version differs from the version received during the get query, somebody has already updated the document concurrently. The transaction is aborted (in the code above, an error is thrown).

Document example:

{
  "_id": {
    "$oid": "654e03210948a61665b7c889"
  },
  "name": "damnatio",
  "description": "Ciminatio totus spiritus suffoco damnatio blanditiis.",
  "status": "completed",
  "createdAt": {
    "$date": "2023-11-10T10:17:05.039Z"
  },
  "__v": 2,
  "updatedAt": {
    "$date": "2023-11-10T10:17:05.064Z"
  }
}

The document above stores the version in property __v.

The next level is a controller. Here is controller example:

const updateTaskById = catchAsync(async (req, res) => {
  const result = await taskService.updateTaskById(req.params.id, req.body);
  if (result.error) {
    switch (result.code) {
      case taskService.errorCodes.AT_LEAST_ONE_UPDATE_REQUIRED_CODE:
        res.status(400).json({ success: false, message: 'at least one update required' });
        return;
      case taskService.errorCodes.INVALID_STATUS_CODE:
        res.status(400).json({ success: false, message: 'invalid status' });
        return;
      case taskService.errorCodes.INVALID_STATUS_TRANSITION_CODE:
        res.status(404).json({ success: false, message: 'task not found' });
        return;
      case taskService.errorCodes.TASK_NOT_FOUND_CODE:
        res.status(400).json({ success: false, message: result.error });
        return;
      case taskService.errorCodes.CONCURRENCY_ERROR_CODE:
        res.status(500).json({ success: false, message: 'concurrency error' });
        return;
      default:
        res.status(500).json({ success: false, message: 'internal server error' });
        return;
    }
  }

res.status(200).json({
    success: true,
    task: toDto(result),
  });
});

It’s responsible for executing application business logic and returning the HTTP response. Controllers are registered in the routes module:

const { Router } = require('express');
const taskController = require('../../../controllers/task');
const taskValidation = require('../../../validation/task');
const validate = require('../../../middlewares/validate');

const router = Router();
router.get('/:id', validate(taskValidation.getTaskById), taskController.getTaskById);
router.put('/', validate(taskValidation.createTask), taskController.createTask);
router.post('/:id', validate(taskValidation.updateTaskById), taskController.updateTaskById);
module.exports = router;

/**
 * @swagger
 * tags:
 *  name: Tasks
 *  description: Task management and retrieval
 * /v1/tasks/{id}:
 *  get:
 *   summary: Get a task by id
 *   tags: [Tasks]
 *   description: Get a task by id
 *   parameters:
 *    - in: path
 *      name: id
 *      schema:
 *       type: string
 *       required: true
 *       description: Task id
 *       example: 5f0a3d9a3e06e52f3c7a6d5c
 *   responses:
 *    200:
 *     description: Task Retrieved
 *     content:
 *      application/json:
 *       schema:
 *        $ref: '#/components/schemas/TaskResult'
 *    404:
 *     description: Task not found
 *     content:
 *      application/json:
 *       schema:
 *        $ref: '#/components/schemas/TaskResult'
 *    500:
 *     description: Internal Server Error
 *  post:
 *   summary: Update a task by id
 *   tags: [Tasks]
 *   description: Update a task by id
 *   parameters:
 *    - in: path
 *      name: id
 *      schema:
 *       type: string
 *       required: true
 *       description: Task id
 *       example: 5f0a3d9a3e06e52f3c7a6d5c
 *   requestBody:
 *    required: true
 *    content:
 *     application/json:
 *      schema:
 *       $ref: '#/components/schemas/UpdateTask'
 *   responses:
 *    200:
 *     description: Task Updated
 *     content:
 *      application/json:
 *       schema:
 *        $ref: '#/components/schemas/TaskResult'
 *     404:
 *      description: Task not found
 *      content:
 *       application/json:
 *        schema:
 *         $ref: '#/components/schemas/TaskResult'
 *     500:
 *      description: Internal Server Error
 * /v1/tasks:
 *  put:
 *   summary: Create a task
 *   tags: [Tasks]
 *   description: Create a task
 *   requestBody:
 *    required: true
 *    content:
 *     application/json:
 *      schema:
 *       $ref: '#/components/schemas/CreateTask'
 *   responses:
 *    201:
 *     description: Task Created
 *     content:
 *      application/json:
 *       schema:
 *        $ref: '#/components/schemas/TaskResult'
 *    500:
 *     description: Internal Server Error
 */

At the bottom, you can see the OpenAPI specification used by Swagger middleware to generate the API documentation page.

Each route registration uses two handlers: the validator and the controller method itself. The validator validates the schema registered in different models. Validator handler:

const Joi = require('joi');
const pick = require('../utils/pick');

function validate(schema) {
  return (req, res, next) => {
    const validSchema = pick(schema, ['params', 'query', 'body']);
    const object = pick(req, Object.keys(validSchema));
    const { value, error } = Joi.compile(validSchema)
      .prefs({ errors: { label: 'key' }, abortEarly: false })
      .validate(object);
    if (error) {
      const errorMessage = error.details.map((details) => details.message).join(', ');
      res.status(400).json({ success: false, message: errorMessage });
      return;
    }
    Object.assign(req, value);
    next();
  };
}
module.exports = validate;

And there is an update request validation schema:

const updateTaskById = {
  params: Joi.object().keys({
    id: objectId.required(),
  }),
  body: Joi.object().keys({
    name: Joi.string().optional(),
    description: Joi.string().optional(),
    status: Joi.string().valid('new', 'active', 'completed', 'cancelled').optional(),
  }),
};

For the update method, I’ve implemented integration tests only. Integration test starts the server and stops it before and after all tests run:

const path = require('path');
const app = require('../../src/app');
const db = require('../../src/db');
const { createConfig } = require('../../src/config/config');
const logger = require('../../src/config/logger');
const setupServer = () => {
  let server;
  const configPath = path.join(__dirname, '../../configs/tests.env');
  const config = createConfig(configPath);
  beforeAll(async () => {
    logger.init(config);
    await db.init(config);
    await new Promise((resolve) => {
      server = app.listen(config.port, () => {
        resolve();
      });
    });
  });
  afterAll(async () => {
    await new Promise((resolve) => {
      server.close(() => {
        resolve();
      });
    });
    await db.destroy();
    logger.destroy();
  });
};
module.exports = {
  setupServer,
};

And there is a test that performs a PUT request (create task) and then a POST request (update task):

describe('should create & update a task', () => {
      const data = [
        {
          name: 'only status update',
          taskName: 'Task 1',
          description: 'Task 1 description',
          newStatus: 'active',
        },
        {
          name: 'english full update',
          taskName: 'Task 1',
          description: 'Task 1 description',
          newTaskName: 'Task 1 New',
          newDescription: 'Task 1 New description',
          newStatus: 'active',
        },
        {
          name: 'english only name update',
          taskName: 'Task 1',
          description: 'Task 1 description',
          newTaskName: 'Task 1 New',
        },
        {
          name: 'english only description update',
          taskName: 'Task 1',
          description: 'Task 1 description',
          newDescription: 'Task 1 New description',
        },
        {
          name: 'japanese full update',
          taskName: 'タスク 1',
          description: 'タスク 1 説明',
          newTaskName: 'タスク 1 新',
          newDescription: 'タスク 1 新 説明',
          newStatus: 'active',
        },
        {
          name: 'japanese only name update',
          taskName: 'タスク 1',
          description: 'タスク 1 説明',
          newTaskName: 'タスク 1 新',
        },
        {
          name: 'japanese only description update',
          taskName: 'タスク 1',
          description: 'タスク 1 説明',
          newDescription: 'タスク 1 新 説明',
        },
        {
          name: 'japanese only status update',
          taskName: 'タスク 1',
          description: 'タスク 1 説明',
          newStatus: 'active',
        },
        {
          name: 'chinese full update',
          taskName: '任务 1',
          description: '任务 1 描述',
          newTaskName: '任务 1 新',
          newDescription: '任务 1 新 描述',
          newStatus: 'active',
        },
        {
          name: 'chinese only name update',
          taskName: '任务 1',
          description: '任务 1 描述',
          newTaskName: '任务 1 新',
        },
        {
          name: 'chinese only description update',
          taskName: '任务 1',
          description: '任务 1 描述',
          newDescription: '任务 1 新 描述',
        },
        {
          name: 'chinese only status update',
          taskName: '任务 1',
          description: '任务 1 描述',
          newStatus: 'active',
        },
        {
          name: 'emoji full update',
          taskName: '👍',
          description: '👍',
          newTaskName: '👍 👍',
          newDescription: '👍 👍 👍',
          newStatus: 'active',
        },
        {
          name: 'emoji only name update',
          taskName: '👍',
          description: '👍',
          newTaskName: '👍 👍',
        },
        {
          name: 'emoji only description update',
          taskName: '👍',
          description: '👍',
          newDescription: '👍 👍',
        },
        {
          name: 'emoji only status update',
          taskName: '👍',
          description: '👍',
          newStatus: 'active',
        },
      ];

data.forEach(({
        name, taskName, description, newTaskName, newDescription, newStatus,
      }) => {
        it(name, async () => {
          let response = await fetch(baseUrl, {
            method: 'put',
            body: JSON.stringify({
              name: taskName,
              description,
            }),
            headers: { 'Content-Type': 'application/json' },
          });
          expect(response.status).toEqual(201);
          const result = await response.json();
          expect(result.task).not.toBeNull();
          expect(result.success).toEqual(true);
          expect(result.task.id).not.toBeNull();
          response = await fetch(`${baseUrl}/${result.task.id}`, {
            method: 'post',
            body: JSON.stringify({
              name: newTaskName,
              description: newDescription,
              status: newStatus,
            }),
            headers: { 'Content-Type': 'application/json' },
          });
          expect(response.status).toEqual(200);
          const result2 = await response.json();
          expect(result2).toEqual({
            success: true,
            task: {
              id: result.task.id,
              name: newTaskName ?? taskName,
              description: newDescription ?? description,
              status: newStatus ?? 'new',
              createdAt: result.task.createdAt,
              updatedAt: expect.any(String),
            },
          });
          expect(new Date() - new Date(result2.task.updatedAt)).toBeLessThan(1000);
        });
      });
    });

To create an application Docker image I’ve defined a simple Dockerfile:

FROM node:20-alpine
WORKDIR /app
COPY package.json yarn.lock ./
RUN yarn install --frozen-lockfile
COPY src /app/src
CMD ["node", "./src/index.js"]

To start application and infrastructure, there is compose.yml definition:

version: '3.9'
services:
    app:
        build: .
        ports:
            - '8081:80'
        depends_on:
            - mongo
        volumes:
            - ./configs/docker.env:/app/configs/.env
            - logs:/app/logs:rw
    mongo:
        image: mongo:5
        restart: always
        ports:
            - 27017:27017
        volumes:
            - mongodata:/data/db
        healthcheck:
            test: echo 'db.runCommand("ping").ok' | mongo localhost:27017/test --quiet
            interval: 10s
            timeout: 2s
            retries: 5
            start_period: 5s
    loki:
        image: grafana/loki:2.9.0
        expose:
            - 3100
        command: -config.file=/etc/loki/local-config.yaml
    promtail:
        image: grafana/promtail:2.9.0
        volumes:
            - logs:/var/log:rw
            - ./infrastructure/promtail.yml:/etc/promtail/config.yml
        command: -config.file=/etc/promtail/config.yml
    prometheus:
        image: prom/prometheus:latest
        volumes:
            - ./infrastructure/prometheus.yml:/etc/prometheus/prometheus.yml
        command:
            - '--config.file=/etc/prometheus/prometheus.yml'
        expose:
            - 9090
    grafana:
        image: grafana/grafana:latest
        volumes:
            - grafanadata:/var/lib/grafana
        environment:
            - GF_PATHS_PROVISIONING=/etc/grafana/provisioning
            - GF_AUTH_ANONYMOUS_ENABLED=true
            - GF_AUTH_ANONYMOUS_ORG_ROLE=Admin
        ports:
            - 3000:3000
volumes:
    mongodata:
    grafanadata:
    logs:

Every Git push triggers GitHub Actions CI to run the build. During CI, I’m installing dependencies, running linter and running all tests:

name: App CI
on:
  push:
    branches:
      - "*"
jobs:
  ci:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v3
      - uses: actions/setup-node@v3
        with:
          node-version: 20
          cache: "yarn"
      - run: yarn install --frozen-lockfile
      - run: yarn run lint
      - run: docker-compose up -d mongo
      - run: yarn test -- --verbose --coverage
      - run: docker-compose build
      - run: docker-compose logs
        if: always()
      - run: docker-compose down --volumes
        if: always()

Conclusion

Node.js is a powerful technology with a vast and active community. While it’s possible to use different stacks when developing a new web service, I recommend first mastering specific technologies within Node.js to become a proficient developer. Only then should you experiment with different stacks.

The stack I’ve used to build web applications in this post is one of the most popular for Node.js web services. There is extensive documentation and numerous libraries available to help you implement various features.

Meeting the non-functional requirements I outlined earlier is straightforward with Node.js. You can create a Docker image of your application and host it with Kubernetes. Kubernetes facilitates rapid scaling of your application by modifying the deployment definition and can adjust scaling based on incoming traffic. Additionally, MongoDB is designed to scale according to your needs, so increased traffic won’t be an issue.

Google’s V8 Node.js engine enhances application performance, impressively translating source code into machine code at high speed.

Numerous resources are available online for building fault-tolerant applications with Node.js. By following best practices, you can ensure robust performance. In my application code, I have implemented measures to restore the MongoDB connection after unexpected failures (such as network issues or MongoDB outages). You can review and adapt this for your applications.

With Jest, you can write various tests for your application and achieve 100% test coverage. Jest also allows you to emulate complex scenarios, enhancing your testing capabilities.

Comments

No comments yet. Why don’t you start the discussion?

Leave a Reply

Your email address will not be published. Required fields are marked *