Node.js Process Exit Strategies: Signals, Errors, and Graceful Shutdowns

Background Introduction

After our service is deployed, it is inevitable that it will be scheduled by the runtime environment (such as containers, PM2, etc.), undergo service upgrades that cause restarts, or encounter various exceptions leading to process crashes. Generally, runtime environments have health monitoring mechanisms for service processes. When a process crashes, the runtime will restart it. During upgrades, rolling upgrade strategies are typically used. However, the scheduling strategies of runtime environments treat our service processes as black boxes, without considering their internal state. Therefore, our service process must proactively detect scheduling actions from the runtime environment and perform necessary cleanup actions before exiting.

Today, we will summarize various scenarios that can cause a Node.js process to exit and discuss what we can do by listening to these process exit events.

Principles

A process exits in one of two ways:

The process voluntarily exits.
The process receives a system signal instructing it to exit.

Exit via System Signals

The official Node.js documentation lists common system signals. We focus on the following:

SIGHUP: Triggered when the terminal is closed directly instead of using Ctrl + C to stop the process.
SIGINT: Triggered when pressing Ctrl + C to stop the process. PM2 also sends this signal to child processes when restarting or stopping them.
SIGTERM: Typically used to gracefully terminate a process. For example, when Kubernetes deletes a pod, it sends a SIGTERM signal to allow the pod to perform cleanup actions within a timeout period (default: 30 seconds).
SIGBREAK: Triggered on Windows systems when Ctrl + Break is pressed.
SIGKILL: Forces the process to exit immediately, preventing any cleanup actions. When running kill -9 pid, the process receives this signal. In Kubernetes, if a pod does not exit within the 30-second timeout, Kubernetes sends a SIGKILL signal to terminate it immediately. Similarly, PM2 sends SIGKILL if a process does not exit within 1.6 seconds during restart or termination.

For non-forceful exit signals, a Node.js process can listen for these signals and define custom exit behaviors. For example, if we have a CLI tool that takes a long time to execute tasks, we can prompt the user before exiting when Ctrl + C is pressed:

const readline = require('readline');

process.on('SIGINT', () => {
  // Simple command-line interaction using readline
  const rl = readline.createInterface({
    input: process.stdin,
    output: process.stdout,
  });

  rl.question('The task is not yet complete. Are you sure you want to exit? ', (answer) => {
    if (answer === 'yes') {
      console.log('Task interrupted. Exiting process.');
      process.exit(0);
    } else {
      console.log('Task continues...');
    }
    rl.close();
  });
});

// Simulate a task that takes one minute to complete
const longTimeTask = () => {
  console.log('Task started...');
  setTimeout(() => {
    console.log('Task completed.');
  }, 1000 * 60);
};

longTimeTask();

This script will display a prompt each time Ctrl + C is pressed:

The task is not yet complete. Are you sure you want to exit? no
Task continues...

The task is not yet complete. Are you sure you want to exit? no
Task continues...

The task is not yet complete. Are you sure you want to exit? yes
Task interrupted. Exiting process.

Voluntary Process Exit

A Node.js process can voluntarily exit due to the following scenarios:

An uncaught error occurs during execution (can be captured using process.on('uncaughtException')).
An unhandled Promise rejection occurs (from Node.js v16 onwards, unhandled rejections cause the process to exit; use process.on('unhandledRejection') to handle them).
An error event is emitted by an EventEmitter but is not handled.
The process explicitly calls process.exit().
The Node.js event loop is empty (i.e., there are no pending tasks), which can be detected using process.on('exit').

PM2 has a daemon process that restarts the service if it crashes. We can implement a similar self-healing mechanism in Node.js using the cluster module, where worker processes are automatically restarted if they crash:

const cluster = require('cluster');
const http = require('http');
const numCPUs = require('os').cpus().length;
const process = require('process');

if (cluster.isMaster) {
  console.log(`Master process started: ${process.pid}`);

  // Create worker processes based on the number of CPU cores
  for (let i = 0; i < numCPUs; i++) {
    cluster.fork();
  }

  // Listen for worker exit events
  cluster.on('exit', (worker, code, signal) => {
    console.log(`Worker ${worker.process.pid} exited with code: ${code || signal}. Restarting...`);
    cluster.fork();
  });
}

if (cluster.isWorker) {
  process.on('uncaughtException', (error) => {
    console.log(`Worker ${process.pid} encountered an error`, error);
    process.emit('disconnect');
    process.exit(1);
  });

  // Create an HTTP server
  http
    .createServer((req, res) => {
      res.writeHead(200);
      res.end('Hello world\n');
    })
    .listen(8000);

  console.log(`Worker process started: ${process.pid}`);
}

Practical Implementation

Now that we have analyzed different scenarios where a Node.js process might exit, let's implement a process exit listener that allows users to define custom exit behaviors.

// exit-hook.js
const tasks = [];

const addExitTask = (fn) => tasks.push(fn);

const handleExit = (code, error) => {
  // Implementation details will be explained below
};

process.on('exit', (code) => handleExit(code));
process.on('SIGHUP', () => handleExit(128 + 1));
process.on('SIGINT', () => handleExit(128 + 2));
process.on('SIGTERM', () => handleExit(128 + 15));
process.on('SIGBREAK', () => handleExit(128 + 21));
process.on('uncaughtException', (error) => handleExit(1, error));
process.on('unhandledRejection', (error) => handleExit(1, error));

For handleExit, we ensure both synchronous and asynchronous tasks are handled properly using process.nextTick():

let isExiting = false;

const handleExit = (code, error) => {
  if (isExiting) return;
  isExiting = true;

  let hasDoExit = false;
  const doExit = () => {
    if (hasDoExit) return;
    hasDoExit = true;
    process.nextTick(() => process.exit(code));
  };

  let asyncTaskCount = 0;
  let asyncTaskCallback = () => {
    process.nextTick(() => {
      asyncTaskCount--;
      if (asyncTaskCount === 0) doExit();
    });
  };

  tasks.forEach((taskFn) => {
    if (taskFn.length > 1) {
      asyncTaskCount++;
      taskFn(error, asyncTaskCallback);
    } else {
      taskFn(error);
    }
  });

  if (asyncTaskCount > 0) {
    setTimeout(() => doExit(), 10 * 1000);
  } else {
    doExit();
  }
};

Graceful Process Exit

When restarting a web server or handling runtime container scheduling (PM2, Docker, etc.), we want to:

Complete ongoing requests.
Clean up database connections.
Log errors and trigger alerts.
Perform other necessary shutdown actions.

Using the exit-hook tool:

const http = require('http');

const server = http
  .createServer((req, res) => {
    res.writeHead(200);
    res.end('Hello world\n');
  })
  .listen(8000);

addExitTask((error, callback) => {
  console.log('Process exiting due to error:', error);
  server.close(() => {
    console.log('Stopped accepting new requests.');
    setTimeout(callback, 5000);
  });
});

Conclusion

By understanding the various scenarios that cause Node.js processes to exit, we can proactively detect and handle abnormal or scheduled terminations. While tools like Kubernetes and PM2 can restart crashed processes, implementing in-code monitoring allows us to detect and resolve issues earlier.