Express Monitoring Setup

When deploying Express applications to production, monitoring becomes a critical aspect of maintaining reliability and performance. This guide will walk you through setting up effective monitoring for your Express applications, helping you detect issues before they impact your users.

Introduction to Express Monitoring

Monitoring provides visibility into your application's health, performance, and usage patterns. With proper monitoring, you can:

• Identify performance bottlenecks
• Track errors and exceptions in real-time
• Monitor resource usage (CPU, memory, disk)
• Understand user behavior and application usage
• Receive alerts when problems occur

By implementing a comprehensive monitoring solution, you'll be able to maintain higher uptime and provide a better experience for your users.

Setting Up Basic Logging

Step 1: Use Morgan for HTTP Request Logging

Morgan is an HTTP request logger middleware for Express that simplifies logging of requests.

First, install Morgan:

bash

npm install morgan

Then implement it in your Express application:

javascript

const express = require('express');
const morgan = require('morgan');
const app = express();

// Use morgan with the 'combined' format for detailed logs
app.use(morgan('combined'));

// Alternative formats:
// app.use(morgan('tiny')); // Minimal output
// app.use(morgan('dev'));  // Colored output for development

app.get('/', (req, res) => {
  res.send('Hello World!');
});

app.listen(3000, () => {
  console.log('Server running on port 3000');
});

Output example (combined format):

::1 - - [10/May/2023:15:23:41 +0000] "GET / HTTP/1.1" 200 11 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36"

Step 2: Implement Custom Logging with Winston

Winston is a versatile logging library that allows for more sophisticated logging configurations:

bash

npm install winston

Basic Winston setup:

javascript

const winston = require('winston');

// Create a logger
const logger = winston.createLogger({
  level: 'info',
  format: winston.format.combine(
    winston.format.timestamp(),
    winston.format.json()
  ),
  transports: [
    // Write logs to console
    new winston.transports.Console(),
    // Write logs to a file
    new winston.transports.File({ filename: 'error.log', level: 'error' }),
    new winston.transports.File({ filename: 'combined.log' })
  ]
});

// Example usage
app.get('/api/data', (req, res) => {
  logger.info('Data endpoint accessed', { 
    userId: req.user?.id,
    query: req.query
  });
  
  try {
    // Your code here
    res.json({ success: true });
  } catch (error) {
    logger.error('Error in data endpoint', { 
      error: error.message,
      stack: error.stack
    });
    res.status(500).json({ error: 'Internal server error' });
  }
});

Error Tracking and Monitoring

Implementing a Global Error Handler

Create a centralized error handling middleware in Express:

javascript

// errorHandler.js
const logger = require('./logger'); // Your Winston logger

module.exports = function(err, req, res, next) {
  // Log the error
  logger.error('Unhandled exception', {
    error: err.message,
    stack: err.stack,
    url: req.originalUrl,
    method: req.method,
    body: req.body,
    user: req.user?.id
  });
  
  // Respond with an error
  res.status(err.status || 500).json({
    error: {
      message: process.env.NODE_ENV === 'production' 
        ? 'Internal server error' 
        : err.message
    }
  });
};

Attach it to your Express app:

javascript

const errorHandler = require('./errorHandler');

// Your routes go here

// Error handler should be the last middleware
app.use(errorHandler);

Using Express-Status-Monitor

Express-status-monitor provides a real-time dashboard for your Express application:

bash

npm install express-status-monitor

Implementation:

javascript

const express = require('express');
const statusMonitor = require('express-status-monitor');
const app = express();

// Add status monitor middleware
app.use(statusMonitor());

// Your routes go here

app.listen(3000);

Now you can access a real-time monitoring dashboard at http://yourapp.com/status.

Advanced Monitoring with APM Tools

Setting up Application Performance Monitoring with New Relic

New Relic provides comprehensive monitoring of your Express applications:

bash

npm install newrelic

Create a newrelic.js configuration file in your project's root:

javascript

'use strict'

exports.config = {
  app_name: ['My Express Application'],
  license_key: 'your_license_key_here',
  logging: {
    level: 'info'
  },
  allow_all_headers: true,
  attributes: {
    exclude: [
      'request.headers.cookie',
      'request.headers.authorization',
      'request.headers.proxyAuthorization',
      'request.headers.setCookie*',
      'request.headers.x*',
      'response.headers.cookie',
      'response.headers.authorization',
      'response.headers.proxyAuthorization',
      'response.headers.setCookie*',
      'response.headers.x*'
    ]
  }
}

Require New Relic at the very beginning of your application:

javascript

// This must be the first line of your main file
require('newrelic');

const express = require('express');
const app = express();
// Rest of your application

Using Prometheus and Grafana for Metrics

Collect and visualize metrics with Prometheus and Grafana:

bash

npm install prom-client express-prom-bundle

Implementation:

javascript

const express = require('express');
const promBundle = require('express-prom-bundle');

const app = express();

// Add prometheus middleware
const metricsMiddleware = promBundle({
  includeMethod: true,
  includePath: true,
  promClient: {
    collectDefaultMetrics: {
      timeout: 5000
    }
  }
});

app.use(metricsMiddleware);

// Your routes go here

app.listen(3000);

This exposes a /metrics endpoint that Prometheus can scrape.

Health Checks for Container Orchestration

Create a health check endpoint to integrate with container platforms:

javascript

app.get('/health', (req, res) => {
  // Perform checks to ensure the app is healthy
  const databaseHealthy = checkDatabaseConnection();
  const cacheHealthy = checkCacheConnection();
  
  if (databaseHealthy && cacheHealthy) {
    return res.status(200).json({ status: 'ok' });
  }
  
  res.status(500).json({ 
    status: 'error',
    checks: {
      database: databaseHealthy ? 'ok' : 'error',
      cache: cacheHealthy ? 'ok' : 'error'
    }
  });
});

// Example check functions
function checkDatabaseConnection() {
  try {
    // Replace with actual database check
    return true;
  } catch (error) {
    logger.error('Database health check failed', { error });
    return false;
  }
}

function checkCacheConnection() {
  try {
    // Replace with actual cache check
    return true;
  } catch (error) {
    logger.error('Cache health check failed', { error });
    return false;
  }
}

Real-world Implementation: Complete Setup

Here's a comprehensive example combining multiple monitoring approaches:

javascript

// monitoring.js - Centralized monitoring setup
const morgan = require('morgan');
const winston = require('winston');
const promBundle = require('express-prom-bundle');

// Configure Winston logger
const logger = winston.createLogger({
  level: process.env.LOG_LEVEL || 'info',
  format: winston.format.combine(
    winston.format.timestamp(),
    winston.format.errors({ stack: true }),
    winston.format.json()
  ),
  defaultMeta: { service: 'express-app' },
  transports: [
    new winston.transports.Console(),
    new winston.transports.File({ filename: 'logs/error.log', level: 'error' }),
    new winston.transports.File({ filename: 'logs/combined.log' })
  ]
});

// Custom morgan stream that writes to Winston
const morganStream = {
  write: (message) => {
    logger.http(message.trim());
  }
};

// Configure Prometheus metrics
const metricsMiddleware = promBundle({
  includeMethod: true,
  includePath: true,
  promClient: {
    collectDefaultMetrics: {
      timeout: 5000
    }
  }
});

// Error handler middleware
const errorHandler = (err, req, res, next) => {
  logger.error('Unhandled exception', {
    error: err.message,
    stack: err.stack,
    path: req.path
  });
  
  res.status(500).json({
    error: process.env.NODE_ENV === 'production' 
      ? 'Internal server error' 
      : err.message
  });
};

module.exports = {
  setupMonitoring: (app) => {
    // HTTP request logging
    app.use(morgan('combined', { stream: morganStream }));
    
    // Metrics endpoint
    app.use(metricsMiddleware);
    
    // Health check endpoint
    app.get('/health', (req, res) => {
      res.status(200).json({ status: 'ok', uptime: process.uptime() });
    });
    
    // Detailed system info (protected endpoint)
    app.get('/system', (req, res) => {
      // In production, this should be protected
      res.json({
        memory: process.memoryUsage(),
        uptime: process.uptime(),
        pid: process.pid,
        versions: process.versions,
        env: process.env.NODE_ENV
      });
    });
    
    return {
      logger,
      errorHandler
    };
  }
};

Main app implementation:

javascript

const express = require('express');
const { setupMonitoring } = require('./monitoring');

const app = express();

// Setup monitoring
const { logger, errorHandler } = setupMonitoring(app);

// Your routes
app.get('/', (req, res) => {
  logger.info('Home route accessed');
  res.send('Hello World!');
});

app.get('/error-test', (req, res, next) => {
  try {
    throw new Error('Test error');
  } catch (error) {
    next(error);
  }
});

// Register error handler
app.use(errorHandler);

// Start server
const PORT = process.env.PORT || 3000;
app.listen(PORT, () => {
  logger.info(`Server started on port ${PORT}`);
});

Summary

Setting up proper monitoring for your Express application involves several components:

1. Request logging with tools like Morgan to track HTTP traffic
2. Application logging with Winston to record application events and errors
3. Performance monitoring with APM tools like New Relic or open-source alternatives
4. Metrics collection with Prometheus for detailed performance data
5. Health checks to integrate with container orchestrators and monitoring systems
6. Error tracking to catch and log exceptions

By implementing these monitoring solutions, you can ensure your Express application remains reliable, performant, and maintainable in production environments.

Additional Resources

• Express.js Documentation
• Winston Documentation
• Morgan Documentation
• Prometheus Node.js Client
• New Relic Node.js Agent

Exercises

1. Implement basic Winston logging in an existing Express application.
2. Create a custom Morgan token that includes the user ID in the logs if the user is authenticated.
3. Set up the Express-status-monitor and customize its dashboard.
4. Create a comprehensive health check endpoint that checks database connectivity, external API dependencies, and system resources.
5. Configure Prometheus metrics and set up a basic Grafana dashboard to visualize them.