Express API Monitoring

In the world of web development, building an API is just the beginning. Once your Express REST API is live, monitoring becomes crucial to ensure everything runs smoothly. This guide will walk you through various approaches to monitoring your Express APIs, from basic logging to advanced monitoring solutions.

Why Monitor Your API?

Before diving into the how, let's understand the why:

• Detect issues before users do: Catch problems early to minimize downtime
• Track performance: Identify bottlenecks and optimize your API
• Understand usage patterns: Learn how clients interact with your API
• Security monitoring: Detect unusual activities that might indicate security threats
• Resource planning: Make informed decisions about scaling based on actual usage data

Basic Logging in Express

The foundation of all monitoring is proper logging. Express makes it easy to implement basic logging with middleware.

Using Morgan for HTTP Request Logging

Morgan is a popular HTTP request logger middleware for Node.js:

bash

npm install morgan

Here's how to integrate it into your Express application:

javascript

const express = require('express');
const morgan = require('morgan');
const app = express();

// Use morgan with predefined format
app.use(morgan('combined'));

app.get('/api/users', (req, res) => {
  res.json({ users: ['John', 'Jane', 'Bob'] });
});

app.listen(3000, () => {
  console.log('Server running on port 3000');
});

When a request is made to /api/users, Morgan will output something like:

::1 - - [23/Apr/2023:14:15:54 +0000] "GET /api/users HTTP/1.1" 200 35 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64)"

This log entry includes the:

• IP address of the client
• Date and time of the request
• HTTP method and path
• HTTP status code
• Response size
• User agent information

Custom Logging Middleware

You can create custom middleware to log specific information:

javascript

// Custom logging middleware
app.use((req, res, next) => {
  const start = Date.now();
  
  // Once the response is finished
  res.on('finish', () => {
    const duration = Date.now() - start;
    console.log(`${req.method} ${req.originalUrl} ${res.statusCode} - ${duration}ms`);
  });
  
  next();
});

This middleware logs:

• The HTTP method
• The requested URL
• The response status code
• The time taken to process the request

Error Monitoring and Handling

Proper error handling is essential for monitoring the health of your API.

Creating an Error-Handling Middleware

javascript

app.use((err, req, res, next) => {
  // Log error details
  console.error(`Error: ${err.message}`);
  console.error(`Stack: ${err.stack}`);
  
  // Send appropriate response to client
  res.status(err.status || 500).json({
    error: {
      message: process.env.NODE_ENV === 'production' 
        ? 'Internal server error' 
        : err.message
    }
  });
});

Centralized Error Tracking

For production applications, consider using error tracking services like:

• Sentry
• Rollbar
• New Relic

Here's an example using Sentry:

javascript

const express = require('express');
const Sentry = require('@sentry/node');

const app = express();

// Initialize Sentry - make sure this is before all other middleware
Sentry.init({ dsn: 'YOUR_SENTRY_DSN' });

// The request handler must be the first middleware
app.use(Sentry.Handlers.requestHandler());

// Your routes
app.get('/api/users', (req, res) => {
  res.json({ users: ['John', 'Jane', 'Bob'] });
});

// Error routes
app.get('/error', (req, res) => {
  throw new Error('This is a test error');
});

// The error handler must be before any other error middleware and after all controllers
app.use(Sentry.Handlers.errorHandler());

// Regular error handlers
app.use((err, req, res, next) => {
  res.status(500).json({ error: err.message });
});

app.listen(3000);

Performance Monitoring

Response Time Monitoring

Tracking how long your API endpoints take to respond helps identify bottlenecks:

javascript

const responseTime = require('response-time');

// Add response time header to all responses
app.use(responseTime((req, res, time) => {
  // You could log this to your monitoring system
  console.log(`${req.method} ${req.url} - Response time: ${time}ms`);
}));

Memory Usage Monitoring

Periodically check your application's memory usage:

javascript

function logMemoryUsage() {
  const memoryUsage = process.memoryUsage();
  console.log('Memory usage:');
  console.log(`- RSS: ${Math.round(memoryUsage.rss / 1024 / 1024)} MB`);
  console.log(`- Heap Total: ${Math.round(memoryUsage.heapTotal / 1024 / 1024)} MB`);
  console.log(`- Heap Used: ${Math.round(memoryUsage.heapUsed / 1024 / 1024)} MB`);
}

// Log memory usage every 5 minutes
setInterval(logMemoryUsage, 5 * 60 * 1000);

Health Check Endpoints

Health checks help external monitoring tools verify that your API is functioning properly.

javascript

app.get('/health', (req, res) => {
  // Basic health check
  res.status(200).json({
    status: 'UP',
    timestamp: new Date().toISOString()
  });
});

// More comprehensive health check
app.get('/api/health-detailed', async (req, res) => {
  try {
    // Check database connection
    const dbStatus = await checkDatabaseConnection();
    
    // Check external services
    const externalServiceStatus = await checkExternalService();
    
    res.status(200).json({
      status: 'UP',
      database: dbStatus ? 'Connected' : 'Disconnected',
      externalService: externalServiceStatus,
      uptime: process.uptime(),
      memoryUsage: process.memoryUsage(),
      timestamp: new Date().toISOString()
    });
  } catch (error) {
    res.status(500).json({
      status: 'DOWN',
      error: error.message,
      timestamp: new Date().toISOString()
    });
  }
});

// Example utility functions
async function checkDatabaseConnection() {
  // Check your database connection here
  return true;
}

async function checkExternalService() {
  // Check any external service your API depends on
  return 'Available';
}

Using APM (Application Performance Monitoring) Tools

For comprehensive monitoring, consider using an APM tool. Here's how to set up New Relic:

bash

npm install newrelic

Create a newrelic.js file in your project root:

javascript

'use strict'
/**
 * New Relic agent configuration.
 */
exports.config = {
  app_name: ['My Express API'],
  license_key: 'YOUR_LICENSE_KEY',
  logging: {
    level: 'info'
  },
  allow_all_headers: true,
  attributes: {
    exclude: [
      'request.headers.cookie',
      'request.headers.authorization',
      'request.headers.proxyAuthorization',
      'request.headers.setCookie*',
      'request.headers.x*',
      'response.headers.cookie',
      'response.headers.authorization',
      'response.headers.proxyAuthorization',
      'response.headers.setCookie*',
      'response.headers.x*'
    ]
  }
}

Then require New Relic at the very top of your main file:

javascript

require('newrelic');
const express = require('express');
// ... rest of your code

Other popular APM options:

• DataDog
• AppDynamics
• Elastic APM

Real-World Example: Complete API Monitoring Setup

Here's a comprehensive example combining various monitoring approaches:

javascript

// First require New Relic if you're using it
require('newrelic');

const express = require('express');
const morgan = require('morgan');
const responseTime = require('response-time');
const Sentry = require('@sentry/node');
const fs = require('fs');
const path = require('path');
const rfs = require('rotating-file-stream');

const app = express();

// Initialize Sentry for error tracking
Sentry.init({ dsn: process.env.SENTRY_DSN });
app.use(Sentry.Handlers.requestHandler());

// Create a rotating write stream for logs
const logDirectory = path.join(__dirname, 'logs');
fs.existsSync(logDirectory) || fs.mkdirSync(logDirectory);
const accessLogStream = rfs.createStream('access.log', {
  interval: '1d',
  path: logDirectory
});

// Setup request logging
app.use(morgan('combined', { stream: accessLogStream }));

// Add response time header
app.use(responseTime());

// Track API usage
let requestCounter = 0;
app.use((req, res, next) => {
  requestCounter++;
  next();
});

// Basic API routes
app.get('/api/users', (req, res) => {
  res.json({ users: ['John', 'Jane', 'Bob'] });
});

// Health check endpoint
app.get('/health', (req, res) => {
  res.status(200).json({
    status: 'UP',
    requests: requestCounter,
    uptime: process.uptime(),
    timestamp: new Date().toISOString()
  });
});

// Metrics endpoint for Prometheus
app.get('/metrics', (req, res) => {
  const memUsage = process.memoryUsage();
  res.set('Content-Type', 'text/plain');
  res.send(`
# HELP process_resident_memory_bytes Memory usage
# TYPE process_resident_memory_bytes gauge
process_resident_memory_bytes ${memUsage.rss}

# HELP process_heap_bytes Heap size
# TYPE process_heap_bytes gauge
process_heap_bytes ${memUsage.heapTotal}

# HELP process_heap_used_bytes Used heap size
# TYPE process_heap_used_bytes gauge
process_heap_used_bytes ${memUsage.heapUsed}

# HELP api_requests_total Total API requests
# TYPE api_requests_total counter
api_requests_total ${requestCounter}
  `);
});

// Error route for testing
app.get('/error', (req, res) => {
  throw new Error('Test error');
});

// Sentry error handler
app.use(Sentry.Handlers.errorHandler());

// Final error handler
app.use((err, req, res, next) => {
  console.error(err.stack);
  res.status(500).json({ 
    error: process.env.NODE_ENV === 'production' ? 'Internal server error' : err.message
  });
});

// Start the server
const PORT = process.env.PORT || 3000;
app.listen(PORT, () => {
  console.log(`API server running on port ${PORT}`);
  
  // Log initial memory usage
  const memoryUsage = process.memoryUsage();
  console.log('Initial memory usage:');
  console.log(`- RSS: ${Math.round(memoryUsage.rss / 1024 / 1024)} MB`);
  console.log(`- Heap Total: ${Math.round(memoryUsage.heapTotal / 1024 / 1024)} MB`);
  console.log(`- Heap Used: ${Math.round(memoryUsage.heapUsed / 1024 / 1024)} MB`);
});

// Periodically log status
setInterval(() => {
  const memoryUsage = process.memoryUsage();
  console.log('STATUS UPDATE:');
  console.log(`- Uptime: ${Math.round(process.uptime() / 60)} minutes`);
  console.log(`- Requests served: ${requestCounter}`);
  console.log(`- Memory RSS: ${Math.round(memoryUsage.rss / 1024 / 1024)} MB`);
}, 15 * 60 * 1000); // Every 15 minutes

Visualization and Dashboards

Once you have monitoring data, you need to visualize it. Popular options include:

1. Grafana: Connect to data sources like Prometheus or InfluxDB to create dashboards
2. Kibana: Used with ELK stack (Elasticsearch, Logstash, Kibana) for log analysis
3. Cloud provider dashboards: AWS CloudWatch, Google Cloud Monitoring, Azure Monitor

Alerting

Set up alerts to be notified when:

• API response times exceed thresholds
• Error rates increase above normal levels
• Memory usage is too high
• Disk space is running low

Alert channels can include:

• Email
• SMS
• Slack/Discord messages
• PagerDuty

Summary

Monitoring your Express APIs is essential for maintaining reliable services. We've covered:

1. Basic logging with Morgan and custom middleware
2. Error monitoring with error-handling middleware and services like Sentry
3. Performance monitoring of response times and memory usage
4. Health check endpoints for status verification
5. APM tools for comprehensive monitoring
6. Real-world examples combining multiple approaches
7. Visualization and alerting to make monitoring data actionable

By implementing these monitoring strategies, you'll be able to maintain a robust, high-performing Express API that meets the needs of your users.

Additional Resources and Exercises

Resources

1. Express.js Documentation
2. Morgan Logger Documentation
3. Sentry Documentation for Node.js
4. Prometheus Node.js Client
5. Grafana Getting Started Guide

Exercises

1. Basic Monitoring: Set up Morgan in your Express API with a custom format that includes response time.

2. Health Check API: Create a detailed health check endpoint that monitors:

   • Database connection status
   • External API dependencies
   • Current memory usage
   • System uptime

3. Dashboard Creation: Use Grafana or a similar tool to create a dashboard for your API metrics.

4. Alert System: Implement an alert system that notifies you (via email or other channel) when your API's error rate exceeds a certain threshold.

5. Performance Testing: Use a tool like Apache Bench or JMeter to stress test your API while monitoring its performance metrics.