Express Temporary Files

Introduction

When building web applications with Express.js, you'll frequently encounter scenarios where you need to work with files temporarily before processing them or storing them permanently. Temporary files are essential when handling user uploads, processing data, or performing operations that require intermediate file storage before reaching a final destination.

In this guide, we'll explore how Express.js handles temporary files, especially during file uploads, and how you can efficiently manage these files to prevent storage issues and optimize your application's performance.

Understanding Temporary Files in Express

Temporary files in Express.js are typically created when:

1. Users upload files through forms
2. Your application needs to buffer data before processing
3. You need to manipulate files before storing them permanently

These temporary files are stored in the system's default temporary directory (like /tmp on Unix systems or C:\Windows\Temp on Windows) unless configured otherwise.

File Upload Middleware

Express itself doesn't handle file uploads natively. Instead, we use middleware like multer, express-fileupload, or formidable to handle multipart/form-data, which creates temporary files during the upload process.

Using Multer for File Uploads

Multer is one of the most popular middleware for handling multipart/form-data in Express applications.

First, let's install Multer:

bash

npm install multer

Here's a basic example of how to use Multer to handle file uploads:

javascript

const express = require('express');
const multer = require('multer');
const path = require('path');
const fs = require('fs');

const app = express();
const port = 3000;

// Configure multer's default settings for temporary storage
const upload = multer({ dest: 'temp/uploads/' });

app.get('/', (req, res) => {
  res.send(`
    <form action="/upload" method="post" enctype="multipart/form-data">
      <input type="file" name="file" />
      <input type="submit" value="Upload" />
    </form>
  `);
});

// Single file upload
app.post('/upload', upload.single('file'), (req, res) => {
  // req.file contains information about the uploaded file
  if (!req.file) {
    return res.status(400).send('No file uploaded');
  }

  console.log('Temporary file details:', req.file);
  
  // The file is now stored in the 'temp/uploads/' directory
  // with a generated filename (no extension)
  res.send(`File uploaded successfully! Temporary path: ${req.file.path}`);
  
  // From here, you could process the file and then move or delete it
});

app.listen(port, () => {
  console.log(`Server running at http://localhost:${port}`);
});

When a user uploads a file through this form, Multer:

1. Creates a temp/uploads/ directory if it doesn't exist
2. Stores the uploaded file there with a randomly generated name (without extension)
3. Makes the file information available via req.file

The output from console.log(req.file) would look something like this:

{
  fieldname: 'file',
  originalname: 'example.txt',
  encoding: '7bit',
  mimetype: 'text/plain',
  destination: 'temp/uploads/',
  filename: '1a2b3c4d5e6f7g8h9i0j',
  path: 'temp/uploads/1a2b3c4d5e6f7g8h9i0j',
  size: 1234
}

Custom Temporary File Storage

You can customize how and where Multer stores temporary files using the storage option:

javascript

const storage = multer.diskStorage({
  destination: function (req, file, cb) {
    // Specify the directory where temporary files will be stored
    cb(null, 'temp/uploads/');
  },
  filename: function (req, file, cb) {
    // Create a custom filename that includes the original extension
    const uniqueSuffix = Date.now() + '-' + Math.round(Math.random() * 1E9);
    cb(null, file.fieldname + '-' + uniqueSuffix + path.extname(file.originalname));
  }
});

const upload = multer({ storage: storage });

Memory Storage for Small Files

For small files, you can avoid writing to disk entirely by using memory storage:

javascript

const memoryStorage = multer.memoryStorage();
const upload = multer({ storage: memoryStorage });

app.post('/upload', upload.single('file'), (req, res) => {
  // req.file.buffer contains the file data in memory
  console.log('File in memory:', req.file.buffer);
  
  // Process the file directly from memory
  res.send('File processed in memory!');
});

This approach is efficient for smaller files but can consume significant memory for large uploads.

Processing Temporary Files

Once you've received a temporary file, you'll typically want to:

1. Validate it (check file type, size, etc.)
2. Process it (resize images, parse data, etc.)
3. Move it to a permanent location or delete it

Here's an example that validates a file is an image before processing it:

javascript

const fs = require('fs');
const path = require('path');
const sharp = require('sharp'); // For image processing

app.post('/upload-image', upload.single('image'), async (req, res) => {
  try {
    if (!req.file) {
      return res.status(400).send('No file uploaded');
    }
    
    // Validate file type
    if (!req.file.mimetype.startsWith('image/')) {
      // Remove invalid file
      fs.unlinkSync(req.file.path);
      return res.status(400).send('Only images are allowed');
    }
    
    // Process the image (resize it)
    const outputPath = path.join('public/images', `processed-${path.basename(req.file.path)}.jpg`);
    
    await sharp(req.file.path)
      .resize(800, 600)
      .jpeg({ quality: 80 })
      .toFile(outputPath);
      
    // Remove the temporary file
    fs.unlinkSync(req.file.path);
    
    res.send(`Image processed and saved to ${outputPath}`);
  } catch (error) {
    console.error('Error processing image:', error);
    res.status(500).send('Error processing the image');
  }
});

Cleaning Up Temporary Files

Temporary files should be removed when they're no longer needed to avoid filling up disk space. There are several strategies for cleanup:

1. Manual Cleanup

Clean up files after processing:

javascript

app.post('/upload', upload.single('file'), (req, res) => {
  // Process the file...
  
  // Then delete it
  fs.unlink(req.file.path, (err) => {
    if (err) console.error('Error deleting temp file:', err);
    console.log('Temporary file deleted');
  });
  
  res.send('File processed');
});

2. Scheduled Cleanup

For applications that handle many uploads, you might want to schedule a periodic cleanup:

javascript

const { scheduleJob } = require('node-schedule');

// Run cleanup job every day at midnight
scheduleJob('0 0 * * *', () => {
  const tempDir = 'temp/uploads/';
  
  fs.readdir(tempDir, (err, files) => {
    if (err) {
      console.error('Error reading temp directory:', err);
      return;
    }
    
    const now = Date.now();
    const oneHourAgo = now - (60 * 60 * 1000); // 1 hour in milliseconds
    
    files.forEach(file => {
      const filePath = path.join(tempDir, file);
      
      fs.stat(filePath, (err, stats) => {
        if (err) {
          console.error(`Error getting stats for ${filePath}:`, err);
          return;
        }
        
        // Delete files older than 1 hour
        if (stats.ctimeMs < oneHourAgo) {
          fs.unlink(filePath, err => {
            if (err) {
              console.error(`Error deleting ${filePath}:`, err);
            } else {
              console.log(`Deleted old temp file: ${filePath}`);
            }
          });
        }
      });
    });
  });
});

Handling Temporary Files During Server Shutdown

To ensure temporary files are cleaned up when your server shuts down, you can add a shutdown handler:

javascript

function cleanupTempFiles() {
  console.log('Cleaning up temporary files...');
  
  // Your cleanup logic here
  const tempDir = 'temp/uploads/';
  
  try {
    const files = fs.readdirSync(tempDir);
    files.forEach(file => {
      fs.unlinkSync(path.join(tempDir, file));
    });
    console.log('Temporary files cleaned up');
  } catch (err) {
    console.error('Error cleaning up temp files:', err);
  }
}

// Handle different termination signals
process.on('SIGINT', () => {
  cleanupTempFiles();
  process.exit(0);
});

process.on('SIGTERM', () => {
  cleanupTempFiles();
  process.exit(0);
});

Real-world Application: File Upload Service

Here's a more comprehensive example of a file upload service that:

1. Accepts multiple file uploads
2. Validates file types
3. Processes images
4. Moves valid files to a permanent storage location
5. Cleans up temporary files

javascript

const express = require('express');
const multer = require('multer');
const path = require('path');
const fs = require('fs');
const sharp = require('sharp');
const { v4: uuidv4 } = require('uuid');

const app = express();
const port = 3000;

// Setup directories
const tempDir = 'temp/uploads/';
const permanentDir = 'public/uploads/';

// Ensure directories exist
[tempDir, permanentDir].forEach(dir => {
  if (!fs.existsSync(dir)) {
    fs.mkdirSync(dir, { recursive: true });
  }
});

// Configure multer
const storage = multer.diskStorage({
  destination: function (req, file, cb) {
    cb(null, tempDir);
  },
  filename: function (req, file, cb) {
    // Use UUID for unique filenames
    cb(null, `${uuidv4()}${path.extname(file.originalname)}`);
  }
});

// File filter function
const fileFilter = (req, file, cb) => {
  // Accept only specific file types
  const allowedTypes = ['image/jpeg', 'image/png', 'application/pdf', 'text/plain'];
  
  if (allowedTypes.includes(file.mimetype)) {
    cb(null, true);
  } else {
    cb(new Error('Unsupported file type'), false);
  }
};

const upload = multer({ 
  storage: storage,
  limits: { fileSize: 10 * 1024 * 1024 }, // 10MB max file size
  fileFilter: fileFilter
});

// Serve static files
app.use('/uploads', express.static(permanentDir));

// Upload form
app.get('/', (req, res) => {
  res.send(`
    <h1>File Upload System</h1>
    <form action="/upload" method="post" enctype="multipart/form-data">
      <input type="file" name="files" multiple />
      <input type="submit" value="Upload Files" />
    </form>
  `);
});

// Handle multiple file uploads
app.post('/upload', upload.array('files', 5), async (req, res) => {
  try {
    if (!req.files || req.files.length === 0) {
      return res.status(400).send('No files uploaded');
    }
    
    const results = [];
    
    // Process each file
    for (const file of req.files) {
      const tempPath = file.path;
      const fileName = path.basename(tempPath);
      const finalPath = path.join(permanentDir, fileName);
      
      // Process images differently from other file types
      if (file.mimetype.startsWith('image/')) {
        // Resize and optimize images
        await sharp(tempPath)
          .resize(1200, 1200, { fit: 'inside', withoutEnlargement: true })
          .jpeg({ quality: 80 })
          .toFile(finalPath);
          
        // Delete the temporary file
        fs.unlinkSync(tempPath);
      } else {
        // For non-image files, just move them
        fs.renameSync(tempPath, finalPath);
      }
      
      results.push({
        originalName: file.originalname,
        storedAs: fileName,
        size: file.size,
        url: `/uploads/${fileName}`
      });
    }
    
    res.json({
      message: 'Files processed successfully',
      files: results
    });
    
  } catch (error) {
    console.error('Error processing uploads:', error);
    
    // Attempt to clean up any temporary files
    if (req.files) {
      req.files.forEach(file => {
        try {
          if (fs.existsSync(file.path)) {
            fs.unlinkSync(file.path);
          }
        } catch (err) {
          console.error(`Failed to clean up ${file.path}:`, err);
        }
      });
    }
    
    res.status(500).send('Error processing uploads');
  }
});

// Start the server
app.listen(port, () => {
  console.log(`Server running at http://localhost:${port}`);
});

Best Practices for Handling Temporary Files

1. Set file size limits to prevent denial-of-service attacks from large uploads
2. Validate file types before processing
3. Use a dedicated directory for temporary files outside your application's public folders
4. Clean up immediately after processing when possible
5. Implement periodic cleanup for files that might be abandoned
6. Add shutdown handlers to clean up when the server stops
7. Monitor disk usage in production environments
8. Consider using memory storage for very small files
9. Log operations related to file creation and deletion for debugging

Summary

Handling temporary files in Express.js is essential for file uploads and data processing scenarios. By using middleware like Multer, you can efficiently manage file uploads, process them, and clean up afterward to maintain a healthy system.

Key points to remember:

• Express doesn't handle file uploads directly; use middleware like Multer
• Temporary files need proper cleanup strategies to avoid disk space issues
• Choose the right storage strategy based on your file sizes and processing needs
• Always validate uploaded files before processing them
• Implement proper error handling for file operations

Additional Resources

• Multer Documentation
• Express.js Documentation
• Node.js File System API
• Sharp Image Processing Library

Exercises

1. Create a file upload system that accepts images, processes them to create thumbnails, and stores both the original and thumbnail in separate directories.

2. Implement a temporary file cleanup system that removes files older than 24 hours from a specific directory.

3. Build an API endpoint that accepts a CSV file upload, processes the data, stores it in a database, and then removes the temporary file.

4. Create a file validation middleware that checks file types, sizes, and potentially harmful content before allowing further processing.

5. Implement a progress tracking system for large file uploads that provides real-time feedback to users during the upload process.