Alexander Gromnitsky's Blog

Writing 4 static servers for Node.js

Latest update: 2023-08-23 10:18:51

Here we write several simple http/1.1 static servers, usable when one need to integrate one of them into a small API server or an SPA.

It's common to include a "last resort" call within http.createServer(), similar to the following:

let server = http.createServer( (req, res) => {
  if (req.method === "POST") {
    ...
  } else if (...) {
    ...
  } else {
    serve_static(req, res)
  }
})

This serve_static() is missing in Node's core. There are, of course, multiple implementations on npm, but they all either have 2^16 dependencies or/and are written as a clunky replacement for

$ ruby -run -e httpd

Furthermore, every API server has different requirements for static content. Some need nothing more than serving an .html page & a couple of .js files. Others expect slightly heavier loads & rely on if-modified-since client requests & compression. Another kind cannot live without byte range calls.

The only npm dependency that could be useful for them is a mime types database, everything else has existed in Node's core for many years.

Before wasting time on this, I tried to find an already written package on npm. Ideally, it would rely solely on mime and nothing more. mime itself has 7411 dependants. Do you know how to search within the dependants for a package? I don't. There is a page that lists a portion of them with an offset of 36 for the next page. So I wrote a primitive scraper with the intention of fetching the full list & searching locally for keywords like "static", but after offset=360 npm stopped delivering the rest of the list, which was quite disappointing.

Anyhow, I found 2 candidates worth mentioning:

• isaacs/st (6 dependencies)
• bahamas10/node-static-route (2 dependencies)

The 2nd package even supports byte range requests. If you don't suffer from the NIH syndrome, hack on one of them.

4 servers--4 modules

Each static server in this blog post is a standalone es6 module intended to be used in the most basic form as:

import http from 'http'
import serve_static from './lib/serverN.js'

let server = http.createServer( (req, res) => {
    serve_static(res, req, {
        headers: { 'server': 'omglol/0.0.1' },
        mime: { '.txt': 'text/plain' },
    })
})

server.listen(process.env.PORT || 3000)

Each subsequent server has more capabilities (& LOC, unfortunately) then the previous one.

$ scc lib/* --by-file | awk '/^l/ {print $5, $1}'
224 lib/server4.js
96 lib/server3.js
51 lib/server2.js
33 lib/server1.js

(You can peruse them on github.)

We start with the smallest server1.js.

Server 1

It has 0 dependencies, 2 functions:

import fs from 'fs'
import path from 'path'

export default function(writable, name, opt = {}) {
    if (/^\/+$/.test(name)) name = "index.html"
    let file = path.join(opt.public_root || process.cwd(), path.normalize(name))

    file = decodeURI(file)
    fs.stat(file, (err, stats) => {
        if (!err && !stats.isFile()) {
            err = new Error("Argument invalide")
            err.code = 'EINVAL'
        }
        if (err) return error(writable, err, opt.verbose)

        let readable = fs.createReadStream(file)
        readable.once('data', () => {
            writable.setHeader('Content-Length', stats.size)
            writable.setHeader('Content-Type', Object.assign({
                '.html': 'text/html',
                '.js': 'application/javascript'
            }, opt.mime)[path.extname(file)] || 'application/octet-stream')
            Object.entries(opt.headers || {}).map(h => writable.setHeader(...h))
        })
        readable.on('error', err => error(writable, err, opt.verbose))
        readable.pipe(writable)
    })
}

function error(writable, err, verbose) {
    if (!writable.headersSent) {
        let codes = { 'ENOENT': 404, 'EACCES': 403, 'EINVAL': 400 }
        writable.statusCode = codes[err?.code] || 500
        if (verbose) try { writable.statusMessage = err } catch {/**/}
    }
    writable.end()
}

This is perhaps the simplest thing you can work with. opt hash provides options for

• the public root directory (defaults to process.cwd() when unset);
• add-ons to the file extension → content type mappings (the defaults cover only .html & .js);
• custom headers;
• a verbose development mode that puts error messages in the HTTP status.

It servers only files, for drawing an index for a directory doesn't make sense for a typical API server.

Server 2

This variant uses mime npm package. Due to its DB structure we change opt.mime format to

{ 'text/markdown': ['txt'] },

This allows us to add/override default mime entries.

Other niceties:

• HEAD requests;
• generate an ETag & set Last-Modified header;
• respond with 304 if a client provided suitable if-modified-since or if-none-match headers.

import fs from 'fs'
import path from 'path'
import mime from 'mime'

export default function(req, writable, name, opt = {}) {
  if (/^\/+$/.test(name)) name = "index.html"
  let file = path.join(opt.public_root || process.cwd(), path.normalize(name))

  file = decodeURI(file)
  fs.stat(file, (err, stats) => {
    if (!err && !stats.isFile()) {
      err = new Error("Argument invalide")
      err.code = 'EINVAL'
    }
    if (err) return error(writable, err, opt.verbose)

    let ims = new Date(req.headers['if-modified-since'])
    if (ims >= new Date(stats.mtime.toUTCString())
      || req.headers['if-none-match'] === etag(stats)) {
      writable.statusCode = 304
      return writable.end()
    }

    let set_hdr = () => set_headers(writable, opt, stats,
                    content_type(file, opt.mime))
    if (req.method === 'HEAD') {
      set_hdr()
      return writable.end()
    }

    let readable = fs.createReadStream(file)
    readable.once('data', () => set_hdr())
    readable.on('error', err => error(writable, err, opt.verbose))
    readable.pipe(writable)
  })
}

function set_headers(writable, opt, stats, content_type) {
  writable.setHeader('Content-Length', stats.size)
  writable.setHeader('Content-Type', content_type)
  writable.setHeader('ETag', etag(stats))
  writable.setHeader('Last-Modified', stats.mtime.toUTCString())
  Object.entries(opt.headers || {}).map( v => writable.setHeader(...v))
}

function etag(s) { return '"'+[s.dev, s.ino, s.mtime.getTime()].join("-")+'"' }

function content_type(file, custom_types) {
  mime.define(custom_types, true)
  return mime.getType(file) || 'application/octet-stream'
}

error() is the same as in server1.js.

Server 3

What else is missing? All browsers typically send Accept-Encoding header with each request, e.g. Chrome 115:

Accept-Encoding: gzip, deflate, br

Node has build-in support for all these encodings--you can compress any readable stream with them. An amiable web server should be able to at least deflate various text/* formats, JS & XML (there is no point in compressing images, musique or videos).

Accept-Encoding may contain q-values like so:

deflate;q=0.5, gzip, *;q=0

This means "gzip is preferred (implicit q=1), but deflate is fine, anything else I don't want to see". With such a request, if a server supports only Brotli, it must respond with 406 Not Accepqtable.

Therefore, dealing with Accept-Encoding involves a 3-step process:

1. parsing the header value;
2. deciding if we can respond;
3. if we do, injecting a zlib Transform stream between the server's readable & writeable streams.

// return [{ name: 'foo', q: 1 }, { name: 'bar', q: 0.5 }, ...]
export function accept_encoding_parse(str) {
  return (str || '').split(',')
    .map( v => v.trim()).filter(Boolean)
    .map( v => {
      let p = v.split(';').map( v => v.trim()).filter(Boolean)
      if (!p[0]) return // invalid algo
      let r = { name: p[0], q: 1 }
      let q = p.find( v => v.slice(0,2) === 'q=')
      if (q) {
        let weight = Number(q.split('=')[1])
        if (weight >= 0) r.q = weight
      }
      return r
    }).filter(Boolean).sort( (a, b) => b.q - a.q)
}

export function accept_encoding_negotiate(algo, enc) {
  let v = enc.find( v => v.name === algo)
  let star = enc.find( v => v.name === '*' || v.name === 'identity')
  if (!v && star?.q === 0) return 'no deal'
  if (v && v.q === 0) return 'no deal'
  if (!v) return 'pass-through'
  return 'compress'
}

function content_encoding(req, writable, content_type) {
  let enc = req.headers['accept-encoding']
  if (!enc || !/(text\/|javascript|json|\+xml)/.test(content_type))
    return [writable] // don't compress binaries

  let r = accept_encoding_negotiate('deflate', accept_encoding_parse(enc))
  if (r === 'no deal') return []
  if (r === 'pass-through') return [writable]

  writable.setHeader('Content-Encoding', 'deflate')
  return [zlib.createDeflate(), writable]
}

content_encoding() here returns a list of streams that we insert into a pipeline after req (which is a readable stream) in our callback to http.createServer().

A negotiation leads to 3 outcomes:

1. a server hasn't heard of a specific encoding, thus it ignores the client's petition & returns an unchanged representation; this is similar to the behaviour of a server that doesn't support any compression, like our server1.js & server2.js;

2. "no deal": the client's list of encodings includes a clause that forbids the server from proceeding when no suitable encoding is found;

3. both parties agree on an encoding. 🍾🥂

The last item implies turning off Content-Length header--we're compressing chunks in a stream & don't know the exact length of all compressed chunks beforehand (unless, of course, you compress a file, put it in a tmp directory & read it from there in the usual fashion; could be beneficial for small files).

Our main exported function is similar to server2.js, the chief difference is the replacement of readable.pipe() with pipeline() to which the list of streams are passed:

import fs from 'fs'
import path from 'path'
import zlib from 'zlib'
import {pipeline} from 'stream'
import mime from 'mime'

export default function(req, writable, name, opt = {}) {
  if (/^\/+$/.test(name)) name = "index.html"
  let file = path.join(opt.public_root || process.cwd(), path.normalize(name))

  file = decodeURI(file)
  fs.stat(file, (err, stats) => {
    if (!err && !stats.isFile()) {
      err = new Error("Argument invalide")
      err.code = 'EINVAL'
    }
    if (err) return error(writable, err, opt.verbose)

    let cnt_type = content_type(file, opt.mime)
    let dest = content_encoding(req, writable, cnt_type)
    if (!dest.length) {
      let err = new Error('Pas acceptable')
      err.code = 'EBADE'
      return error(writable, err, opt.verbose)
    }

    let ims = new Date(req.headers['if-modified-since'])
    if (ims >= new Date(stats.mtime.toUTCString())
      || req.headers['if-none-match'] === etag(writable, stats)) {
      writable.statusCode = 304
      return writable.end()
    }

    if (req.method === 'HEAD') {
      set_headers(writable, opt, stats, cnt_type)
      return writable.end()
    }

    let readable = fs.createReadStream(file)
    readable.once('data', () => set_headers(writable, opt, stats, cnt_type))
    readable.on('error', err => error(writable, err, opt.verbose))
    pipeline(readable, ...dest, () => {/* all streams are closed */})
  })
}

function set_headers(writable, opt, stats, content_type) {
  if (!writable.getHeader('content-encoding'))
    writable.setHeader('Content-Length', stats.size)
  writable.setHeader('Content-Type', content_type)
  writable.setHeader('ETag', etag(writable, stats))
  writable.setHeader('Last-Modified', stats.mtime.toUTCString())
  Object.entries(opt.headers || {}).map( v => writable.setHeader(...v))
}

A little detail about ETag that some implementations forget about: its value for a compressed & initial representations should be different. A common choice is to append the name of a compression method to the value (Apache does exactly that).

function etag(writable, s) {
  let suffix = writable.getHeader('content-encoding')
  return '"' + [s.dev, s.ino, s.mtime.getTime(), suffix]
    .filter(Boolean).join`-` + '"'
}

function error(writable, err, verbose) {
  if (!writable.headersSent) {
    let codes = { 'ENOENT':404, 'EACCES':403, 'EINVAL':400, 'EBADE':406 }
    writable.statusCode = codes[err?.code] || 500
    if (verbose) try { writable.statusMessage = err } catch {/**/}
  }
  writable.end()
}

content_type() is the same as in server2.js.

Server 4

If your serve audio/video, your server should support byte range requests. When a client sees the following header from the server:

Accept-Ranges: bytes

It can ask for a portion of a remote file via adding

Range: bytes=-500

This instructs the server to return HTTP/1.1 206 Partial Content with only last 500 bytes of the file.

Range: bytes=0-0,-1

returns the first & the last bytes. These types of requests are called multipart/byteranges, all major web servers respond to them, but I can't find a living web browser that uses this feature, which is a pity.

If you search for byte range requests implementations on npm, almost all of them imply the usage of a single range:

Range: bytes=100-200
Range: bytes=5-
Range: bytes=-5

This is sufficient for playing .mp3 in a web browser, for example, but if we're parsing Range header, it would be a shame to pass on multipart/byteranges. The code for it is too extensive for a blog post (see the github repo), but here are a few notes:

• we turn off compression for byte ranges requests--neither we nor a client has any idea what byte offset in a gzip stream would correspond to a position in an unpacked file;

• we don't need to manually calculate how many bytes have passed through a readable stream--fs.createReadStream() allows us to specify a range;

• a client can send If-Range header (alongside Range), with a value of an ETag, for example; the server should respond with a proper 206 only if a file hasn't changed, otherwise return the full content of the file;