Alexander Gromnitsky's Blog

Home streaming & inetd-style servers

Latest update:

The easiest way to stream a movie is to serve it using a static HTTP server that supports range requests. For this, even Ruby's Webrick will do the job. Type this in a directory with your The Sopranos collection:

$ ruby -run -ehttpd . -b 127.0.0.1 -p 8000

& point mpv or vlc to a particular episode:

$ mpv http://127.0.0.1/s01e01.mp4

This should work as if you're playing a local file. To play a movie with a web browser, make sure the web server returns correct Content-Type headers. A container format counts too: e.g., Chrome doesn't like mkv.

Can we do something similar without the HTTP server? Depending on the container format, it's possible to feed mpv with a raw TCP stream. We'll lose seeking, but if we were creating, say, a YouTube Shorts or Facebook Reels competitor, this won't matter, for consumers of these kind of clips don't care much about that.

The most primitive solution requires only 2 utils:

  1. ncat, that can listen on a socket & fork an external program when someone connects to the former:

    $ cat mickeymousetube
#!/bin/sh

export movie="${1:?Usage: ${0##*/} file.mkv [port]}"
port=${2:-61001}
type pv ncat || exit 1

__dirname=$(dirname "$(readlink -f "$0")")
ncat -vlk -e "$__dirname/pv.sh" 127.0.0.1 $port

  2. pv, the famous pipe monitor that can limit a transfer rate; without the limiter, mpv eats all available bandwidth:

    $ cat pv.sh
#!/bin/sh
pv -L2M "$movie"

    The -L2M option means max 2MB/s.

Then run mickeymousetube in one terminal & mpv tcp://127.0.0.1:61001 in another to play a clip.

tcplol

How hard may it be to replace ncat with our custom script? What ncat does with -e option is akin to what inetd did back in the day:

Steps performed by inetd

(The illustration is from Stevens' UNIX Network Programming.)

Instead of creating a server that manages sockets, one writes a program that simply reads from stdin and outputs to stdout. All the intricacies of properly handling multiple clients are managed by the super-duper-server.

There is no (x)inetd package in modern distros like Fedora, as systemd has superseded it with socket activation.

Suppose we have a script that asks a user for his nickname & greets him in return:

$ cat hello.sh
#!/bin/sh
uname 1>&2
while [ -z "$name" ]; do
    printf "Nickname? "
    read -r name || exit 1
done
echo "Hello, $name!"

To expose it to a network, we can either write 2 systemd unit files & place them in ~/.config/systemd/user/, or opt for a tiny 37 LOC Ruby script instead:

require 'socket'

usage = 'Usage: tcplol [-2v] [-h 127.0.0.1] -p 1234 program [args...]'
…

server = TCPServer.new opt['h'], opt['p']
loop do
  client = server.accept
  cid = client.remote_address.ip_unpack.join ':'

  warn "Client #{cid}"
  pid = fork do
    $stdin.reopen client
    $stdout.reopen client
    $stderr.reopen client if opt['2']
    client.close
    exec(*ARGV)
  end
  client.close
  Thread.new(cid, pid) do
    Process.wait pid
    warn "Client #{cid}: disconnect"
  end
end

This is a classic fork server that uses a thread for each fork to watch out for zombies. The linked tcplol script performs an additional clean-up in case the server gets hit with a SIGINT, for example.

ncat, on the other hand, operates quite differently:

  1. it creates 2 pipes;
  2. after each new connection, it forks itself;
  3. it connects the 2 pipes to the child's stdin/stdout;
  4. (in the parent process) it listens on a connected socket using select(2) syscall and transfers data to/from the child using the 2 pipes; we'll talk about select(2) and the concept of multiplexing later on.

Anyhow, if we run our much simpler "super-server":

$ ./tcplol -v -p 1234 ./hello.sh

& connect to it with 2 socat clients, the process tree under Linux would look like:

$ pstree `pgrep -f ./tcplol` -ap
ruby,259576 ./tcplol -v -p 8000 ./hello.sh
  ├─hello.sh,259580 ./hello.sh
  ├─hello.sh,259587 ./hello.sh
  ├─{ruby},259583
  ├─{ruby},259588
  └─{ruby},259589

The dialog:

$ socat - TCP4:127.0.0.1:8000
Nickname? Dude
Hello, Dude!

(Why socat? We can use ncat as well, but the latter doesn't close its end of a connection; it hangs in CLOSE_WAIT until one presses Ctrl-D.)

To play a movie, run

$ ./tcplol -v -p 8000 ./pv.sh file.mkv

using a modified version of pv.sh script:

#!/bin/sh
echo Streaming "$1" 1>&2
pv -L2M "${1?Usage: pv.sh file}"

Then connect to the server with

$ mpv tcp://127.0.0.1:8000

Mickey mouse SOCKS4 server

inetd-style services can perform various actions, not just humbly write to stdout. Nothing prevents such a service from opening a connection to a different machine and relaying bytes from it to the tcplol clients.

To illustrate the perils of the low-level socket interface, let's write a crude, allow-everyone socks4 service and test it with curl. The objective is to retrieve security.txt file from Google using a TLS connection like so:

$ curl -L https://google.com/.well-known/security.txt --proxy socks4://127.0.0.1:8000

As a socks4 client, curl sends a request to 127.0.0.1:8000 with an IP+port to which it wants our service to establish a connection (meaning we don't have to resolve google.com domain name ourselves). We decode this and promptly send an acknowledgment reply. This is the 1st part of socks4.rb which we are going run under tcplol:

$stdout.sync = true

req = $stdin.read 8 + 1
ver, command, port, ip = req.unpack 'CCnN' # 8 bytes
abort 'Invalid CONNECT' unless ver == 4 && command == 1

ip = ip.to_s(16).scan(/.{2}/).map(&:hex) # [a,b,c,d]
res = [0, 90].pack('C*') +               # request granted
      [port].pack('n') + ip.pack('C*')
$stdout.write res

What should we do next? As soon as curl gets the value of 'res' variable, it eagerly starts sending a TLS ClientHello message to 127.0.0.1:8000. At this point, we don't need to analyse exactly what it sends--our primary concern is relaying traffic to and fro as quickly as possible without losing bytes.

To temporarily test that we have correctly negotiated SOCKS4, we can conclude the script with the ncat call:

exec "ncat", "-v", ip.join('.'), port.to_s

It should work. However, we can also rewrite that line in pure Ruby using the Kernel.select method. What we need here is to monitor 2 file descriptors in different modes to react to changes in their state:

  1. in reading mode: stdin and a TCP socket to google.com;
  2. in writing mode: the TCP socket to google.com.

(We assume that stdout is always available.) This kind of programming--being notified when an IO connection is ready (for reading or writing, for example) on a set of file descriptors--is called IO multiplexing. Most web programmers never encounter it because the socket interface is many levels below the stack they are working in, but it may be interesting sometimes to see how the sausage is made.

Replace exec "ncat" line with:

require 'socket'

s = TCPSocket.new ip.join('.'), port
wbuf = []
BUFSIZ = 1024 * 1024
loop do
  sockets = select [$stdin, s], [s], [], 5

  sockets[0].each do |socket|   # readable
    if socket == $stdin
      input = $stdin.readpartial BUFSIZ
      wbuf << input
    else
      input = socket.readpartial BUFSIZ
      $stdout.write input
    end
  end

  sockets[1].each do |socket|   # writable
    wbuf.each { |v| socket.write v }
    wbuf = []
  end
end

We're establishing a connection to google.com and then initiating the monitoring of two file descriptors in an endless loop. The select method blocks until one or both of these file descriptors become available for reading or writing. The last argument to it is a timeout in seconds.

When select unblocks, sockets[0] contains an array of file descriptors available for reading. If it's stdin, we read the data the OS kernel thinks is obtainable & save such a chunk to wbuf array. If it is a socket to google.com, we read some bytes from it & immediately write them to stdout for curl to consume.

sockets[1] contains an array of file descriptors available for writing. We only have 1 google.com socket here, to which we write the contents of wbuf array.

The script terminates when $stdin.readpartial returns an EOFError. This indicates to curl that the other party has closed its connection.

If you run socks4.rb under tcplol:

./tcplol -v -p 8000 ./socks4.rb

and observe errors tcplol prints from socks4.rb, you'll see that curl makes 2 requests to google.com, for the first one yields 301.

$ curl -sLI https://google.com/.well-known/security.txt --proxy socks4://127.0.0.1:8000 | grep -E '^HTTP|content-length'
HTTP/2 301
content-length: 244
HTTP/2 200
content-length: 246

Tags: ойті
Authors: ag