If such a container has apt-get, but you lack permission to run it, there is a reliable way to force apt-get to download a .deb file with all its dependencies under a regular user, but we won't discuss that here.

I got curious about how hard it would be to write a primitive HTTP get-only client in Bash, as Bash is typically compiled with "network" redirection support:

$ exec 3<> /dev/tcp/www.gnu.org/80
$ printf "%s\r\n" 'HEAD /robots.txt HTTP/1.1' >&3
$ printf "%s\r\n\r\n" 'Host: www.gnu.org' >&3
$ cat <&3
HTTP/1.1 200 OK
Date: Sun, 11 Feb 2024 07:02:40 GMT
Server: Apache/2.4.29
Content-Type: text/plain
Content-Language: non-html
…

This could've been useful before the days of TLS everywhere, but it won't suffice now: to download a statically compiled curl binary from Github, we need TLS support and proper handling of 302 redirections. Certainly, it's possible to cheat: put the binary on our web server and serve it under plain HTTP, but that would be too easy.

What if we use ncat+openssl as a forward TLS proxy? ncat may serve as an initd-like super-server, invoking "openssl s_client" on each connection:

$ cat proxy.sh
#!/bin/sh
read -r host
openssl s_client -quiet -no_ign_eof -verify_return_error "$host"
$ ncat -vk -l 10.10.10.10 1234 -e proxy.sh

The 1st thing we need in the bash-http-get client is URL parsing. It wouldn't have been necessary if Github served files directly from "Releases" pages, but it does so through redirects. Therefore, when we grab Location header from a response, we need to disentangle its hostname from a pathname.

Ideally, it should work like URL() constructor in JavaScript:

$ node -pe 'new URL("https://q.example.com:8080/foo?q=1&w=2#lol")'
URL {
  href: 'https://q.example.com:8080/foo?q=1&w=2#lol',
  origin: 'https://q.example.com:8080',
  protocol: 'https:',
  username: '',
  password: '',
  host: 'q.example.com:8080',
  hostname: 'q.example.com',
  port: '8080',
  pathname: '/foo',
  search: '?q=1&w=2',
  searchParams: URLSearchParams { 'q' => '1', 'w' => '2' },
  hash: '#lol'
}

StackOverflow has various examples of how to achieve that using regular expressions, but none of them were able to parse the example above. I tried asking ChatGPT to repair the regex, but it only made it worse. Miraculously, Google's Gemini supposedly fixed the regex on the second try (I haven't tested it extensively).

$ cat lib.bash
declare -A URL

url_parse() {
    local pattern='^(([^:/?#]+):)?(//((([^:/?#]+)@)?([^:/?#]+)(:([0-9]+))?))?(/([^?#]*))?(\?([^#]*))?(#(.*))?'
    [[ "$1" =~ $pattern ]] && [ "${BASH_REMATCH[2]}" ] && [ "${BASH_REMATCH[4]}" ] || return 1
    URL=(
        [proto]=${BASH_REMATCH[2]}
        [host]=${BASH_REMATCH[4]}
        [hostname]=${BASH_REMATCH[7]}
        [port]=${BASH_REMATCH[9]}
        [pathname]=${BASH_REMATCH[10]:-/}
        [search]=${BASH_REMATCH[12]}
        [hash]=${BASH_REMATCH[14]}
    )
}

Next, we need to separate headers from a response body. This means looking for the 1st occurrence of \r\n\r\n. Sounds easy,

grep -aobx $'\r' file | head -1

until you decide to port the client to a BusyBox-based system like Alpine Linux. The latter has grep that doesn't support -ab options. There are some advices on employing od(1), but no examples. If we print a file using a 2-column format:

0000000 68
0000001 20
0000002 3a
…

where the left column is a decimal offset, we can convert the 1st 32KB of the response into a single line and search for the pattern using grep -o:

od -N $((32*1024)) -t x1 -Ad -w1 -v "$tmp" | tr '\n' ' ' | \
    grep -o '....... 0d ....... 0a ....... 0d ....... 0a' | \
    awk '{if (NR==1) print $7+0}'

Here's the full version of the client that supports only URLs with the https protocol. It saves the response in a temporary file and looks for the \r\n\r\n offset. If the HTTP status code was 200, it prints the body to stdout. If it was 302, it extracts the value of the Location header and recursively calls itself with a new URL.

#!/usr/bin/env bash

set -e -o pipefail
. "$(dirname "$(readlink -f "$0")")/lib.bash"

tmp=`mktemp fetch.XXXXXX`
trap 'rm -f $tmp' 0 1 2 15
eh() { echo "$*" 1>&2; exit 2; }

[ $# = 3 ] || eh Usage: fetch.bash proxy_host proxy_port url
proxy_host=$1
proxy_port=$2
url=$3

get() {
    url_parse "$1"; [ "${URL[proto]}" = https ] || return 1

    exec 3<> "/dev/tcp/$proxy_host/$proxy_port" || return 1
    echo "${URL[hostname]}:${URL[port]:-443}" >&3
    printf "GET %s HTTP/1.1\r\n" "${URL[pathname]}${URL[search]}${URL[hash]}" >&3
    printf '%s: %s\r\n' Host "${URL[hostname]}" Connection close >&3
    printf '\r\n' >&3
    cat <&3
}

get "$url" > "$tmp" || eh ':('
[ -s "$tmp" ] || eh 'Empty reply, TLS error?'

offset_calc() {
    if echo 1 | grep -aobx 1 >/dev/null 2>&1; then # gnu-like grep
        grep -aobx $'\r' "$tmp" | head -1 | tr -d '\r\n:' | \
            xargs -r expr 1 +
    else                                      # busybox?
        od -N $((32*1024)) -t x1 -Ad -w1 -v "$tmp" | tr '\n' ' ' | \
            grep -o '....... 0d ....... 0a ....... 0d ....... 0a' | \
            awk '{if (NR==1) print $7+0}'
    fi || echo -1
}
offset=`offset_calc`
headers() { head -c "$offset" "$tmp" | tr -d '\r'; }
hdr() { headers | grep -m1 -i "^$1:" | cut -d' ' -f2; }

status=`head -1 "$tmp" | cut -d' ' -f2`
case "$status" in
    200) [ "$offset" = -1 ] && offset=-2 # invalid responce, dump all
         tail -c+$((offset + 2)) "$tmp"
         [ "$offset" -gt 0 ] ;;
    302) headers 1>&2; echo 1>&2
         hdr location | xargs "$0" "$1" "$2" ;;
    *)   headers 1>&2; exit 1
esac

It should work even on Alpine Linux of FreeBSD:

$ ./fetch.bash 10.10.10.10 1234 https://github.com/stunnel/static-curl/releases/download/8.6.0/curl-linux-arm64-8.6.0.tar.xz > curl.tar.xz
HTTP/1.1 302 Found
Location: https://objects.githubusercontent.com/…
…
$ file curl.tar.xz
curl.tar.xz: XZ compressed data, checksum CRC64

E.g., an unpacked tarball of Emacs 29.2 source code consists of 6791 files with a total size of 276MB. If you were to distribute it as a .tar.something archive, which compression tool would be the optimal choice? We can easily write a small utility that answers this question.

$ ./comprtest ~/opt/src/emacs/emacs-29.2 | tee table
tar: Removing leading `/' from member names
szip             0.59   56.98        126593557
gzip             9.21   72.70         80335332
compress         3.57   57.45        125217137
bzip2           17.28   78.08         64509672
rzip            17.61   79.50         60336377
lzip           113.61   81.67         53935898
lzop             0.67   57.14        126121462
xz             111.03   81.89         53295220
brotli          13.10   78.14         64336399
zstd             1.13   73.77         77179446

comprtest is a 29 LOC long shell script. The 2nd column here indicates time in seconds, the 3rd column displays $100(1-\frac{\mathrm{compressed}}{\mathrm{orig}})$, representing space saving in % (higher % is better), & the 4th column shows the final result in bytes.

Then we can sort the table by the 3rd column & draw a bar chart:

$ sort -nk3 table | cpp -P plot.gp | gnuplot -persist

If you're wondering why all of a sudden the C preprocessor becomes part of it, read on.

comprtest expects either a file as an argument or a directory (in which case it creates a plain .tar of it first). Additional optional arguments specify which compressors to use:

$ ./comprtest /usr/libexec/gdb gzip brotli
gzip             0.60   61.17          6054706
brotli           1.17   65.84          5325408

The gist of the script involves looping over a list of compressors:

archivers='szip gzip compress bzip2 rzip lzip lzop xz brotli zstd'
…
for c in ${@:-$archivers}; do
    echo $c
    case $c in
        szip   ) args='< "$input" > $output' ;;
        rzip   ) args='-k -o $output "$input"' ;;
        brotli ) args='-6 -c "$input" > $output' ;;
        *      ) args='-c "$input" > $output'
    esac

    eval "time -p $c $args" 2>&1 | awk '/real/ {print $2}'
    osize=`wc -c < $output`

    echo $isize $osize | awk '{print 100*(1-$2/($1==0?$2:$1))}'
    echo $osize
    rm $output
done | xargs -n4 printf "%-8s  %11.2f  %6.2f  %15d\n"

• Not every archive tool has gzip-compatible CLI.
• We are using a default compression level for each tool with the exception of brotli, as its default level 11 is excruciatingly slow.
• szip is an interface to the Snappy algorithm. Your distro probably doesn't have it in its repos, hence run cargo install szip. Everything else should be available via dnf/apt.

Bar charts are generated by a gnuplot script:

$ cat plot.gp
$data <<E
#include "/dev/stdin"
E
set key tmargin
set xtics rotate by -30 left
set y2tics
set ylabel "Seconds"
set y2label "%"
set style data histograms
set style fill solid
plot $data using 2 axis x1y1 title "Time", \
     "" using 3:xticlabels(1) axis x1y2 title "Space saving"

Here is where the C preprocessor comes in handy: without an injected "datablock" it won't be possible to draw a graph with 2 ordinates when reading data from stdin.

In an attempt to demonstrate that xz is not always the best choice, I benchmarked a bunch of XML files (314MB):

$ ./comprtest ~/Downloads/emacs.stackexchange.com.tar
szip             0.59   63.70        119429565
gzip             7.18   77.59         73724710
compress         4.03   67.17        108015563
bzip2           21.37   83.36         54751478
rzip            17.42   85.93         46304199
lzip           119.70   85.06         49151518
lzop             0.67   63.63        119667058
xz             125.80   85.55         47559464
brotli          13.56   82.52         57509978
zstd             1.07   79.40         67766890

Creating an .cpio or .tar.xz won't cut it: file archivers such as 7-Zip are free & easy to install. Furthermore, sending an ext4 image, generated as follows:

$ truncate -s 10M file.img
$ mkfs.ext4 file.img
$ sudo mount -o loop file.img /somewhere
$ sudo cp something /somewhere
$ sudo umount /somewhere

doesn't help nowadays, for 7-Zip opens them too¹. Although disk cloning utils like FSArchiver can produce an image file from a directory, they are exclusive to Linux.

It boils down to this: which filesystems can be read across Linux/MacOS/FreeBSD that Windows file archivers don't recognise? This rules out fat/ntfs/udf, for they are too common, or f2fs/nilfs2, for they are Linux-only.

The only viable candidate I found is XFS. Btrfs was a contender, but I'm unsure how to mount it on Mac.

Below is a script to automate the creation of prank archives. It takes any zip/tar.gz (or anything else that bsdtar is able to parse) & outputs an image file in the format specified by the output file extension:

sudo ./mkimg file.zip file.xfs

It requires sudo, for mount -o loop can't be done under a regular user.

#!/bin/sh

set -e

input=$1
output=$2
type=${2##*.}
[ -r "$input" ] && [ "$output" ] && [ "`id -u`" = 0 ] || {
    echo Usage: sudo mkimg file.zip file.ext2 1>&2
    exit 1
}
mkfs=mkfs.$type
cmd() { for c; do command -v $c >/dev/null || { echo no $c; return 1; }; done; }
cmd bsdtar "$mkfs"

cleanup() {
    set +e
    umount "$mnt" 2>/dev/null
    rm -rf "$mnt" "$log"
    [ "$ok" ] || rm -f "$output"
}

trap cleanup 0 1 2 15
usize=`bsdtar tvf "$input" | awk '{s += $5} END {print s}'`
mnt=`mktemp -d`
log=`mktemp`

case "$type" in
    msdos|*fat) size=$((1024*1024 + usize*2)); opt_tar=--no-same-owner ;;
    ext*|udf  ) size=$((1024*1024 + usize*2)) ;;
    f2fs      ) size=$((1024*1024*50 + usize*2)) ;;
    btrfs     ) size=$((114294784 + usize*2)) ;;
    nilfs2    ) size=$((134221824 + usize*2)) ;;
    xfs       ) size=$((1024*1024*300 + usize*2)) ;;
    jfs       ) size=$((1024*1024*16 + usize*2)); opt=-q ;;
    hfsplus   )
        size=$((1024*1024 + usize*2))
        [ $((size % 4096)) != 0 ] && size=$((size + (4096-(size % 4096)))) ;;
    *) echo "$type is untested" 1>&2; exit 1
esac
rm -f "$output"
truncate -s $size "$output"
$mkfs $opt "$output" > "$log" 2>&1 || { cat "$log"; exit 1; }

mount -o loop "$output" "$mnt"
bsdtar -C "$mnt" $opt_tar --chroot -xf "$input"
[ "$SUDO_UID" ] && chown "$SUDO_UID:$SUDO_GID" "$output"
ok=1

.xfs files start at a size of 300MB, even if you place a 0-length file in it, but bzip2 compresses such an image into 6270 bytes.

To mount an .xfs under a regular user, use libfsxfs.

1. 7z -i prints all supported formats.

$ ruby -run -ehttpd . -b 127.0.0.1 -p 8000

& point mpv or vlc to a particular episode:

$ mpv http://127.0.0.1/s01e01.mp4

This should work as if you're playing a local file. To play a movie with a web browser, make sure the web server returns correct Content-Type headers. A container format counts too: e.g., Chrome doesn't like mkv.

Can we do something similar without the HTTP server? Depending on the container format, it's possible to feed mpv with a raw TCP stream. We'll lose seeking, but if we were creating, say, a YouTube Shorts or Facebook Reels competitor, this won't matter, for consumers of these kind of clips don't care much about that.

The most primitive solution requires only 2 utils:

1. ncat, that can listen on a socket & fork an external program when someone connects to the former:

   $ cat mickeymousetube
   #!/bin/sh
   
   export movie="${1:?Usage: ${0##*/} file.mkv [port]}"
   port=${2:-61001}
   type pv ncat || exit 1
   
   __dirname=$(dirname "$(readlink -f "$0")")
   ncat -vlk -e "$__dirname/pv.sh" 127.0.0.1 $port
   

2. pv, the famous pipe monitor that can limit a transfer rate; without the limiter, mpv eats all available bandwidth:

   $ cat pv.sh
   #!/bin/sh
   pv -L2M "$movie"
   

   The -L2M option means max 2MB/s.

Then run mickeymousetube in one terminal & mpv tcp://127.0.0.1:61001 in another to play a clip.

tcplol

How hard may it be to replace ncat with our custom script? What ncat does with -e option is akin to what inetd did back in the day:

(The illustration is from Stevens' UNIX Network Programming.)

Instead of creating a server that manages sockets, one writes a program that simply reads from stdin and outputs to stdout. All the intricacies of properly handling multiple clients are managed by the super-duper-server.

There is no (x)inetd package in modern distros like Fedora, as systemd has superseded it with socket activation.

Suppose we have a script that asks a user for his nickname & greets him in return:

$ cat hello.sh
#!/bin/sh
uname 1>&2
while [ -z "$name" ]; do
    printf "Nickname? "
    read -r name || exit 1
done
echo "Hello, $name!"

To expose it to a network, we can either write 2 systemd unit files & place them in ~/.config/systemd/user/, or opt for a tiny 37 LOC Ruby script instead:

require 'socket'

usage = 'Usage: tcplol [-2v] [-h 127.0.0.1] -p 1234 program [args...]'
…

server = TCPServer.new opt['h'], opt['p']
loop do
  client = server.accept
  cid = client.remote_address.ip_unpack.join ':'

  warn "Client #{cid}"
  pid = fork do
    $stdin.reopen client
    $stdout.reopen client
    $stderr.reopen client if opt['2']
    client.close
    exec(*ARGV)
  end
  client.close
  Thread.new(cid, pid) do
    Process.wait pid
    warn "Client #{cid}: disconnect"
  end
end

This is a classic fork server that uses a thread for each fork to watch out for zombies. The linked tcplol script performs an additional clean-up in case the server gets hit with a SIGINT, for example.

ncat, on the other hand, operates quite differently:

1. it creates 2 pipes;
2. after each new connection, it forks itself;
3. it connects the 2 pipes to the child's stdin/stdout;
4. (in the parent process) it listens on a connected socket using select(2) syscall and transfers data to/from the child using the 2 pipes; we'll talk about select(2) and the concept of multiplexing later on.

Anyhow, if we run our much simpler "super-server":

$ ./tcplol -v -p 1234 ./hello.sh

& connect to it with 2 socat clients, the process tree under Linux would look like:

$ pstree `pgrep -f ./tcplol` -ap
ruby,259576 ./tcplol -v -p 8000 ./hello.sh
  ├─hello.sh,259580 ./hello.sh
  ├─hello.sh,259587 ./hello.sh
  ├─{ruby},259583
  ├─{ruby},259588
  └─{ruby},259589

The dialog:

$ socat - TCP4:127.0.0.1:8000
Nickname? Dude
Hello, Dude!

(Why socat? We can use ncat as well, but the latter doesn't close its end of a connection; it hangs in CLOSE_WAIT until one presses Ctrl-D.)

To play a movie, run

$ ./tcplol -v -p 8000 ./pv.sh file.mkv

using a modified version of pv.sh script:

#!/bin/sh
echo Streaming "$1" 1>&2
pv -L2M "${1?Usage: pv.sh file}"

Then connect to the server with

$ mpv tcp://127.0.0.1:8000

Mickey mouse SOCKS4 server

inetd-style services can perform various actions, not just humbly write to stdout. Nothing prevents such a service from opening a connection to a different machine and relaying bytes from it to the tcplol clients.

To illustrate the perils of the low-level socket interface, let's write a crude, allow-everyone socks4 service and test it with curl. The objective is to retrieve security.txt file from Google using a TLS connection like so:

$ curl -L https://google.com/.well-known/security.txt --proxy socks4://127.0.0.1:8000

As a socks4 client, curl sends a request to 127.0.0.1:8000 with an IP+port to which it wants our service to establish a connection (meaning we don't have to resolve google.com domain name ourselves). We decode this and promptly send an acknowledgment reply. This is the 1st part of socks4.rb which we are going run under tcplol:

$stdout.sync = true

req = $stdin.read 8 + 1
ver, command, port, ip = req.unpack 'CCnN' # 8 bytes
abort 'Invalid CONNECT' unless ver == 4 && command == 1

ip = ip.to_s(16).scan(/.{2}/).map(&:hex) # [a,b,c,d]
res = [0, 90].pack('C*') +               # request granted
      [port].pack('n') + ip.pack('C*')
$stdout.write res

What should we do next? As soon as curl gets the value of 'res' variable, it eagerly starts sending a TLS ClientHello message to 127.0.0.1:8000. At this point, we don't need to analyse exactly what it sends--our primary concern is relaying traffic to and fro as quickly as possible without losing bytes.

To temporarily test that we have correctly negotiated SOCKS4, we can conclude the script with the ncat call:

exec "ncat", "-v", ip.join('.'), port.to_s

It should work. However, we can also rewrite that line in pure Ruby using the Kernel.select method. What we need here is to monitor 2 file descriptors in different modes to react to changes in their state:

1. in reading mode: stdin and a TCP socket to google.com;
2. in writing mode: the TCP socket to google.com.

(We assume that stdout is always available.) This kind of programming--being notified when an IO connection is ready (for reading or writing, for example) on a set of file descriptors--is called IO multiplexing. Most web programmers never encounter it because the socket interface is many levels below the stack they are working in, but it may be interesting sometimes to see how the sausage is made.

Replace exec "ncat" line with:

require 'socket'

s = TCPSocket.new ip.join('.'), port
wbuf = []
BUFSIZ = 1024 * 1024
loop do
  sockets = select [$stdin, s], [s], [], 5

  sockets[0].each do |socket|   # readable
    if socket == $stdin
      input = $stdin.readpartial BUFSIZ
      wbuf << input
    else
      input = socket.readpartial BUFSIZ
      $stdout.write input
    end
  end

  sockets[1].each do |socket|   # writable
    wbuf.each { |v| socket.write v }
    wbuf = []
  end
end

We're establishing a connection to google.com and then initiating the monitoring of two file descriptors in an endless loop. The select method blocks until one or both of these file descriptors become available for reading or writing. The last argument to it is a timeout in seconds.

When select unblocks, sockets[0] contains an array of file descriptors available for reading. If it's stdin, we read the data the OS kernel thinks is obtainable & save such a chunk to wbuf array. If it is a socket to google.com, we read some bytes from it & immediately write them to stdout for curl to consume.

sockets[1] contains an array of file descriptors available for writing. We only have 1 google.com socket here, to which we write the contents of wbuf array.

The script terminates when $stdin.readpartial returns an EOFError. This indicates to curl that the other party has closed its connection.

If you run socks4.rb under tcplol:

./tcplol -v -p 8000 ./socks4.rb

and observe errors tcplol prints from socks4.rb, you'll see that curl makes 2 requests to google.com, for the first one yields 301.

$ curl -sLI https://google.com/.well-known/security.txt --proxy socks4://127.0.0.1:8000 | grep -E '^HTTP|content-length'
HTTP/2 301
content-length: 244
HTTP/2 200
content-length: 246

I think today, when every desktop OS has a decent terminal emulator, we can sent little shell scripts instead. E.g., if Bob receives a file

$ fold -w34 postcard
tail -c+37 "$0"|base64 -d|gzip -cd
#H4sIALdOkGUAA62XO47bMBCG+1xBDbNNu
sBeCnDhA+QQrlwY2c1GyCIrJ0jHQoUKFbI
...
EkmUw04ZNvxUwGDoSYCTCG+IUYwfpg/7MA
v8BGNZ3QFsTAAA=

he won't be able to immidiately see what it is about unless he runs it:

$ chmod +x postcard
$ ./postcard

The "payload" in the "postcard" is modified ANSI art from 1992 (RN-BART.MIR in mirage01.zip artpack), converted to UTF-8.

We can generate such a program with a simple script that expects raw payload form stdin:

$ cat mkpostcard
#!/bin/sh

set -e
prefix() { printf 'tail -c+37 "$0"|base64 -d|gzip -cd\n#'; }

script=`mktemp`
trap 'rm -f "$script"' 1 2 15

prefix > "$script"
gzip -c | base64 -w0 >> "$script"
chmod +x "$script"
mv "$script" "${1:-postcard}"

Easy-peasy. We are not adding a shebang, as in this case, it may be omitted.

We can also go further & generate a password-protected postcard. This presents some difficulties, though: POSIX doesn't mention any command line utility for (en|de)cryption, & openssl could be missing on a target machine. Thus we either

• write a xor-cipher in awk or
• embed a source code of a C implementation of ChaCha20, but again, the target machine may not have a compiler installed or
• embed an αcτµαlly pδrταblε εxεcµταblε what will take care of decryption.

Everybody has heard about Cosmopolitan Libc toolchain during the Covid19 days, but personally, I haven't had any use for it.

It's an amazing peace of work. E.g., this minimal rc4 cipher implementation (I believe it's public domain)

$ cat rc4.c
#define S ,t=s[i],s[i]=s[j],s[j]=t /* rc4 hexkey <file */
unsigned char k[256],s[256],i,j,t;main(c,v,e)char**v;{++v;while(++i)s[
i]=i;for(c=0;*(*v)++;k[c++]=e)sscanf((*v)++-1,"%2x",&e);while(j+=s[i]
+k[i%c]S,++i);for(j=0;c=~getchar();putchar(~c^s[t+=s[i]]))j+=s[++i]S;}

compiles into

$ file rc4
rc4: DOS/MBR boot sector; partition 1 : ID=0x7f, active, start-CHS (0x0,0,1), end-CHS (0x3ff,255,63), startsector 0, 4294967295 sectors
$ du rc4
404K    rc4

that runs on Linux (including aarch64, I specifically checked that!) and FreeBSD.

rc4.c has a few deficiencies:

• if its argv[1] is null or an empty string, it coredumps;
• it requires a hex-formatted string, not a plain-text password.

At first, I fixed the bug, & added an str2hex conversion. I could continue & add everything else: embed an encrypted message, read the password from terminal, &c, but why? Writing shell scrips is easier & a postcard recipent then gets a text file, not a binary¹. Hence, I reverted to the "stock" rc4.c, & decided to use Ruby's erb instead:

$ cat message.erb
#<%= rc4=File.read(rc4) %>
#<%= message=File.read(message) %>
slice() { tail -c+$1 "$0" | head -c $2 | base64 -d | gzip -cd; }
cleanup() { rm -f "$rc4"; stty echo; }
set -e
rc4=`mktemp`
trap cleanup 0
trap 'cleanup; echo 1>&2; exit 1' 1 2 15
slice 2 '<%= rc4.length %>' > "$rc4"
chmod +x "$rc4"
stty -echo
printf 'Password: ' 1>&2; read -r password; printf "\n" 1>&2
[ -n "$password" ]
hexkey=`printf '%s' "$password" | od -A n -t x1 -v | tr -d ' \n'`
slice '<%= rc4.length+4 %>' '<%= message.length %>' | "$rc4" $hexkey

It generates a "password-protected" shell script, that

1. extracts the αcτµαlly pδrταblε rc4 εxεcµταblε from its 1st comment line;
2. asks a user for a password;
3. converts the password to a hex string;
4. extracts the "message" from its 2nd comment line and decrypts it with rc4.

Makefile, that assists in creating such a script:

password := monkey
message := payload/hello1.txt
out := _out
CC := ~/opt/s/cosmocc/bin/cosmocc

all: $(out)/message

$(out)/%: %.c
    @mkdir -p $(dir $@)
    $(CC) --std=c89 -o $@ $<

$(out)/message: message.erb $(out)/rc4
    gzip -c < $(out)/rc4 | base64 -w0 > rc4.text
    $(out)/rc4 $(hexkey) < $(message) | gzip -c | base64 -w0 >message.text
    erb rc4=rc4.text message=message.text $< > $@
    chmod +x $@
    rm *.text

.DELETE_ON_ERROR:
hexkey = $(shell printf '%s' $(call se,$(password)) | od -A n -t x1 -v | tr -d ' \n')
se = '$(subst ','\'',$1)'

You'll need to adjust CC variable which expects the compiler from Cosmopolitan Libc toolchain.

Then

$ make password=12345 message=payload/TO-GZ.txt
...
$ du _out/message
304K    _out/message
$ _out/message
Password:

I should end with a remainder that rc4 is not a moderd, state-of-the-art cipher, & you should absolutely not blindly run "encrypted" postcards from unknown sources.

1. Whilst αcτµαlly pδrταblε εxεcµταblεs are also shell scripts that cleverly pretend otherwise, they are not plain-text files.

25m<sup>3</sup>

Some text editors have a special machanism for entering characters not present in the current keyboard layout. E.g., in Emacs C-x 8 ^ 2 inserts superscript ², but this won't work for writing a formula like y = x^2a, as C-x 8 ^ a injects the â character.

Unicode should've had at least a super/subscript version of Latin characters, but that never happend. It has a subset of it, spreaded across seemingly random sections. For instance, a subscript letter j sits in the Latin Extended-C block between a Finno-Ugric ∃ (that remainds me of existential quantification quantifier) & a superscript capital V.

Unicode has the Superscripts and Subscripts section, but it appears more like an afterthought, incomplete & abandoned:

If one peruses Unicode blocks, it's possible to assemble most of the Latin characters in super/subscript variants to write a simple script for character substitution. Then, any decent text editor could use such a script to replace a chunk of text with a superscript version of it.

I also thought of a slightly more interesting feature: selecting a chunk of text with HTML tags (like <i>y = x<sup>2a</sup></i>) in the text editor to transform only <sup> or <sub> nodes.

Script 1: supsub

It reads its input from the stdin line-by-line:

$ lsb_release -d | ./supsub sub
dₑₛ𞁞ᵣᵢₚₜᵢₒₙ:    fₑdₒᵣₐ ᵣₑₗₑₐₛₑ ₃₉ ₍ₜₕᵢᵣₜᵧ ₙᵢₙₑ₎
$ lsb_release -d | ./supsub sub | ./supsub invert
ᵈ𞀵ˢ𞀿ʳⁱ𞀾ᵗⁱ𞀼ⁿ:    ᶠ𞀵ᵈ𞀼ʳ𞀰 ʳ𞀵ˡ𞀵𞀰ˢ𞀵 ³⁹ ⁽ᵗʰⁱʳᵗʸ ⁿⁱⁿ𞀵⁾
$ lsb_release -d | ./supsub sub | ./supsub invert | ./supsub restore
dеsсrірtіоn:    fеdоrа rеlеаsе 39 (thіrty nіnе)

(Some mobile browsers have a lot of trouble rendering even Latin superscript/subscript characters.)

$ cat supsub
#!/usr/bin/env -S ruby --disable-gems

db = DATA.read.split(/\s+/).filter {|v| v}
$supa = db.map {|v| [v[0], v[1]] }.to_h
$sub = db.map {|v| [v[0], v[2]] }.to_h

mode = 'sup|sub|invert|restore'
abort "Usage: supsub #{mode} < file.txt" unless ARGV[0] =~ /^#{mode}$/

def tr mode, chars
  case mode
  when 'sup'
    chars.map { |ch| $supa[ch.downcase] || ch }.join
  when 'sub'
    chars.map { |ch| $sub[ch.downcase] || ch }.join
  when 'invert'
    supa_v = $supa.invert
    sub_v = $sub.invert
    chars.map do |ch|
      if supa_v[ch]
        $sub[supa_v[ch]] || ch
      elsif sub_v[ch]
        $supa[sub_v[ch]] || ch
      else
        ch
      end
    end.join
  else # restore
    supa_v = $supa.invert
    sub_v = $sub.invert
    chars.map { |ch| supa_v[ch] || sub_v[ch] || ch}.join
  end
end

while (line = STDIN.gets)
  print tr(ARGV[0], line.chars)
end

__END__
0⁰₀ 1¹₁ 2²₂ 3³₃ 4⁴₄ 5⁵₅ 6⁶₆ 7⁷₇ 8⁸₈ 9⁹₉ +⁺₊ -⁻₋ =⁼₌ (⁽₍ )⁾₎
aᵃₐ bᵇb cᶜ𞁞 dᵈd eᵉₑ fᶠf gᵍg hʰₕ iⁱᵢ jʲⱼ kᵏₖ lˡₗ mᵐₘ nⁿₙ oᵒₒ
pᵖₚ q𐞥q rʳᵣ sˢₛ tᵗₜ uᵘᵤ vᵛᵥ wʷw xˣₓ yʸᵧ zᶻz
а𞀰ₐ б𞀱𞁒 в𞀲𞁓 г𞀳𞁔 ґґ𞁧 д𞀴𞁕 е𞀵ₑ ж𞀶𞁗 з𞀷𞁘 и𞀸𞁙 іⁱ𞁨 їїї ййй
к𞀹𞁚 л𞀺𞁛 м𞀻ₘ нᵸн о𞀼ₒ п𞀽𞁝 р𞀾ₚ с𞀿𞁞 т𞁀т у𞁁𞁟 ф𞁂𞁠 х𞁃𞁡 ц𞁄𞁢
ч𞁅𞁣 ш𞁆𞁤 щщщ ьꚝь ю𞁉ю яяя

Script 2: supsub-xml, version 1

$ echo '<sub>test</sub> <i>lol</i> <sup>haha</sup> but <sup>it works</sup>!' | ./supsub-xml
ₜₑₛₜ <i>lol</i> ʰᵃʰᵃ but ⁱᵗ ʷᵒʳᵏˢ!

With the help of 鋸, we parse the stdin as an XML fragment, & replace <sup> or <sub> nodes with the result of supsub script.

#!/usr/bin/env ruby

require 'nokogiri'

cmd = ARGV[0] || File.join(__dir__, 'supsub')
doc = Nokogiri::HTML.fragment STDIN.read

doc.css('sup,sub').each do |node|
  IO.popen("#{cmd} #{node.name}", 'w+') do |t|
    t.write node.text
    t.close_write
    node.replace t.gets
  end
end

print doc.to_s

The script works, & for the most practical purposes one may leave it as is, but it has 1 issue I find quite barbaric: it forks the external program each time it needs to transform a string.

Script 2: supsub-xml, version 2

No, we are not going to rewrite supsub as a library; we are going to invoke it from supsub-xml as a coprocess.

What is a coprocess? Stevens, in his APUE, described it as a program that runs alongside a parent process & communicates with it via 2 one-way pipes.

Think of it as a local microservice: you write to it using one pipe, then read a responce using another pipe.

A simple (but contrived) example of a Ruby script that drives another program is to use the tr(1) utility for upcasing a string:

$ cat coprocess # broken, read below
#!/usr/bin/env ruby

parent_in, parent_out = IO.pipe
child_in, child_out = IO.pipe
spawn "tr '[a-z]' '[A-Z]'", in: child_in, out: parent_out
parent_out.close
child_in.close

child_out.puts "lol"
print parent_in.gets
child_out.puts "haha"
print parent_in.gets

If we run it, though, we won't get LOL and HAHA--the script would hang indefinitely on the print parent_in.gets line. The reason for that is the usage of the libc fwrite(2) by the tr utility (at least in the coreutils version under Linux). fwrite(2) uses a libc stream that is buffered by default. What happens is that tr(1) eats "lol" string and prints "LOL" into its stdout, but our coprocess script doesn't see the result, for "LOL" is stuck in a buffer.

We can fix the script by executing tr under the stdbuf(1) utility:

spawn "stdbuf -i0 -o0 tr '[a-z]' '[A-Z]'", in: child_in, out: parent_out

then bytes should start moving freely through the pipes:

$ ./coprocess
LOL
HAHA

Unfortunately, this fix won't work if the coprocess is a Ruby script. Ruby has its own IO mechanism that stdbuf(1) cannot affect.

The general solution for this kind of problems is to trick a coprocess in thinking it is connected to a (pseudo) terminal.

Thankfully, Ruby has a nifty built-in pty extension that abstracts away most of the communication with a pseudo terminal device.

$ cat supsub-xml
#!/usr/bin/env ruby

require 'nokogiri'
require 'pty'

cmd = ARGV[0] || File.join(__dir__, 'supsub')

class Coprocess
  def initialize cmd
    @master, slave = PTY.open
    read, @write = IO.pipe
    spawn cmd, in: read, out: slave
    read.close
    slave.close
  end

  def puts str; @write.puts str; end
  def gets; @master.gets; end
end

transforms = {
  "sup" => Coprocess.new("#{cmd} sup"),
  "sub" => Coprocess.new("#{cmd} sub")
}

doc = Nokogiri::HTML.fragment STDIN.read
doc.css('sup,sub').each do |node|
  tr = transforms[node.name]
  tr.puts node.text
  node.replace tr.gets.chomp
rescue
  warn "transforming `#{node.text}` failed: #{$!}"
end

print doc.to_s

If we add sleep to the very end of the script, than we may examine how this arrangement works:

$ $$
bash: 42396: command not found
$ echo '<sub>test</sub> <i>lol</i> <sup>haha</sup>!' | ./supsub-xml
ₜₑₛₜ <i>lol</i> ʰᵃʰᵃ!

While the script sleeps, from another terminal:

$ pstree -p 42396 -al
bash,42396
  └─ruby,100032 ./supsub-xml
      ├─ruby,100033 --disable-gems ... sup
      └─ruby,100034 --disable-gems ... sub

$ file /proc/100032/fd/* # supsub-xml
...
$ file /proc/100033/fd/* # supsub sup
...

Chrome displays it almost correctly--except for the "formulas" block. Judging from the shape of the glyphs, the default monospace font I use in Fedora

$ fc-match monospace
LiberationMono-Regular.ttf: "Liberation Mono" "Regular"

doesn't contain all the characters from UTF-8-demo.txt; hence, Chrome does font fallback. In its devtools it reports:

Liberation Mono     — Local file (5,438 glyphs)
DejaVu Sans         — Local file (1,179 glyphs)
Droid Sans Thai     — Local file (415 glyphs)
Droid Sans Ethiopic — Local file (320 glyphs)
Segoe UI Historic   — Local file (45 glyphs)
Noto Sans Math      — Local file (7 glyphs)
Droid Sans Fallback — Local file (5 glyphs)
Segoe UI Symbol     — Local file (1 glyph)
Noto Color Emoji    — Local file (1 glyph)
Times New Roman     — Local file (1 glyph)

(Your list would be, of course, completely different.)

Interestingly, xterm, gvim and gedit

... render the file better than Chrome (they don't mangle the "formulas" block) and Emacs 29.1 not only fails the "formulas" test but incorrectly aligns pseudo-graphics at the end of the file.

Anyhow, I became curious about how many installed monospace fonts I have can render that file without font substitution (spoiler: none).

Is there a way to disable font fallback? We can provide a custom fontconfig config via

$ FONTCONFIG_FILE=~/tmp/lol.conf gedit

or convert UTF-8-demo.txt to the pdf format using a tool that, by default, doesn't know about font substitution:

# input output font
txt2pdf() {
  awk '{print "    " $0}' < "${1:-/dev/null}" | \
    pandoc --pdf-engine=xelatex \
    -V "monofont:${3:-Roboto Mono}" -V "mainfont:${3:-Roboto Mono}" \
    -V geometry:"top=1cm,left=1cm,bottom=1.5cm,right=1cm" \
    -t pdf -o "${2:-${1%.*}.pdf}"
}

(Yes, we trick pandoc into thinking the input is markdown; IRL you'd probably want to add -V papersize=a4 and -V fontsize=12pt options to it.)

Then we can type:

$ txt2pdf UTF-8-demo.txt lol.pdf 'Ubuntu Mono'

and examine lol.pdf to see that a typeface, created "to complement the Ubuntu tone of voice", not only has "a contemporary style" but also "conveys a precise, reliable and free attitude":

How to do the same for all installed fixed-width fonts? First, we obtain the list of such fonts:

$ type fc.mono
fc.mono is aliased to `fc-list :mono family | awk -F, "{print \$1}" | sort -u'
$ fc.mono
Bitstream Vera Sans Mono
Courier 10 Pitch
Courier New
Cursor
DejaVu Sans Mono
Droid Sans Mono
Inconsolata
Liberation Mono
Ligconsolata
Material Icons
Material Icons Outlined
Material Icons Round
Material Icons Sharp
Material Icons Two Tone
Nimbus Mono PS
Noto Color Emoji
Source Code Pro
Terminus
Ubuntu Mono

(I have Roboto Mono v2 installed as well, but fontconfig doesn't recognise it as a monospace font.)

then

$ (IFS=$'\n'; for fn in `fc.mono`; do txt2pdf UTF-8-demo.txt "$fn".pdf "$fn"; done)
$ ls *pdf -1
'Bitstream Vera Sans Mono.pdf'
'Courier New.pdf'
'DejaVu Sans Mono.pdf'
'Droid Sans Mono.pdf'
Inconsolata.pdf
'Liberation Mono.pdf'
Ligconsolata.pdf
'Nimbus Mono PS.pdf'
'Source Code Pro.pdf'
'Ubuntu Mono.pdf'

In my case, not every .txt→.pdf conversion was successful, but among those that succeeded, 'DejaVu Sans Mono.pdf' produced the best result, glyph-wise.

Here's a high precision simulator:

1. Load module-simple-protocol-tcp module into a PulseAudio server.
2. The module can use any sink you prefer, typically the default one.
3. Initiate a regular playback using the selected sink (with mpv, vlc, a web browser, whatever).
4. Connect to the server from an Android phone using Simple Protocol Player.

For an additional milieu, put the phone in a bucket. $100 saved, "greater dynamic range" achieved.

The scheme is compatible with PipeWire too.

Coincidentally this can be used as a rustic spying technique: you can listen to everything a machine with a running module-simple-protocol-tcp module plays.

A script (requires dialog(1) and jq(1)) that draws a menu with available sinks and starts listening on a socket:

#!/usr/bin/make -f

# the standard pulseaudio tcp port
port := 4713
answer := $(shell mktemp -u)

status:; pactl list | grep tcp -B1
stop:; -pactl unload-module module-simple-protocol-tcp

$(answer):
    pactl -f json list sinks \
     | jq -r '.[] | @sh "\(.index) \(.description)"' \
     | xargs dialog --keep-tite --menu "Local sink" 0 0 0 2>$(answer)

start: $(answer) stop
    pactl load-module module-simple-protocol-tcp rate=44100 format=s16le channels=2 source=`cat $<` record=true port=$(port)
    rm $<

.DELETE_ON_ERROR:

Run it as

$ android-pa-share start

Beware that module-simple-protocol-tcp module doesn't support authentication, hence protect the port (4713 in the script above) from the WAN.