A Minimalistic Weather App
Latest update:
This tweet caught my attention:
Programmer Memes ~
@iammemeloper, 10:39 PM ยท Jul 25, 2023
When you ask backend developer to make frontend
Most comments from the folks I follow were of a "THAT'S ME" nature.
I liked the mockup too, yet I had a couple of questions. Why do we
need a button? Presumably, there is no JavaScript on the page at
all--it's a classic HTML form. That's fine, there's nothing wrong with
that, but what happens when a user wants data for Paris in Illinois?
Would the app return a list of links then?
What if we, contrary to the perception of a grumpy 'backend developer'
in a vacuum, use JavaScript? Then we can get rid of the button, listen
on the 'input' event, & even display popup completions. Can a guy
write such an app without using any web frameworks, any web
components, or any third-party code & still keep the app tiny?
This is what I came up with
(I think it's way too big):
$ wc -lc *js index.html
24 682 mkdb.js
121 3566 server.js
96 3334 web.js
21 634 index.html
262 8216 total
$ scc *.js index.html | awk '/Total/ { print $6 }'
227
$ terser web.js -mc | openssl zlib | wc -c
1127
The server part is written in Nodejs with 0 npm dependencies:
$ cat package.json
{"type": "module"}
This is how it looks:
It saves its state in the url search params, thus it's possible to
bookmark a location.
Data sources
Where does a guy gets weather data? There are, of course, a multitude
of subscription-based options, but for casual use--checking the
temperature a couple of times a day--they seem superfluous.
At first, I recalled that >10 years ago I was using a dockapp called
wmweather. There is no such program in Fedora repos, but one can fetch
it from Debian. It retrieves weather data from noaa.gov, downloading
a METAR report for a specific airport, if you feed it with an ICAO
location indicator. E.g., for an airport near Kyiv, it's UKBB, so I
typed
$ wmweather -s UKBB
and saw strange results (the screenshot is heavily magnified):
It's the 2nd part of summer. Why is it showing 1ยฐC? Looking at the
exact URL that wmweather used, it immediately becomes clear why:
$ curl https://tgftp.nws.noaa.gov/data/observations/metar/decoded/UKBB.TXT
Boryspil, Ukraine (UKBB) 50-20N 030-58E 122M
Feb 23, 2022 - 10:30 PM EST / 2022.02.24 0330 UTC
Wind: from the S (190 degrees) at 2 MPH (2 KT):0
Visibility: 3 mile(s):0
Sky conditions: mostly cloudy
Weather: light rain; mist
Temperature: 33 F (1 C)
Dew Point: 33 F (1 C)
Relative Humidity: 100%
Pressure (altimeter): 30.12 in. Hg (1020 hPa)
ob: UKBB 240330Z 19001MPS 5000 -RA BR SCT008 BKN016 01/01 Q1020 TEMPO 1000 BR
cycle: 3
Yeah. Just out of curiosity, I checked some Russian airports, and
noaa.gov happily returned fresh, up-to-date data, hinting that
everything is going well over there, which made me quite disgusted.
Then I recalled that GNOME has libgweather library. I doubt they pay
some company for an API key (otherwise it would've been abused
immediately), hence there must be a free solution. There is one: the
Norwegian Meteorological Institute.
It works like this:
- you obtain, by whatever means, a pair of lat/lon coordinates for a
location;
- you ask
api.met.no to return a forecast for the lat/lon pair;
- keep your requests rate <= 20 queries/sec;
- limit your geolocation coordinates to a max of 4 decimals.
In a more sane world, each country would have a free-for-everyone
endpoint to examine weather data in a standardised format for its own
locations only. Then it would be possible to maintain a list of such
endpoints without relying on an intermediary.
libgweather also conveniently maintains a list of cities with
corresponding coordinates. Unfortunately, it's in XML format, which
introduces difficulties into our no-3rd-party-code JS game. However,
there are other lists of cities on GitHub already available in JSON
format. I selected what I thought was one of the most comprehensive
ones & augmented it with country flags emojis (maybe it's too
comprehensive: e.g., they have "West New York", "East New York", &
"New York City" entries).
Server
Do we need a server part? Unfortunately, we do, for api.met.no
doesn't allow CORS requests.
We also need geo data. At first, just like any simpleton would've
done, I loaded a pre-generated module locations.js, that contained
144,579 entries of city names & their coordinates, into
memory. Searching was very fast (duh), but the process was
consuming > 300MB of RAM, which is insane.
Then I had 3 choices:
- use sqlite;
- write a primitive b-tree backed key-value storage (certanly doable,
but it would've been the biggest/ugliest part of the app);
- put all 144,579 entries into
locations.jsonl & do a linear search
line-by-line.
Guess what did I choose as the most simple approach.
Nodejs isn't your friend when you search through a 7MB file. Just
interating with fs.readLines() (& doing nothing with the data) was
consuming ~140ms. For contrast: grepping with 'qwerty*q' (the RE that
could not match anything) in the same file took 6ms. Maybe I should
run grep for each query--even with the fork overhead, that would be
faster that anything I can write in JavaScript. (I didn't do that.)
Then I tried reading the file chunk-by-chunk with
fs.readSync(). When doing nothing but looking for the last \n in
each chunk to reposition the next fs.readSync() call, the operation
took 37ms.
I guess a user can live with that?
The next discovery was that JSON.parse() isn't free, & having
locations.txt with the following structure:
$ grep '^.. Paris' _out/locations.txt | head -3
๐ง๐ท Parisi; Brazil|-20.2655|-50.0390
๐จ๐ฆ Paris; Ontario; Canada|43.2000|-80.3833
๐ซ๐ท Paris; Ile-de-France; France|48.8534|2.3486
was noticebly faster to parse. Fruitlessly looping through the
"DB" & sending a network reply took ~60ms. Not great, but acceptable.
$ time curl -s 'http://127.0.0.1:8080/api?city=qwerty'
[]
real 0m0.059s
user 0m0.001s
sys 0m0.007s
These numbers, of course, should be multiplied by 5 when using a VPS
with a virtual potato instead of a CPU.
Anyway, how would a little server work? It needs to do 3 things:
act as a static server, to return index.html & web.js files;
respond with a list of completions when a user types something,
e.g.:
$ curl -s 'http://127.0.0.1:8080/api?city=paris;' | json
[
"๐จ๐ฆ Paris; Ontario; Canada",
"๐ซ๐ท Damparis; France",
"๐ซ๐ท Paris; Ile-de-France; France",
"๐ต๐ฆ Paris; Herrera Province; Panama",
"๐บ๐ธ Paris; Arkansas; United States",
"๐บ๐ธ Paris; Idaho; United States",
"๐บ๐ธ Paris; Illinois; United States",
"๐บ๐ธ New Paris; Indiana; United States",
"๐บ๐ธ Paris; Kentucky; United States",
"๐บ๐ธ Paris; Maine; United States"
]
I hardcoded max 10 items as a limit.
return weather data for an item from the completion:
$ curl -s 'http://192.168.197.98:8080/api?l=%F0%9F%87%AB%F0%9F%87%B7+Paris;+Ile-de-France;+France' | json
{
"time": "2023-08-02T13:00:00Z",
"details": {
"air_pressure_at_sea_level": 995.2,
"air_temperature": 22.4,
"cloud_area_fraction": 39.8,
"relative_humidity": 52.5,
"wind_from_direction": 237.2,
"wind_speed": 10.6
},
"co": {
"lat": "48.8534",
"lon": "2.3486"
}
}
This is a small chunk of what api.met.no returns.
The 'router' of the server:
let server = http.createServer( (req, res) => {
let url; try {
url = new URL(req.url, `http://${req.headers.host}`)
} catch {
return usage(res)
}
if (req.method === 'GET' && url.pathname === '/api') {
let city = url.searchParams.get('city')
let location = url.searchParams.get('l')
if (city != null) {
cities(res, city)
} else if (location != null) {
weather(res, location)
} else
usage(res)
} else
serve_static(res, url.pathname)
})
usage() (that sets 400 status code & ends a connection) &
serve_static() are not relevant. cities() returns completions:
function cities(res, query) {
let r = find(query, 10, (v, q) => {
v = v.slice(0, v.indexOf('|')) // faster then v.split("|")[0]
return v.toLowerCase().includes(q) ? v : null
})
res.setHeader("Expires", new Date(Date.now() + 300*1000).toUTCString())
res.end(JSON.stringify(r))
}
find() does the linear search:
function find(q, limit, callback) {
let r = []
q = (q || '').trim().toLowerCase(); if (!q) return r
let buf = Buffer.alloc(1024*50)
let pos = 0
while (fs.readSync(lfd, buf, 0, buf.length, pos)) {
let ln = buf.lastIndexOf(10)
if (ln < 0) throw new Error("better not happen")
pos += ln+1
let lines = buf.subarray(0, ln).toString().split("\n")
for (let line of lines) {
let match = callback(line, q)
if (match) {
r.push(match)
limit--
}
if (!limit) break
}
if (!limit) break
}
return r
}
weather() makes a request to api.met.no & reformats the reply:
function weather(res, query) {
let co = find(query, 1, (v, q) => {
if (v.slice(0, v.indexOf('|')).toLowerCase() === q) {
v = v.split("|")
return {lat: v[1], lon:v[2]}
}
})[0]
if (!co) return err(res, 'invalid location')
fetch(`https://api.met.no/weatherapi/locationforecast/2.0/compact?lat=${co.lat}&lon=${co.lon}`).then( async v => {
if (!v.ok) throw new Error(v.statusText)
return { expires: v.headers.get('expires'), json: await v.json() }
}).then( v => {
let r = v.json?.properties?.timeseries?.[0]
if (v.expires) res.setHeader("Expires", v.expires)
res.end(JSON.stringify({
time: r?.time,
details: r?.data?.instant?.details,
co
}))
}).catch( e => err(res, `MET Norway: ${e.message}`, 500))
}
Another blunder: the value for the /api?l= url search parameter must
match exactly the entry in the "DB". You can see a problem here when
an impatient user decides to type "paris" & press Enter (without
waiting for a completion popup) only to see an 'invalid location'
error. A more useful app would employ an approximate string matching
algorithm.
Folks from MET Norway recommend saving the Last-Modified timestamp
from an api.met.no request & including If-Modified-Since when
repeating the same query. Our weather() does nothing of the
sort, it's as dumb as it gets: we simply copy the value of the
Expires header to a reply & hope that a web browser abides by it.
I'm not discussing the serving of static files here, only want to note
that in a more complex app, instead of the half-witted serve_static(),
it's better to use isaacs/st or cloudhead/node-static or
search for a similar npm module that ideally has exactly 1 dependency:
broofa/mime (surely, there must be something sleek among 7364
dependants of it).
Client
<p id="form">
<span>City:</span>
<input type="search" list="cities" spellcheck="false">
<datalist id="cities"></datalist>
</p>
<p id="result"></p>
<script type="module" src="web.js"></script>
The main part of the app's UI is a completion widget. HTML provides us
with <datalist> element, which practically nobody uses due to its
behavior differing enough in various browsers to discard it. A classic
vicious circle follows: developers ignore it because it's
"unfinished," & web browser vendors don't try to fix subtle bugs
because hardly anyone uses it.
This following is a quote from Lea Verou. It's from 2015(!) &
everything she was writing about <datalist> back than is relevant &
true in 2023:
"the more I played with it [<datalist>], the more my excitement
was dying a slow death, taking my open web standards dreams and
hopes along with it. Not only itโs incredibly inconsistent across
browsers (e.g. Chrome matches only from the start, Firefox
anywhere!), itโs also not hackable or customizable in any way. Not
even if I got my hands dirty and used proprietary CSS, I still
couldnโt do anything as simple as changing how the matching happens,
styling the dropdown or highlighting the matching text!"
Still, it's fine for such a simple app. So far, it works ~reliably in
- Chrome for Desktop
- Chrome for Android
- Firefox for Desktop
It's missing on Firefox for Android & it has too many bugs in WebKit.
Chrome for Desktop doesn't render emojis in the completion menu. For
Android it draws a horizontally scrollable widget at the bottom a
screen, that is small, annoying, & easily missed.
The most bothersome differences between Chrome & Firefox boil down to
event handling. While Chrome immediately fires 'change' event for
<input> when a user has selected an item, Firefox emits
'insertReplacementText'. Sometimes 'change' is fired even if the user
inserted the same completion, &c.
let city = document.querySelector("#form input")
let datalist = document.querySelector("#form datalist")
let result = document.querySelector("#result")
city.addEventListener("input", debounce(function(evt) {
log(evt.inputType)
if (evt.inputType === "insertReplacementText") { // firefox
this.dispatchEvent(new Event('change'))
datalist.innerHTML = ''
} else if (!evt.inputType?.match(/^(insertText|delete|insertFromPaste)/)) {
datalist.innerHTML = ''
} else {
completion(this.value).then( v => datalist.innerHTML = v)
.catch(e => err(this, e))
}
}, 250))
let prev_change_value
city.addEventListener("change", function() {
if (prev_change_value === this.value) return
prev_change_value = this.value
log("CHANGE")
result.innerText = "Loading..."
this.disabled = true
fetch_json('api?l='+this.value).then( v => {
// here we render 'v' into a <table>
// ...
let params = (new URL(location.href)).searchParams
params.set('l', this.value)
history.replaceState({}, '', '?'+params.toString())
}).catch(e => {
err(this, e)
}).finally( () => {
this.disabled = false
this.focus()
})
})
completion() only fetches json & returns new <option> tags:
async function completion(query) {
return (await fetch_json('api?city='+query)).map( v => {
return html`<option value="${v}">`
}).join`\n`
}
What can one do differently?
Use a 3rd-party completion widget instead of <datalist>.
Render a full forecast that api.met.no provides, not just numbers
from 1 hour ago.
Instead of a database with 144,579 entries, use a much smaller list
that can be loaded into memory. E.g., Locations.xml from
libgweather has < 5000 cities, & nobody is complaining how
unsatisfactory that is.
Rewrite server.js in C. GNOME libsoup can handle gory HTTP
details (+ Gio library underneath automatically creates a thread
pool).
Utilize SQLite's FTS.
Do #4 + #5, & charge a $20 monthly subscription for it.
SQLite update
It reduces the worst-case scenario from 60ms to 7ms. Check out sqlite
branch branch of the github repo. Using its FTS simplifies queries:
api?l=paris+france matches the real EU city. The only downside is
completion: while sqlite supports "pari*" queries, & api?city=pari*
returns 10 entries, <datalist> won't show them, for, say, "Paringa;
Australia" string doesn't contain "pari*" substring.
Tags: ะพะนัั
Authors: ag