How the Web Gallery of Art's Data Was Processed

The Web Gallery of Art has called itself a searchable database for most of its thirty years, but there has never been a database behind it. Its source is 53,000 JPEGs, about 8.4 GB of them, and each image carries its own catalogue entry in the file’s comment field: the artist, title, date, technique, location and commentary. A small tool called jpegcw reads those comments and generates the gallery’s pages. The 5,600 artist biographies, the tours and the glossary are hand-written HTML, each with conventions of its own.

Rebuilding the gallery meant turning all of that into one database, and getting the text out was the easy part. Bertholet Flémal’s Allegory of the Foundation of Rome is dated “c. 1670”. An importer built for a tidy database turns that into the number 1670 and drops the “c.”, which is the curator saying that nobody knows the year exactly. Most of the work of processing the gallery’s data went into not doing that.

Two copies of every record

Each artwork exists twice in the source. One copy is the comment inside the JPEG, the one jpegcw reads. Here is Flémal’s, with the commentary shortened:

1FLÉMAL, Bertholet
2(b. 1614, Liège, d. 1675, Liège)
3
4Allegory of the Foundation of Rome
5c. 1670
6Oil on canvas, 144 x 195 cm
7Private collection
8
9This work is a rare interpretation of Rome's founding; …
10
11--- Keywords: --------------
12
13Author: FLÉMAL, Bertholet
14Title: Allegory of the Foundation of Rome
15Time-line: 1651-1700
16School: Flemish
17Form: painting
18Type: mythological

The other copy is the gallery’s catalogue, a spreadsheet with one row for each of its 52,868 artworks and eleven columns: author, born–died, title, date, technique, location, URL, form, type, school and timeframe. Flémal’s row says the same things as the comment, without the commentary.

The top of the comment is positional: a line is the title or the date only because of where it sits. The keywords block underneath is labelled. The catalogue puts every field in its own column, so the importer treats it as the canonical record, and the artworks, artists, schools, forms and types all come from it. The commentary exists only in the comment, so that’s where the importer takes it from. The comment is stored in full beside the row. Only the fields the importer recognises with confidence are parsed out of it, into a separate JSON column, and where a parsed value disagrees with the catalogue, the catalogue keeps its value.

Dates

The catalogue’s date column is free text, and its 52,868 rows write dates in 5,784 different ways. Most follow a handful of patterns:

The importer keeps the text exactly as written and stores four fields beside it: a start year, an end year, whether the date is approximate, and a qualifier such as “begun” or “completed”. The text is what visitors see. The fields are what the timeline and the date filters use. An abbreviated range like “1574-88” takes its century from the start year. A decade or a century becomes the whole span, marked approximate, so the 1490s never turn into 1490.

Among the “something else” rows, “June 1888, Arles” is van Gogh, whose works the gallery dates to the month and the town. The parser takes the year, and the place stays in the text. Where nothing else matches, the parser picks out any four-digit year and marks the date approximate, because a guess shouldn’t look like a fact. “1296 (completed)” lands there, as a circa 1296 that loses the qualifier from the structured fields. The text still says it.

Names and lives

The catalogue files each artist under a sort key in capitals: “AACHEN, Hans von”, “DÜRER, Albrecht”, but also “LEONARDO da Vinci” and “ANDREA DEL CASTAGNO”, who have no surname to sort by. The importer keeps the key and builds a display name from it. The part before the comma moves to the end, capitals become title case, and particles such as da, del and von go lower case. That gives Hans von Aachen, Albrecht Dürer, Leonardo da Vinci and Andrea del Castagno.

Next to the name is a line of life details, such as “(b. 1614, Liège, d. 1675, Liège)”. 3,305 of the 5,768 artists follow exactly that shape. The rest write “ca.”, say “active 1470-1490 in Huesca” or “d. after 1498”, or leave a word out. “(b. 1767, Paris, 1842, Paris)” has no “d.”, so nothing actually says that 1842 is a death year. The importer fills a structured year only when it’s a plain four-digit year after “b.” or “d.”, or one marked “ca.”, and flags the approximate ones. Everything else stays empty, and the whole line is kept as written. So “d. after 1498” gives no death year at all, because an empty field is more accurate than 1498.

Places

The location column is free text as well, with 5,551 different strings across the 52,805 artworks that have one. Most of the Louvre’s works say “Musée du Louvre, Paris”, but the column also has “Louvre, Paris”, “Palais du Louvre, Paris”, “Cour Carré, Louvre, Paris” and “Musée du Louvre, Paris (central), Galleria Sabauda, Turin (wings)” for one work split between two museums.

A location becomes a database record only through a reviewed list of 957 places. Each entry has a city, whether it’s a museum, whether the public can visit, and the aliases someone has checked mean that place. “Private coll.”, “Priv. collection” and “Priv. coll.” all lead to one non-public Private Collection, for example. Before comparing, the importer lower-cases the text and strips accents and punctuation, so “Musée du Louvre, paris” still matches. Nothing looser than that counts as a match.

That resolves 44,590 artworks. The other 8,215, spread over about 4,500 strings, go into a review report with the nearest museum names by edit distance as suggestions for a person to check. The suggestions show why they aren’t applied automatically: the closest museum to “Kenwood House, Hampstead Heath, London” is the Rockox House in Antwerp. An unresolved artwork keeps its location text, so visitors still see where it is, even though the database can’t yet connect it to other works in the same place. Countries are filled in the same cautious way, from a reviewed list that so far covers 17 cities.

Biographies

The 5,593 biographies are pages from the framed site, and most of each file is frame. Scripts reload the page inside the frameset when someone opens it on its own, and there’s a printing link and a footer. The biography itself sits between two HTML comments, <!-- Comment Start --> and <!-- Comment End -->, which the old tooling used as markers. The importer takes what’s between them, keeps it as it was, and does its work on a copy.

Most of that work is links. 5,033 of the biographies link somewhere, about 20,000 links in all: 12,040 to other biographies, 6,819 to artwork pages, and the rest mostly to tours and the glossary. They’re written in every form the old site allowed: https://www.wga.hu/html/a/aachen/adonis.html, /html/a/aachen/adonis.html, or just adonis.html relative to the page it’s on. The importer turns each one into a single canonical old path and looks it up in a table built from every imported artist and artwork. The table gives the new address, such as /artists/<artist-slug>-<id>/<artwork-slug>-<id>. If two records claim the same old path, the run stops instead of picking one. A link to a page nothing was imported for is unwrapped to its text rather than left pointing nowhere, and links outside the gallery stay as they are.

What software added

Everything so far is deterministic: the same source gives the same database on every run. That includes the one thing the importer works out for itself, the colour palettes. A fixed sample of each image’s pixels gives a weighted palette, stored with the version of the method and a hash of the image, so the same image always gets the same palette.

Language models only come in afterwards, in a separate stage that runs on the finished records and keeps what it adds apart from what the gallery recorded. They write one thing, image descriptions, and only for accessibility: someone using a screen reader should get more than a title, and there’s nobody available to describe 52,000 images by hand. They’re written by a model running locally that sees only the pixels, with no title, artist or commentary, so it can’t repeat catalogue facts as if it had seen them in the picture. Every result is cached by its exact input, so a rebuild never asks a model the same question twice, and a result that fails validation isn’t stored.

Tours

The guided tours aren’t in the source folder at all, so the importer reads them from the live site, as it does the music, the guestbook and the bibliography. Asked for a page that no longer exists, the live site can answer with HTTP 200 and a page titled “Web Gallery of Art has been moved to a new address!”, which a scraper would happily store as part of a tour. The tours scraper recognises that page and refuses it. The nine times it turns up in a full crawl are all links the tours themselves get wrong: list13.html.html, list23html, chroonology.html.

What the scraper keeps is evidence: each page’s raw HTML, its canonical old address, a hash, and its text, images, image maps, citations and links, in order. Turning that into a page-by-page reader takes judgement about which page follows which, and what’s a picture page and what’s a list. So it goes through a curation file that a person approves, and the importer publishes only what’s in that file. The file is still empty, so for now every tour stays in its original form.

Old addresses

Thirty years of links point at the old site: bookmarks, links from other sites, search results. Every record the importer creates keeps the address it had on the old site, and the table that rewrites the links inside biographies maps each old path to its new one. We’re collecting all of the old addresses, from artists and artworks to tours and the pages around them, so that each one leads somewhere on the new site and the switch breaks as few links as possible.

Wrapping up

A program can now read the gallery’s data, though the data itself is no tidier than it was. Flémal’s allegory still says “c. 1670” on the new site, and underneath, the database knows that means around 1670. Where the source states something plainly, it becomes a field. Where the source hedges, misspells a word or leaves one out, the text stays exactly as written, and the fields beside it stay empty or say they’re approximate. The searchable database the Web Gallery of Art has always described itself as now exists, built from the same source the old site was generated from.