CARIBBEIN
← All notes

Engineering note · pipeline

How you catalogue a sea

The marketplace's catalogue was not scraped and dumped. It was built island by island, in waves of small scripts with two properties that matter more than any algorithm: every destructive pass is reversible, and every claim about a venue gets judged before it publishes.

Summer 2026 · directory numbers probed

providers in the live directory
islands covered
ingest & enrichment scripts
entries with first-party imagery

Sources have a rank, and it is not negotiable

The island's official tourism source outranks every aggregator, every time. Aggregators copy each other, which means their errors correlate — the same closed restaurant alive on four sites is not four confirmations, it is one stale record with reach. The official list is where changes actually land first. Aggregators still get read, but as candidates to verify against first-party surfaces, never as truth.

Then enrichment happens venue by venue, from the venue's own published surfaces — its site, its social pages, its listed phone. That is slower than trusting a reseller's copy, and it is the difference between a directory and a rumour.

Small scripts, run as waves, all reversible

The pipeline is small scripts, not one big one, run in waves per island group: discover, enrich, geolocate, describe, photograph, verify, wire into the page. Two conventions hold across all of them, and they are the actual engineering content of this note:

  • Every pass takes a dry-run flag, and the dry run prints exactly what the wet run would do. You read the list, then you run it again without the flag.
  • Every destructive pass writes its own undo file before touching anything — the ids it changed and the prior values, as data. A watermark sweep that nulls flagged photos saves every (id, previous URL) pair. A re-categorisation saves the moved ids so restore is one batch request. Rollback is not a procedure someone remembers under stress; it is a file the pass already wrote.

Honesty passes

A surprising share of the pipeline exists to make the catalogue more honest, not bigger. Three examples that actually ran:

  • Broker demotion. Real-estate brokerages had drifted into the “places to stay” bucket. A traveller opening “stays” should see bookable villas, not agencies — so a conservative pass moved unambiguous brokerages to services, matching on clear brokerage naming only, and deliberately leaving villa-rental and property-management companies that actually list bookable units. Conservative, name-pattern-only, reversible.
  • Synthetic and duplicate prefilters. Dedup runs before publish, and entries that look manufactured rather than observed get filtered before they ever reach the page. A catalogue's credibility dies by a thousand almost-duplicates.
  • QC verdict sheets. Photos get laid out on contact sheets and judged tile by tile; the verdicts become data, and an apply pass executes them. A watermarked or wrong photo gets nulled — and the card then renders an honest branded placeholder rather than a stock look-alike. A photoless listing that says so beats a pretty lie; that rule is why the first-party imagery number below the title is a real number.

Imagery is rehosted, and that is a policy

We do not hotlink whatever loads. Imagery enters the catalogue as the venue publishes it, under an explicit rights model that is formalised when the venue joins as a provider — and it is rehosted, so an island page never waits on forty venues' servers, and a venue's redesign never breaks our card. Rehosting also makes the QC pass possible at all: you can only judge what you hold.

Bilingual at the data layer

The catalogue does not get translated at render time — it is bilingual in the rows themselves. An event carries title_en and title_fr, a provider carries descriptions in both languages, and the page just picks. Translation done at ingest is translation done once, reviewed once, cached forever; translation done at render is a tax on every page view and a bug on the cache key.

AI earns its keep here — drafting descriptions, judging photo tiles, classifying phone numbers as mobile or landline so a WhatsApp link is only offered where it can work. All of it behind the QC verdicts, none of it between a traveller and a person on their island. That split — AI in the tooling, humans in the product — is stated on /geek as taste; this note is what it looks like as work.

Check it yourself

The directory numbers on this page are re-derivable right now: probe.mjs counts the same public directory the marketplace reads, the same way. The endpoints it calls are documented in the API note.