# Related topics, by meaning

2026-08-11. Port #4 of the 43, `edrelated` → `convoro-related`. Fourth by
benefit: discovery is the bottleneck Ernest has named repeatedly, and a forum's
own archive is the cheapest traffic it has.

---

## Why this one is worth porting at all

Every forum has "related topics" and almost all of them match on **tags**.
Convoro's core does too. That works exactly as well as your tagging discipline,
which on a real forum is: not very.

This matches on **meaning**. Each topic is embedded once, and similarity is
cosine distance between vectors. On the Invision original, running on Ernest's
own forum, *Workflows* matched *Connect for Flarum 2* at **79%** — different
platform, different category, **zero shared tags**. No tagging scheme produces
that, because nobody would ever have tagged them alike.

## 🚨 The decision the whole port exists to carry

**The threshold is corpus-dependent, so shipping a number as a default is a
lie.**

On a forum where every topic announces a product, *unrelated* pairs still score
75–78 — the writing is that similar. On a general forum the same pair scores
40. A "sensible default" is sensible for exactly one corpus and wrong for every
other, and the operator has no way to know which they have.

So the settings screen does not ask for a number in the abstract. It shows
**real matches from their own content, with real scores, ignoring the
threshold**, and greys out the ones that fall below the line as they move it.
They pick by looking at their own forum.

That is the feature. Everything else here is plumbing.

## What it must not do

**Never block posting.** Embedding is queued. A provider outage delays
indexing; it never delays a person pressing Reply. This is Convoro's standing
rule and a live site went down proving it.

**Never show an empty box.** Nothing to suggest means nothing rendered — no
heading over a blank list, no "no related topics found". A permanent empty
region is furniture, and furniture is what teaches people to stop looking.

**Never compare across models.** If the operator changes embedding model, the
old vectors have different dimensions and are meaningless against the new ones.
Those entries are **skipped, not compared**. Fewer results beats confident
nonsense.

**Never re-embed unchanged text.** The text is hashed; editing a typo costs
nothing. On a busy forum this is the difference between a bill and a rounding
error.

## What changes coming to Convoro

| | |
|---|---|
| **One row per topic, not per chunk** | carried across unchanged. The AI Assistant chunks posts because it answers questions from fragments; this compares whole topics to whole topics, and chunking blurs exactly the signal being measured. |
| **Packed float32, not JSON** | also carried. 1024 dimensions is 4KB packed against ~20KB as text, and every stored vector is read on every lookup. |
| **No licensing phone-home** | `sources/Usage/Usage.php` does not come across, as with every port. |
| **Renders through `topic.posts.after`** | Convoro has a real hook there. The Invision version injected markup through head JS because it had nowhere better; that was a workaround, not a design. |
| **Its own provider settings** | Convoro has no LLM seam yet — OpenSearch embeds cluster-side and is a different thing. If Convoro later grows one, this should borrow it so there is one key and one bill. |

## The PHP trap, recorded because it cost real time

`pack('g*', ...$vector)` **throws if the array has string keys** — PHP 8 reads
a spread of a string-keyed array as named arguments and `pack()` rejects it. It
must be `...array_values(array_map('floatval', $vector))`. It only surfaced
against a real API response, because every fixture had integer keys.

## Scope

In: indexing on create/edit, backfill, the similarity lookup, the display
block, the settings screen with the live preview.

Out, deliberately, and named rather than left to be discovered: no cross-content
matching (topics to blog entries), no per-forum thresholds, no "more like this"
search. Each is a real feature and none is needed to prove the idea.

## What building it turned up

Written after the fact, because each of these is a gap in Convoro rather than
in this extension, and the next port will hit them too.

A third one is recorded here and then withdrawn: I wrote that `__()` cannot
count, and that pluralising needed two keys. It does not. Convoro has `@choice`
with locale-aware plural rules, so `'1 reply|{count} replies'` is one key and a
language needing three or six forms gets three or six. The block uses it. Worth
leaving the correction visible rather than quietly deleting the claim — the two
English keys I shipped first would have made every other language wrong.

**An extension cannot ship a CSS or JavaScript file.** There is no assets
convention and `ext:package` has no notion of one, so this screen's slider
behaviour and the cut-off rule are an inline `<script>` and `<style>` in the
template. The SEO port worked around the same gap through the `head.after`
hook. It is honest and it is small, but forty ports remain and several will
want real assets. Worth building into core.

**`settings` is `(key, value)` and nothing else.** No `id`, no timestamps. So
`insertGetId()` cannot work on it and adding `created_at` is an unknown column
— both are a 500 on the very first save, on a fresh install, which is the one
moment nobody is watching a log. The idiom is a raw `INSERT`, which the
Developer Sign-In port had already found. A settings helper on the container
would stop the third port rediscovering it.

## Two things the screenshots changed

**The strikethrough had to go.** The first preview struck out everything below
the cut-off. On the shipped default of 75% that meant every row on the screen
was struck out the moment the page opened, which is indistinguishable from "this
is broken". It is now a labelled rule that slides between the rows — the same
information, stated as a decision rather than as damage.

**The slider moved.** It was in Settings, three cards below the table it
changes. A control and the thing it controls have to be in the same eyeline or
dragging it teaches nobody anything, which was the entire justification for
building the preview in the first place.

Both were only findable by looking at the rendered page. Neither would have
failed a test.


## 1.1.0 — OpenSearch as a second backend

Ernest, 2026-08-11: *"since we are using OpenSearch for embeddings for the AI
Assistant wouldn't it also make sense to use it for related as well?"* — then,
clarifying: *"I wasn't wanting OpenSearch to be a requirement… was just asking
shouldn't it be an option for those that may want to use OpenSearch to save
money by not having to use Voyage."* It has since become a standing rule for
every module that takes an embedding key.

He is right, and the reason is money. A site running OpenSearch with an ML
Commons model has already paid for embeddings, in hardware. Billing it a
provider a second time — per topic, and again on every backfill — is charging
for something its own infrastructure is doing anyway.

**It is an option, never a requirement.** OpenSearch is optional in Convoro and
the model is optional on top of that; `embeddingModel` empty is a documented,
supported state. The provider backend stays the default and remains the only
path on a site without either.

### What it buys, and what it costs

| | |
|---|---|
| **No key, no bill** | the cluster embeds on ingest, so a lookup costs a query |
| **No second index** | rebuilding search rebuilds the vectors; coverage is free |
| **Permissions for free** | the search index already carries a `gate` field |
| **A smaller model** | all-MiniLM-L6-v2 at 384 dimensions, against voyage-3.5-lite |
| **Shared fate** | the vectors live and die with the search index |

### The decision that took the most care

**The cut-off is stored per backend.** A cosine from Voyage and a `neural`
relevance score from OpenSearch are different numbers about different spaces.
Carrying 0.75 across a switch would silently mean something else — usually
"show nothing" or "show everything" — and the operator would be looking at a
broken block with a setting that reads correct. So `related_threshold_direct`
and `related_threshold_opensearch` are separate, the old single key survives as
a fallback so a 1.0.0 upgrade keeps its number, and the preview always shows
scores from whichever backend is actually answering.

There is no honest formula converting one to the other, and the code says so
rather than inventing one.

### Two things the front end had to learn

**Null is not empty.** A backend returning `null` means "I could not answer";
`[]` means "nothing is similar". The block treats them the same way — it does
not render — but the settings screen has to tell them apart, and so does
anything that later decides whether to fall back.

**The chosen backend is not always the answering one.** A cluster that went
away falls back to the provider rather than taking the block down on every
topic, and the settings screen says so out loud. Silently changing which
numbers a threshold refers to is the one thing this feature must never do.

### What this needed from core

`OpenSearchIndex` grew `embeds()` and `similar()`. They belong in the search
module rather than in this extension: reaching into somebody else's cluster
from an extension would be the wrong seam, and "more like this" is not a
question only Related asks.

That makes 1.1.0 depend on core 1.3.4 **for this backend only**. The manifest
still declares `^1.3.3`, deliberately — the provider backend works perfectly on
1.3.3 and refusing to install would be worse than a disabled option that
explains itself. An older core is named as its own reason, distinct from "no
OpenSearch module", because they send somebody to fix different things.
