Hybrid search for Persian RAG: a field guide

Hybrid search for Persian (Farsi) RAG on PostgreSQL and pgvector: safe normalisation, RRF over dense and lexical ranks, no locale filter, bilingual tests.

Hybrid search for Persian (Farsi) RAG means running two retrievers side by side, a dense embedding search and a lexical search, and fusing their rankings with reciprocal rank fusion (RRF). You need both because Persian breaks the quiet assumptions of English recipes: the same word can be stored with Arabic or Persian letters, a zero-width non-joiner sits inside words, and neither Snowball nor PostgreSQL ships a Persian stemmer. The short version: normalise on write, keep the lexical side stemmer-free, fuse ranks rather than scores, never filter by locale, chunk along the document's own structure, and test on a bilingual question set before trusting any of it.

This guide draws on two systems. One is this site's assistant, which I built and maintain. The other is Kartez Contract, where I worked as an AI engineer; its copilot had to find the right clauses in Persian contracts, numbered by article, clause and note. The retrieval design described here (dense plus lexical, rank fusion, pgvector) is this site's; from Kartez I take only what its case study states. Neither system is benchmarked here.

Why English RAG recipes fail on Persian text

The typical failure is not a wrong answer but no answer: a search misses a word the user can see in the document, because two strings that look identical differ underneath.

One word, two strings: Arabic and Persian Yeh and Kaf

Persian and Arabic share a script, not every code point. The Persian ی (U+06CC ARABIC LETTER FARSI YEH) and the Arabic Yeh (U+064A ARABIC LETTER YEH) are two different characters, as are the Persian ک (U+06A9 ARABIC LETTER KEHEH) and the Arabic Kaf (U+0643 ARABIC LETTER KAF). The W3C's Arabic and Persian Layout Requirements name the Yeh pair directly: the letter is "represented with two different characters". A passage stored with one form does not match a question typed with the other; the normalise step in the diagram above shows both pairs.

This site folds Arabic Yeh, Alef Maksura and Kaf, among other characters, to their Persian forms before a passage enters the retrieval index, strips invisible bidi marks from that copy, and runs the same folding over every question.

The zero-width non-joiner is part of the word

می‌شود, میشود and می شود are three different strings. The first separates the prefix from the verb with a zero-width non-joiner (ZWNJ, U+200C), which stops the letters connecting without adding a space. The Unicode Standard describes this non-default joining "as is sometimes required in writing Persian using the Arabic script", and the W3C requirements say control characters "should be preserved when storing and displaying texts".

Strip the ZWNJ, as an English pipeline might, and می‌شود becomes میشود; turn it into a space and one word becomes two tokens. The normaliser in hazm, a Persian NLP toolkit, includes "ZWNJ correction", turning کتاب های into کتاب‌های. On this site, stored text keeps the ZWNJ, and only the folding used inside comparisons treats it as a space.

Three sets of digits

Persian writes ۱۲۳ with Eastern Arabic-Indic digits (U+06F0 to U+06F9), Arabic writes the same number with Arabic-Indic digits (U+0660 to U+0669), and technical text often writes 123.

Fold digits for comparison only, never in stored text; on this site, rewriting stored Persian digits would change how the Persian CV renders. Folding has to run on both sides, though: fold only the question's digits, and a question in Persian digits no longer matches the passage it was copied from. The article on streaming LLM guardrails applies the same folding to numbers a model emits.

Diacritics are rare, so one stray mark breaks an exact match

The W3C requirements put it plainly: "In Arabic script text it is unusual to use diacritics for vowel information and for consonant lengthening." So a single passage with a kasra (a short-vowel mark), or a question pasted with a tashdid (the doubling mark), fails every exact match against the unmarked spelling. In the Persian text-to-speech research I worked on for MCI, the same property was the hard part from the speech side: short vowels and the ezāfe, which links a noun to its modifier, are not written. For retrieval, strip diacritics inside comparisons and keep them in stored text.

Users ask colloquially; documents are written formally

"Persian is one of the languages with many differences between its formal and informal styles of writing," write Tajalli, Shamsfard and Kalantari, introducing a parallel corpus of 50,000 sentence pairs. A user types می‌تونم where formal text writes می‌توانم. My reading of the consequence: a lexical matcher sees little overlap between a colloquial question and a formal clause, so the dense retriever is the one with a chance, and a test set of well-formed questions will hide the gap.

Normalisation that is safe for Persian

• Fold Arabic Yeh and Kaf to Persian ی and ک on write, in Persian content only.

• Keep the ZWNJ in stored text.

• Strip bidi marks from the copy you index and compare, not from displayed text.

• Fold digits and strip diacritics only inside comparisons, on both sides.

• Run exactly the same function on questions as on passages.

• Never apply Persian folding to Arabic text, where Arabic Yeh (U+064A) and Kaf (U+0643) are the correct letters.

This site's retriever does not meet that list yet: it folds Persian letterforms in every language, Arabic included; its lexical list can miss a passage written in Persian digits; and Arabic-script diacritics are stripped from the question's terms and from every comparison the answer gate makes, but not from the stored text the keyword search scans. Read the list as the target.

Persian full-text search in PostgreSQL: no built-in configuration

Full-text search is the obvious lexical half of a hybrid, and PostgreSQL's documentation says it "comes with predefined configurations for many languages". Persian is not one of them.

Snowball has an Arabic stemmer, not a Persian one

Snowball's list of stemming algorithms includes Arabic and has no entry for Persian or Farsi. PostgreSQL builds its stemming dictionaries from Snowball: the language list in the PostgreSQL 18 source includes arabic and no persian, and the documentation's dictionary listing shows arabic_stem and simple with no Persian dictionary. An Arabic stemmer shares Persian's script, not its grammar. Outside the database, hazm ships a Persian stemmer and lemmatiser; inside it, there is none.

What the simple configuration gives you, and what it does not

The simple dictionary template, in the words of PostgreSQL's dictionary documentation, works by "converting the input token to lower case and checking it against a file of stop words". The built-in simple dictionary has no stop-word file, so every token is kept, and there is no stemming: قرارداد and قراردادها stay two lexemes, a loss for recall and a gain for predictability.

Before any dictionary runs, the parser decides what a word is, and its notion of a letter "is determined by the database's locale setting, specifically lc_ctype". So do not assume how your database splits Persian, least of all words joined by a ZWNJ. Ask it, with the ts_debug function from the text search debugging documentation:

-- How does this database tokenise Persian? Check; do not assume.
SELECT alias, token, lexemes
FROM ts_debug('simple', 'قراردادها را می‌توان فسخ کرد');

This site's keyword search uses simple for Persian rows and english for English rows. The semantic vs keyword search demo runs it next to the assistant's hybrid retriever over this site's content.

Trigram similarity for typos

The pg_trgm extension compares three-character sequences, so a misspelt word still shares trigrams with the correct one. It "ignores non-word characters (non-alphanumerics) when extracting trigrams", so check before relying on it: SELECT show_trgm('قرارداد'); should return trigrams made of Persian letters. The site's keyword search adds a word_similarity arm, which matches a short query against the best continuous stretch of a long passage.

Whole-word coverage: the lexical list inside this site's retriever

The assistant's retriever does not use full-text search for its lexical list, since there is no Persian configuration. It counts how many of the question's content words appear in a passage as whole words, using regular-expression word boundaries. It removes common words in each of the site's languages, skips very short terms, keeps Hindi vowel signs inside the word they belong to, and does no stemming. The result is a coverage ratio between 0 and 1.

Word boundaries matter: a substring match on "is" hits "his", "this" and "Isfahan", and every passage starts to look like evidence. The bounded ratio matters too, because a similarity floor can be compared against it. The ratio counts toward the refusal floor only when at least two different content words match.

Learned sparse vectors: the documented option

Learned sparse retrieval matches terms like a lexical search, with weights a model learns. In BGE-M3 the terms are tokeniser pieces (its model card weighs 'De', 'fin' and 'ation'), and the vector is "of size equal to the vocabulary", in the words of the paper that introduced it: 250,002 in the model's configuration. This site's production retriever does not use them: it runs dense plus lexical, and the embeddings API it calls returns no sparse weights. If you add them, read the off-by-one section below first.

Hybrid search with reciprocal rank fusion: fuse ranks, not scores

A dense retriever and a lexical retriever return ranked lists whose scores mean different things. Reciprocal rank fusion ignores the scores and combines the positions.

The formula, and where k = 60 comes from

Cormack, Clarke and Büttcher define a document's fused score as the sum over rankings of 1/(k + r(d)), where r(d) is its rank in each list. The constant is less delicate than it looks: "k = 60 was fixed during a pilot investigation and not altered during subsequent validation", and the pilot "indicated that k = 60 was near-optimal, but that the choice was not critical".

This site's retriever uses k = 60, and each list contributes more candidates than the answer will use, so a passage in the middle of one list and near the top of the other can still surface. In outline:

K = 60  # Cormack, Clarke and Büttcher

def fuse(lists, k=K):
    """lists: (weight, passage ids best-first) for each retriever."""
    scores = {}
    for weight, ranked in lists:
        for rank, passage in enumerate(ranked, start=1):
            scores[passage] = scores.get(passage, 0.0) + weight / (k + rank)
    return scores

def order(scores, locales_of, query_locale):
    # The fused score decides. Locale only separates exactly equal
    # scores, and the id keeps the order identical on every run.
    return sorted(scores, key=lambda p: (-scores[p], query_locale not in locales_of[p], str(p)))

fused = fuse([(1.0, dense_ids), (0.5, lexical_ids)])
passages = order(fused, locales_of, query_locale="fa")[:top_k]

Why score sums need weights you have to tune

The alternative is to add scores, as the paper behind BGE-M3 does: s_rank ← w1·s_dense + w2·s_lex + w3·s_mul, "where the values of w1, w2 and w3 depend on the downstream scenario". For dense plus sparse the authors set w1 = 1 and w2 = 0.3 on MIRACL, and w1 = 0.2 and w2 = 0.8 on MLDR, a long-document benchmark.

A cosine similarity, a sparse inner product and a coverage ratio live on different scales, and the exchange rate between them moves with the corpus. Rank fusion sidesteps that calibration. It is a trade-off, not a free win: the paper reports strong results with weighted sums, and RRF cannot tell a landslide from a narrow win.

Why the lexical list gets half weight

On this site the lexical list is there to promote passages with exact strings a dense model can blur: a domain name such as booqai.app, an acronym, a programme name such as SAMPAD. That is the usual reason to keep one. In Anthropic's Contextual Retrieval write-up, an embedding model "might find content about error codes in general, but could miss the exact 'TS-999' match."

The lexical list contributes at half weight, which lifts a passage the dense list also returned without letting keyword overlap outvote the dense ranking. The price is real: with k = 60 and this site's candidate counts, a passage only the lexical list finds ranks below every dense candidate, so fusion cannot bring back a passage the dense model missed entirely. The one exception is the passage with the highest similarity, which is always returned, as the next section explains. If the embedding call fails, retrieval falls back to the lexical list alone, and the similarity floor still applies.

An RRF score is not a confidence

An RRF score is built from positions alone, so a corpus of pure noise still produces a first-ranked passage with the best fused score on offer. This site's refusal floor is therefore compared with a real similarity for each passage, never with the fused score: the better of its cosine similarity to the question as asked (or, for a follow-up, to the question joined to the one before it, with the item it points at between the two when the follow-up is worded as a position in a list, or the start of the previous answer when that item cannot be read) and its whole-word coverage, which counts only when at least two content words match. The floor is checked before the language model is called.

The passage with the highest similarity among all candidates is always among those returned, even when fusion ranked it too low to make the cut. Below the floor, a question that is not about me is refused without a model call, and a question about me is given the site's records instead; the article on RAG hallucination guardrails covers both.

Never filter retrieval by locale

A locale filter looks like tidiness: Persian questions search Persian passages. On a bilingual corpus it deletes evidence.

Your best evidence can be in the other language

On this site, passages in every language sit side by side in one table, and a question searches all of them. The Persian CV is the source document and the English CV an adaptation, so an English question about a role can have its best evidence in Persian. Kartez Contract had the same shape in reverse: its drafting wizard had to carry Persian requirements into a library of English contract-drafting texts.

Locale as a tiebreak that only breaks exact ties

Locale still has one use in ranking: between two passages with exactly the same fused score, the one available in the question's language comes first, then a stable id decides, as in the sort key above. With one dense list and the lexical list at half weight, exact ties practically never happen. They can when a question is searched in more than one version, as described below, and two passages collect the same ranks from those versions' dense lists, which carry equal weight; even then, locale decides only when just one of the two exists in the question's language. That is deliberate: locale must never outrank a score. Locale has one more job: choosing which translation of a passage the reader sees.

A small bonus for same-language passages is not a tiebreak. Adjacent contributions from one full-weight list differ by 1/((k + r)(k + r + 1)), a gap that shrinks further down the list, and sums from two lists can sit closer still. A fixed bonus is safe only while it stays below every gap your list depths and weights can produce, a limit that moves whenever either changes. Once the bonus exceeds that limit, results that were not tied swap places mid-list, among exactly the candidates fusion is meant to lift.

One passage, ten translations

Pages on this site are published in up to ten languages, and copies of one passage in different languages could take several of the eight places handed to the model. Retrieval now treats the translations of a passage as one candidate: it is scored by its best translation, ranked once, and shown in the reader's language when that translation exists. Articles are grouped by section rather than by chunk, because a long section splits into a different number of chunks in each language; a translation whose sections do not line up with the English version's is kept apart rather than grouped wrongly. The reader's language decides which translation is shown; beyond an exact tie, it never decides where the passage ranks.

Persian questions over English documents

Only the dense side can cross languages: a Persian question and an English passage share almost no words, apart from Latin terms such as RAG. BGE-M3's own evaluation does not settle it for Persian: its cross-lingual benchmark, MKQA, retrieves English Wikipedia passages for queries in 25 non-English languages, Arabic included and Persian not. Test your own Persian–English pairs.

Questions with no keywords

A visitor's "What did he do recently?" carries no word the corpus can match, and "And before that?" carries nothing at all. When a question is about me, it is also searched with my name attached, and a follow-up is also searched joined to the question before it, with the item it points at between them when it is worded as a position in a list, such as "the second one", or the start of the previous answer when that item cannot be read. All the versions are embedded in one call and fused by rank. The version with my name attached orders the passages but never counts toward the refusal floor, because my name alone can make an unrelated question about me look close to my CV. Questions about my work as a whole do not rely on retrieval: they are answered from dated records the site builds from my CV, case studies, articles and services, as the article on RAG hallucination guardrails describes.

Chunking Persian contracts by article, clause and note

Persian contracts have their own structure of articles (ماده), clauses (بند) and notes (تبصره). In the Kartez Contract copilot, the user's own contract was split along that structure, and only the sections relevant to a question were used to answer it; answers "draw only on the selected clauses". The Kartez Contract case study covers the rest of the project, and what to build against legal AI hallucinations covers the domain.

The same principle drives this site's chunker: one chunk is one semantic record. A CV role is never split from its dates, a case study splits at its authored sections, a post at its headings, and a long section breaks only at a paragraph boundary. Every section chunk carries the document title, so a chunk retrieved alone still says where it came from.

For Persian contracts, that suggests a few rules, offered as general advice rather than a description of how Kartez was built:

• Keep the number with the text. "ماده ۵" and its body belong in one chunk.

• Carry the parent article's heading into every clause and note chunk.

• Never cut a clause at a token count. A note separated from its clause can change what the clause appears to say.

• Store the structural path, article › clause › note, so an answer can point a professional to the exact place.

pgvector: exact scan or HNSW?

pgvector answers half of this itself: "By default, pgvector performs exact nearest neighbor search, which provides perfect recall." An approximate index "trades some recall for speed".

Filters are the less obvious cost. "With approximate indexes, filtering is applied after the index is scanned. If a condition matches 10% of rows, with HNSW and the default hnsw.ef_search of 40, only 4 rows will match on average." pgvector 0.8.0 and later can scan the index iteratively until enough rows match, once you enable it.

This site's retrieval table has no approximate index: the vector search is an exact scan. Publication state is part of the SQL WHERE clause, never a filter applied after LIMIT, where it would silently shrink the result set. Start with an exact scan, add an index when latency demands it, and compare its top-k with an exact scan on your own questions before trusting it.

The pgvector sparsevec off-by-one

This matters only if you add learned sparse vectors to PostgreSQL, and it fails silently.

The text format for sparse vectors in pgvector is {index1:value1,index2:value2}/dimensions, "and indices start at 1 like SQL arrays". Tokeniser ids start at 0. The Python client takes 0-based indices ("Indices start at 0") and adds 1 itself when it writes the text form.

from pgvector import SparseVector

weights = {0: 0.05, 2: 0.14, 4: 0.04}  # tokeniser ids, 0-based

# Client library: pass 0-based ids; it adds 1 when it writes SQL text.
vec = SparseVector(weights, 6)

# Hand-built SQL literal: shift exactly once, here.
literal = "{" + ",".join(f"{i + 1}:{w}" for i, w in weights.items()) + "}/6"
# "{1:0.05,3:0.14,5:0.04}/6"

Shift exactly once, where a SQL literal is built by hand. pgvector's text input converts each index to 0-based and raises "sparsevec index out of bounds" only when the result is below 0 or at or above the dimension. So a double shift fails only for the top of the vocabulary, a missing shift only for token id 0, and every other weight quietly lands on the neighbouring token.

Choosing an embedding model for Persian (Farsi)

What MIRACL says about Persian and Arabic

MIRACL is a retrieval benchmark across 18 languages, Persian and Arabic among them. The paper behind BGE-M3 reports nDCG@10, a score for ranking quality in the top ten results, on its development set (ACL version; the authors updated these results in July 2024):

On Persian, dense retrieval scores far above this BM25, learned sparse sits between the two, and combining all three modes adds a little on top of dense. Four caveats travel with the numbers:

• This BM25 uses subword tokens. The authors ran it on XLM-RoBERTa's tokeniser to match the model; their appendix shows Lucene's analyser raising the MIRACL average across languages from 31.9 to 38.5. MIRACL's own BM25 baseline, with language-specific analysers where they exist, reaches 0.333 nDCG@10 on Persian and 0.481 on Arabic, on a 0 to 1 scale.

• MIRACL is monolingual. It is "designed to support monolingual retrieval tasks, where the queries and the corpora are in the same language", so it says nothing about Persian questions over English documents.

• Persian and Arabic are not a controlled comparison. They use different questions and Wikipedia corpora; the project page lists 632 development queries for Persian and 2,896 for Arabic, so the gap is a signal, not a measure of difficulty.

• BGE-M3 was fine-tuned on MIRACL's training data, so its rows are in-domain scores.

FaMTEB and Hakim

FaMTEB, published in Findings of EMNLP 2025, is a Persian benchmark built on MTEB, with "63 datasets spanning seven different tasks", retrieval among them.

Hakim is a Persian embedding model described in an arXiv preprint, first posted in May 2025 and revised in October 2025. Its authors report an 8.5% improvement over existing approaches on FaMTEB. Two facts belong next to that result: Hakim's authors are also FaMTEB's authors, and Hakim's second-stage training data lists MIRACL_fa. Neither makes the result wrong; both are reasons to test on your own data.

What to test before you choose

• Your own questions in Persian and English, colloquial phrasing included.

• Cross-lingual pairs: Persian questions answered by English text, and the reverse.

• Spelling variants of one question: Yeh and Kaf forms, ZWNJ or space, each digit set.

• The licence. BGE-M3's model card says MIT.

• Maximum input length against your chunk size, in tokens. The BGE-M3 authors describe long documents of up to 8,192 tokens, and Persian can need more tokens than English for the same content; the Persian token tax found that premium small on BGE-M3's own tokeniser and much larger on the generation models' tokenisers it measured.

• The vector dimension. On this site it is fixed in the schema, so changing it means a migration and a full reindex.

The site's embedding explorer is deliberately not neural: its character n-gram vectors group words by spelling, not meaning, which is the gap a dense model has to close for colloquial Persian.

Evaluating Persian RAG on a bilingual golden set

A golden set for Persian RAG needs questions in both languages: answerable ones with the passages they should retrieve, must-refuse ones with no evidence in the corpus, and ones whose best evidence is in the other language. Measure recall@k on the answerable questions, check that every no-evidence question falls below the similarity floor, and add a test that fails if no answer draws on a passage in the other language. It catches a locale filter the day someone adds one back.

On this site, the golden set, in all ten of the site's languages with most questions in English and Persian, runs in CI through the real fusion, grouping, ordering and top-k code, but against an in-memory corpus and a bag-of-tokens surrogate embedder, not the production one. The surrogate also feeds a sparse list production does not run, and the no-evidence set is small. CI catches regressions in the ranking machinery, not the live assistant's recall. A retrieval-only command, which never calls the language model, checks the refusal floor against the deployed embedder; no recall run against that embedder has been made, so I publish no recall figure for this retriever.

What carries over to Arabic

I have not designed or evaluated retrieval for Arabic text. This site's retriever indexes its Arabic-language content, but nothing in it was built or tested for Arabic. This section is what the Persian design implies on paper.

Most of it carries over: rank fusion, no locale filter, structure-aware chunking, stripping diacritics inside comparisons (the W3C statement covers Arabic script generally), an exact scan first and a bilingual golden set. Three things differ:

• PostgreSQL does ship Arabic. Snowball has an Arabic stemmer and PostgreSQL an arabic_stem dictionary; whether stemming helps is for your golden set to decide.

• The Persian letter folding must not run. In Arabic, Yeh (U+064A) and Kaf (U+0643) are the correct letters, so a mixed corpus has to normalise by language.

• The digits differ. Arabic-Indic digits are U+0660 to U+0669, not the U+06F0 range Persian uses.

Arabic also has more benchmark coverage, in MIRACL and MKQA. In the MKQA table of BGE-M3's paper, Arabic questions over English Wikipedia reach a Recall@100 of 71.1 with BGE-M3 dense, against 18.9 for the paper's subword BM25; Recall@100 counts a question as found when a passage containing its answer is in the top 100. Persian has no such cross-lingual number.

Checklist: hybrid search for Persian RAG

• Fold Persian letterforms on write, in Persian content only; strip bidi marks from the retrieval copy.

• Run the same normaliser on every question; keep the ZWNJ in stored text.

• Keep the lexical side stemmer-free; add trigram similarity for typos.

• Fuse dense and lexical ranks with RRF, k = 60, the lexical list at half weight, knowing that half weight cannot rescue a passage the dense list missed.

• Put the refusal floor on a similarity, never on the RRF score.

• Never filter by locale; let locale decide only exact ties and which translation of a passage is shown.

• Chunk along the document's structure, with the title in every chunk.

• Start with an exact scan, and check any approximate index against it.

• Evaluate on a golden set in every language you serve, with cross-language and no-evidence questions.

• Test embedding models on your own Persian questions, colloquial ones included.

If you are planning retrieval over Persian or mixed-language content, Project Diagnostics turns five questions into a first-pass architecture.

Frequently asked questions

Is BM25 enough for Persian search?

Not on its own. BM25 catches exact names, numbers and codes but misses paraphrase, and a colloquial Persian question can share little wording with a formal document. On MIRACL's Persian set, published dense results sit well above BM25, whether BM25 uses subword tokens or Lucene's language analysers. Use BM25, or whole-word coverage, as the lexical half of a hybrid.

Should I translate Persian queries into English before retrieval?

Not by default. A multilingual embedding model already maps both languages into one vector space, and translation adds a model call that can mangle names and exact terms. The cross-lingual benchmark BGE-M3's authors used includes Arabic but not Persian, so test both approaches on your own Persian–English pairs.

Which embedding model is best for Persian RAG?

Benchmarks give a shortlist, not an answer: MIRACL results cover BGE-M3 and multilingual E5 on Persian, FaMTEB spans 63 datasets, and Hakim's authors report gains on FaMTEB. Test the shortlist on your own colloquial and cross-lingual questions, after checking the licence and vector dimension.

Do I need a separate vector database for Persian RAG?

Not to start. pgvector in your existing PostgreSQL performs exact nearest-neighbour search by default, with perfect recall, and keeps filters such as publication state inside the SQL query. Add an approximate index only when latency requires it.

Why does my Persian search miss a word that is plainly in the text?

Usually because the strings differ in a way the eye cannot see: Arabic Yeh or Kaf (U+064A, U+0643) against Persian ی or ک, a ZWNJ written as a space or left out, a different digit set, or a stray diacritic. Normalise stored passages and questions with the same function, and fold digits and diacritics on both sides of the comparison.

Sources

• G. V. Cormack, C. L. A. Clarke and S. Büttcher, "Reciprocal Rank Fusion outperforms Condorcet and individual Rank Learning Methods", SIGIR 2009.

• Chen, Xiao, Zhang, Luo, Lian and Liu, "M3-Embedding: Multi-Linguality, Multi-Functionality, Multi-Granularity Text Embeddings Through Self-Knowledge Distillation", Findings of ACL 2024.

• BAAI, model card for BGE-M3.

• Zhang et al., "MIRACL: A Multilingual Retrieval Dataset Covering 18 Diverse Languages", TACL 2023 (PDF with the baseline table), and the MIRACL project page.

• Zinvandi et al., "FaMTEB: Massive Text Embedding Benchmark in Persian Language", Findings of EMNLP 2025.

• Sarmadi et al., "Hakim: Farsi Text Embedding Model", arXiv preprint, 2025.

• pgvector README, pgvector sparse vector source and pgvector-python README.

• Snowball stemming algorithms.

• PostgreSQL documentation: text search introduction, dictionaries, psql support, parsers, testing and debugging text search and pg_trgm.

• PostgreSQL source (REL_18_STABLE): snowball_create.pl and pg_ts_dict.dat.

• W3C, Arabic & Persian Layout Requirements, Group Draft Note, 2 October 2025.

• The Unicode Standard 18.0, Chapter 23: Special Areas and Format Characters, and the Unicode Character Database.

• hazm, a Persian NLP toolkit.

• Tajalli, Shamsfard and Kalantari, "Developing an Informal-Formal Persian Corpus: Highlighting the Differences between Two Writing Styles", AbjadNLP 2025.

• Anthropic, "Introducing Contextual Retrieval", 19 September 2024.