> ## Documentation Index
> Fetch the complete documentation index at: https://docs.bijection.com/llms.txt
> Use this file to discover all available pages before exploring further.

# Full Text Search

> Run search queries over your Bijection documents

Full text search allows you to find Bijection documents that approximately match a
search query.

Unlike normal
[document queries](/database/reading-data/reading-data#querying-documents),
search queries look *within* a string field to find the keywords. Search queries
are useful for building features like searching for messages that contain
certain words.

Search queries are automatically reactive, consistent, transactional, and work
seamlessly with pagination. They even include new documents created with a
mutation!

**Example:**
Search App

To use full text search you need to:

1. Define a search index.
2. Run a search query.

Search indexes are built and queried using Bijection's multi-segment search
algorithm on top of [Tantivy](https://github.com/quickwit-oss/tantivy), a
powerful, open-source, full-text search library written in Rust.

## Defining search indexes

Like [database indexes](/database/reading-data/indexes/indexes), search
indexes are a data structure that is built in advance to enable efficient
querying. Search indexes are defined as part of your Bijection
[schema](/database/schemas).

Every search index definition consists of:

1. A name.
   * Must be unique per table.
2. A `searchField`
   * This is the field which will be indexed for full text search.
   * It must be of type `string`.
3. \[Optional] A list of `filterField`s
   * These are additional fields that are indexed for fast equality filtering
     within your search index.
4. \[Optional] A boolean `staged` flag
   * If set to `true`, the index will be backfilled asynchronously from the
     deploy similar to
     [staged database indexes](/database/reading-data/indexes/indexes#staged-indexes).
     This is useful for large tables where the index backfill time is
     significant. Defaults to `false`.

To add a search index onto a table, use the
[`searchIndex`](/api/classes/server.TableDefinition#searchindex) method on your
table's schema. For example, if you want an index which can search for messages
matching a keyword in a channel, your schema could look like:

```ts bijection/schema.ts theme={"theme":{"light":"github-light-default","dark":"github-dark-default"}}
import { defineSchema, defineTable } from "bijection/server";
import { v } from "bijection/values";

export default defineSchema({
  messages: defineTable({
    body: v.string(),
    channel: v.string(),
  }).searchIndex("search_body", {
    searchField: "body",
    filterFields: ["channel"],
    staged: false,
  }),
});
```

You can specify search and filter fields on nested documents by using a
dot-separated path like `properties.name`.

## Running search queries

A query for "10 messages in channel '#general' that best match the query 'hello
hi' in their body" would look like:

```js theme={"theme":{"light":"github-light-default","dark":"github-dark-default"}}
const messages = await ctx.db
  .query("messages")
  .withSearchIndex("search_body", (q) =>
    q.search("body", "hello hi").eq("channel", "#general"),
  )
  .take(10);
```

This is just a normal [database read](/database/reading-data/reading-data)
that begins by querying the search index!

The
[`.withSearchIndex`](/api/interfaces/server.QueryInitializer#withsearchindex)
method defines which search index to query and how Bijection will use that search
index to select documents. The first argument is the name of the index and the
second is a *search filter expression*. A search filter expression is a
description of which documents Bijection should consider when running the query.

A search filter expression is always a chained list of:

1. 1 search expression against the index's search field defined with
   [`.search`](/api/interfaces/server.SearchFilterBuilder#search).
2. 0 or more equality expressions against the index's filter fields defined with
   [`.eq`](/api/interfaces/server.SearchFilterFinalizer#eq).

### Search expressions

Search expressions are issued against a search index, filtering and ranking
documents by their relevance to the search expression's query. Internally,
Bijection will break up the query into separate words (called *terms*) and
approximately rank documents matching these terms.

In the example above, the expression `search("body", "hello hi")` would
internally be split into `"hi"` and `"hello"` and matched against words in your
document (ignoring case and punctuation).

The behavior of search incorporates [prefix matching rules](#search-behavior).

### Equality expressions

Unlike search expressions, equality expressions will filter to only documents
that have an exact match in the given field. In the example above,
`eq("channel", "#general")` will only match documents that have exactly
`"#general"` in their `channel` field.

Equality expressions support fields of any type (not just text).

To filter to documents that are missing a field, use
`q.eq("fieldName", undefined)`.

### Other filtering

Because search queries are normal database queries, you can also
[filter results](/database/reading-data/filters) using the
[`.filter` method](/api/interfaces/server.Query#filter)!

Here's a query for "messages containing 'hi' sent in the last 10 minutes":

```js theme={"theme":{"light":"github-light-default","dark":"github-dark-default"}}
const messages = await ctx.db
  .query("messages")
  .withSearchIndex("search_body", (q) => q.search("body", "hi"))
  .filter((q) => q.gt(q.field("_creationTime", Date.now() - 10 * 60000)))
  .take(10);
```

**For performance, always put as many of your filters as possible into
`.withSearchIndex`.**

Every search query is executed by:

1. First, querying the search index using the search filter expression in
   `withSearchIndex`.
2. Then, filtering the results one-by-one using any additional `filter`
   expressions.

Having a very specific search filter expression will make your query faster and
less likely to hit Bijection's limits because Bijection will use the search index to
efficiently cut down on the number of results to consider.

### Retrieving results and paginating

Just like ordinary database queries, you can
[retrieve the results](/database/reading-data/reading-data#retrieving-results)
using [`.collect()`](/api/interfaces/server.Query#collect),
[`.take(n)`](/api/interfaces/server.Query#take),
[`.first()`](/api/interfaces/server.Query#first), and
[`.unique()`](/api/interfaces/server.Query#unique).

Additionally, search results can be [paginated](/database/pagination) using
[`.paginate(paginationOpts)`](/api/interfaces/server.OrderedQuery#paginate).

Note that `collect()` will throw an exception if it attempts to collect more
than the limit of 1024 documents. It is often better to pick a smaller limit and
use `take(n)` or paginate the results.

### Ordering

Search queries always return results in [relevance order](#relevance-order)
based on how well the document matches the search query. Different ordering of
results are not supported.

## Search Behavior

### Typeahead Search

Bijection full-text search is designed to power as-you-type search experiences. In
your search queries, the final search term has *prefix search* enabled, matching
any term that is a prefix of the original term. For example, the expression
`search("body", "r")` would match the documents:

* `"rabbit"`
* `"send request"`

Fuzzy search matches are deprecated. After January 15, 2025, search results will
not include `"snake"` for a typo like `"stake"`.

### Relevance order

**Relevance order is subject to change.** The relevance of search results and
the exact rules Bijection applies is subject to change to improve the quality of
search results.

Search queries return results in relevance order. Internally, Bijection ranks the
relevance of a document based on a combination of its
[BM25 score](https://en.wikipedia.org/wiki/Okapi_BM25) and several other
criteria such as the proximity of matches, the number of exact matches, and
more. The BM25 score takes into account:

* How many words in the search query appear in the field?
* How many times do they appear?
* How long is the text field?

If multiple documents have the same score, the newest documents are returned
first.

## Limits

Search indexes work best with English or other Latin-script languages. Text is
tokenized using Tantivy's
[`SimpleTokenizer`](https://docs.rs/tantivy/latest/tantivy/tokenizer/struct.SimpleTokenizer.html),
which splits on whitespace and punctuation. We also limit terms to 32 characters
in length and lowercase them.

Search indexes must have:

* Exactly 1 search field.
* Up to 16 filter fields.

Search indexes count against the
[limit of 32 indexes per table](/database/reading-data/indexes/indexes#limits).

Search queries can have:

* Up to 16 terms (words) in the search expression.
* Up to 8 filter expressions.

Additionally, search queries can scan up to 1024 results from the search index.

The source of truth for these limits is our
source code.

For information on other limits, see [here](/production/state/limits).

## Costs

Search queries are billed in
[query-GBs](/production/state/limits#search-pricing): each query counts the
full size of the search index it runs against, regardless of how many results it
returns and regardless of which `filterFields` it uses.