Full Text Search
Use for product search: FtsSearchRepo, pgsearch/pglike/Elasticsearch, mapping migrations, projections, Outbox sync, reindexing and performance. Excludes agent web search.
mkdir -p ~/.claude/skills/full-text-search && curl -fsSL 'https://raw.githubusercontent.com/lobehub/lobehub/canary/.agents/skills/full-text-search/SKILL.md' -o ~/.claude/skills/full-text-search/SKILL.md
Get the complete skill folder, including its scripts and reference files.
When to use this skill
Use Full Text Search when an AI agent needs a reusable SKILL.md workflow for this job: Use for product search: FtsSearchRepo, pgsearch/pglike/Elasticsearch, mapping migrations, projections, Outbox sync, reindexing and performance. Excludes agent web search.
When not to use it
Skip Full Text Search when the task is outside the coding category, or when a more specific skill in this directory already covers the same workflow with clearer triggers.
How to install
Where will you use it?
mkdir -p ~/.claude/skills/full-text-search && curl -fsSL 'https://raw.githubusercontent.com/lobehub/lobehub/canary/.agents/skills/full-text-search/SKILL.md' -o ~/.claude/skills/full-text-search/SKILL.md
- Claude Code personal install: place the skill and its bundled files at ~/.claude/skills/full-text-search/SKILL.md. For a shared project, commit the folder at .claude/skills/full-text-search/ instead.
- Current Claude Code detects SKILL.md changes in watched directories during the session. If you just created a top-level skills directory, run /reload-skills. Check that the skill is listed before testing its trigger.
- For Claude chat or Cowork in the Desktop app, manage uploaded skills in Customize > Skills; copying files into a local Claude Code folder is not a Desktop install. Other agents may use different install locations.
Full install guide for Claude, Cursor, and Codex →
Try a starter prompt
After installing the skill, paste this starter prompt into your agent to verify it triggers on a small task.
Use the Full Text Search skill for this task: Use Full Text Search when this documented workflow matches the task. Confirm any required inputs or files before making changes.
What supports this recommendation?
71/100 provisional content score. This measures documentation, practical detail and source signals. It is not a measured task success rate or a security certification.
- Documentation
- 77/100
- Practical detail
- 68/100
- Source evidence
- 63/100
- Source signals
- 72/100
Model compatibility evidence
- claudeuntested
No model-specific signal or recorded compatibility test was found.
- chatgptuntested
No model-specific signal or recorded compatibility test was found.
- geminiuntested
No model-specific signal or recorded compatibility test was found.
- copilotuntested
No model-specific signal or recorded compatibility test was found.
- llamauntested
No model-specific signal or recorded compatibility test was found.
- perplexityuntested
No model-specific signal or recorded compatibility test was found.
- mistraluntested
No model-specific signal or recorded compatibility test was found.
- grokuntested
No model-specific signal or recorded compatibility test was found.
Inferred compatibility is based on content signals. Check the app, its available tools and setup requirements before running a skill.
What this skill does
# Product Full-Text Search
This skill covers search over LobeHub-owned product data such as agents, topics, messages, files, knowledge bases, documents, chat groups, and memories. It does not cover the agent's external web search providers under `apps/server/src/services/search/` or builtin web-browsing tools.
## Architecture
Product reads follow one stable path:
```text router/service -> createFtsSearchRepo -> FtsSearchRepo -> selected backend -> existing result schema ```
- `apps/server/src/services/ftsSearch/` owns request-scoped provider selection, Elasticsearch configuration, its HTTP client, and backend telemetry. Routers and domain services must call `createFtsSearchRepo`; they must not construct providers themselves. - `packages/database/src/repositories/ftsSearch/` owns the provider-neutral contract and the concrete PostgreSQL and Elasticsearch implementations. Keep public result shapes stable in `FtsSearchRepo`. - `packages/types/src/ftsSearch.ts` owns the shared searchable-entity list and search domain types. - `packages/database/src/repositories/ftsSearchDocument/` owns Elasticsearch document schemas, mappings, queryable fields, and source-to-document projection. - `packages/database/src/schemas/ftsSearchSyncOutbox.ts` and `packages/database/src/repositories/ftsSearchSyncOutbox/` own durable change capture, claims, retries, dead letters, leases, revision fences, and capture-definition validation. - `scripts/elasticsearchReindex/` owns the resumable full-backfill command and its operational runtime. Shared database source queries and document construction remain in `packages/database/src/repositories/ftsSearchDocument/`. `apps/server/src/services/ftsSearchSync/` and `scripts/elasticsearchSync/` own continuous incremental draining. - `packages/env/src/ftsSearch.ts` owns generic Elasticsearch environment variables.
## Provider and Permission Invariants
- `FTS_SEARCH_PROVIDER` is a deployment-level provider selector with current values `pg_search`, `elasticsearch`, and `pg_like`. It is not a feature flag or a user rollout. Add another enum value only when its provider covers all entities in `FTS_SEARCH_BACKEND_ENTITIES` end to end, including `mode: 'candidates'` when the provider enables `ftsSearchCandidateEnabled`. - Provider errors, missing configuration, and unsupported candidate behavior must remain visible. Never silently retry through another provider. `pg_like` is an explicitly selected lightweight provider, not an implicit fallback: a missing extension or unreachable service must still fail. - `packages/database/src/repositories/ftsSearch/postgres/` owns the query, permission, pagination, and hydration layer shared by PostgreSQL providers. `pgSearch/` and `pgLike/` own only their provider adapter, dialect, and provider-specific candidate behavior. Neither provider directory may import the other, so either provider can be retired without owning shared functionality. - Before selecting Elasticsearch, require coverage tests proving that it supports every entity in the provider-neutral backend contract. Do not add per-entity routing between providers. - Preserve `userId`, `workspaceId`, and caller-agent visibility throughout every provider. Candidate retrieval must not broaden the caller's scope. - Where Elasticsearch only supplies candidate IDs, PostgreSQL hydration and parent checks remain the authoritative permission boundary. Do not return raw Elasticsearch hits directly when the existing result path requires database authorization or hydration. - Provider implementations return the existing hydrated response types and ordering contract. Do not make routers understand provider-specific result shapes.
## Changing a Searchable Entity
### Scope for pg_like
`pg_like` primarily serves individual users with small datasets. Review it for correct matching, permissions, and usable result ordering at that scale. For richer search quality or larger datasets, recommend self-hosted Elasticsearch or Elastic Cloud. Do not add indexes or change the database schema as part of pg_like optimization. Keep repairs bounded; distinguish new implementation bugs from shared provider limitations and deliberate lightweight-search trade-offs. Do not infer a personal-user performance problem from large shared development datasets without representative measurements.
### Entity changes
Treat an entity addition or projection change as one cross-layer change. Inspect and update every applicable item:
1. `FTS_SEARCH_DOCUMENT_ENTITIES` and shared request/result types. 2. Search document schema, mapping, text fields, filters, fixtures, and mapping parity tests. 3. `FtsSearchDocumentBuilder`, including soft deletion and fanout from related source rows. 4. PostgreSQL and Elasticsearch backend behavior, permission hydration, pagination, and ranking. 5. Capture functions/triggers or fanout queries in `captureInfrastructure.ts`. 6. Reindex checkpoints, Outbox draining, metrics, and self-host documentation.
Schema fields, mappings, builders, and fixed fixtures must agree. Current mapping fields plus explicit `FTS_SEARCH_RETAINED_SOURCE_PROPERTIES` must equal the current Zod fields, without overlap. Retained source properties keep older open indexes writable during field removals; keep the builder producing them until those indexes are closed. They must not appear in current query metadata. A field outside the document schema must not appear in a mapping or query field list.
Elasticsearch `multi_match` query length is bounded by a shared leaf-clause budget divided by the selected query-field count. Adding a field reduces that entity's safe query length, and changing a query analyzer to emit multiple terms per Unicode code point requires revisiting the budget and its field-count regression tests.
## Capture, Reindex, and Sync
- Regular database migrations own durable schema: the revision sequence, Outbox table and indexes, and schema-managed source indexes such as the Memory fanout GIN index. - Capture functions and triggers are installed only when an operator explicitly enables the Elasticsearch path. `installCaptureInfrastructure()` is transactional, definition-checked, idempotent for an exact installation, and fail-closed for partial or altered definitions. - Do not install capture for every PostgreSQL-only instance. Do not move a normal schema index into the runtime installer merely because it supports capture. - The Outbox coalesces by `(entity, document_id)`. A newer capture resets retry/dead-letter state and allocates a new revision only after locking the conflicting row, preserving same-document commit order. - Sequence allocation is non-transactional. Use the existing write fences and committed revision boundary; never treat `last_value` alone as proof that all earlier Outbox rows are visible. - Claims use the precise lease timestamp as a fencing token. A stale worker must not acknowledge, fail, or release work reclaimed by another worker. - Permanent failures and exhausted retries become durable dead letters. A drain that creates or observes dead work must fail instead of continuing to publish a successful cutover signal. - Full backfill does not replace continuous sync. For the initial PostgreSQL-to-Elasticsearch cutover, keep the application on PostgreSQL until aliases are ready and the Outbox is empty and stable, then switch explicitly. Later mapping upgrades keep searches on the existing Elasticsearch alias until promotion.
The supported operator entrypoints below run from the OSS repository root. A wrapping repository may expose different package scripts; check its `package.json` before invoking these or the mapping migration commands linked below.
```bash bun run db:install-fts-search-capture bun run fts-search:reindex -- --status bun run fts-search:reindex -- --apply --yes bun run fts-search:sync -- --max-steps=8 --yes bun run scripts/pgSearchCleanup/index.ts --status bun run scripts/pgSearchCleanup/index.ts --apply --yes ```
## Mapping Generations
Docker startup runs `fts-search-elasticsearch-reindex.cjs --startup --yes` after PostgreSQL migrations when the selected provider is Elasticsearch. It blocks serving until index migration and catch-up succeed. Persist the same checkpoint volume across application and migration containers; failed namespace locks still require explicit recovery. Continuous sync remains a separate worker. Hosted and manual workflows keep their explicit migration commands.
For mapping changes, generation operations, repeat/resume behavior, or deployment integration, read [Mapping migration workflow](references/mapping-migrations.md). It routes to the public command guide and adds recovery, automation, and local Docker rehearsal rules.
- Treat managed and Serverless Elasticsearch APIs as capability-constrained. Create operational metadata and control indexes with the smallest portable request, and add topology, storage, or index settings only after verifying that the target service supports them; settings accepted by a self-managed cluster may be rejected by a managed service. - Make idempotent Elasticsearch resource creation depend on the structured error type, such as `resource_already_exists_exception`, not a broad HTTP status range. Preserve any other creation failure and stop before follow-up reads or mutations instead of turning a rejected request into a misleading verification error.
Elasticsearch cannot change an existing field's type or index-time analyzer in place. The code declares the target and Elasticsearch records the live state, per entity:
- `FTS_SEARCH_INDEX_DEFINITIONS[entity].schemaVersion` is the declared generation; the fingerprint is `sha256` of the mapping plus the shared analysis. Add complete changed-entity definitions in a new `ftsSearchDocument/migration/NNNN-meaningful-name/` batch, register it and update current pointers in `migration/index.ts`. Append expected fingerprints in `__tests__/schemaSnapshots.json`; preserve published batches and baselines. `mappings.test.ts` and `migration/index.test.ts` check current parity, history, version bumps and fingerprint changes. See `migration/README.md` for ownership. Shared analysis changes alter every entity fingerprint and classify as breaking for every entity; bump all affected versions and rebuild rather than using in-place upgrades. - Every physical index `<alias>-v<n>` or same-schema rebuild `<alias>-v<n>-r<runId>` carries `_meta.{reindex_run_id, schema_version, schema_fingerprint}`; the alias marks the live generation. A rebuild suffix is physical identity, not a schema-version bump. Indexes created before fingerprints existed may omit the fingerprint, but still need a valid run ID and schema version for sync readiness. `_meta.schema_version` wins over the `-v<n>` suffix because an in-place upgrade advances `_meta` without renaming the index. - The sync runtime accepts an alias that still serves an older generation (upgrade in progress) and refuses one that serves a newer generation or a different fingerprint of the declared version. It writes every change to all open `<alias>-v*` indexes plus the alias write index, pruning each document to the fields that index maps (every generation is `dynamic: strict`), so a new generation can be backfilled beside the live one; a bulk work item is acknowledged only when every existing generation accepted it (2xx or 409 conflict). - Canonical migration checkpoints remain keyed by `(namespace, schemaVersion)`. Explicit current-version rebuild checkpoints add `reindexRunId`, so repeated v1 rebuilds never reuse a completed v1 cursor. `--apply` groups the requested entities by declared version, treats existing aliases as an upgrade (no `--fresh-run`), leaves existing aliases in place, and emits `promotion_pending`. A completed first install creates aliases. Promoting a newer generation requires a completed checkpoint, a fingerprint match when targeting the declared version, and an idle target-entity Outbox. Rollback to an older stamped generation can use its metadata without a retained checkpoint; `--retire` requires `in_sync` and only closes old generations; explicit `--purge` installs exact-index templates forbidding auto-creation before deleting eligible closed generations. `--in-place` requires `mappingChange: additive`, widens the live index with `PUT _mapping`, pins the checkpoint to that index, and backfills with `external_gte` so concurrent sync writes win. - Use `--apply --rebuild-current --entity=<entity> --yes` when projection or capture semantics changed without a physical mapping change and historical documents must be regenerated from PostgreSQL. Deploy runtime support first: sync must recognize `-r<runId>` generations before one is created. Resume with the reported `--run-id=<uuid>`. Promote with the exact `--generation=<alias>-v<n>-r<uuid>` because version-only selection is ambiguous. The old generation remains dual-written for rollback and becomes same-version retirement-eligible only after the promotion records which live run superseded it. - Checkpoints are local files, not Drizzle migration history. Preserve `ES_REINDEX_STATE_DIR` across invocations. Completed runs skip backfill; incomplete runs resume from saved cursors. Mutating CLI commands share a non-expiring Elasticsearch namespace lock, independent of checkpoint location. A failed command retains its lock when its outcome may be uncertain. Before `--release-lock=<owner> --yes`, stop the previous process and resolve pending requests; never assume age proves it stopped. The lock does not serialize old binaries or external operator actions. - Reconciliation is exact only on a first install; once an alias serves an entity, concurrent sync writes make a higher Elasticsearch count legitimate and only a shortfall fails. - `scripts/elasticsearchReindex/runtime/generationService.ts` owns classification (`missing`, `unmanaged`, `in_sync`, `drift`, `upgrade_available`, `rollback_required`), promotion, and retirement; keep them free of Cloud-specific policy.
Read `docs/self-hosting/advanced/neon-pg-search-migration.mdx` or its Chinese counterpart before changing the operational sequence. When database rollout or index cost affects the design, also use the `db-migrations` skill and measure the relevant operation on the actual Dev database before adding manual or deferred release steps.
## Observability and Product Analytics
- Server provider metrics and traces live in `apps/server/src/services/ftsSearch/observability.ts`. Keep labels bounded: entity, provider, operation, outcome, and coarse error type are acceptable; raw queries, user IDs, document IDs, and index contents are not. - User-perceived search behavior lives with the open-source Command Menu in `src/features/CommandMenu/analytics.ts`. Product analytics may cover end-to-end duration, rendered result counts, empty results, result clicks, and abandonment without a Cloud business slot. - Measurement or analytics failures must never alter the selected provider's result or error. - Backend metrics explain provider cost and latency; frontend events explain what the user actually experienced. Do not substitute one for the other.
## Validation
- Add or update focused tests beside every changed provider, mapping, builder, capture, sync, or analytics module. - Preserve boundary tests proving callers use `FtsSearchRepo`, providers cannot leak raw result shapes, and telemetry failures do not affect search behavior. - Test pagination after authorization drops candidates, delete/scope-change priority, concurrent Outbox claims and stale settlements, retry/dead-letter behavior, reindex resume identity, and capture-definition mismatch whenever those paths change. - Run `bun run check <changed-files...>` from the repository root. For migration or database-runtime changes, follow the `db-migrations` and `testing` skills and verify against the actual Dev database.
Intended uses
- Use Full Text Search when this documented workflow matches the task.
Implementation notes
- Review the source instructions and adapt inputs before running the workflow.
Try it, then tell us how it went
A copied prompt does not tell us whether the task worked. Report your result after checking the output. We collect your tool and outcome, never your files or prompt.