Semantic text field typeedit
This functionality is in beta and is subject to change. The design and code is less mature than official GA features and is being provided as-is with no warranties. Beta features are not subject to the support SLA of official GA features.
The semantic_text
field type automatically generates embeddings for text
content using an inference endpoint.
The semantic_text
field type specifies an inference endpoint identifier that will be used to generate embeddings.
You can create the inference endpoint by using the Create inference API.
This field type and the semantic
query type make it simpler to perform semantic search on your data.
Using semantic_text
, you won’t need to specify how to generate embeddings for
your data, or how to index it. The inference endpoint automatically determines
the embedding generation, indexing, and query to use.
PUT my-index-000001 { "mappings": { "properties": { "inference_field": { "type": "semantic_text", "inference_id": "my-elser-endpoint" } } } }
Parameters for semantic_text
fieldsedit
-
inference_id
- (Required, string) Inference endpoint that will be used to generate the embeddings for the field. Use the Create inference API to create the endpoint.
Inference endpoint validationedit
The inference_id
will not be validated when the mapping is created, but when documents are ingested into the index.
When the first document is indexed, the inference_id
will be used to generate underlying indexing structures for the field.
Automatic text chunkingedit
Inference endpoints have a limit on the amount of text they can process.
To allow for large amounts of text to be used in semantic search, semantic_text
automatically generates smaller passages if needed, called chunks.
Each chunk will include the text subpassage and the corresponding embedding generated from it. When querying, the individual passages will be automatically searched for each document, and the most relevant passage will be used to compute a score.
semantic_text
structureedit
Once a document is ingested, a semantic_text
field will have the following structure:
"inference_field": { "text": "these are not the droids you're looking for", "inference": { "inference_id": "my-elser-endpoint", "model_settings": { "task_type": "sparse_embedding" }, "chunks": [ { "text": "these are not the droids you're looking for", "embeddings": { (...) } } ] } }
The field will become an object structure to accommodate both the original text and the inference results. |
|
The |
|
Model settings, including the task type and dimensions/similarity if applicable. |
|
Inference results will be grouped in chunks, each with its corresponding text and embeddings. |
Refer to this tutorial to learn more about
semantic search using semantic_text
and the semantic
query.
Customizing semantic_text
indexingedit
semantic_text
uses defaults for indexing data based on the inference endpoint
specified. It enables you to quickstart your semantic search by providing
automatic inference and a dedicated query so you don’t need to provide further
details.
In case you want to customize data indexing, use the
sparse_vector
or dense_vector
field
types and create an ingest pipeline with an
inference processor to generate the embeddings.
This tutorial walks you through the process.
Updates to semantic_text
fieldsedit
Updates that use scripts are not supported when the index contains a semantic_text
field.
copy_to
supportedit
The semantic_text
field type can be the target of
copy_to
fields. This means you can use a single semantic_text
field to collect the values of other fields for semantic search. Each value has
its embeddings calculated separately; each field value is a separate set of chunk(s) in
the resulting embeddings.
This imposes a restriction on bulk updates to documents with semantic_text
.
In bulk requests, all fields that are copied to a semantic_text
field must have a value to ensure every embedding is calculated correctly.