One of the things that the
GIST
family of models fell short on is the performance on retrieval tasks. We propose a method that produces improved retrieval performance while maintaining independence on crafting arbitrary instructions, a trending paradigm in embedding models for retrieval tasks, when encoding a query.
Technical details of the model will be published shortly.
Usage
from typing importUnionimport torch
import torch.nn.functional as F
from transformers import AutoModel, AutoTokenizer
model = AutoModel.from_pretrained("avsolatorio/NoInstruct-small-Embedding-v0")
tokenizer = AutoTokenizer.from_pretrained("avsolatorio/NoInstruct-small-Embedding-v0")
defget_embedding(text: Union[str, list[str]], mode: str = "sentence"):
model.eval()
assert mode in ("query", "sentence"), f"mode={mode} was passed but only `query` and `sentence` are the supported modes."ifisinstance(text, str):
text = [text]
inp = tokenizer(text, return_tensors="pt", padding=True, truncation=True)
with torch.no_grad():
output = model(**inp)
# The model is optimized to use the mean pooling for queries,# while the sentence / document embedding uses the [CLS] representation.if mode == "query":
vectors = output.last_hidden_state * inp["attention_mask"].unsqueeze(2)
vectors = vectors.sum(dim=1) / inp["attention_mask"].sum(dim=-1).view(-1, 1)
else:
vectors = output.last_hidden_state[:, 0, :]
return vectors
texts = [
"Illustration of the REaLTabFormer model. The left block shows the non-relational tabular data model using GPT-2 with a causal LM head. In contrast, the right block shows how a relational dataset's child table is modeled using a sequence-to-sequence (Seq2Seq) model. The Seq2Seq model uses the observations in the parent table to condition the generation of the observations in the child table. The trained GPT-2 model on the parent table, with weights frozen, is also used as the encoder in the Seq2Seq model.",
"Predicting human mobility holds significant practical value, with applications ranging from enhancing disaster risk planning to simulating epidemic spread. In this paper, we present the GeoFormer, a decoder-only transformer model adapted from the GPT architecture to forecast human mobility.",
"As the economies of Southeast Asia continue adopting digital technologies, policy makers increasingly ask how to prepare the workforce for emerging labor demands. However, little is known about the skills that workers need to adapt to these changes"
]
# Compute embeddings
embeddings = get_embedding(texts, mode="sentence")
# Compute cosine-similarity for each pair of sentences
scores = F.cosine_similarity(embeddings.unsqueeze(1), embeddings.unsqueeze(0), dim=-1)
print(scores.cpu().numpy())
# Test the retrieval performance.
query = get_embedding("Which sentence talks about concept on jobs?", mode="query")
scores = F.cosine_similarity(query, embeddings, dim=-1)
print(scores.cpu().numpy())
Support for the Sentence Transformers library will follow soon.
Runs of dansporici NoInstruct-small-Embedding-v0-doc-cls on huggingface.co
139
Total runs
25
24-hour runs
30
3-day runs
46
7-day runs
52
30-day runs
More Information About NoInstruct-small-Embedding-v0-doc-cls huggingface.co Model
More NoInstruct-small-Embedding-v0-doc-cls license Visit here:
NoInstruct-small-Embedding-v0-doc-cls huggingface.co is an AI model on huggingface.co that provides NoInstruct-small-Embedding-v0-doc-cls's model effect (), which can be used instantly with this dansporici NoInstruct-small-Embedding-v0-doc-cls model. huggingface.co supports a free trial of the NoInstruct-small-Embedding-v0-doc-cls model, and also provides paid use of the NoInstruct-small-Embedding-v0-doc-cls. Support call NoInstruct-small-Embedding-v0-doc-cls model through api, including Node.js, Python, http.
NoInstruct-small-Embedding-v0-doc-cls huggingface.co is an online trial and call api platform, which integrates NoInstruct-small-Embedding-v0-doc-cls's modeling effects, including api services, and provides a free online trial of NoInstruct-small-Embedding-v0-doc-cls, you can try NoInstruct-small-Embedding-v0-doc-cls online for free by clicking the link below.
dansporici NoInstruct-small-Embedding-v0-doc-cls online free url in huggingface.co:
NoInstruct-small-Embedding-v0-doc-cls is an open source model from GitHub that offers a free installation service, and any user can find NoInstruct-small-Embedding-v0-doc-cls on GitHub to install. At the same time, huggingface.co provides the effect of NoInstruct-small-Embedding-v0-doc-cls install, users can directly use NoInstruct-small-Embedding-v0-doc-cls installed effect in huggingface.co for debugging and trial. It also supports api for free installation.
NoInstruct-small-Embedding-v0-doc-cls install url in huggingface.co:
