Text Representation

• Huggingface Transformers
  • BaseTransformer
• Encoders
  • TransformerEncoder
  • TransformerTokensEncoder
  • TransformerTextEncoderAdapter
  • DualTransformerEncoder
  • SentenceTransformerTextEncoder
  • OneHotHuggingFaceEncoder
  • DualDuoBertTransformerEncoder
  • TransformerVocab
  • TransformerTokensEncoderWithMLMOutput
• Tokenizers
  • OneHotHuggingFaceEncoder
  • HuggingfaceTokenizer
• Masked-LM
  • MLMEncoder
• Hooks
  • LayerSelector
• Word vectors
  • WordvecVocab
  • WordvecHashVocab
  • WordvecUnkVocab

The text module groups classes and configurations that compute a representation of text – this includes word embeddings as well as contextual word embeddings and document embeddings.

XPM Configxpmir.text.encoders.Tokenizer

Bases: Config

Represents a vocabulary and a tokenization method

batch_tokenize(texts: List[str], batch_first=True, maxlen=None, mask=False) → TokenizedTexts

Returns tokenized texts

Parameters:

mask – Whether a mask should be computed

id2tok(idx: int) → str

Converts an integer id to a token

lexicon_size() → int

Returns the number of items in the lexicon

tok2id(tok: str) → int

Converts a token to an integer id

XPM Configxpmir.text.encoders.TokensEncoder

Bases: Tokenizer, Encoder

Represent a text as a sequence of token representations

forward(tok_texts: TokenizedTexts)

Returns embeddings for the tokenized texts.

tok_texts: tokenized texts

XPM Configxpmir.text.encoders.Encoder

Bases: Module, EasyLogger

Base class for all word and text encoders

XPM Configxpmir.text.encoders.MeanTextEncoder(*, encoder)

Bases: TextEncoder

Returns the mean of the word embeddings

encoder: xpmir.text.encoders.TokensEncoder

XPM Configxpmir.text.encoders.TripletTextEncoder

Bases: Encoder

The generic class for triplet encoders: query-document-document

This encoding is used in models such as DuoBERT that compute whether a pair is preferred to another

XPM Configxpmir.text.encoders.TextEncoder

Bases: Encoder

Vectorial representation of a text - can be dense or sparse

forward(texts: List[str]) → torch.Tensor

Returns a matrix encoding the provided texts

XPM Configxpmir.text.encoders.DualTextEncoder

Bases: Encoder

Vectorial representation for a pair of texts

This is used for instance in the case of BERT models that represent the (query, document couple)

forward(texts: List[Tuple[str, str]]) → torch.Tensor

Computes the representation of a list of pair of texts