Text representation

The text module provides the building blocks for converting raw text into numerical representations – from tokenisation to contextual embeddings and document-level vectors. These encoders are used by the neural models in Neural models and can be composed freely.

HuggingFace Transformers

Base encoders

Abstract interfaces for encoders and token-level encoders.

XPM Configxpmir.text.encoders.Encoder[source]

Bases: Module, EasyLogger, ABC

Base class for all word and text encoders

XPM Configxpmir.text.encoders.TokensEncoder[source]

Bases: Tokenizer, Encoder

(deprecated) Represent a text as a sequence of token representations

forward(tokenized: TokenizedTexts)[source]

Returns embeddings for the tokenized texts.

tokenized: tokenized texts

Tokenizers

Tokenizers split text into token sequences and manage the vocabulary mapping.

XPM Configxpmir.text.tokenizers.Tokenizer[source]

Bases: Config

Represents a vocabulary and a tokenization method

Deprecated: Use TokenizerBase instead

batch_tokenize(texts: List[str], batch_first=True, maxlen=None, mask=False) → TokenizedTexts[source]

Returns tokenized texts

Parameters:: mask – Whether a mask should be computed

id2tok(idx: int) → str[source]: Converts an integer id to a token

lexicon_size() → int[source]: Returns the number of items in the lexicon

tok2id(tok: str) → int[source]: Converts a token to an integer id

XPM Configxpmir.text.tokenizers.TokenizerBase[source]

Bases: Config, Initializable, Generic[TokenizerInput, TokenizerOutput], ABC

Base tokenizer

Text encoders

Encoders that map a text string (or a pair of texts) to a dense representation.

XPM Configxpmir.text.encoders.TextEncoderBase[source]

Bases: Module, Generic[InputType, EncoderOutput]

Base class for all text encoders

XPM Configxpmir.text.encoders.TextEncoder[source]

Bases: TextEncoderBase

Encodes a text into a vector

Deprecated since version 1.3: Use TextEncoderBase directly

XPM Configxpmir.text.encoders.DualTextEncoder[source]

Bases: TextEncoderBase

Encodes a pair of text into a vector

Deprecated since version 1.3: Use TextEncoderBase directly

XPM Configxpmir.text.encoders.TripletTextEncoder[source]

Bases: TextEncoderBase

Encodes a triplet of text into a vector

Deprecated since version 1.3: Use TextEncoderBase directly

This is used in models such as DuoBERT where we encode (query, positive, negative) triplets.

XPM Configxpmir.text.encoders.TokenizedTextEncoderBase[source]: Bases: TextEncoderBase

XPM Configxpmir.text.encoders.TokenizedEncoder[source]

Bases: Encoder, Generic[EncoderOutput, TokenizerOutput]

Encodes a tokenized text into a vector

Tokenizer-based encoders

XPM Configxpmir.text.encoders.TokenizedTextEncoder(*, tokenizer, encoder)[source]

Bases: TokenizedTextEncoderBase, Generic[InputType, EncoderOutput, TokenizerOutput]

Encodes a tokenizer input into a vector

This pipelines two objects:

A tokenizer that segments the text;
An encoder that returns a representation of the tokens in a vector space

tokenizer: xpmir.text.tokenizers.TokenizerBase[InputType, TokenizerOutput]

encoder: xpmir.text.encoders.TokenizedEncoder

Adapters

Adapters transform or aggregate encoder outputs.

XPM Configxpmir.text.adapters.MeanTextEncoder(*, encoder)[source]

Bases: TokenizedTextEncoderBase

Returns the mean of the word embeddings

encoder: xpmir.text.encoders.TokenizedTextEncoderBase

XPM Configxpmir.text.adapters.TopicTextConverter[source]

Bases: Converter[TextRecord, str]

Extracts the text from a topic