Overview of Word Embedding using Embeddings from Language Models (ELMo)

Word embeddings are numerical representations of words that enable machine learning and deep learning models to process and understand natural language. Early embedding techniques such as Word2Vec and GloVe represent each word using a single fixed vector regardless of the context in which it appears.

• To address this limitation, ELMo (Embeddings from Language Models) introduced contextual word embeddings, where the representation of a word changes depending on its surrounding words.
• By leveraging a deep bidirectional language model, ELMo captures both syntactic and semantic information, producing richer and more context-aware embeddings.

Word2Vec vs ELMo vs BERT

Working of ELMo

Step 1: Character-based Word Representation

• Instead of directly assigning a fixed vector to each word, ELMo first represents every word as a sequence of characters.
• A Character Convolutional Neural Network (Character CNN) extracts features such as prefixes, suffixes, and word morphology to create an initial word representation.
• This enables ELMo to effectively handle rare and out-of-vocabulary words.

Step 2: Bidirectional Language Modelling

The character-based representations are then passed through a Bidirectional Language Model (biLM), which consists of two LSTM networks working in opposite directions, as illustrated in the figure.

• The Forward LSTM reads the sentence from left to right and learns the context by predicting the next word.
• The Backward LSTM reads the sentence from right to left and learns the context by predicting the previous word.

Step 3: Multi-layer Contextual Embeddings

• ELMo does not rely only on the final LSTM output.
• Instead, it combines representations from multiple layers of the bidirectional language model, including the character-based embedding layer and the hidden states from each BiLSTM layer.
• Lower layers generally capture syntactic information, while higher layers capture semantic meaning.
• A weighted combination of these layers forms the final ELMo embedding.

Step 4: Integration with Downstream Tasks

• The generated ELMo embeddings are used as input features for downstream NLP models such as text classifiers, named entity recognition systems, question answering models, and sentiment analysis models.
• During training, the downstream model learns how much importance to assign to each ELMo layer, while the pretrained language model can either remain fixed or be fine-tuned depending on the application.

Implementation of ELMo Embeddings

In this implementation, we use TensorFlow and TensorFlow Hub to load the pretrained ELMo model and generate contextualized word embeddings for input sentences.

Step 1: Install Required Libraries

Install TensorFlow and TensorFlow Hub, which provide the pretrained ELMo model and the APIs required to generate contextual embeddings.

pip install tensorflow tensorflow_hub

Step 2: Import Libraries and Load ELMo

• Import TensorFlow for tensor operations and TensorFlow Hub for loading the pretrained ELMo model.
• tensorflow is imported as tf to create tensors and execute the model.
• tensorflow_hub is imported as hub to download and load the pretrained ELMo model.

import tensorflow as tf
import tensorflow_hub as hub

Step 3: Load the Pretrained ELMo Model

• Load the pretrained ELMo model directly from TensorFlow Hub.
• hub.load() downloads and loads the pretrained ELMo model.
• The loaded model generates 1024-dimensional contextual embeddings for every word in the input sentence.

elmo = hub.load("https://tfhub.dev/google/elmo/3")

Step 4: Define a Function to Generate ELMo Embeddings

• Create a function that accepts one or more sentences and returns their contextual word embeddings.
• tf.constant() converts the sentences into TensorFlow tensors.
• elmo.signatures["default"] passes the sentences through the pretrained model.
• ["elmo"] extracts the contextual embedding tensor produced by the model.

def get_elmo_embeddings(sentences):
    embeddings = elmo.signatures["default"](
        tf.constant(sentences)
    )["elmo"]
    return embeddings

Step 5: Create Sample Sentences

• Create two sentences containing the word bank used in different contexts.
• The word bank refers to a financial institution in the first sentence and the edge of a river in the second.
• ELMo generates different embeddings for the word based on its surrounding context.

sentences = [
    "The bank approved the loan.",
    "He sat on the bank of the river."
]

Step 6: Generate and Display the Embeddings

The output tensor has three dimensions:

• 2 → Number of input sentences.
• 7 → Maximum number of words (after padding) among the input sentences.
• 1024 → Size of the contextual embedding generated for each word.
• Although the word "bank" appears in both sentences, ELMo produces different embedding vectors because it considers the surrounding words while generating the representation.

embeddings = get_elmo_embeddings(sentences)

print("Embedding Shape:", embeddings.shape)

Output:

Embedding Shape: (2, 8, 1024)

You can download the complete code form here.

Applications

1. Sentiment Analysis: Identifies the sentiment of words based on their context, improving opinion classification.
2. Named Entity Recognition (NER): Distinguishes entities such as people, organizations, and locations using contextual information.
3. Question Answering: Produces more accurate answers by understanding the meaning of words within the given passage.
4. Machine Translation: Improves translation quality by selecting the contextually appropriate meaning of ambiguous words.
5. Text Classification: Enhances document and intent classification by generating richer contextual representations.
6. Information Retrieval: Improves search relevance by matching documents based on contextual meaning rather than exact keywords.

Advantages

• Generates different embeddings for the same word based on its surrounding text.
• Correctly distinguishes words with multiple meanings in different contexts.
• Learns representations from characters, making it effective for rare and unseen words.
• Combines information from multiple BiLSTM layers to learn both grammatical and semantic features.
• Can be added as pretrained embeddings to existing NLP models with minimal architectural changes.

Limitations

• BiLSTM layers process tokens sequentially, making training and inference slower than Transformer models.
• Requires more memory and processing time than static embedding methods such as Word2Vec.
• LSTMs may struggle to capture dependencies across very long sentences.
• The 1024-dimensional embeddings increase storage requirements and model size.
• Modern Transformer-based models like BERT and RoBERTa generally achieve better performance on most NLP benchmarks.