Word Embeddings for Natural Language Processing

• 6 Oct 2016
• Natural Language Processing, Word2Vec, FastText

Word embedding is a technique that treats words as vectors whose relative similarities correlate with semantic similarity. This technique is one of the most successful applications of unsupervised learning. Natural language processing (NLP)systems traditionally encode words as strings, which are arbitrary and provide no useful information to the system regarding the relationships that may exist between different words. Word embedding is an alternative technique in NLP whereby words or phrases from the vocabulary are mapped to vectors of real numbers in a low-dimensional space relative to the vocabulary size, and the similarities between the vectors correlate with the words’ semantic similarity.

Word embedding is a technique that treats words as vectors whose relative similarities correlate with semantic similarity.

For example, let’s take the words woman, man, queen, and king. We can get their vector representations and use basic algebraic operations to find semantic similarities. Measuring similarity between vectors is possible using measures such as cosine similarity. So when we subtract the vector of the word man from the vector of the word woman, then its cosine distance would be close to the distance between the word queen minus the word king (see Figure 1).

W("woman")−W("man") ≃ W("queen")−W("king")

Word2Vec

Continuous bag-of-words (CBOW)

In the CBOW architecture, the model predicts the current word from a window of surrounding context words. Mikolov et al. thus use both the n words before and after the target word w to predict it.
A sequence of words is equivalent to a set of items. Therefore, it is also possible to replace the terms word and item, which allows for applying the same method for Collaborative Filtering and recommender systems. CBOW is several times faster to train than the skip-gram model and has slightly better accuracy for the words that appear frequently (see Figure 2).

Predicting the word given its context

Skip-gram

In the skip-gram model, instead of using the surrounding words to predict the center word, it uses the center word to predict the surrounding words (see Figure 3). According to Mikolov et al., skip-gram works well with a small amount of the training data and does a good job of representing even rare words and phrases.

Predicting the context given a word

Coding

Looking to Wikipedia for a big dataset

• Find the ISO 639 code for your desired language: List of ISO 639 codes
• Go to: https://dumps.wikimedia.org/{{LANGUAGE CODE}}wiki/latest/
• Download {{LANGUAGE CODE}}wiki-latest-pages-articles.xml.bz2

pip install --upgrade gensim

from gensim.corpora import WikiCorpus

language_code = "he"
inp = language_code+"wiki-latest-pages-articles.xml.bz2"
outp = "wiki.{}.text".format(language_code)
i = 0

print("Starting to create wiki corpus")
output = open(outp, 'w')
space = " "
wiki = WikiCorpus(inp, lemmatize=False, dictionary={})
for text in wiki.get_texts():
  article = space.join([t.decode("utf-8") for t in text])

  output.write(article + "\n")
  i = i + 1
  if (i % 1000 == 0):
    print("Saved " + str(i) + " articles")

output.close()
print("Finished - Saved " + str(i) + " articles")
	

Training the model

• size: The dimensionality of the vectors (bigger size values require more training data, but can lead to more accurate models)
• window: The maximum distance between the current and predicted word within a sentence
• min_count: Ignore all words with total frequency lower than this.

import multiprocessing
from gensim.models import Word2Vec
from gensim.models.word2vec import LineSentence

language_code = "he"
inp = "wiki.{}.text".format(language_code)
out_model = "wiki.{}.word2vec.model".format(language_code)
size = 100
window = 5
min_count = 5

start = time.time()

model = Word2Vec(LineSentence(inp), sg = 0, # 0=CBOW , 1= SkipGram
	     size=size, window=window, min_count=min_count, workers=multiprocessing.cpu_count())

# trim unneeded model memory = use (much) less RAM
model.init_sims(replace=True)

print(time.time()-start)

model.save(out_model)

FastText

pip install fasttext

start = time.time()

language_code = "he"
inp = "wiki.{}.text".format(language_code)
output = "wiki.{}.fasttext.model".format(language_code)
model = fasttext.cbow(inp,output)

print(time.time()-start)

Evaluating Embeddings: Analogies

W("woman") ≃ W("man")+ W("queen")− W("king")

import numpy as np
from gensim.matutils import unitvec

def test(model,positive,negative,test_words):

  mean = []
  for pos_word in positive:
    mean.append(1.0 * np.array(model[pos_word]))

  for neg_word in negative:
    mean.append(-1.0 * np.array(model[neg_word]))

  # compute the weighted average of all words
  mean = unitvec(np.array(mean).mean(axis=0))

  scores = {}
  for word in test_words:

    if word not in positive + negative:

      test_word = unitvec(np.array(model[word]))

      # Cosine Similarity
      scores[word] = np.dot(test_word, mean)

  print(sorted(scores, key=scores.get, reverse=True)[:1])

positive_words = ["queen","man"]

negative_words = ["king"]

# Test Word2vec
print("Testing Word2vec")
model = word2vec.getModel()
test(model,positive_words,negative_words,model.vocab)

# Test Fasttext
print("Testing Fasttext")
model = fasttxt.getModel()
test(model,positive_words,negative_words,model.words)

Testing Word2vec
	['woman']
Testing Fasttext
	['woman']
	

W("woman") ≃ W("man")+ W("queen")− W("king")

Deep Solutions wishes you a great deep day!