Lecture 2 – Word Vectors and Word Senses

---

1. Review: Main idea of word2vec

Word2vec parameters and computations

Word2vec maximizes objective function by putting similar words nearby in space

 

2. Optimization: Gradient Descent

Gradient Descent

Stochastic Gradient Descent

Stochastic gradients with word vectors!

1b. Word2vec: More details

So far, we have looked at two main classes of methods to find word embeddings. The first set are count-based and rely on matrix factorization (e.g. LSA, HAL). While these methods effectively leverage global statistical information, they are primarily used to capture word similarities and do poorly on tasks such as word analogy, indicating a sub-optimal vector space structure. The other set of methods are shallow window-based (e.g. the skip-gram and the CBOW models), which learn word embeddings by making predictions in local context windows. These models demonstrate the capacity to capture complex linguistic patterns beyond word similarity, but fail to make use of the global co-occurrence statistics.

The skip-gram model with negative sampling (HW2)

In comparison, GloVe consists of a weighted least squares model that trains on global word-word co-occurrence counts and thus makes efficient use of statistics. The model produces a word vector space with meaningful sub-structure. It shows state-of-the-art performance on the word analogy task, and outperforms other current methods on several word similarity tasks. 

---

3. But why not capture co-occurrence counts directly?

Example: Window based co-occurrence matrix

Window based co-occurrence matrix

Problems with simple co-occurrence vectors

Solution: Low dimensional vectors

Method 1: Dimensionality Reduction on X (HW1)

Simple SVD word vectors in Python

Hacks to X (several used in Rohde et al. 2005)

Interesting syntactic patterns emerge in the vectors

Count based vs. direct prediction

How to evaluate word vectors?

Intrinsic word vector evaluation

Glove Visualizations

Glove Visualizations: Company - CEO

Glove Visualizations: Superlatives

Details of intrinsic word vector evaluation

Analogy evaluation and hyperparameters

 Analogy evaluation and hyperparameters

Another intrinsic word vector evaluation

Closest words to “Sweden” (cosine similarity)

Correlation evaluation

 Word senses and word sense ambiguity

pike

Improving Word Representations Via Global Context And Multiple Word Prototypes (Huang et al. 2012)

Linear Algebraic Structure of Word Senses, with Applications to Polysemy

Extrinsic word vector evaluation

Course plan: coming weeks

Office Hours / Help sessions