Welcome to the new version of CaltechAUTHORS. Login is currently restricted to library staff. If you notice any issues, please email coda@library.caltech.edu
Published June 7, 2018 | Published + Accepted Version
Book Section - Chapter Open

Probabilistic FastText for Multi-Sense Word Embeddings


We introduce Probabilistic FastText, a new model for word embeddings that can capture multiple word senses, sub-word structure, and uncertainty information. In particular, we represent each word with a Gaussian mixture density, where the mean of a mixture component is given by the sum of n-grams. This representation allows the model to share statistical strength across sub-word structures (e.g. Latin roots), producing accurate representations of rare, misspelt, or even unseen words. Moreover, each component of the mixture can capture a different word sense. Probabilistic FastText outperforms both FastText, which has no probabilistic model, and dictionary-level probabilistic embeddings, which do not incorporate subword structures, on several word-similarity benchmarks, including English RareWord and foreign language datasets. We also achieve state-of-art performance on benchmarks that measure ability to discern different meanings. Thus, the proposed model is the first to achieve multi-sense representations while having enriched semantics on rare words.

Additional Information

© 2018 The Association for Computational Linguistics. Materials published in or after 2016 are licensed on a Creative Commons Attribution 4.0 License.

Attached Files

Accepted Version - 1806.02901.pdf

Published - P18-1001.pdf


Files (965.8 kB)
Name Size Download all
527.8 kB Preview Download
438.0 kB Preview Download

Additional details

August 19, 2023
August 19, 2023