A Caltech Library Service

Probabilistic FastText for Multi-Sense Word Embeddings

Athiwaratkun, Ben and Wilson, Andrew Gordon and Anandkumar, Anima (2018) Probabilistic FastText for Multi-Sense Word Embeddings. In: Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). Association for Computational Linguistics , Stroudsburg, PA, Art. No. P18-1001 . ISBN 978-1-948087-32-2.

[img] PDF - Published Version
Creative Commons Attribution.

[img] PDF - Accepted Version
See Usage Policy.


Use this Persistent URL to link to this item:


We introduce Probabilistic FastText, a new model for word embeddings that can capture multiple word senses, sub-word structure, and uncertainty information. In particular, we represent each word with a Gaussian mixture density, where the mean of a mixture component is given by the sum of n-grams. This representation allows the model to share statistical strength across sub-word structures (e.g. Latin roots), producing accurate representations of rare, misspelt, or even unseen words. Moreover, each component of the mixture can capture a different word sense. Probabilistic FastText outperforms both FastText, which has no probabilistic model, and dictionary-level probabilistic embeddings, which do not incorporate subword structures, on several word-similarity benchmarks, including English RareWord and foreign language datasets. We also achieve state-of-art performance on benchmarks that measure ability to discern different meanings. Thus, the proposed model is the first to achieve multi-sense representations while having enriched semantics on rare words.

Item Type:Book Section
Related URLs:
URLURL TypeDescription Paper
Additional Information:© 2018 The Association for Computational Linguistics. Materials published in or after 2016 are licensed on a Creative Commons Attribution 4.0 License.
Record Number:CaltechAUTHORS:20190327-085800530
Persistent URL:
Usage Policy:No commercial reproduction, distribution, display or performance rights in this work are provided.
ID Code:94177
Deposited By: George Porter
Deposited On:28 Mar 2019 22:11
Last Modified:03 Oct 2019 21:01

Repository Staff Only: item control page