CaltechAUTHORS
  A Caltech Library Service

MEGATRON-CNTRL: Controllable Story Generation with External Knowledge Using Large-Scale Language Models

Xu, Peng and Patwary, Mostofa and Shoeybi, Mohammad and Puri, Raul and Fung, Pascale and Anandkumar, Anima and Catanzaro, Bryan (2020) MEGATRON-CNTRL: Controllable Story Generation with External Knowledge Using Large-Scale Language Models. . (Unpublished) https://resolver.caltech.edu/CaltechAUTHORS:20201106-120144603

[img] PDF - Submitted Version
See Usage Policy.

693Kb

Use this Persistent URL to link to this item: https://resolver.caltech.edu/CaltechAUTHORS:20201106-120144603

Abstract

Existing pre-trained large language models have shown unparalleled generative capabilities. However, they are not controllable. In this paper, we propose MEGATRON-CNTRL, a novel framework that uses large-scale language models and adds control to text generation by incorporating an external knowledge base. Our framework consists of a keyword predictor, a knowledge retriever, a contextual knowledge ranker, and a conditional text generator. As we do not have access to ground-truth supervision for the knowledge ranker, we make use of weak supervision from sentence embedding. The empirical results show that our model generates more fluent, consistent, and coherent stories with less repetition and higher diversity compared to prior work on the ROC story dataset. We showcase the controllability of our model by replacing the keywords used to generate stories and re-running the generation process. Human evaluation results show that 77.5% of these stories are successfully controlled by the new keywords. Furthermore, by scaling our model from 124 million to 8.3 billion parameters we demonstrate that larger models improve both the quality of generation (from 74.5% to 93.0% for consistency) and controllability (from 77.5% to 91.5%).


Item Type:Report or Paper (Discussion Paper)
Related URLs:
URLURL TypeDescription
http://arxiv.org/abs/2010.00840arXivDiscussion Paper
ORCID:
AuthorORCID
Fung, Pascale0000-0002-0628-7132
Additional Information:This work was done during the internship of Peng Xu at NVIDIA.
Record Number:CaltechAUTHORS:20201106-120144603
Persistent URL:https://resolver.caltech.edu/CaltechAUTHORS:20201106-120144603
Usage Policy:No commercial reproduction, distribution, display or performance rights in this work are provided.
ID Code:106481
Collection:CaltechAUTHORS
Deposited By: George Porter
Deposited On:06 Nov 2020 20:35
Last Modified:06 Nov 2020 20:35

Repository Staff Only: item control page