CaltechAUTHORS
  A Caltech Library Service

Prismer: A Vision-Language Model with An Ensemble of Experts

Liu, Shikun and Fan, Linxi and Johns, Edward and Yu, Zhiding and Xiao, Chaowei and Anandkumar, Anima (2023) Prismer: A Vision-Language Model with An Ensemble of Experts. . (Unpublished) https://resolver.caltech.edu/CaltechAUTHORS:20230316-153658096

[img] PDF - Submitted Version
See Usage Policy.

5MB

Use this Persistent URL to link to this item: https://resolver.caltech.edu/CaltechAUTHORS:20230316-153658096

Abstract

Recent vision-language models have shown impressive multi-modal generation capabilities. However, typically they require training huge models on massive datasets. As a more scalable alternative, we introduce Prismer, a data- and parameter-efficient vision-language model that leverages an ensemble of domain experts. Prismer only requires training of a small number of components, with the majority of network weights inherited from readily-available, pre-trained domain experts, and kept frozen during training. By leveraging experts from a wide range of domains, we show that Prismer can efficiently pool this expert knowledge and adapt it to various vision-language reasoning tasks. In our experiments, we show that Prismer achieves fine-tuned and few-shot learning performance which is competitive with current state-of-the-art models, whilst requiring up to two orders of magnitude less training data. Code is available at https://github.com/NVlabs/prismer.


Item Type:Report or Paper (Discussion Paper)
Related URLs:
URLURL TypeDescription
http://arxiv.org/abs/2303.02506arXivDiscussion Paper
https://github.com/NVlabs/prismerRelated ItemProject website
ORCID:
AuthorORCID
Fan, Linxi0000-0001-7393-3125
Yu, Zhiding0000-0003-1776-996X
Xiao, Chaowei0000-0002-7043-4926
Anandkumar, Anima0000-0002-6974-6797
Record Number:CaltechAUTHORS:20230316-153658096
Persistent URL:https://resolver.caltech.edu/CaltechAUTHORS:20230316-153658096
Usage Policy:No commercial reproduction, distribution, display or performance rights in this work are provided.
ID Code:120079
Collection:CaltechAUTHORS
Deposited By: George Porter
Deposited On:16 Mar 2023 22:18
Last Modified:16 Mar 2023 22:18

Repository Staff Only: item control page