A Caltech Library Service

Utilizing image and caption information for biomedical document classification

Li, Pengyuan and Jiang, Xiangying and Zhang, Gongbo and Trelles Trabucco, Juan and Raciti, Daniela and Smith, Cynthia and Ringwald, Martin and Marai, G. Elisabeta and Arighi, Cecilia and Shatkay, Hagit (2021) Utilizing image and caption information for biomedical document classification. Bioinformatics, 37 (S1). i468-i476. ISSN 1367-4803. PMCID PMC8346654. doi:10.1093/bioinformatics/btab331.

[img] PDF - Published Version
Creative Commons Attribution Non-commercial.


Use this Persistent URL to link to this item:


Motivation: Biomedical research findings are typically disseminated through publications. To simplify access to domain-specific knowledge while supporting the research community, several biomedical databases devote significant effort to manual curation of the literature—a labor intensive process. The first step toward biocuration requires identifying articles relevant to the specific area on which the database focuses. Thus, automatically identifying publications relevant to a specific topic within a large volume of publications is an important task toward expediting the biocuration process and, in turn, biomedical research. Current methods focus on textual contents, typically extracted from the title-and-abstract. Notably, images and captions are often used in publications to convey pivotal evidence about processes, experiments and results. Results: We present a new document classification scheme, using both image and caption information, in addition to titles-and-abstracts. To use the image information, we introduce a new image representation, namely Figure-word, based on class labels of subfigures. We use word embeddings for representing captions and titles-and-abstracts. To utilize all three types of information, we introduce two information integration methods. The first combines Figure-words and textual features obtained from captions and titles-and-abstracts into a single larger vector for document representation; the second employs a meta-classification scheme. Our experiments and results demonstrate the usefulness of the newly proposed Figure-words for representing images. Moreover, the results showcase the value of Figure-words, captions and titles-and-abstracts in providing complementary information for document classification; these three sources of information when combined, lead to an overall improved classification performance. Availability and implementation: Source code and the list of PMIDs of the publications in our datasets are available upon request.

Item Type:Article
Related URLs:
URLURL TypeDescription CentralArticle
Li, Pengyuan0000-0002-8205-7611
Trelles Trabucco, Juan0000-0002-0367-9763
Smith, Cynthia0000-0003-3691-0324
Ringwald, Martin0000-0002-5696-5421
Arighi, Cecilia0000-0002-0803-4817
Shatkay, Hagit0000-0001-6953-3970
Additional Information:© The Author(s) 2021. Published by Oxford University Press. This is an Open Access article distributed under the terms of the Creative Commons Attribution Non-Commercial License (, which permits non-commercial re-use, distribution, and reproduction in any medium, provided the original work is properly cited. Accepted: 06 May 2021; Published: 12 July 2021. This work was partially supported by National Institutes of Health (NIH)/National Library of Medicine (NLM) awards [R56LM011354A and R01LM012527]; NIH/National Institute of Child Health and Human Development (NICHD) award [P41 HD062499 to M.R.]. Conflict of Interest: none declared.
Funding AgencyGrant Number
NIHP41 HD062499
Issue or Number:S1
PubMed Central ID:PMC8346654
Record Number:CaltechAUTHORS:20210721-224759383
Persistent URL:
Official Citation:Pengyuan Li, Xiangying Jiang, Gongbo Zhang, Juan Trelles Trabucco, Daniela Raciti, Cynthia Smith, Martin Ringwald, G Elisabeta Marai, Cecilia Arighi, Hagit Shatkay, Utilizing image and caption information for biomedical document classification, Bioinformatics, Volume 37, Issue Supplement_1, July 2021, Pages i468–i476,
Usage Policy:No commercial reproduction, distribution, display or performance rights in this work are provided.
ID Code:109967
Deposited By: Tony Diaz
Deposited On:26 Jul 2021 22:43
Last Modified:11 Aug 2021 22:08

Repository Staff Only: item control page