CaltechAUTHORS
  A Caltech Library Service

Panoptic SegFormer: Delving Deeper into Panoptic Segmentation with Transformers

Li, Zhiqi and Wang, Wenhai and Xie, Enze and Yu, Zhiding and Anandkumar, Anima and Alvarez, Jose M. and Luo, Ping and Lu, Tong (2021) Panoptic SegFormer: Delving Deeper into Panoptic Segmentation with Transformers. . (Unpublished) https://resolver.caltech.edu/CaltechAUTHORS:20220714-224718853

[img] PDF - Submitted Version
Creative Commons Attribution.

7MB

Use this Persistent URL to link to this item: https://resolver.caltech.edu/CaltechAUTHORS:20220714-224718853

Abstract

Panoptic segmentation involves a combination of joint semantic segmentation and instance segmentation, where image contents are divided into two types: things and stuff. We present Panoptic SegFormer, a general framework for panoptic segmentation with transformers. It contains three innovative components: an efficient deeply-supervised mask decoder, a query decoupling strategy, and an improved post-processing method. We also use Deformable DETR to efficiently process multi-scale features, which is a fast and efficient version of DETR. Specifically, we supervise the attention modules in the mask decoder in a layer-wise manner. This deep supervision strategy lets the attention modules quickly focus on meaningful semantic regions. It improves performance and reduces the number of required training epochs by half compared to Deformable DETR. Our query decoupling strategy decouples the responsibilities of the query set and avoids mutual interference between things and stuff. In addition, our post-processing strategy improves performance without additional costs by jointly considering classification and segmentation qualities to resolve conflicting mask overlaps. Our approach increases the accuracy 6.2% PQ over the baseline DETR model. Panoptic SegFormer achieves state-of-the-art results on COCO test-dev with 56.2% PQ. It also shows stronger zero-shot robustness over existing methods. The code is released at {this https URL https://github.com/zhiqi-li/Panoptic-SegFormer}.


Item Type:Report or Paper (Discussion Paper)
Related URLs:
URLURL TypeDescription
https://arxiv.org/abs/2109.03814arXivDiscussion Paper
https://github.com/zhiqi-li/Panoptic-SegFormerRelated ItemCode
ORCID:
AuthorORCID
Anandkumar, Anima0000-0002-6974-6797
Additional Information:Attribution 4.0 International (CC BY 4.0). This work is supported by the Natural Science Foundation of China under Grant 61672273 and Grant 61832008. Ping Luo is supported by the General Research Fund of HK No.27208720 and 17212120. Wenhai Wang and Tong Lu are corresponding authors.
Funders:
Funding AgencyGrant Number
National Natural Science Foundation of China61672273
National Natural Science Foundation of China61832008
General Research Fund of Hong Kong27208720
General Research Fund of Hong Kong17212120
Record Number:CaltechAUTHORS:20220714-224718853
Persistent URL:https://resolver.caltech.edu/CaltechAUTHORS:20220714-224718853
Usage Policy:No commercial reproduction, distribution, display or performance rights in this work are provided.
ID Code:115613
Collection:CaltechAUTHORS
Deposited By: George Porter
Deposited On:15 Jul 2022 15:17
Last Modified:15 Jul 2022 15:17

Repository Staff Only: item control page