GLIP: Introducing Language-Picture Pre-Coaching to Object Detection | by Sascha Kirch | Sep, 2023

Paper Abstract: Grounded Language-Picture Pre-training

Today we’ll dive right into a paper that builds upon the good success of CLIP in language-image pre-training and extends it to the duty of object detection: GLIP — Grounded Language-Image Pre-training. We are going to cowl the important thing ideas and findings of the paper and make them simple to know by offering additional context and including annotations to pictures and experiment outcomes. Let’s go!

Paper: Grounded Language-Image Pre-training

Code: https://github.com/microsoft/GLIP

First Revealed: 7 Dec. 2021

Authors: Liunian Harold Li, Pengchuan Zhang, Haotian Zhang, Jianwei Yang, Chunyuan Li, Yiwu Zhong, Lijuan Wang, Lu Yuan, Lei Zhang, Jenq-Neng Hwang, Kai-Wei Chang, Jianfeng Gao

Class: illustration studying, object detection, phrase-grounding, multi-modal deep studying, pc vison, pure language processing, basis fashions

1. Context & Background
2. Claimed Contributions
3. Methodology
4. Experiments
5. Additional Readings & Assets

GLIP (Grounded Language-Image Pre-training) is a multi-modal language-image mannequin. Just like CLIP (Contrastive Language-Image Pre-Coaching), it performs contrastive pre-training to study semantically wealthy representations and aligns them throughout its modalities. Whereas CLIP learns these illustration on a picture stage, which suggests one sentence describes your entire picture, GLIP goals to increase this method to object-level representations, that means one sentence would possibly correspond to a number of objects inside the picture. The duty of figuring out correspondences between single tokens in a text-prompt and objects or areas in a picture is named phrase grounding. Therefore the phrase “Grounded” in GLIP.

Subsequently, GLIP goals to:

1. Unify phrase grounding and object detection for large-scale pre-training.
2. Present a versatile framework for zero-shot object detection, the place versatile means it’s not restricted to a hard and fast set of lessons.
3. Construct one pre-trained mannequin that seamlessly transfers to numerous duties and domains, in a zero-shot or few-shot method.

What are you able to do with such a mannequin? You possibly can use textual content prompts to search out objects or areas of curiosity inside a given enter picture. And the most effective half: you aren’t restricted to pre-defined lessons.

You possibly can additional course of these detections (e.g. feeding these right into a monitoring system) or create a customized dataset with sure lessons of curiosity and use these to coach your individual supervised detection system. Not solely that you may cowl uncommon or very particular lessons, however you may additionally save numerous money and time for the creation of guide labels. As we’ll see later, the authors of GLIP had an analogous concept to spice up the efficiency even additional by introducing a teacher-student framework.

GLIP has been adopted by many different tasks and domains in deep studying. For instance, GLIGEN (Grounded-Language-to-Image-Generation) makes use of GLIP as to situation the picture technology of a latent diffusion mannequin to extend the controllability. Moreover, GLIP has been mixed with different basis fashions corresponding to DINO (Self Distilation with no Labels) and SAM (Segment Anything) to GroundingDINO and Grounded-Segment-Anything respectively. GLIPv2 extends the preliminary GLIP mannequin with vision-language understanding to not solely enhance phrase grounding but in addition allow visible query answering duties.

1. Giant scale pre-training for mixed phrase grounding and object detection
2. Offering a unified view on object detection and phrase grounding
3. Deep cross-modality fusion to study high-quality language-aware visible representations and to attain superior switch studying efficiency.
4. Presenting that prompt-tuning is simpler in deep vision-language fusion (e.g. GLIP) as in shallow fused networks (e.g. CLIP)

Having a tough concept of what might be carried out with GLIP, let’s have a more in-depth look into the main points of the paper.

Architectural Overview

On a excessive stage, GLIP’s structure is sort of just like CLIP’s in a way that it additionally consists of a textual content encoder, a picture encoder and a few type of contrastive studying on the similarity of textual content and picture options. The structure of GLIP is proven in Fig. 2.

GLIP provides a language-image conscious deep fusion module after the textual content and picture encoder. This module performs cross-modal consideration and extracts additional options. A cosine similarity is calculated over the ensuing area options and phrase options. Throughout coaching, the similarity of matching pairs is maximized, whereas minimized for incorrect pairs. In distinction to CLIP, the place the matching pairs are positioned on the diagonal of the similarity matrix, in GLIP the matching shouldn’t be carried out on sentence stage, however on (sub)phrase stage leading to normally off-diagonal positions.

Phrase Grounding Formulated as Object Detection Downside

The authors famous that the issue of phrase grounding (= associating phrases with objects/areas in a picture) might be formulated as Object detection Goal, the place the usual loss goal is:

The localization loss is anxious with the standard of the anticipated bounding field, which relying on the format, is perhaps the scale and placement of the field. The classification loss is the important thing half within the unification. By calculating the logits over the similarity rating of text-image options as an alternative of over the logits from a picture classifier, the identical loss goal can be utilized for coaching.

Totally different Mannequin Variants

5 completely different fashions are educated to point out the impact of the authors’ design decisions and mannequin scale:

Trainer-Pupil Pre-Coaching

To spice up the efficiency of GLIP, the authors prepare the GLIP-T (C) mannequin (see Fig.3) on human annotated knowledge, known as GoldG, to generate grounding knowledge from text-image pairs from the web. They name this mannequin the instructor mannequin and subsequently prepare a scholar mannequin feeding it the with the information used to coach the instructor plus the information the instructor generated. See Fig. 4 for an illustration.

Be aware: Though the phrases instructor and scholar are used, it’s not the identical course of as in information distillation, the place a smaller scholar mannequin is educated to match the output of a bigger instructor mannequin.

Apparently, as we’ll see within the experiments, the scholar surpasses the instructor on many (however not all) datasets for each; zero-shot and few-shot detection. Why is that? The paper hypothesizes, that eventhough the instructor offers a prediction with low confidence (they name it an “educated guess”), it turns into the bottom fact (they name it “supervised sign”) within the generated dataset consumed by the scholar.

The GLIP paper presents varied experiments and ablation research, primarily involved with:

1. Zero-Shot Area Switch
2. Knowledge Effectivity
3. Immediate Engineering

I’ve some doubts for a few of the outcomes and the best way they’re introduced, and I’ll level them out within the annotations. I don’t wish to diminish the achievements of GLIP and slightly view it with a vital eye.

Now let’s bounce into the main points!

Zero-Shot Area Switch

First, we’ll take a look into the outcomes from the zero-shot area switch. On this activity the target is to investigate how effectively the pre-trained GLIP fashions carry out on a special dataset (i.e. COCO and LVIS) as used throughout pre-training and examine it towards a baseline with fashions which were educated in a supervised vogue. Then, the pre-trained GLIP is additional fine-tuned and evaluated on the dataset beneath take a look at.

In Fig.5 we see the outcomes from the zero-shot area switch on COCO. We see that every one GLIP fashions have a greater 0-shot efficiency as a supervised Quicker RCNN. We’re additionally introduced with the outcome, that GLIP-L outperforms the earlier SOTA (on the time of the paper’s launch). We see that the bigger scholar GLIP-L outperforms the instructor mannequin GLIP-T (C).

Following I checklist my doubts when studying these outcomes and the claims made within the paper, the place it’s mentioned that GLIP-L surpasses the most effective supervised mannequin SoftTeacher.

1. The mannequin that has higher metrics than SoftTeacher is GLIP-L, which is healthier by 0.2 factors. This small margin may not be the results of the brand new technique of GLIP however is perhaps as a consequence of some variations in coaching hyperparameters.
2. GLIP-L doesn’t even use the information (Cap4M or Cap24M) generated from instructor mannequin which they introduced as an excellent resolution.
3. GLIP-L has been educated on a a lot bigger corpus of coaching knowledge as SoftTeacher.

For my part the outcomes evaluating the completely different GLIP fashions and the DyHead-T they educated themselves are fully nice, I simply have my doubts basically when completely different strategies and fashions are in contrast beneath unclear or completely different constraints.

In Fig.6, we see the zero-shot area switch efficiency on LVIS dataset. We are able to see that the most important GLIP mannequin, GLIP-L, outperforms all different introduced supervised fashions.

Lastly, GLIP has been in contrast on its phrase grounding efficiency on the Flickr30K entities towards MDETR (see Fig.7). Each scholar fashions, GLIP-T and GLIP-L, surpass the MDETR baselines.

Knowledge Effectivity

One other experiment is anxious with the information effectivity. This experiment goals to point out how the efficiency (when it comes to common precision) modifications when fine-tuning a pre-trained mannequin on a sure variety of activity particular knowledge. In Fig.8, the fashions are evaluated on 13 completely different datasets and their efficiency is reported as common precision averaged over the 13 datasets. Outcomes are reported for 0-shot, 1-shot, 3-shot, 5-shot, 10-shot and “all”-shot (I doubt that’s an official time period for full fine-tuning, however I assume you get the purpose 😅).

Immediate Engineering

Comparable as in CLIP, the authors additionally report a correlation of the mannequin’s efficiency and the formulation of the enter textual content immediate. They suggest two strategies to enhance the efficiency of a pre-trained mannequin, with out the necessity to retrain the mannequin’s weights:

1. Guide immediate tuning
2. Immediate Tuning

The thought of guide immediate tuning is to supply additional context in type of further descriptive phrases, see Fig. 9:

Guide immediate tuning can at all times be used to enhance the efficiency, that means it doesn’t matter if the mannequin is absolutely fine-tuned or if the mannequin is utilized in a zero-shot or few-shot situation.

The second method, immediate tuning, requires entry to floor fact labels of a downstream activity and is very appropriate for eventualities, the place every detection activity has a single immediate (e.g. “Detect automotive”). In that situation, this immediate would first be translated right into a function embedding utilizing the textual content encoder. Then, the picture encoder and the deep fusion module are frozen and solely the enter embedding is optimized utilizing the bottom fact labels. The optimized embeddings would then function enter to the mannequin and the textual content encoder may very well be eliminated.

Fig.10 exhibits the results of this immediate tuning for varied GLIP fashions. When utilized to fashions which have a deep fusion module, immediate tuning achieves virtually the identical efficiency as fine-tuning the mannequin’s weights.

As talked about firstly of this text, GLIP has been extensively adopted by an enormous variety of tasks.

Following an inventory of papers that constructed upon GLIP:

1. GLIPv2: Unifying Localization and Vision-Language Understanding
2. GLIGEN: Open-Set Grounded Text-to-Image Generation
3. Grounding DINO: Marrying DINO with Grounded Pre-Training for Open-Set Object Detection