init

Author: siyuanliii

.gitignore

@@ -0,0 +1,7 @@
+tools/euler/*
+.gitignore
+tools/bsub_*
+tools/lint.sh
+.idea
+*.log
+teta/scripts/lint.sh

LICENSE

@@ -186,7 +186,7 @@
       same "printed page" as the copyright notice for easier
       identification within third-party archives.
 
-   Copyright [yyyy] [name of copyright owner]
+   Copyright [2022] [Siyuan Li]
 
    Licensed under the Apache License, Version 2.0 (the "License");
    you may not use this file except in compliance with the License.

README.md

@@ -1,3 +1,78 @@
-# Track Every Thing in the Wild, ECCV 2022
+# Track Every Thing in the Wild [ECCV2022]
 
-In construction, please check back later.
+This is the offical implementation of paper [Track Every Thing in the Wild](https://arxiv.org/abs/2207.12978).
+
+Our project website contains more information: [vis.xyz/pub/tet](https://www.vis.xyz/pub/tet/).
+
+
+## Abstract
+
+Current multi-category Multiple Object Tracking (MOT) metrics use class labels to group tracking results for per-class evaluation. Similarly, MOT methods typically only associate objects with the same class predictions. 
+These two prevalent strategies in MOT implicitly assume that the classification performance is near-perfect. 
+However, this is far from the case in recent large-scale MOT datasets, which contain large numbers of classes with many rare or semantically similar categories. Therefore, the resulting inaccurate classification leads to sub-optimal tracking and inadequate benchmarking of trackers.
+We address these issues by disentangling classification from tracking.
+We introduce a new metric, Track Every Thing Accuracy (TETA), breaking tracking measurement into three sub-factors: localization, association, and classification, allowing comprehensive benchmarking of tracking performance even under inaccurate classification. TETA also deals with the challenging incomplete annotation problem in large-scale tracking datasets. We further introduce a Track Every Thing tracker (TETer), that performs association using Class Exemplar Matching (CEM). Our experiments show that TETA evaluates trackers more comprehensively, and TETer achieves significant improvements on the challenging large-scale datasets BDD100K and TAO compared to the state-of-the-art.
+
+## TETA
+[TETA](teta/README.md) builds upon the HOTA metric, while extending it to better deal with
+multiple categories and incomplete annotations. TETA evaluate trackers based on a novel local cluster design. TETA consists of three parts: a
+localization score, an association score, and a classification score, which enable
+us to evaluate the different aspects of each tracker properly.
+
+<img src="figures/teta-teaser.png" width="400">
+
+## TETer
+TETer follows an Associate-Every-Thing (AET) strategy.
+Instead of only associating objects in the same class, we associate every object in neighboring frames. 
+We introduce Class Exemplar Matching (CEM), where the learned class exemplars incorporate valuable class information in a soft manner.
+In this way, we effectively exploit semantic supervision on large-scale detection datasets while not relying on the often incorrect classification output.
+
+<img src="figures/teaser-teter.png" width="800">
+
+## Main results
+Our method outperforms the states of the art on BDD100K, and TAO benchmarks.
+
+### BDD100K val set
+
+| Method                                              | backbone  | mMOTA | mIDF1 | TETA | LocA | AssocA | ClsA |
+|-----------------------------------------------------|-----------|-------|-------|------|------|--------|------|
+| [QDTrack(CVPR21)](https://arxiv.org/abs/2006.06664) | ResNet-50 | 36.6  | 51.6  | 47.8 | 45.9 | 48.5   | 49.2 |
+| TETer (Ours)                                        | ResNet-50 | 39.1  | 53.3  | 50.8 | 47.2 | 52.9   | 52.4 |
+
+
+### BDD100K test set
+
+| Method                                              | backbone  | mMOTA | mIDF1 | TETA | LocA | AssocA | ClsA |
+|-----------------------------------------------------|-----------|-------|-------|------|------|--------|------|
+| [QDTrack(CVPR21)](https://arxiv.org/abs/2006.06664) | ResNet-50 | 35.7  | 52.3  | 49.2 | 47.2 | 50.9   | 49.2 |
+| TETer (Ours)                                        | ResNet-50 | 37.4  | 53.3  | 50.8 | 47.0 | 53.6   | 50.7 |
+
+
+### TAO val set
+
+| Method                                              | backbone   | TETA | LocA | AssocA | ClsA |
+|-----------------------------------------------------|------------|------|------|--------|------|
+| [QDTrack(CVPR21)](https://arxiv.org/abs/2006.06664) | ResNet-101 | 30.0 | 50.5 | 27.4   | 12.1 |
+| TETer (Ours)                                        | ResNet-101 | 33.3 | 51.6 | 35.0   | 13.2 |
+| TETer-swinT (Ours)                                  | SwinT      | 34.6 | 52.1 | 36.7   | 15.0 |
+
+## Installation
+
+Please refer to [INSTALL.md](docs/INSTALL.md) for installation instructions.
+
+
+## Usages
+Please refer to [GET_STARTED.md](docs/GET_STARTED.md) for dataset preparation and running instructions.
+
+
+## Citation
+
+```
+@InProceedings{trackeverything,
+  title = {Track Every Thing in the Wild},
+  author = {Li, Siyuan and Danelljan, Martin and Ding, Henghui and Huang, Thomas E. and Yu, Fisher},
+  booktitle = {Proceedings of the European Conference on Computer Vision (ECCV)},
+  month = {Oct},
+  year = {2022}
+}
+```

configs/_base_/faster_rcnn_r50_fpn.py

@@ -0,0 +1,108 @@
+# model settings
+model = dict(
+    type='FasterRCNN',
+    backbone=dict(
+        type='ResNet',
+        depth=50,
+        num_stages=4,
+        out_indices=(0, 1, 2, 3),
+        frozen_stages=1,
+        norm_cfg=dict(type='BN', requires_grad=True),
+        norm_eval=True,
+        style='pytorch',
+        init_cfg=dict(type='Pretrained', checkpoint='torchvision://resnet50')),
+    neck=dict(
+        type='FPN',
+        in_channels=[256, 512, 1024, 2048],
+        out_channels=256,
+        num_outs=5),
+    rpn_head=dict(
+        type='RPNHead',
+        in_channels=256,
+        feat_channels=256,
+        anchor_generator=dict(
+            type='AnchorGenerator',
+            scales=[8],
+            ratios=[0.5, 1.0, 2.0],
+            strides=[4, 8, 16, 32, 64]),
+        bbox_coder=dict(
+            type='DeltaXYWHBBoxCoder',
+            target_means=[.0, .0, .0, .0],
+            target_stds=[1.0, 1.0, 1.0, 1.0]),
+        loss_cls=dict(
+            type='CrossEntropyLoss', use_sigmoid=True, loss_weight=1.0),
+        loss_bbox=dict(type='L1Loss', loss_weight=1.0)),
+    roi_head=dict(
+        type='StandardRoIHead',
+        bbox_roi_extractor=dict(
+            type='SingleRoIExtractor',
+            roi_layer=dict(type='RoIAlign', output_size=7, sampling_ratio=0),
+            out_channels=256,
+            featmap_strides=[4, 8, 16, 32]),
+        bbox_head=dict(
+            type='Shared2FCBBoxHead',
+            in_channels=256,
+            fc_out_channels=1024,
+            roi_feat_size=7,
+            num_classes=80,
+            bbox_coder=dict(
+                type='DeltaXYWHBBoxCoder',
+                target_means=[0., 0., 0., 0.],
+                target_stds=[0.1, 0.1, 0.2, 0.2]),
+            reg_class_agnostic=False,
+            loss_cls=dict(
+                type='CrossEntropyLoss', use_sigmoid=False, loss_weight=1.0),
+            loss_bbox=dict(type='L1Loss', loss_weight=1.0))),
+    # model training and testing settings
+    train_cfg=dict(
+        rpn=dict(
+            assigner=dict(
+                type='MaxIoUAssigner',
+                pos_iou_thr=0.7,
+                neg_iou_thr=0.3,
+                min_pos_iou=0.3,
+                match_low_quality=True,
+                ignore_iof_thr=-1),
+            sampler=dict(
+                type='RandomSampler',
+                num=256,
+                pos_fraction=0.5,
+                neg_pos_ub=-1,
+                add_gt_as_proposals=False),
+            allowed_border=-1,
+            pos_weight=-1,
+            debug=False),
+        rpn_proposal=dict(
+            nms_pre=2000,
+            max_per_img=1000,
+            nms=dict(type='nms', iou_threshold=0.7),
+            min_bbox_size=0),
+        rcnn=dict(
+            assigner=dict(
+                type='MaxIoUAssigner',
+                pos_iou_thr=0.5,
+                neg_iou_thr=0.5,
+                min_pos_iou=0.5,
+                match_low_quality=False,
+                ignore_iof_thr=-1),
+            sampler=dict(
+                type='RandomSampler',
+                num=512,
+                pos_fraction=0.25,
+                neg_pos_ub=-1,
+                add_gt_as_proposals=True),
+            pos_weight=-1,
+            debug=False)),
+    test_cfg=dict(
+        rpn=dict(
+            nms_pre=1000,
+            max_per_img=1000,
+            nms=dict(type='nms', iou_threshold=0.7),
+            min_bbox_size=0),
+        rcnn=dict(
+            score_thr=0.05,
+            nms=dict(type='nms', iou_threshold=0.5),
+            max_per_img=100)
+        # soft-nms is also supported for rcnn testing
+        # e.g., nms=dict(type='soft_nms', iou_threshold=0.5, min_score=0.05)
+    ))

configs/_base_/qdtrack_faster_rcnn_r50_fpn.py

@@ -0,0 +1,43 @@
+_base_ = './faster_rcnn_r50_fpn.py'
+model = dict(
+    type='QDTrack',
+    rpn_head=dict(
+        loss_bbox=dict(type='SmoothL1Loss', beta=1.0 / 9.0, loss_weight=1.0)),
+    roi_head=dict(
+        type='QuasiDenseRoIHead',
+        track_roi_extractor=dict(
+            type='SingleRoIExtractor',
+            roi_layer=dict(type='RoIAlign', output_size=7, sampling_ratio=0),
+            out_channels=256,
+            featmap_strides=[4, 8, 16, 32]),
+        track_head=dict(
+            type='QuasiDenseEmbedHead',
+            num_convs=4,
+            num_fcs=1,
+            embed_channels=256,
+            norm_cfg=dict(type='GN', num_groups=32),
+            loss_track=dict(type='MultiPosCrossEntropyLoss', loss_weight=0.25),
+            loss_track_aux=dict(
+                type='L2Loss',
+                neg_pos_ub=3,
+                pos_margin=0,
+                neg_margin=0.1,
+                hard_mining=True,
+                loss_weight=1.0))),
+    train_cfg=dict(
+        embed=dict(
+            assigner=dict(
+                type='MaxIoUAssigner',
+                pos_iou_thr=0.7,
+                neg_iou_thr=0.3,
+                min_pos_iou=0.5,
+                match_low_quality=False,
+                ignore_iof_thr=-1),
+            sampler=dict(
+                type='CombinedSampler',
+                num=256,
+                pos_fraction=0.5,
+                neg_pos_ub=3,
+                add_gt_as_proposals=True,
+                pos_sampler=dict(type='InstanceBalancedPosSampler'),
+                neg_sampler=dict(type='RandomSampler')))))