NeighborTrack

Update: Add model speed and some experiment in CVPRW:

NeighborTrack: Single Object Tracking by Bipartite Matching With Neighbor Tracklets and Its Applications to Sports

Old version:

NeighborTrack: Improving Single Object Tracking by Bipartite Matching with Neighbor Tracklets

This paper was accepted by the 9th International Workshop on Computer Vision in Sports (CVsports) 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPR)

Single Object Tracking(SOT) post-processing method by using cycle consistency and neighbor(python version)

Some SOT model codes are from OSTrack, votchallenge, Ocean, TransT, pytracking, and Mixformer. Thanks to these projects a lot.

Website: OSTrack, TransT, Votchallenge, Ocean, pytracking, Mixformer,

KalmanFilter implement

SoftNMS implement

Demo videos

Results

Models and source results link

LaSOT,GOT10K,TrackingNet,UAV123,OTB100 (baseline from OSTrack github code)

LaSOT	AUC	OP50	OP75	Precision	Norm Precision
OSTrack384	71.90	82.91	72.50	77.65	81.40
OSTrack384_NeighborTrack	72.25	83.33	72.70	78.05	81.82

GOT-10K	AO	SR0.50	SR0.75	Hz
OSTrack384	73.94	83.63	72.16	7.00 fps
OSTrack384_NeighborTrack	75.73	85.72	73.29	2.99 fps
OSTrack384_gottrainonly	74.19	83.98	71.58	3.88 fps
OSTrack384_gottrainonly_NeighborTrack	74.53	84.25	71.54	4.07 fps

TrackingNet	Success	Precision	Normalized Precision	Coverage
OSTrack384	83.58	82.94	88.05	100
OSTrack384_NeighborTrack_tau=9	83.73	83.16	88.23	100
OSTrack384_NeighborTrack_tau=18	83.79	83.24	88.30	100

UAV123	AUC	OP50	OP75	Precision	Norm Precision	FPS
OSTrack384	72.17	87.24	68.09	92.59	88.06	3.83
OSTrack384_NeighborTrack_tau=9	71.52	86.41	67.47	91.86	87.27	2.11
OSTrack384_NeighborTrack_tau=27	72.56	87.75	68.15	93.37	88.51	1.31

Note: UAV123 has some long-term tracking videos, and it needs more temporal information, if use standard setting tau=9, it cannot improve AUC, we set tau=27 on the whole dataset

OTB100	AUC	OP50	OP75	Precision	Norm Precision	FPS
OSTrack384	69.27	85.42	56.39	89.62	84.38	3.91
OSTrack384_NeighborTrack_tau=9	69.54	85.52	56.40	90.21	84.68	1.98
OSTrack384_NeighborTrack_tau=27	69.74	85.88	56.49	90.42	84.87	1.23

Votchallenge

VOT2022-ST	EAO	A	R
OSTrack384	0.538	0.779	0.824
OSTrack384_NeighborTrack	0.564	0.779	0.845
Ocean	0.484	0.703	0.823
Ocean_NeighborTrack	0.486	0.703	0.822
TransT_N2	0.493	0.780	0.775
TransT_N2_NeighborTrack	0.519	0.781	0.808
TransT_N4	0.486	0.779	0.771
TransT_N4_NeighborTrack	0.518	0.777	0.810
Normal Cross Correlation tracker(NCC)	0.102	0.564	0.208
NCC_NeighborTrack	0.127	0.549	0.266

Bibtex

@InProceedings{Chen_2023_CVPR,
    author    = {Chen, Yu-Hsi and Wang, Chien-Yao and Yang, Cheng-Yun and Chang, Hung-Shuo and Lin, Youn-Long and Chuang, Yung-Yu and Liao, Hong-Yuan Mark},
    title     = {NeighborTrack: Single Object Tracking by Bipartite Matching With Neighbor Tracklets and Its Applications to Sports},
    booktitle = {Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) Workshops},
    month     = {June},
    year      = {2023},
    pages     = {5138-5147}
}

Quick start

1. Install Environment

my driver version: NVIDIA-SMI 465.19.01 Driver Version: 465.19.01 Python 3.7.7 (default, Mar 23, 2020, 22:36:06) torch.version.cuda=10.1

pip install munkres==1.1.4
pip install shapely

Other environments depend on your base model, e.g. OSTrack:

Example of my Environment please see This file.

cd trackers/ostrack
sh example_ostrack_install.sh

2. Download the dataset and models, then put them on each path

Models and source results link

More information for model paths

Get results from NeighborTrack with OSTrack

Work space is in 'NeighborTrack/trackers/ostrack/', please remember to change the dataset and model's root.

More information :OSTrack user's guide

LaSOT, GOT-10K

cd /your_path/trackers/ostrack/
sh test.sh
#or
#lasot example
python tracking/test.py ostrack vitb_384_mae_ce_32x4_ep300_neighbor --dataset lasot --threads 24 --num_gpus 8 --neighbor 1
#python tracking/analysis_results.py 

#got-10K example
python tracking/test.py ostrack vitb_384_mae_ce_32x4_ep300_neighbor --dataset got10k_test --threads 16 --num_gpus 8 --neighbor 1 
#to use got-10K train_from_got10K_only
python tracking/test.py ostrack vitb_384_mae_ce_32x4_got10k_ep100_neighbor --dataset got10k_test --threads 16 --num_gpus 8 --neighbor 1 

VOT challenge

vot test ostrackNeighbor
vot test ostrackNeighborAR
vot evaluate --workspace ./vot2022st ostrackNeighbor
vot analysis --workspace vot2022st ostrackNeighbor

vot evaluate --workspace ./vot2021 ostrackNeighborAR
vot analysis --workspace vot2021 ostrackNeighborAR

setting vot workspace example VOT trackers example:trackers.ini, ostrack_384_vot_neighbor.py

If you want to know how to create workspace of vot2022st vot2020 vot2021 dataset, please seen Votchallenge:

In your own video

sh video_test.sh
# or
python tracking/video_demo_neighbor.py ostrack vitb_384_mae_ce_32x4_ep300_neighbor ./cup1.avi  \
   --optional_box 1109 531 82 135 --save_results --debug 1 --save_img
#optional_box is GT in first frame.

How to use NeighborTrack in your own SOT tracker:

1. Create dependent functions:

There is a simple code from the votchallenge NCC tracker, add 3 functions to use our method(initialize, track_neighbor, and update_center). Please see:

NeighborTrack/trackers/example_ncc_tracker.py

Line 14 in c889695

class NCCTracker(object):

After adding functions are seems like:

NeighborTrack/trackers/example_ncc_tracker.py

Line 51 in c889695

class NCCTracker_neighbor(object):

Remember, the tracker should have 2 independent models forward/reverse because all of the SOT methods will forget the tracking target after initialization, if just 1 forward/backward tracker, it cannot switch forward/backward mission and ensure the forward answer doesn't have any change (even didn't use our method to change output, just use the same tracker to track any other object, your forward output will not come back to original answer, because the memory of tracker is changed.)

Other example: ostrack add 3 functions

NeighborTrack/trackers/ostrack/lib/test/evaluation/tracker.py

Line 328 in c889695

class NeighborTracker(Tracker):

More details:

2.Usage:

Init

NeighborTrack/trackers/ostrack/tracking/ostrack_384_vot_neighbor.py

Line 63 in c889695

self.ntracker = neighbortrack(self.tracker,image,region[:2],region[2:],invtracker=self.invtracker)

Get tracking answer

NeighborTrack/trackers/ostrack/tracking/ostrack_384_vot_neighbor.py

Line 70 in c889695

state = self.ntracker._neighbor_track(img_RGB)

Tracker and invtracker are original ostrack, you can change them by your SOT tracker.

region = [x,y,w,h],(x y = top left)

image = image by your model input, for example ostrack's image = numpy.array(img[h,w,3(RGB)])

No module named xxxx

If you see this error, please add 3 paths on tracking/test.py

{}\NeighborTrack\trackers\ostrack\lib\test\tracker

{}\NeighborTrack\trackers\ostrack\tracking

{}   (= NeighborTrack\..\)