The increasing progress of transportation systems caused a dramatic increase in demand for smart systems capable of monitoring traffic and street safety.
Fundamental to these applications are algorithms for multi-object detection and multi-object tracking. It is thus important for a practitioner to know the pros and cons of different works in these categories.
It is goal of this Challenge to provide a comprehensive performance evaluation to state-of-the-art detection and tracking algorithms.
The challenge is based on UA-DETRAC, a real-world multi-object detection and multi-object tracking benchmark. The dataset is composed of traffic video sequences annotated with vehicle bounding boxes and trajectories at various challenging levels.
The challenge has been divided in two different levels of difficult: beginner and experienced (targeted to the research groups or people with richer experience).
The challenge will be held in conjunction with the International Workshop on Traffic and Street Surveillance for Safety and Security (T4S), in order to guarantee the publishing of the papers whose results will be over the baseline thresholds.
All the participants must submit the obtained results on UA-DETRAC dataset and a 6 pages long (including references) paper describing the applied methodology.
All the works, whose results are above the set thresholds, will be automatically published in the workshop proceeding (in association with the AVSS 2018 conference) and the best ones will be presented in the workshop day session.
Results and authorship of the papers that participated to the challenge and that scored more than the threshold will be summarized in a final paper in the main conference.
Submissions to the workshop that do not participate the challenge are permitted, but are subject to the standard peer review process.
Camera ready for both challenge and independent paper are due by October 15, 2018.
Challenge organization and evaluation
In order to allow a wider participation to the challenge a ‘task’/‘degree of difficult’ division has been considered leading to 4 different tracks as in figure below
Figure 2: Challenge tasks and levels of difficulty scheme. Each branch reports prize and metric baseline
Dataset specifics: The dataset consists of 10 hours of videos captured with a Cannon EOS 550D camera at 24 different locations at Beijing and Tianjin in China.
The videos are recorded at 25 frames per seconds (fps), with resolution of 960×540 pixels. There are more than 140 thousand frames in the UA-DETRAC dataset and 8250 vehicles that are manually annotated, leading to a total of 1.21 million labeled bounding boxes of objects.
The UA-DETRAC dataset is divided into training (UA- DETRAC-train) and testing (UA-DETRAC-test) sets, with 60 and 40 sequences, respectively. Training videos are taken at different locations from the testing videos, but similar traffic conditions and attributes are ensured.
The UA-DETRAC dataset contains videos with large variations in scale, pose and illumination, occlusion, and background clutters making useful to classify the test sequences in different level of difficulty (easy, medium and hard)
Detection: test sequences have been labeled based on the detection rate of the EdgeBox (more details and reference in the UA-DETRAC paper). This classification lead to 10 easy sequences, 20 medium sequences and 10 hard sequences.
Tracking: test sequences have been labeled based on the average PR-MOTA scores of six bench- marked object tracking methods (more details and reference in the UA-DETRAC paper). More precisely the 40 test sequences have been divided into 10 easy sequences, 20 medium sequences and 10 hard sequences.
Tasks: Following the UA-DETRAC organization the challenge is divided in two main task, the detection and tracking. The participation is disjoint: people can partecipate to the detection one, to the tracking one (employing known detection algorithm) or both of them in case where both elements in the proposed solution are innovative.
The solutions on detection task must present some innovative aspect.
The solutions on tracking task are allowed to exploit known detection algorithms keeping an innovation element in the tracking stage.
Some known detection and tracking algorithms are available (code too) in the UA-DETRAC website (http://detrac-db.rit.albany.edu).
Levels of difficulty: the challenge has been divided in two different levels of difficult: beginner and experienced (targeted to the research groups or people with richer experience).
The difference depends on the video sequence considered:
- Beginner: people participating to the beginner degree of difficult must submit only the results referred to test video sequences marked as easy independently on the task (detection or tracking)
- Experienced: people participating to the experienced level of difficult must submit the results referred to the overall test video sequences set for the detection task and the results referred to the test video sequences marked as hard and medium for the tracking task.
Each task/level will be evaluated separately and will be subject to specific metrics and baselines
An accurate definition of the employed metrics and level of difficult can be found in the UA-DETRAC website and paper:
L. Wen, D. Du, Z. Cai, Z. Lei, M. Chang, H. Qi, J. Lim, M. Yang, and S. Lyu. UA-DETRAC: A new benchmark and protocol for multi-object tracking. CoRR, abs/1511.04136, 2015. [PDF]
All the material is available at: http://detrac-db.rit.albany.edu
Result and paper submission
The result must be submitted by means of the UA-DETRAC web site. Participants have to:
- Register an account at http://detrac-db.rit.albany.edu/auth/register and activate it by averification email.
- Upload their detection/tracking result files with the format described at http://detrac-db.rit.albany.edu/instructions.
- Upload the paper related with the challenge submission. A template will be provided to theauthors by considering the guidelines given by AVSS 2018 Workshop chairs.
- Check email for the evaluation results for DETRAC-test set.
Challenge papers are 6 pages long (including references), with up to two more pages (attracting extra page charges).