Detailed Performance of FSD

[Abyssaledge]

Performance of fsd_waymoD1_1x_submission.py (test set): submission
Performance of fsd_waymoD1_1x.py (validation set):

OBJECT_TYPE_TYPE_VEHICLE_LEVEL_1: [mAP 0.792293] [mAPH 0.787888]                                                                                              
OBJECT_TYPE_TYPE_VEHICLE_LEVEL_2: [mAP 0.704975] [mAPH 0.700932]
OBJECT_TYPE_TYPE_PEDESTRIAN_LEVEL_1: [mAP 0.825951] [mAPH 0.773279]
OBJECT_TYPE_TYPE_PEDESTRIAN_LEVEL_2: [mAP 0.739378] [mAPH 0.690696]
OBJECT_TYPE_TYPE_SIGN_LEVEL_1: [mAP 0] [mAPH 0]     
OBJECT_TYPE_TYPE_SIGN_LEVEL_2: [mAP 0] [mAPH 0]                   
OBJECT_TYPE_TYPE_CYCLIST_LEVEL_1: [mAP 0.77096] [mAPH 0.759925]   
OBJECT_TYPE_TYPE_CYCLIST_LEVEL_2: [mAP 0.743786] [mAPH 0.73312]    
RANGE_TYPE_VEHICLE_[0, 30)_LEVEL_1: [mAP 0.928933] [mAPH 0.925183] 
RANGE_TYPE_VEHICLE_[0, 30)_LEVEL_2: [mAP 0.916398] [mAPH 0.91269]    
RANGE_TYPE_VEHICLE_[30, 50)_LEVEL_1: [mAP 0.782089] [mAPH 0.777451]  
RANGE_TYPE_VEHICLE_[30, 50)_LEVEL_2: [mAP 0.714469] [mAPH 0.710126]
RANGE_TYPE_VEHICLE_[50, +inf)_LEVEL_1: [mAP 0.593706] [mAPH 0.586526]
RANGE_TYPE_VEHICLE_[50, +inf)_LEVEL_2: [mAP 0.461929] [mAPH 0.456148]
RANGE_TYPE_PEDESTRIAN_[0, 30)_LEVEL_1: [mAP 0.865141] [mAPH 0.823203]
RANGE_TYPE_PEDESTRIAN_[0, 30)_LEVEL_2: [mAP 0.82562] [mAPH 0.784474]
RANGE_TYPE_PEDESTRIAN_[30, 50)_LEVEL_1: [mAP 0.814913] [mAPH 0.756908]
RANGE_TYPE_PEDESTRIAN_[30, 50)_LEVEL_2: [mAP 0.738424] [mAPH 0.684682]
RANGE_TYPE_PEDESTRIAN_[50, +inf)_LEVEL_1: [mAP 0.74548] [mAPH 0.662026]
RANGE_TYPE_PEDESTRIAN_[50, +inf)_LEVEL_2: [mAP 0.588333] [mAPH 0.5195]
RANGE_TYPE_SIGN_[0, 30)_LEVEL_1: [mAP 0] [mAPH 0]
RANGE_TYPE_SIGN_[0, 30)_LEVEL_2: [mAP 0] [mAPH 0]
RANGE_TYPE_SIGN_[30, 50)_LEVEL_1: [mAP 0] [mAPH 0]
RANGE_TYPE_SIGN_[30, 50)_LEVEL_2: [mAP 0] [mAPH 0]
RANGE_TYPE_SIGN_[50, +inf)_LEVEL_1: [mAP 0] [mAPH 0]
RANGE_TYPE_SIGN_[50, +inf)_LEVEL_2: [mAP 0] [mAPH 0]
RANGE_TYPE_CYCLIST_[0, 30)_LEVEL_1: [mAP 0.854262] [mAPH 0.843677]
RANGE_TYPE_CYCLIST_[0, 30)_LEVEL_2: [mAP 0.848135] [mAPH 0.837626]
RANGE_TYPE_CYCLIST_[30, 50)_LEVEL_1: [mAP 0.733669] [mAPH 0.722557]
RANGE_TYPE_CYCLIST_[30, 50)_LEVEL_2: [mAP 0.695306] [mAPH 0.684749]
RANGE_TYPE_CYCLIST_[50, +inf)_LEVEL_1: [mAP 0.616862] [mAPH 0.602011]
RANGE_TYPE_CYCLIST_[50, +inf)_LEVEL_2: [mAP 0.576674] [mAPH 0.562745]

[shawnding]

Hi, @Abyssaledge

I noticed there is a 6-frame FSD++ on the WOD leaderboard here.

Do you mind publishing the detailed performance of this 6f model on the evaluation set?

Thanks!

[Abyssaledge]

FSD++ is our new work, which does not simply concatenate 6-frame point clouds. We are going to release it one month later.

[shawnding]

I see. Do you have the performance of the plain FSD on WOD validation set using 6 past frames? Thanks

[Abyssaledge]

Below is the validation results of plain FSD uses 6 past frames, trained with 6 epochs. I have not tried the better 12-epoch schedule yet. Hope it helps.

OBJECT_TYPE_TYPE_VEHICLE_LEVEL_1: [mAP 0.799491] [mAPH 0.794361]
OBJECT_TYPE_TYPE_VEHICLE_LEVEL_2: [mAP 0.715469] [mAPH 0.710729]
OBJECT_TYPE_TYPE_PEDESTRIAN_LEVEL_1: [mAP 0.852234] [mAPH 0.823217]
OBJECT_TYPE_TYPE_PEDESTRIAN_LEVEL_2: [mAP 0.780457] [mAPH 0.752212]
OBJECT_TYPE_TYPE_SIGN_LEVEL_1: [mAP 0] [mAPH 0]
OBJECT_TYPE_TYPE_SIGN_LEVEL_2: [mAP 0] [mAPH 0]
OBJECT_TYPE_TYPE_CYCLIST_LEVEL_1: [mAP 0.799318] [mAPH 0.790754]
OBJECT_TYPE_TYPE_CYCLIST_LEVEL_2: [mAP 0.773707] [mAPH 0.765401]
RANGE_TYPE_VEHICLE_[0, 30)_LEVEL_1: [mAP 0.928514] [mAPH 0.924518]
RANGE_TYPE_VEHICLE_[0, 30)_LEVEL_2: [mAP 0.917057] [mAPH 0.913095]
RANGE_TYPE_VEHICLE_[30, 50)_LEVEL_1: [mAP 0.790543] [mAPH 0.78467]
RANGE_TYPE_VEHICLE_[30, 50)_LEVEL_2: [mAP 0.725964] [mAPH 0.720454]
RANGE_TYPE_VEHICLE_[50, +inf)_LEVEL_1: [mAP 0.614176] [mAPH 0.605479]
RANGE_TYPE_VEHICLE_[50, +inf)_LEVEL_2: [mAP 0.483056] [mAPH 0.475891]
RANGE_TYPE_PEDESTRIAN_[0, 30)_LEVEL_1: [mAP 0.877269] [mAPH 0.851449]
RANGE_TYPE_PEDESTRIAN_[0, 30)_LEVEL_2: [mAP 0.841557] [mAPH 0.81575]
RANGE_TYPE_PEDESTRIAN_[30, 50)_LEVEL_1: [mAP 0.84102] [mAPH 0.812239]
RANGE_TYPE_PEDESTRIAN_[30, 50)_LEVEL_2: [mAP 0.777104] [mAPH 0.749143]
RANGE_TYPE_PEDESTRIAN_[50, +inf)_LEVEL_1: [mAP 0.804108] [mAPH 0.772521]
RANGE_TYPE_PEDESTRIAN_[50, +inf)_LEVEL_2: [mAP 0.673028] [mAPH 0.643962]
RANGE_TYPE_SIGN_[0, 30)_LEVEL_1: [mAP 0] [mAPH 0]
RANGE_TYPE_SIGN_[0, 30)_LEVEL_2: [mAP 0] [mAPH 0]
RANGE_TYPE_SIGN_[30, 50)_LEVEL_1: [mAP 0] [mAPH 0]
RANGE_TYPE_SIGN_[30, 50)_LEVEL_2: [mAP 0] [mAPH 0]
RANGE_TYPE_SIGN_[50, +inf)_LEVEL_1: [mAP 0] [mAPH 0]
RANGE_TYPE_SIGN_[50, +inf)_LEVEL_2: [mAP 0] [mAPH 0]
RANGE_TYPE_CYCLIST_[0, 30)_LEVEL_1: [mAP 0.853747] [mAPH 0.845311]
RANGE_TYPE_CYCLIST_[0, 30)_LEVEL_2: [mAP 0.848122] [mAPH 0.83974]
RANGE_TYPE_CYCLIST_[30, 50)_LEVEL_1: [mAP 0.782653] [mAPH 0.773833]
RANGE_TYPE_CYCLIST_[30, 50)_LEVEL_2: [mAP 0.745297] [mAPH 0.736878]
RANGE_TYPE_CYCLIST_[50, +inf)_LEVEL_1: [mAP 0.683078] [mAPH 0.6742]
RANGE_TYPE_CYCLIST_[50, +inf)_LEVEL_2: [mAP 0.642995] [mAPH 0.634601]