Avito Duplicate Ads Detection¶
luoq08@gmail.com OR hzluoqiang@corp.netease.com
data¶
size¶
- train pair: 2991396
- test pair: 1044196
- train info: 3344613
- train test: 1315205
- image: 11380670
result analysis¶
parameter tuning¶
grid searchand fit on leaderboard- max_depth: 5 -> 10 -> 15(auc increase in local, auc decrease in leaderboard)
- subsample=0.8, colsample_bytree=0.8
- min_child_weight = 1
- learning_rate=0.05, n_estimators=1000
feature size¶
features_train = pd.concat((
simple_features_train,
aggregation_features_train,
title_features_train, description_features_train, ncd_features_train,
image_features_train,
corpus_based_features_train,
dummy_features_train,
categoryID_shuffle_features_train,
), axis=1)| feature group | size |
|---|---|
| simple | 26 |
| aggregation | 20 |
| text simple | 18 |
| image | 44 |
| text vector space | 110 |
| dummy | 51 |
| total | 290 |
time¶
- simple, aggregation, text simple: hours
- image histogram, hash: each 1day with 8(16?) cores
- image mxnet: about 5days with GPU(Titan)
- text vector space: overnight
- model training, 32 threads
CPU times: user 3d 10h 19min 40s, sys: 2min 51s, total: 3d 10h 22min 32s Wall time: 3h 4min 33s
feature importance¶
not representative for leaderboard
- by weight(the number of times a feature is used to split the data across all trees)
180 title_word_1_2gram_dtm_0_predict_log_price__1 0.015352
181 title_word_1_2gram_dtm_0_predict_log_price__2 0.015343
107 mxnet_bn_batch_mean_sim 0.014696
4 description_length_max 0.014154
5 description_length_min 0.013829
11 price_diff 0.013002
184 title_description_dtm_0_predict_log_price__1 0.012954
185 title_description_dtm_0_predict_log_price__2 0.012500
105 mxnet_bn_batch_max_sim 0.011103
14 price_min 0.010932
30 locationID_1_freq 0.010878
13 price_max 0.010721- by gain (the average gain of the feature when it is used in trees)
image_phash_hamming_0_min 2875.218537 mxnet_bn_batch_max_sim 1869.406305 categoryID_112 1714.816033 title_word_dtm_0_1__binary_tfidf__cosine 1053.358729 image_dhash_hamming_0_min 811.860253 title_word_dtm_0_1__tfidf__cosine 359.690265 categoryID_9 311.754781 categoryID_33 275.383379 categoryID_111 254.657611 price_diff_ratio 254.372805
auc by categoryID¶
- not much image in category 112
112 0.858464
85 0.903141
33 0.913381
111 0.913763
101 0.927559
105 0.930247
31 0.932722
99 0.934950
10 0.940427
23 0.942282
97 0.944655
19 0.947516
26 0.948816
98 0.949131
34 0.949929
25 0.950278
32 0.951069
115 0.951840
42 0.951955
86 0.954261
21 0.957213
40 0.958290
24 0.958903
87 0.959818
84 0.960464
39 0.960946
96 0.963323
102 0.966269
29 0.966634
38 0.966994
36 0.967149
90 0.967512
83 0.968200
27 0.969401
28 0.971950
94 0.974298
114 0.974501
20 0.974637
89 0.976800
9 0.978176
106 0.980815
88 0.981223
30 0.982920
93 0.983322
81 0.984331
14 0.985742
82 0.987740
92 0.988143
11 0.989218
91 0.989520
116 0.993231feature contribution¶
impact on final result without imagehash and mxnet features
In [2]:
pd.read_csv('auc.old.csv')
Out[2]:
leaderboard¶
progress¶
In [3]:
leaderboard = pd.read_csv('avito-duplicate-ads-detection_public_leaderboard.csv',
parse_dates=['SubmissionDate'])
# remove bad data
leaderboard = leaderboard[leaderboard['TeamId']!=334028]
leaderboard[leaderboard.TeamName=='luoq']
Out[3]:
top 20¶
In [4]:
leaderboard.groupby('TeamName')['Score'].max().sort_values(ascending=False).iloc[:20]
Out[4]:
In [10]:
top20_team = leaderboard.groupby('TeamName')['Score'].max().sort_values(ascending=False).iloc[:20].index.tolist()
p = TimeSeries(leaderboard[leaderboard.TeamName.apply(lambda x: x in top20_team)],
x='SubmissionDate', y='Score', color='TeamName',
plot_width=1000, legend=False)
show(p)
Out[10]:
In [11]:
p = TimeSeries(leaderboard[(leaderboard.TeamName.apply(lambda x: x in top20_team)) & (leaderboard.Score>=0.9)],
x='SubmissionDate', y='Score', color='TeamName',
plot_width=1000, legend=False)
show(p)
Out[11]:
interesting finding¶
- gap between train and test
- not iid
- ordered by time
- text feature
- bad result for some category
- bad case analysis: not useful