ãªããŒã¿ã§ãããã€ãã¿ãŒã§äººå·¥ç¥èœã®ããšãä»åªäœã§æžããŠããèšäºãªã© ã玹ä»ããŠããŸãã®ã§ã人工ç¥èœã®ããšããã£ãšç¥ãããæ¹ãªã©ã¯æ°è»œã«@omiita_atiimoããã©ããŒããŠãã ããïŒ
ãã©ã¡ãŒã¿æ°10åïŒææ°ã®å·šå€§ç»åèªèã¢ãã«ãBiTãçèª & 解説
2019幎12æ24æ¥ã®ã¯ãªã¹ãã¹ã€ãã«arxiväžã§Google Brainããæ°ããªç»åèªèã¢ãã«ãçºè¡šãããŸããããã®åã BiT(=Big Transfer)ããã®æ§èœã¯2019幎ã«GoogleãåºããEfficientNet(æè解説èšäº)ãæ§ã ãªããŒã¿ã»ããã§è¶ ããã»ã©ã§çŸåšã®State-of-The-Art ã«ãªã£ãŠããŸããé©ãã¹ãã¯ããã ãã§ãªãããªããšãã®ã¢ãã«ããã©ã¡ãŒã¿æ°ã10åã«ãããã¶å·šå€§ãªã¢ã³ã¹ã¿ãŒã¢ãã« ã«ãªã£ãŠããŸãããããªBiTã«ã€ããŠæ©éãã®èšäºã§è§£èª¬ããŠãããããšæããŸãããããããŒã ãããããã¢ãŠããWeight Decayãªã©ã䜿çšããŠããªããšããã ä»ãŸã§ã®ç»åèªèã¢ãã«ã§ã¯ããã¡ã¯ãã§äœ¿ã£ãŠããæè¡ãã¡ã䜿ããã«SoTAãéæããã®ãããããã ã§ããBiT-Mã¢ãã«ãå ¬éããã ããã§ãã
èªãã§å°ãã§ãäœãåŠã¹ããšæããã ããããã ã ãã³ã¡ã³ãã ããããããšããããã®å±ã¿ã«ãªããŸãïŒãããããé¡ãããŸãïŒ
æµã:
- å¿ããæ¹ãž
- è«æè§£èª¬
- ãŸãšããšææ
- çšèªãšè£è¶³
- åè
åè«æ: Large Scale Learning of General Visual Representations for Transfer, Kolesnikov, A. et al.(2019)
0. å¿ããæ¹ãž
- 300äžæã®ç»åãå«ã巚倧ãªããŒã¿ã»ããã§ResNetv2-151ã®å¹ ã4åã«ãããã©ã¡ãŒã¿æ°10åã®å·šå€§ãªã¢ãã«ãäºååŠç¿ããŠãããåã¿ã¹ã¯ã«ãã¡ã€ã³ãã¥ãŒãã³ã°ãããSoTAå©ãåºããŸãã£ããã
- 䜿çšããããŒã¿ã»ããã®å€§ããã«ãã£ãŠBiT-S, BiT-M, BiT-Lã®ã¢ãã«ãããããSoTAã¯BiT-Lã ããBiT-Mã¯å ¬é ããããã
- ResNet-151ã§ã¯ãããããŒãã©ã€ãŒãŒã·ã§ã³ã§ã¯ãªãã**ãã°ã«ãŒãããŒãã©ã€ãŒãŒã·ã§ã³ + éã¿æšæºåã**ã䜿ã£ãŠãããã
- ã€ãã§ã«ããããã¢ãŠããšWeight Decay䜿ã£ãŠããªããã
- ãã¡ã€ã³ãã¥ãŒãã³ã°ã®æã«èª¿æŽãããã€ããŒãã©ã¡ãŒã¿ã¯ãã£ãã®3ã€ã ãã«çµã£ããã
- å ¥åç»åã®å€§ãã
- MixUpã®äœ¿çšã®æç¡
- åŠç¿çã¹ã±ãžã¥ãŒã«
- äžã®3ã€ã BiTãã€ããŒãã©ã¡ãŒã¿ ãšåŒãã§ãããã
1.ãè«æLarge Scale Learning of General Visual Representations for Transfer解説
1.0 èŠçŽ
å€§èŠæš¡ããŒã¿ã§äºååŠç¿ãããŠãããåã¿ã¹ã¯ã§ãã¡ã€ã³ãã¥ãŒãã³ã°ããããšããåŸæ¥ã®æ¹æ³ã䜿ã£ãŠããããã®ãã¡ãäºååŠç¿ã®ã¹ã±ãŒã«ã¢ãã ãš Big Transfer(BiT)ãã€ããŒãã©ã¡ãŒã¿ã®äœ¿çš ã«ããã倧尿§ã ãªç»åããŒã¿ã§SoTAãéæããã®è«æã§ã¯äºååŠç¿ã®ã¹ã±ãŒã«ã¢ãããšBiTãã€ããŒãã©ã¡ãŒã¿ã«ã€ããŠè©³çްã«èª¿æ»ããŠããã
1.1 å°å ¥
ãã£ãŒãã©ãŒãã³ã°ã¯ç»ååéã§å€§ããææãåºããŠãããããã®ããã«ã¯å€§éã®ããŒã¿ã«ããåŠç¿ãå¿ èŠã§èšç®éãè«å€§ã§ããããã®ãããåã¿ã¹ã¯ã§ã®èšç®éãæ¥µç«¯ã«å€§ãããªã£ãŠããŸããããã解決ããŠããã®ã転移åŠç¿(Transfer Learning)ã§ããã転移åŠç¿ãšã¯å€§èŠæš¡ããŒã¿ã«ãã£ãŠäºååŠç¿ããããã®éã¿ãåæå€ãšããŠåã¿ã¹ã¯ã§å床åŠç¿(Fine Tuning)ãããŠãã ãšãããã®ã§ãããããã§ã¯ããããäžæµ(Upstream)ãšäžæµ(Downstream)ãšåŒã¶ããšã«ããããã®äºååŠç¿+ãã¡ã€ã³ãã¥ãŒãã³ã°ãšããæŠç¥ã«ã¯å€§ãã2ã€å©ç¹ãããã
- äžæµã¿ã¹ã¯ã«ãããŠå°ãªãããŒã¿ã§é«ãææãåŸããã
- ãã£ãŒãã©ãŒãã³ã°ã®è«å€§ãªèšç®éãäžæµã¿ã¹ã¯ã«ã ãéäžããããã
ãã®è«æã§ã¯ãäºååŠç¿+ãã¡ã€ã³ãã¥ãŒãã³ã° ã®æŠç¥ã«æ³šç®ããŠããã
çŸåšã®ãã£ãŒãã©ãŒãã³ã°ã«ã¯ãEfficientNetã®ãããªã¢ãŒããã¯ãã£ãAdamããMixUpããããããŒãã©ã€ãŒãŒã·ã§ã³ãªã©æçšãªãã¯ããã¯ãå€ãããããã®ãããä»åã¯ãã®ãã¯ããã¯ã®ãã¡ããæå¹ãªãã®ã ãã«çµã£ãŠäœ¿çšãããããããŠæå¹ãªãã¯ããã¯ã ããçµã£ãŠäœ¿çšããã¢ãã«ã®ããšãBig Transfer(BiT) ãšåŒã¶ã
Large Scale Learning of General Visual Representations for Transfer, Kolesnikov, A. et al.(2019)
JFT-300MããŒã¿ã»ããã§äºååŠç¿ããã¢ãã«ã®ããšãBiT-LãšåŒã¶ããã®BiT-LãSoTAãå©ãåºããŸãã£ã(äžå³)ãããŒã¿ã»ãã5ã€ã®çµæã瀺ããããããæ£ã°ã©ããèŠããš3ã€ã®ã¢ãã«(BiT-L, SoTA, ResNet-50(ããŒã¹ã©ã€ã³))ãæ¯èŒããŠãããããããBiT-LãSoTAã¢ãã«ããããããç·ã°ã©ãã¯Few-shot Learningã®ããã«åã¯ã©ã¹ãããã®ç»åæ°ãæžãããæã®BiT-Lã®çµæã§æšªè»žã¯1ã¯ã©ã¹ãããã®ç»åæ°ã衚ããŠããã1ã¯ã©ã¹ãããå°ãªãç»åæ°ã§é«ã粟床ãåºããŠãã ããšãããããããå°ãå°ããImageNet-21kããŒã¿ã»ããã§äºååŠç¿ããã¢ãã«ãBiT-MãšåŒã¶ã
ããã§éèŠãªããšã¯ãBiT-Lã¯äºååŠç¿ã1床ã ãã§è¯ã ãšããããšããã¡ã€ã³ãã¥ãŒãã³ã°ããäžæµã¿ã¹ã¯ã¯èšç®éãå°ãããŠè¯ãã
ãŸãã転移åŠç¿ã®ããã®ãã€ããŒãã©ã¡ãŒã¿ãèšå®ãããã¥ãŒãªã¹ãã£ãã¯ããã®è«æã§çŽ¹ä» ããã
ãã®è«æã®ç®ç㯠SoTAã¢ãã«ã«æ°ããèŠçŽ ãªã©ã远å ããããšã§ã¯ãªããæå°éã®ã³ãã§ãäºååŠç¿+ãã¡ã€ã³ãã¥ãŒãã³ã°ãã«ããæ§èœãæå€§éã«åŒãåºãããš ã«ããã
ãã®ããã«æ³šç®ããã®ã¯ä»¥äžã®3ã€ã ãã
- åŠç¿ã®èŠæš¡
- ã©ã®ã¢ãã«ã
- åŠç¿æã®ãã€ããŒãã©ã¡ãŒã¿
1.2 BiTã®æ§æ
1.2.1 äžæµã®äºååŠç¿
泚ç®ããããšã¯2ã€ãåŠç¿ã®èŠæš¡ ãšãããããŒãã©ã€ãŒãŒã·ã§ã³ã®ä»²éã§ãã ã°ã«ãŒãããŒãã©ã€ãŒãŒã·ã§ã³ã®äœ¿çšã
倧ããããŒã¿ã»ããã«ãããŠã¯é·æéã§åŠç¿çã調æŽããå¿
èŠãããããšãããã£ããããŒã¿ã®çµ±èšæ
å ±ã䜿ããããããŒã ã¯ãç°ãªãããŒã¿ã䜿ã転移åŠç¿ã«åããªãã代ããã«ãã°ã«ãŒãããŒã +éã¿æšæºå(Weight Standardization)ãã䜿ã£ãããšã§ç²ŸåºŠãåäžããã
1.2.2 äžæµã¿ã¹ã¯ãžã®è»¢ç§»
æéã®ããããã€ããŒãã©ã¡ãŒã¿ãµãŒãã¯ãããããŠã次ã®3ã€ã ãã䜿çšããã¥ãŒãªã¹ãã£ãã¯ãªå€ãææ¡ã
- å ¥åç»åãµã€ãº
- MixUpã®äœ¿çšã®æç¡
- åŠç¿ã¹ãããæ°
ã¿ã¹ã¯ã倧äžå°ã§å®çŸ©ããããããã«åã£ãå€ã䜿ãã ãã詳现ã¯1.3.3.2ã«ãŠèª¬æã
ç»åã®ååŠçã¯ãããªãµã€ãºãããã¯ããããããããªãããã®3ã€ã ãã(ç»åã®æ¹åãæå³ãæã€ã¿ã¹ã¯ã§ã¯ããªãããããŠããªãã)
ãŸããWeight Decayããã³ããããã¢ãŠãã¯äœ¿ããªãã»ãã粟床ãããã£ãã
1.3 å®éš
1.3.1 äžæµã®äºååŠç¿ã«ãããããŒã¿ã»ãã
äºååŠç¿ã«çšããããŒã¿ã®å€§ããã®éãã§ã¢ãã«ã3çš®é¡ã«åããã
以äžã®è¡šã«ãŸãšããã
| ã¢ãã« | ããŒã¿ã»ãã | ç»åæ° | ã¯ã©ã¹æ° |
|---|---|---|---|
| BiT-S | ILSVRC-2012 | 1.28 M | 1 k |
| BiT-M | ImageNet-21k | 14.2 M | 21 k |
| BiT-L | JFT-300M | 300 M | 18 k |
1.3.2 äžæµã¿ã¹ã¯çšã®ããŒã¿ã»ãã
ILSVRC-2012, CIFAR-10/100, OxfordIIIT Pet, Oxford Flowers-102ãªã©ã®ããŒã¿ã»ãããšã19åã®ã¿ã¹ã¯ãå«ãVTABãã³ãããŒã¯ã䜿ã£ãŠBiTã®æ§èœã枬ã£ãã
1.3.3 ãã€ããŒãã©ã¡ãŒã¿ã®è©³çް
1.3.3.1 äžæµã®äºååŠç¿
ã¢ãã«ã¯ãããã ãããããŒã ãã°ã«ãŒãããŒã ããã³Weight Standardizationã«å€æŽããResNet-v2 ã䜿ã£ããããããµã€ãºã¯4096ã(ã§ãããã)
| ã¢ãã« | ã¢ãŒããã¯ã㣠|
|---|---|
| BiT-S | ResNet101x3 |
| BiT-M | ResNet101x3 |
| BiT-L | ResNet152x4 |
ããã§x3ãx4ãšããã®ã¯å ã ã®ã¢ãã«ã®å¹ (åã¬ã€ã€ãŒã®ãµã€ãº)ããã®ãŸãŸ3åã4åãããšããããšãBiT-Lã¯9.3ååã®ãã©ã¡ãŒã¿æ°ãæã€ã
1.3.3.2 äžæµã¿ã¹ã¯ãžã®ãã¡ã€ã³ãã¥ãŒãã³ã°
ã¿ã¹ã¯ããšã®ãã€ããŒãã©ã¡ãŒã¿ãªã©ã®èª¿æŽã¯æå°éã«æããããã®ã®ãã¿ã¹ã¯ã«ãã£ãŠå ¥åç»åãµã€ãºãããŒã¿ã»ããã®å€§ãããç°ãªãããå šã¿ã¹ã¯ã«åããã€ããŒãã©ã¡ãŒã¿ã䜿ãã®ã¯ããŸãè³¢ããªããããã§åé ã§ãè¿°ã¹ãããã«ã調æŽãããã€ããŒãã©ã¡ãŒã¿ã3ã€ã«çµã£ã ããŸããããããã®å€ã¯å ¥åç»åãµã€ãºãšããŒã¿ã»ããã®å€§ããã«ãã£ãŠæ±ºå® ããããšã«ãããããã BiTãã€ããŒãã©ã¡ãŒã¿ ãšåŒã¶ãBiTãã€ããŒãã©ã¡ãŒã¿ã®ããŒãã«ã¯ä»¥äžã
Large Scale Learning of General Visual Representations for Transfer, Kolesnikov, A. et al.(2019)ãããšã«äœæ
-
ç»åãµã€ãº:
äžæµã¿ã¹ã¯ã«ãããŠå ¥åç»åãµã€ãºã96px以äžã§ããã°ã160pxã«ç»åããªãµã€ãºããã®ã¡128pxã®å€§ããã§ã¯ããããããšããæå³ã§ããã96px以äžã«ãããŠãåæ§ã«æžããŠããåŠçãè¡ãã -
MixUp:
ããŒã¿ã»ããã®å€§ããã«ãã£ãŠMixUpãè¡ããè¡ããªãããæ±ºãããMixUpã®ä¿æ° $\alpha=0.1$ ãšããã -
åŠç¿ã¹ãããæ°:
ããŒã¿ã»ããã®å€§ããã«ãã£ãŠãã¡ã€ã³ãã¥ãŒãã³ã°ã§ã®ã¹ãããæ°ãæ±ºãããããã§åŠç¿çã¯åŠç¿ã¹ãããã®30 %, 60 %, 90 %ã§ $\frac{1}{10}$ ãšãªãã
1.3.4 ç»åèªèãã³ãããŒã¯ã§ã®ã¢ãã«è©äŸ¡
Large Scale Learning of General Visual Representations for Transfer, Kolesnikov, A. et al.(2019)
BiT-Lããžã§ãã©ãªã¹ãã¢ãã«ã®SoTAãšã¹ãã·ã£ãªã¹ãã¢ãã«ã®SoTAäž¡æ¹ãšæ¯èŒããã
ããã§ãžã§ãã©ãªã¹ãã¢ãã«ãšã¯EfficientNetã®ãããªæ§ã
ãªããŒã¿ã»ããã§é«ã粟床ãåºãããšãç®çã«äœãããã¢ãã«ã®ããšã§ãäžæ¹ã¹ãã·ã£ãªã¹ãã¢ãã«ãšã¯1ã€ã®ããŒã¿ã»ããã§é«ã粟床ãåºãããšãç®çã«äœãããã¢ãã«ã®ããšã
äžã®è¡šãèŠããšãããããã«ãBiT-Lã¯ãžã§ãã©ãªã¹ãã¢ãã«ãã¡ããã¯ãã¡ããã¹ãã·ã£ãªã¹ãã¢ãã«ãã¡ãšæ¯ã¹ãŠãããè¯ã粟床ãå©ãåºããŠãããããããç¹å¥ãªãã€ããŒãã©ã¡ãŒã¿ãã¥ãŒãã³ã°ãªã(ã€ãŸã BiTãã€ããŒãã©ã¡ãŒã¿ã ã)ã§éæããŠããã
Large Scale Learning of General Visual Representations for Transfer, Kolesnikov, A. et al.(2019)
ãŸããã¢ãŒããã¯ãã£ã¯åãã§äºååŠç¿ã®ããŒã¿ã»ãããç°ãªãã ãã®BiT-SãšBiT-Mãæ¯èŒããŠèŠããšãäºååŠç¿ã®ããŒã¿ã»ãã(ã®å€§ãã)ã®éãã«ãã£ãŠãã¡ã€ã³ãã¥ãŒãã³ã°ã¿ã¹ã¯ã®ç²ŸåºŠã倧ããå€ãã ããšããããã
1.3.5 ããŒã¿æ°ãå°ãªãå Žåã®ã¢ãã«è©äŸ¡
人工ç¥èœãããç ç©¶è ãæã€ç®æšãšã㊠ã1ã¯ã©ã¹ãããã®ããŒã¿æ°ãå°ãªããŠãé«ç²ŸåºŠããããã ããããã ãšãããã®ããããããã§ã¯ 1ã¯ã©ã¹ãããã®ããŒã¿æ°ãå€ããããšã§ã©ãã ãBiT-Lã¢ãã«ã®ç²ŸåºŠãå€ããã ãèŠãã
Large Scale Learning of General Visual Representations for Transfer, Kolesnikov, A. et al.(2019)
äžå³ãããããããã«ãBiT-Lã¯1ã¯ã©ã¹ãããã®ããŒã¿æ°ãå°ãªããŠãé«ã粟床ãå©ãåºããŠãããç¹çãã¹ãã¯ãILSVRC-2012ããŒã¿ã»ãã(äžçªå·Š)ã«ãããŠã¯1ã¯ã©ã¹ããã1æããç»åããªãå Žåã§ã粟床74.1 % ãå©ãåºããŠããããšã ã
1.4 詳ããåæ
ããã§ã¯ç¹ã«ä»¥äžã®5ã€ã®ããšã«ã€ããŠãã詳ãã調æ»ããŠããã
- ã¢ãã«ãšããŒã¿ã»ããã®å€§ãã
- 倧ããããŒã¿ã»ããã«å¯Ÿããæé©åã¢ã«ãŽãªãºã
- ã°ã«ãŒãããŒãã©ã€ãŒãŒã·ã§ã³ããã³éã¿æšæºå
- 転移åŠç¿ã«ããããã€ããŒãã©ã¡ãŒã¿ãã¥ãŒãã³ã°
- äºååŠç¿ãšäžæµã¿ã¹ã¯éã§ã®ç»åã®éè€ã«ãã圱é¿
1.4.1 ã¢ãã«ãšããŒã¿ã»ããã®å€§ãã
ããã§ã¯ã¢ãã«ãµã€ãºãšäºååŠç¿ããŒã¿ã»ãããµã€ãºã«ãã粟床ã®å€åãèŠãã
4ã€ã®ããŒã¿ã»ããã«å¯ŸããŠèª¿æ»ããã
Large Scale Learning of General Visual Representations for Transfer, Kolesnikov, A. et al.(2019)
åã°ã©ããäžæµã¿ã¹ã¯ã«å¯Ÿå¿ãã粟床ã«ãªã£ãŠãããæšªè»žãäºååŠç¿ããŒã¿ã»ãã瞊軞ããã®ç²ŸåºŠã§ãããããããã®ããããã¯äœ¿çšããã¢ãã«ã«å¯Ÿå¿ããã
å šäœãšããŠèšããã®ã¯ã倧ããããŒã¿ã»ããã§å€§ããã¢ãã«ãäºååŠç¿ãããšç²ŸåºŠãäžãã ãšããããšã§ããããŸãããã®ã°ã©ãããåãããããããããšã¯äž»ã«2ã€ã
- å¿ ãããããŒã¿ãå€ããŠãã¢ãã«ãå°ãããšç²ŸåºŠãäžãããªã
- 倧ããã¢ãã«ã¯å°ããäžæµã¿ã¹ã¯ã§ãé«ã粟床ãå©ãåºã
1ã€ç®ã«é¢ããŠã¯ã倧ããäºååŠç¿ããŒã¿ã§é«ã粟床ãåºãã«ã¯ãã®ããŒã¿ãµã€ãºã«å¿ãããµã€ãºã®ã¢ãã«ãå¿ èŠ ãšããããšã§ããã(äŸãã°äžã®å³ã§ResNet-50x1(å°ããéè²)ã3ã€ã®ããŒã¿ã»ããåã ã§äºååŠç¿ãããæã倧ãµã€ãºã®ããŒã¿ã»ããã§åŠç¿ãããæãããäžãµã€ãºã®ããŒã¿ã»ããã§äºååŠç¿ãããæ¹ãé«ã粟床ãåºããŠããã)
2ã€ç®ã«é¢ããŠã¯ã倧ããã¢ãã«ãVTAB-1kãšããå°ããã¿ã¹ã¯ã§é«ã粟床ãåºããŠãã ããšããèšãããVTAB-1kãšã¯19åã®å°ããã¿ã¹ã¯(åã¿ã¹ã¯ç»å1000æã®ã¿)ãå«ããã³ãããŒã¯ã§ãåã¿ã¹ã¯ã«å¯Ÿãã粟床ã®å¹³åå€ããšã£ããã®ãæçµçãªã¹ã³ã¢ãšãªãã
1.4.2 倧ããããŒã¿ã»ããã«å¯Ÿããæé©åã¢ã«ãŽãªãºã
ILSVRC-2012ã®ãããªã¹ã¿ã³ããŒããªããŒã¿ã»ããã«å¯ŸããåŠç¿ã¯åºãè¡ãããŠããããåŠç¿æ¹æ³ãžã®ãã¹ããã©ã¯ãã£ã¹ãããããBiT-Lã«çšããŠãããããªå·šå€§ãªäºååŠç¿ããŒã¿ã»ããã«å¯Ÿãããã¹ããã©ã¯ãã£ã¹ã¯ãŸã 確ç«ããŠããªãããã®ããããã§ã¯å·šå€§ãªããŒã¿ã»ããã«å¯ŸããåŠç¿æ¹æ³ã®ãã¹ããã©ã¯ãã£ã¹ã«ã€ããŠèª¿æ»ããã
ãã®çµæå·šå€§ãªäºååŠç¿ããŒã¿ã»ããã«å¯Ÿãããã¹ããã©ã¯ãã£ã¹ãšããŠããã£ãããšã¯ä»¥äžã
- ããªãé·ãåŠç¿ãèŠãã(æ°é±é粟床åäžãèŠããªããŠãæ°ã¶æåäœã§èŠããšç²ŸåºŠãäžããç¶ããŠãã)
- Weight Decayãå°ããæ¹( $1e^{-5}$ )ãåæãæ©ãããæçµçãªç²ŸåºŠã¯Weight Decayã倧ããæ¹( $1e^{-4}$ )ãããã£ãã
- æé©åã¢ã«ãŽãªãºã ã¯ã·ã³ãã«ãªã¢ãŒã¡ã³ã¿ã SGDãäžçªããã£ãã
1.4.3 ã°ã«ãŒãããŒãã©ã€ãŒãŒã·ã§ã³ããã³éã¿æšæºå
ä»å㯠ããããµã€ãºã4096ã§ãã巚倧 ãTPUã®Accelerator chipã128å䜿çšããã®å ŽåããããããŒãã©ã€ãŒãŒã·ã§ã³(BN)ãããã ãã°ã«ãŒãããŒãã©ã€ãŒãŒã·ã§ã³(GN)+éã¿æšæºå(WS)ãã®æ¹ãããã£ããããããµã€ãºã倧ããæ¹ãBNãããŸãããã¯ãã ããGN+WSãæ¡çšããæµãã¯ä»¥äžã
巚倧ã¢ãã«ã®äœ¿çš â ããã®ããã§ãããã«èšå€§ãªã¡ã¢ãªãå¿ èŠ â ããããããã®ããããµã€ãºãå°ããããŠç·©å â ããã€ã¹æ¯ã«BN䜿ã£ãããµã€ãºãå°ããããå®åçºæ®ã§ããªã â å šããã€ã¹ã«å æ¬çãªBN â æ¬ ç¹2ã€: 1.ããããµã€ãºã倧ãããããšéã«ãããªã 2.BNãèšç®ããé å»¶ãããã â ä»£æ¿æ¡ã®GN+WSãå¶åºŠããã£ãã
Large Scale Learning of General Visual Representations for Transfer, Kolesnikov, A. et al.(2019)
GN+WSãäºååŠç¿(Table 4)ããã³äžæµã¿ã¹ã¯(Table 5)ãããã«ãããŠãäžçªè¯ã粟床ãåºããŠãã ã®ããããã
1.4.4 転移åŠç¿ã«ããããã€ããŒãã©ã¡ãŒã¿ãã¥ãŒãã³ã°
ãã®è«æã§ã¯ ãã€ãã©ãã¥ãŒãã³ã°ããªãã¹ãé¿ããããšããçç±ããBiTãã€ããŒãã©ã¡ãŒã¿ãäžè²«ããŠäœ¿ã£ãŠãããããããèšç®ãªãœãŒã¹ã«äœè£ãããã®ã§ããã° åã¿ã¹ã¯æ¯ã«ãã€ãã©ãã¥ãŒãã³ã°ãããã«è¶ããããšã¯ãªãããã®ããããã§ã¯ã©ã³ãã ãµãŒãã䜿ã£ãŠãã€ãã©ãã¥ãŒãã³ã°ãè¡ã£ãŠã¿ããã¢ãã«ã¯BiT-Lãçšããäžæµã¿ã¹ã¯ãšããŠVTAB-1kã䜿ã£ãããã®çµæã¯ä»¥äžå³ã
Large Scale Learning of General Visual Representations for Transfer, Kolesnikov, A. et al.(2019)
ã©ã³ãã ãµãŒãã«ãã£ãŠãã€ãã©ãã¥ãŒãã³ã°ãããæ¹ãBiTãã€ããŒãã©ã¡ãŒã¿ãããè¯ãçµæã«ãªã£ãŠããããšãããããç¹çãã¹ãã¯ããã€ãã©ã®æ¢çޢ忰ã¯10~20åçšåºŠã§è¯ã ããšã
1.4.5 äºååŠç¿ãšäžæµã¿ã¹ã¯éã§ã®ç»åã®éè€ã«ãã圱é¿
äºååŠç¿ã§äœ¿ã£ãããŒã¿ã»ãããšäžæµã®ã¿ã¹ã¯ã§äœ¿ã£ãããŒã¿ã»ãããšã§ç»åãéè€ããŠãããšããŠãã粟床ã«ã»ãŒåœ±é¿ããªã ããšãããã£ããããŒã¿ã»ããå šäœã䜿ã£ãå Žåã®çµæãšããŒã¿ã»ãããã䌌ãç»åãé€ããå Žåã®çµæã¯äžè¡šãããã§Fullãå šäœã䜿ã£ãçµæãDedupãéè€ãé€ããçµæãDupsãšããã®ã"éè€"ãšå€æãããç»åæ°ã
Large Scale Learning of General Visual Representations for Transfer, Kolesnikov, A. et al.(2019)
ããããFullãšDedupãæ¯èŒãããšãããããçµæã«ã»ãŒåœ±é¿ããŠããªãããšãããã£ãã
1.5 é¢é£ç ç©¶
ããã§ã¯4åéã®é¢é£ç ç©¶ã玹ä»ãããŠãããããããã®è«ç¹ã ããåæããã
-
å€§èŠæš¡åŒ±æåž«ãã衚çŸåŠç¿:
転移åŠç¿ã«ãããŠã¯äºååŠç¿ã§å€§ããªããŒã¿ã»ããããã³ã¢ãã«ã䜿ãããšã§ç²ŸåºŠããããããšããã®è«æãéããŠããã£ãããã JFT-300Mããã倧ããããŒã¿ã»ãã3.5B InstagramããŒã¿ã»ãããResNeXtã¢ãã«ã«å¯ŸããŠäœ¿ã£ãŠããé¢é£ç ç©¶ããããããããããä»åã®ãResNet + JFT-300M + BiTãã€ãã©ãããæåŒ·ã -
ã¹ãã·ã£ãªã¹ãã¢ãã«:
äžæµã®ã¿ã¹ã¯ã«åããããªããŒã¿ã»ããã䜿ã£ãŠäºååŠç¿ããã¹ãã·ã£ãªã¹ãã¢ãã«ã¯é«ã粟床ãã§ããã®ã®ã以äžã®2ã€æ¬ ç¹ããããããBiTã®ãããªãžã§ãã©ãªã¹ãã¢ãã«ãè¯ãã- åãã£ãŠäžæµã®ã¿ã¹ã¯ãç¥ã£ãŠããªããšãããªãã
- äžæµã®åã¿ã¹ã¯ã«ããããäºååŠç¿ãå¿ èŠãªããèšç®éã倧ããã
-
æåž«ãªã衚çŸåŠç¿ãšåæåž«ãã衚çŸåŠç¿
æåž«ãªãåŠç¿ãåæåž«ããåŠç¿ãèªå·±æåž«ããåŠç¿ãæåž«ããåŠç¿ãªã©è²ã ãªææ³ã®è¡šçŸåŠç¿ãç ç©¶ããé¢é£ç ç©¶ããããããã§ã¯ãæåž«ããåŠç¿+èªå·±æåž«ãã衚çŸåŠç¿ããäžçªãšãããŠããããä»åã®è«æã§ã¯ãå€§èŠæš¡ããŒã¿ã«å¯ŸããŠã®æåž«ããäºååŠç¿ããæå¹ã§ããããšãããã£ã ã -
Few-shot Learning:
ä»åã®è«æã§ã¯Few-shot Learningã«ã¯ãã©ãŒã«ã¹ããããããŸã§ã ãžã§ãã©ãªã¹ãã¢ãã«ãäœãããšã«æ³šåããã
1.6 èå¯
BiT-Lãééããçµæãèå¯ãããããã§ã¯CIFAR-10ã®ééããå šéšãšILSVRC-2012ã®ééããäžéšã瀺ãã
Large Scale Learning of General Visual Representations for Transfer, Kolesnikov, A. et al.(2019)
å·ŠãCIFAR-10ããŒã¿ã»ããã«ãããŠééããç»åå šéšã§ãå³ãILSVRC-2012ããŒã¿ã»ããã«ãããŠééããç»åã®äžéšã§ããã1ã€1ã€èŠããšããããããã¹ããã¯ããã¹ã§ãªã ããšãããããééããŠããåå ãã¢ãã«ã«ããã®ã§ã¯ãªããç»åã®ææ§ããã©ãã«ãã€ãºã«ãããã®ã§ããããã®ããã«ãCIFAR-10ãªã©ã®çŸåšã¹ã¿ã³ããŒãã§äœ¿ãããŠããããŒã¿ã»ããã¯ãã¯ãæ»ç¥æžã¿ ãšèšããã
2. ãŸãšããšææ
ä»åã®è«æã¯ç¹ã«æ°ããã¢ãã«ãèŠçŽ ãåãå ¥ããŠããã®ã§ã¯ãªããã転移åŠç¿ãã«ãã©ãŒã«ã¹ããŠç²ŸåºŠããããŠããã®ããããããããã®ããã«ã巚倧ãªäºååŠç¿+ã¿ã¹ã¯æ¯ã®ãã¡ã€ã³ãã¥ãŒãã³ã°ããšããã®ã§SoTAãçãæµãã¯NLPã®æ¹ã§ãèŠãŠåããã®ã§ãç»åèªèãNLPããããªã£ãŠããã®ã ããããå人çã«ã¯æ°ããã¢ãã«ãšãèŠçŽ ãå ¥ã£ãŠãããæ¹ãããããããã
ãã€ãã¿ãŒã§äººå·¥ç¥èœã®ããšãä»åªäœã§æžããŠããèšäºãªã© ã玹ä»ããŠããŸãã®ã§ã人工ç¥èœã®ããšããã£ãšç¥ãããæ¹ãªã©ã¯æ°è»œã«ãã©ããŒããŠãã ããïŒ ããããã³ã¡ã³ãããåŸ ã¡ããŠãããŸãïŒ
3. çšèªãšè£è¶³
3.1 MixUp
æ¬è«æäžã§ã¯äžæµã¿ã¹ã¯ãžã®ãã¡ã€ã³ãã¥ãŒãã³ã°ã®éã®ãã€ããŒãã©ã¡ãŒã¿ãBiTãã€ããŒãã©ã¡ãŒã¿ãã«ãŠMixUpã䜿çšãããŠããã
MixUpãšã¯Zhang, H. et al.(2017)ã§ææ¡ãããæ°ããªData Augmentationææ³ãã¢ãã«ã®æ±åæ§èœãGANã®æ§èœã®å®å®æ§ãåäž ããã广ããããKaggleã§ããã䜿ãããŠãã(ãããã) (ãœãŒã¹)
äžèšã§èšãã°ãã2ã€ã®ããŒã¿ãç·åœ¢è£éãæ°ããªããŒã¿ãçæããããšããããšã
äŸãã°ãããã©ãã«ãæã€ç»åããŒã¿ã2ã€ãããšãããããã§ãããã $(x_1, y_1)$, $(x_2, y_2)$ ãšãããããã§ $y$ ã¯one-hot衚çŸããã®æ MixUpãšã¯æ°ããªããŒã¿ $(\lambda x_1 + (1-\lambda)x_2, \lambda y_1 + (1-\lambda)y_2)$ ãäœã ããšãããã§ $\lambda$ ã¯ããŒã¿ååžãããµã³ããªã³ã°ããã[0,1]ã®ç¯å²å ã®å€ã§ãFacebookã«ããPyTorchã®å®è£ ã§ã¯ååžã¯ $\mathrm{Beta}(1.0, 1.0)$ ãããã©ã«ããšããŠããã
3.2 éã¿æšæºå(Weight Standardization)
æ¬è«æäžã§ã¯BiTã¢ãã«(巚倧ãªResNet)ã«ãŠãããããŒã ã®ä»£ããã«ã°ã«ãŒãããŒã ãšå ±ã«éã¿æšæºå(WS)䜿ãããŠããã
éã¿æšæºåãšã¯Qiao, S. et al.(2019)ã§ææ¡ãããæ°ããªNormalizationææ³ãåçŽã«CNNã«ãããéã¿(Filter)ãå ¥åãã£ãã«æ¹åã§æšæºåããã ãã以äžè«æããã®åŒçšã $C_{in}, C_{out}$ ã¯ããããConvå±€ã«ãããå ¥åãã£ãã«æ°ãšåºåãã£ãã«æ°ã瀺ãã
Weight Standardization, Qiao, S. et al. (2019)
3.3 ã©ã³ãã ãµãŒã
æ¬è«æäžã§ã¯è»¢ç§»åŠç¿ã«ãããBiTãã€ããŒãã©ã¡ãŒã¿ã§ã¯ãªãå®éã«ãã€ããŒãã©ã¡ãŒã¿ãæ¢çŽ¢ããå Žåã®ç²ŸåºŠå€åãèŠãéã«ãã©ã³ãã ãµãŒãããã®æ¢çŽ¢æ¹æ³ãšããŠçšããããŠããã
ã©ã³ãã ãµãŒããšã¯Bergstra, J. et al.(2012)ã«ãŠæçšæ§ã説ããããã€ããŒãã©ã¡ãŒã¿ã®æ¢çŽ¢æ¹æ³ããã«ãŒããã©ãŒã¹ãªã°ãªãããµãŒããããäœãèšç®ã³ã¹ãã§ãé«ç²ŸåºŠãªãã€ãã©ãç²åŸã§ããã
ããçšåºŠåŠç¿çãã¬ã€ã€ãŒå±€ãªã©ã®ãã€ãã©ã®å€ããããã(äŸãã°5åãã€)決ããå šéãã®çµã¿åããã§ã¢ãã«ã®ç²ŸåºŠãèŠãŠã¿ãã°ãªãããµãŒããšã¯ããšãªããã©ã³ãã ãµãŒããšã¯åãã€ãã©ã®åãããç¯å²ã ããæå®ããŠããšã¯ã©ã³ãã ã§çµã¿åãããä»»æã®æ°è©Šãã¢ãã«ç²ŸåºŠãèŠãæ¢çŽ¢æ¹æ³ ã®ããšã
4. åè
-
Large Scale Learning of General Visual Representations for Transfer
(è«æ) åè«æã -
A Newbieâs Guide to Stochastic Gradient Descent With Restarts
(è±èª) åŠç¿çã¢ããŒãªã³ã°ã«ã€ããŠã®èª¬æã -
Mixup data augmentation
(è±èª) fast.aiã®ã¹ã¬ãããMixUpã«ã€ããŠå ·äœäŸã䜿ããªããããããããè°è«ãããŠããã -
mixup: Data-Dependent Data Augmentation
(è±èª) MixUpã®æŠèŠãšæ°åŒçãªèª¬æã -
Random Search for Hyper-parameter Optimization
(YouTube) ã©ã³ãã ãµãŒãã®èª¬æãã€ã³ãè±èªã -
In a CNN, does each new filter have different weights for each input channel, or are the same weights of each filter used across input channels?
(è±èª) StackExchangeã®è³ªåãCNNã®ãã£ã«ã¿ãŒã¯åºåãã£ãã«æ°çšæããã®ã¯ãã¡ããå ¥åãã£ãã«æ¯ã«ãå¥ã ã®ãã£ã«ã¿ãŒãçšæããã













