ãªããŒã¿ã§ãããã€ãã¿ãŒã§äººå·¥ç¥èœã®ããšãä»åªäœã§æžããŠããèšäºãªã© ã玹ä»ããŠããŸãã®ã§ã人工ç¥èœã®ããšããã£ãšç¥ãããæ¹ãªã©ã¯æ°è»œã«@omiita_atiimoããã©ããŒããŠãã ããïŒ
ãæ±ºå®çãã¹ãŒããŒããããããæé©åã¢ã«ãŽãªãºã
深局åŠç¿ãç¥ãã«ããã£ãŠãæé©åã¢ã«ãŽãªãºã (Optimizer)ã®çè§£ã¯é¿ããŠéããŸããã
ãã æé©åã¢ã«ãŽãªãºã ãçè§£ããããšãããšæ°åŒãåºãŠæ¥ãŠãããåŸé
éäžæ³ããã¢ãŒã¡ã³ã¿ã ããAdamãããçš®é¡ãå€ãããè€éã«èŠããŠããŸããŸãã
å®ã¯ãããããäœãããã®ã«ã¯ãã£ãããšããæµããããããããçè§£ããã° ç°¡åã«æé©åã¢ã«ãŽãªãºã ãçè§£ããããšãã§ããŸã ã
ããã§ã¯ããããã®æé©åã¢ã«ãŽãªãºã ãšæå€±é¢æ°ã®æå³ããå
¥ããææ¥éäžæ³ããæé©åã¢ã«ãŽãªãºã ã®å€§å®çªã®AdamãããŠäºé埮åã®ãã¥ãŒãã³æ³ãŸã§é ã远ã£ãŠ å³ããµãã ãã«äœ¿ããªããäžå¯§ã«è§£èª¬ ããŠãããŸãã
ããã§ã¯æ©éæé©åã¢ã«ãŽãªãºã ãšã¯äœãªã®ããèŠãŠã¿ãŸãããïŒ
èªãã§å°ãã§ãäœãåŠã¹ããšæããã ããããã ã ãã³ã¡ã³ãã ããããããšããããã®å±ã¿ã«ãªããŸãïŒãããããé¡ãããŸãïŒ
(现ããå³å¯æ§ãããçè§£ã®ããã®ãããããããåªå ããŠããã®ã§è«æã®ãããªå³å¯æ§ãæ±ããŠããå Žåã¯æé©åã¢ã«ãŽãªãºã ãã¡ã®åè«æãåœãã£ãŠããã ããŸããããé¡ãããŸãã)
Image Credit: Alec Radford
0. å¿ããæ¹ãž
- ææ¥éäžæ³: å šããŒã¿ ã䜿ã£ãп倱颿°ãæå°å€ã«ãªãããã«ãåŸé ã䜿ã£ãŠãã©ã¡ãŒã¿æŽæ°ãããã
- SGD: ããŒã¿1ã€ã ã ããµã³ãã«ããŠäœ¿ãããšã§ãææ¥éäžæ³ã«ã©ã³ãã æ§ãå ¥ãããã
- ãããããSGD: ããŒã¿ã16åãšã32å ãšã䜿ãããšã§äžŠåèšç®ã§ããããã«ããSGDã ãã
- ã¢ãŒã¡ã³ã¿ã : SGDã« ç§»åå¹³å ãé©çšããŠãæ¯åãæå¶ãããã
- NAG: ã¢ãŒã¡ã³ã¿ã ã§æå€±ãèœã¡ãããã«ä¿èšŒããŠãããã
- RMSProp: åŸé ã®å€§ããã«å¿ã㊠åŠç¿çãèª¿æŽ ããããã«ããŠãæ¯åãæå¶ãããã
- Adam: ã¢ãŒã¡ã³ã¿ã + RMSProp ã ããä»ã§ã¯è³ãæã§äœ¿ãããŠãããã
- ãã¥ãŒãã³æ³: äºé埮å ã䜿ã£ãŠããã®ã§ããã®ãããéãã§åæãããããã èšç®éãèšå€§ãã㊠å®çšãããŠããªã ãã
1. æé©åã¢ã«ãŽãªãºã ãšã¯ïŒ
SGDããã¢ãŒã¡ã³ã¿ã ããAdamããè²ã
ãªæé©åã¢ã«ãŽãªãºã ãèããŸããã
ãããã®æé©åã¢ã«ãŽãªãºã ã¯å
šãŠææ¥éäžæ³ã®æŽŸçã§ãææ¥éäžæ³ããçè§£ããã°ä»ã®æé©åã¢ã«ãŽãªãºã ã¯çè§£ã§ããŠããŸã ã®ã§ãã
ãã¥ãŒã©ã«ãããã¯ãŒã¯ãã¯ãããšããå
šãŠã®æ©æ¢°åŠç¿ã§å°éããããŽãŒã«ã¯äœã§ããããã
ããã¯ã極端ã«èšãã°ãæå€±ããŒãã«ãã ãšããããšã§ãããæå€±ããŒãã«ããããšãããŽãŒã«ããªãã¹ãå¹çããå°éããŠããããã®ãæé©åã¢ã«ãŽãªãºã ã§ãã
ãã®ããšã«ã€ããŠè©³ãã説æããŠãããŸãã
1.1 ããããæå€±ãšã¯ïŒ
ãããã æå€±ãšã¯ãã¥ãŒã©ã«ãããã¯ãŒã¯ã«ããäºæž¬å€ãšæ£è§£å€ãšã®å·® ã®ããšã§ãããã®ããããã¥ãŒã©ã«ãããã¯ãŒã¯ã®äºæž¬å€ãå€ããã°ãæå€±ãå€ãããŸãããã®ãã¥ãŒã©ã«ãããã¯ãŒã¯ã®äºæž¬å€ã¯ãã©ã¡ãŒã¿(éã¿ããã³ãã€ã¢ã¹)ã§å€åããŸããã€ãŸããã®æå€±ãå°ããããã«ã¯ã ãã¥ãŒã©ã«ãããã¯ãŒã¯ãæ£è§£å€ãåºãããã«ã®ãã®äžã®ãã©ã¡ãŒã¿ãã¡ã®å€ããããããšã€ãžãããã®ã§ãããã® ãã©ã¡ãŒã¿ãšããã®ã¯ãã¥ãŒã©ã«ãããã¯ãŒã¯ã®éã¿ãšãã€ã¢ã¹ã®ããš ã§ããã以äžã®åŒã§ãããš$w$ãš$b$ã§ããã$a$ ã¯ä»»æã®æŽ»æ§å颿°(SigmoidãšãReLUãšãã)ã§ãã
y = a(wx+b)
ãããããšæ¬¡ã®åé¡ãåºãŠããŸãããããã¯ãã¥ãŒã©ã«ãããã¯ãŒã¯ã®ãã©ã¡ãŒã¿ãã©ããããã°ããã®ãããšããããšã§ãããã®åã«ãããªãçè§£ã®ããã«å°ãæå€±ãšãã¥ãŒã©ã«ãããã¯ãŒã¯ã«ã€ããŠæ°åŒã§èŠãŠã¿ãŸãããã
ç¹°ãè¿ãã«ãªããŸããã æå€±ã¯äºæž¬å€ãšæ£è§£å€ã®å·® ãªã®ã§åŒã§è¡šããšã1ã€ã®ããŒã¿ã«å¯ŸããŠæå€±ã¯ä»¥äžã®ããã«ãªããŸãã
\mathcal{L} = (y - \hat{y})^2
ããã§ $y, \hat{y}$ ã¯ããããæ£è§£å€ãšãã¥ãŒã©ã«ãããã¯ãŒã¯ã«ããäºæž¬å€ã§ãã2ä¹ã«ããŠããã®ã¯åçŽã«æå€±ã®ç¬Šå·ãæ£ã«ãããã ã§ãã(ãã®äºä¹èª€å·®ã¯ç¹ã«MSE(Mean Squared Error)ãšèšãããŸãã)
1.2 æå€±ãšãã¥ãŒã©ã«ãããã¯ãŒã¯
ãã¥ãŒã©ã«ãããã¯ãŒã¯ã¯å ¥åã«å¯ŸããŠåºåãåºãã®ã§ããã¥ãŒã©ã«ãããã¯ãŒã¯ã颿°$f$ãšèŠããšãäºæž¬å€ $\hat{y}$ ã¯å ¥åããŒã¿$x$ã«å¯Ÿãããã¥ãŒã©ã«ãããã¯ãŒã¯ $f$ ã®åºåãªã®ã§ã$\hat{y} = f(x)$ ãšè¡šããããšãããããŸãã
ãã¥ãŒã©ã«ãããã¯ãŒã¯ãããå°ãå ·äœçã«å®çŸ©ããŠã¿ãŸããããããã§ã¯sigmoid颿°ã掻æ§å颿°ãšãã1å±€ã®ãã¥ãŒã©ã«ãããã¯ãŒã¯ã䜿ããŸããããããããã®ãããã€ã¢ã¹ $b$ ã¯é€ããŸããå³ã§è¡šããšä»¥äžã«ãªããŸãã
æ°åŒã§è¡šãã°ã
f(x) = \mathrm{sigmoid}(w_1x) =\frac{1}{1+e^{-w_1x}}
ãšãªãã®ã§ãå ã»ã©ã®æå€± $\mathcal{L}$ ã®åŒã¯ä»¥äžã®ããã«è¡šããŸãã
\mathcal{L} = (y - \hat{y})^2 = (y - \frac{1}{1+e^{-w_1x}})^2
ããã§å ¥åããŒã¿ $x=2$, æ£è§£å€ $y=1$ ã®ããŒã¿ããã£ããšãããšäžåŒã¯ãã©ã¡ãŒã¿ $w_1$ (éã¿)ãã®é¢æ°ãšããŠèŠãããšãã§ããŸããã€ãŸãã
\mathcal{L(w_1)} = (1 - \frac{1}{1+e^{-2w_1}})^2
= 1-\frac{2}{1+e^{-2w_1}}+\frac{1}{1+2e^{-2w_1}+e^{-4w_1}}
ãããæå€±é¢æ°ã§ããã®æå€±é¢æ°ãæå°å€ãåãããã« $w$ ãããããã°ãçµæçã«æé«ã®ãã©ã¡ãŒã¿ $w$ ãæã£ããã¥ãŒã©ã«ãããã¯ãŒã¯ãç²åŸã§ãããã§ã ã
ã€ãŸããæå€±é¢æ°ãæå°å€ãåããããªãã©ã¡ãŒã¿ãæ±ããã°ããã®ã§ãã
次ã«å³ã§çè§£ããŠã¿ãŸããä»åºŠã¯ãã¥ãŒã©ã«ãããã¯ãŒã¯ã2ã€ã®ãã©ã¡ãŒã¿$w_1, w_2$ãæã£ãŠãããšããŸããå³ã§è¡šãã°ä»¥äžã®ãããªãã¥ãŒã©ã«ãããã¯ãŒã¯ã§ãã
ãããæ°åŒã§è¡šããšã
f(x) = \sigma(w_2\sigma(w_1x))
ããã§ $\sigma(\cdot)$ ã¯sigmoid颿°ã®ããšã§ãããã®$w_1, w_2$ã®å€ãè²ã ãšèª¿æŽããããšã§ãã¥ãŒã©ã«ãããã¯ãŒã¯ã«ããäºæž¬å€ãå€ãã£ãŠãããŸããé©åœã«ããããšãäºæž¬å€ãæ£è§£å€ã«è¿ã¥ãæå€±ãå°ãããªã£ãããã¯ããŸãäºæž¬å€ãæ£è§£å€ããããæå€±ã倧ãããªã£ããããŸããå ã»ã©åæ§ãå ¥åããŒã¿ $x=2$, æ£è§£å€ $y=1$ ã®ããŒã¿ã§ã®æå€±é¢æ°ã¯ä»¥äžã«ãªããŸãã
\mathcal{L(w)} = (1 - \sigma(w_2\sigma(w_1\cdot2)))^2
ããã $w_1, w_2, \mathcal{L}(w)$ ã軞ãšããã°ã©ãã§è¡šããšäžå³ã®ããã«ãªããŸãã
æå€±ãå°ããããã°ããããã§ããããå³ã®äžã§äžçªå°ããæå€±å€(éãéšå)ããšããã©ã¡ãŒã¿ $w_1, w_2$ ããã®ãã¥ãŒã©ã«ãããã¯ãŒã¯ãç²åŸããããã©ã¡ãŒã¿ã®å€ ã§ãããã®æé©ãªãã©ã¡ãŒã¿ããªãã¹ãå¹ççã«ç²åŸããŠããããã®ãããæé©åã¢ã«ãŽãªãºã ãªã®ã§ãã
1.3 æå€±ãšæé©åã¢ã«ãŽãªãºã
ããŠããããŸã§ã§æé©ãª $w_1, w_2$ ãç²åŸããããšã®æå³ãçè§£ã§ããããšæããŸããã§ã¯å
·äœçã«ãã¥ãŒã©ã«ãããã¯ãŒã¯ã¯ã©ããã£ãŠãã®æé©ãªãã©ã¡ãŒã¿ãç²åŸããã®ã§ããããã
ããã§åºãŠããã®ã埮å ã§ãã
埮åã¯é¢æ°ã®ããç¹ã§ã®åŸããããã ãšããåªããã®ã§ãããããã¥ãŒã©ã«ãããã¯ãŒã¯ã§ã¯ããã®åªããæ§è³ªãæã€åŸ®åãäœåºŠãç¹°ãè¿ããã©ã€ã¢ã«&ãšã©ãŒã§æé©ãªãã©ã¡ãŒã¿ãç²åŸããŸãã
ã§ããã埮åãªããèšããŸã©ãã£ãããæ¹æ³ã䜿ããªããŠãäžã®ã°ã©ããèŠãã°ãããšæå°å€ãããããããäžã®ãããªã°ã©ããæžããŠæå°å€ãåãå€ãæé©ãª $w_1, w_2$ ãšããã°ãããããªãããããšæã£ãŠããŸãããã§ãã
ãã ãããèããŠã¿ãŠãã ããããã®ã°ã©ããç²åŸããã«ã¯ãæå€±é¢æ°ã«ããããªçµã¿åããã® $w_1, w_2$ ã倧éã«ä»£å
¥ããŠã¿ãŠãããããã§å€ãæ±ããŠå€§éã«ããããããŠããããšã§äžã®ãããªã°ã©ããæžããŸãããã®è²ã
ãªå€ã倧éã«èšç®ãããšããã®ã¯èŠã¯ãã«ãŒããã©ãŒã¹ã§å
šãè³¢ããããŸãããäžã®ã°ã©ãã®ãããª2次å
($w_1, w_2$)ãããã©ã¡ãŒã¿ããªããªãããªããã§ããããšæãæ¹ããããããããŸããããå®éã®æè¿ã®ãã¥ãŒã©ã«ãããã¯ãŒã¯ã®ãã©ã¡ãŒã¿æ°ã¯2次å
ã©ããã§ã¯ãªãã1000äžä»¥äžãã®ãã©ã¡ãŒã¿ãããããšãçãããããŸããããã¯ã1000äžã®æ°åã®çµã¿åããããæå€±ãïŒã€èšç®ããã®ã ãã§ãæéããããããã§ããããã«ãŒããã©ãŒã¹ã§ã¯ããã倧éã«èšç®ããã®ã§èšç®éã¯ãã¯ãç¡éã«ãªã£ãŠããŸããŸãã
ããã§è³¢ãæå€±é¢æ°ã®æå°å€ãæ±ããŠã¿ããããšããã®ã埮åã䜿ã£ãæ¹æ³ã§ãã
ãã®æ¹æ³ã®ååãææ¥éäžæ³ ã§ãã
2. ææ¥éäžæ³
ãŸããåèªã®æå³ã§ãããææ¥éäžãšããã®ã¯ãæå€±é¢æ°ã«ãããŠããç¹ããæå€±ãæãæ¥ã«éäžããã(æ¹å)ãšããããšã§ãã
ã€ãŸãããã©ã¡ãŒã¿ã§æå€±é¢æ°ã埮åããããšã§æå€±ãéäž(å°ãã)ããæ¹åã鿬¡çã«æ¢ããŠãã®æ¹åã«ãã©ã¡ãŒã¿ã調æŽãããŠããæ¹æ³ ã®ããšãè±èªã§Gradient DescentãšèšãããåŸé
éäžæ³ãšãèšãããŸãã
2.1 ææ¥éäžæ³ã®æé
- ãã¥ãŒã©ã«ãããã¯ãŒã¯ã«å ¥åããŒã¿ãå šéšå ¥ããŠäºæž¬å€ãåºãã
- æ£è§£å€ãšäºæž¬å€ã®äºä¹èª€å·®ã§ããèª€å·®é¢æ°ãå®çŸ©
- èª€å·®é¢æ°ããã©ã¡ãŒã¿ã§åŸ®å
- 埮åããŠåºãå€ã§ãã©ã¡ãŒã¿æŽæ°
- 1.ã«æ»ãã(ãã©ã¡ãŒã¿æŽæ°ã®å€åéãããªãå°ãããã°çµäºã)
ãã®æé ãæ°åŒã§è¡šããšãäžã®ãããªç°¡æœãªåŒãç¹°ãè¿ãããšãšçãããªããŸãã
$$
w_{t+1} = w_{t} - \alpha\nabla_w\mathcal{L}(w)
$$
ããã§ $w_t, \alpha, \nabla_w, \mathcal{L}(w)$ ã¯ãããããæŽæ°$t$åç®ã®ãã©ã¡ãŒã¿ã åŠç¿çã ãã©ã¡ãŒã¿ã§ã®åŸ®åãæå€±é¢æ°ã§ãããã©ã¡ãŒã¿ãæŽæ°ããããã®åŒã¯ç¹ã« æŽæ°åŒ ãšèšãããŸãã
2.2 ææ¥éäžæ³ã®æŽæ°åŒã®æå³
$$
w_{t+1} = w_{t} - \alpha\nabla_w\mathcal{L}(w)
$$
ã«ã€ããŠããå°ãå ·äœçã«èª¬æããŸãã
ããŒã¿(å
¥åå€ãšæ£è§£ã©ãã«(å€)ã®ãã¢)ã1000åãããšããŸãããã¥ãŒã©ã«ãããã¯ãŒã¯ã¯åãã¯ã©ã³ãã ã« $w_1$ ãš $w_2$ ãæã£ãŠããŸãã
(æé 1)ãã®æã®ããŒã¿1000åã䜿ã£ãŠãŸãã¯äºæž¬å€ã1000ååºããŸãã(æé 2)æ£è§£å€ãšã®æå€±ãMSEã§å®çŸ©ããŸããããã§ãæé 3ã®åŸ®åã«ç§»ãåã«å®éã«1000åã®æå€±ãèšç®ãããšããŠããã®1000åã®æå€±ã®ç·åãå
šäœã®æå€±ãšããŠå³ã«ããããããŠã¿ãŸãããã

æé 3ããã¯çŸåšã® $w_1, w_2$ ã«ããæå€±ãããå°ããæå€±ãšãªãããã«ã $w_1$ ãš $w_2$ ã埮調æŽããŸãããã®åŸ®èª¿æŽã«ã¯æ¬¡ã®2ç¹ã決ããå¿
èŠããããŸãã
- $w_1$ ãš $w_2$ ãããããå¢ããããããšãæžãããã
- ããããã©ãã ãå€åããããã
å ã«çµè«ãèšããšããããã解決ããæ¹æ³ã¯
- å¢ãããæžãããã¯åŸ®å ã«ããæ±ãã
- ã©ãã ãå€åããããã¯ãããããç§ãã¡ããªããšãªãåæã«æ±ºãã¡ãã
ã®ã§ãã
ç§ãã¡ããªããšãªãåæã«æ±ºãã¡ãããããªãã©ã¡ãŒã¿ã®ããšã ãã€ããŒãã©ã¡ãŒã¿ ãšèšããä»ã«ãã€ããŒãã©ã¡ãŒã¿ã®ä»²éãšããŠãããã¥ãŒã©ã«ãããã¯ãŒã¯ã®å±€ã®æ°ããã1å±€ã«ããããã¥ãŒãã³(ãŠããã)ã®æ°ããªã©ããããŸãã
ä»åã®ã©ãã ãå€åãããããæ±ºå®ãããã€ããŒãã©ã¡ãŒã¿ã®ããšã åŠç¿ç ãšèšããŸãã
ããããããã®ãããäžã®2åã®ãã€ã³ãããããã
- 埮å
- åŠç¿ç
ãšããŸãã
ããã§ã¯ææ¥éäžæ³ã«ãããããã©ã¡ãŒã¿å€ãå¢ãããæžããããæ±ºããããã»ã¹ãèŠãŠãããŸãã
2.2.1 [埮å]å¢ãããæžããã
赀䞞ãçŸåšã®ãã©ã¡ãŒã¿ã«å¯Ÿããæå€±ã衚ããŠããŸããå³ã®ãããªäœçœ®ã«èµ€äžžãæå€±é¢æ°ã«äœçœ®ãããšã㯠$w_1$ ãš $w_2$ ãšãã«å°ããããæ¹ãè¯ãããã§ãã
çŸåšã®æå€±(赀䞞)ãäžã®ããã«äœçœ®ããã®ã§ããã° $w_1$ ã¯å€§ãããã $w_2$ ã¯å°ããããæ¹ãè¯ãããã§ãã
ãããããäžå³ã®ãããªå
šäœã衚ãã°ã©ããæã«å
¥ããªãã®ã¯åè¿°ããéãã§ãããæäœã§ãä»ã® $w_1$ ãš $w_2$ ã®äœçœ®ãã©ãããåŸãã«ãªã£ãŠãããã ããèŠããã° $w_1$ ãš $w_2$ ãã©ãå€åãããã°ãããã¯ãããããã§ãã
ããç¹ã§åŸããåããšããã°ããŸãã«åŸ®å ã§ããã
ãŸã㯠$w_1$ ã§å埮åããŠçŸåšã® $w_1$ ãš $w_2$ ã代å
¥ããã° $w_1$ æ¹åã®åŸãããããã
$w_2$ ã§å埮åããŠçŸåšã® $w_1$ ãš $w_2$ ã代å
¥ããã° $w_2$ æ¹åã®åŸããããããŸãã
ããã«ãã£ãŠã $w_1$ ãš $w_2$ è¿èŸºã®åŸãã¯äžå³ã®ããã«ããããæå€±ãäžããããã«ã¯$w_1$ ãš $w_2$ ã倧ããããã®ããŸãã¯å°ããããã®ããããããŸããã
2.2.2 [åŠç¿ç]ã©ãã ãå€åãããã
ãšãã£ãŠãããããããæ±ºããŠãããåŠç¿ç(0~1ã®å€)ã䜿ãã ã ã§ãããã $10^{-5}$ ã $10^{-2}$ ãªã©ãèŠåããããŸãããã®åŠç¿çãå
ã»ã©åŸ®åããŠåºãŠããå€ã«æããã ãã
åŠç¿çã¯å¥åã¹ããããµã€ãº(step size)ãšãåŒã°ãããã©ã¡ãŒã¿ã®1åã®æŽæ°(ã¹ããã)ã®å€§ããã衚ãå€ ã«ãªã£ãŠããŸãã
åŠç¿çã倧ãããããšã¹ãããã倧ãããªããããŠæå°å€ãéãè¶ããŠããŸã(=ãªãŒããŒã·ã¥ãŒã)ãéã«åŠç¿çãå°ãããããš1åã®æŽæ°ã§å°ãããåããªããªãåæãŸã§æãŠããªãæéãããã£ãŠããŸããŸãã
2.2.3 ææ¥éäžæ³ã®æŽæ°åŒã®ãŸãšã
ããŒã¿å
šéšã䜿ã£ãŠãã®æã®æå€±ãããããããŸããæå€±é¢æ°ãä»ã®ãã©ã¡ãŒã¿å€ã§åŸ®åããæ¹åãæ±ããŸããåºãŠããæ¹åã«åŠç¿çãããããããã®åã ããã©ã¡ãŒã¿ $w_1$ ãš $w_2$ ãæŽæ°ããŸãã
æŽæ°ãã $w_1$ ãš $w_2$ ããã€ãã¥ãŒã©ã«ãããã¯ãŒã¯ã«åã³ããŒã¿å
šéšå
¥ããŠæå€±ãåºããã®çµæãããã©ã¡ãŒã¿ãæŽæ°ããããšããã®ãç¹°ãè¿ãããã®æŽæ°éãä»»æã®ãããå€ãããå°ãããªã£ããåŠç¿ãåæãããã®ãšããŠçµããã§ãã
æåŸã«ããäžåºŠæŽæ°åŒãèŠãŠèŠãŸãããã第äžé ãçŸåšã®ãã©ã¡ãŒã¿ã§ç¬¬äºé ããããæŽæ°ãããå€ã§ããããã§ $\alpha$ ãåŠç¿çã $\nabla \mathcal{L}(w)$ ããã©ã¡ãŒã¿ãæŽæ°ããæ¹åã衚ããŠããŸãã
$$
w_{t+1} = w_{t} - \alpha\nabla_w\mathcal{L}(w)
$$
2.2.4 åŠç¿ã®åæã«ã€ããŠ
æå°å€ããé ããã°åŸé
ã¯å€§ãããªããããäžåŒã§ãã©ã¡ãŒã¿ãžã®æŽæ°ã¯å€§ãããªããŸãã
äžæ¹ã§æå°å€ä»è¿åŸããå°ããããåŸé
ãå°ãããªããŸããã€ãŸãããã©ã¡ãŒã¿ã®æŽæ°ãå°ãããªããŸããæå°å€ã§ã¯ãã¯ãåŸãã¯0ã«ãªããã©ã¡ãŒã¿ã¯å
šãæŽæ°ãããªããªããŸãã
ãã®ããã« æå°å€ä»è¿ã§ã¯ãã©ã¡ãŒã¿ãå
šãæŽæ°ãããªãããããã©ã¡ãŒã¿ã®æŽæ°ãå°ãããã°åŠç¿çµäºããšã¿ãªãããšèšãããšã§ãã
2.3 ææ¥éäžæ³ã®æ¬ ç¹
ããŒã¿å šéšã®ããšãããããšèšããŸããææ¥éäžæ³ã§ã¯1åã®ãã©ã¡ãŒã¿æŽæ°ã§ããŒã¿å šéšã䜿ã£ãŠãããããäžæ°ã«ãã©ã¡ãŒã¿ãæŽæ°ã§ããŸãããèšç®éã倧ãããã€æé©è§£ã§ã¯ãªã極å°å€ã«é¥ã£ãŠããŸã£ãå Žåæãåºããªãããšããæ¬ ç¹ããããŸãããã èšç®éã倧ããããšã¯ã䞊åèšç®ã§è§£æ±ºã§ããŸããå šããŒã¿ã«å¯ŸããåŸé ãèšç®ããã©ã¡ãŒã¿ãæŽæ°ããã®ã§ããã®å šããŒã¿ã«å¯ŸããåŸé ã®èšç®ãè€æ°ã®ããœã³ã³ã§äžŠåèšç®ããã°è¯ãã®ã§ããããã§ã¯2ã€ç®ã®æ¥µå°å€ã«é¥ã£ãå Žåã«æãåºããªãããšããã®ã¯ã©ãããããšã§ãããããäŸãã°ä»¥äžã®å³ã®ãããªæå€±ã®ãšãã«
極å°å€ã§åžžã«åŸ®åã0ã«ãªã£ãŠããŸããŸãããããæ¥µå°å€ã«é¥ã£ãŠããŸããšèšãããšã§ãããããæã£ãŠãããã®ãã©ã³ãã æ§ ã§ãã
ãã®ã©ã³ãã æ§ãå«ãã ææ¥éäžæ³ããããSGD(=確ççåŸé
éäžæ³)ã®ããš ã§ãã
3. SGD
æ£åŒåç§°ã¯Stochastic Gradient Descentãšãã°ãã確ççåŸé
éäžæ³ã𿥿¬èªã§ã¯ããèšãããŸãã
ãã®SGDãšã¯ã©ã³ãã æ§ãå«ãã åŸé
éäžæ³ã®ããšãšèšããŸããããã©ã³ãã æ§ãã©ãå
¥ããŠããã®ãã
ããã¯è³ã£ãŠç°¡åã§ããå
ã»ã©ãŸã§ã¯äžåã®ãã©ã¡ãŒã¿æŽæ°ã«å
šããŒã¿ã䜿ã£ãŠããŸããããSGDã¯äžåã®æŽæ°ã§ã¯ããŒã¿ã1ã€ãã䜿ããŸããã
ææ¥éäžæ³ãšSGDã®éãã¯ããã©ã¡ãŒã¿ã®1åã®æŽæ°ã«å
šããŒã¿ã§ã¯ãªããã©ã³ãã ã«ããã¯ã¢ãããã1ã€ã®ããŒã¿ã䜿ããšããããšã ã ã§ãã
ããã§SGDã®èª¬æã¯çµãããªã®ã§ããããªãããã§æ¥µå°å€ã«é¥ãããšã解決ããŠãããã®ãããšããããšã«å°ãè§ŠããŸãã
ããã¯æ¯åã©ã³ãã ã«éãããŒã¿ã䜿ã£ãŠããããã1ã€åã®ããŒã¿ã§ããŒã«ã«ã«é¥ã£ããšããŠã次ã«ã©ã³ãã ã«éžãã ããŒã¿ã§ã¯æå€±ã倧ãããªã(ãŸãã¯å°ãããªãå¯èœæ§ããããã)ãããåã³ãã©ã¡ãŒã¿ã倧ããæŽæ°ããæ¥µå°å€ããè±åºã§ããŸãã
æ¯åã®ãã©ã¡ãŒã¿æŽæ°ã§åãããŒã¿ã䜿ã£ãŠãããšãäžå極å°å€ã«é¥ã£ããå
šãåãããŒã¿ãå
¥ããŠããã ããªã®ã§æ¬¡ãããã«é¥ãç¶ããçµå±æãåºããªãã®ã§ãã
3.1 SGDã®æ¬ ç¹
1ã€ã®ããŒã¿ã§åŸé
ãèšç®ãæŽæ°ããŠã次ã®ããŒã¿ã§åŸé
ãèšç®ãæŽæ°ããããšããã®ã鿬¡çã«è¡ããšããã«åé¡ããããŸããã€ãŸãã1ã€ã®ããŒã¿ã«ããåŸé
ã§æŽæ°ããªããšæ¬¡ã®ããŒã¿ã«ã¯ç§»ããªããšããããšã§ããããã¯äžŠååãã§ããŸããã
䞊ååãã§ããããã«ã©ãããã°ããã§ããããã
次é
ã®ãããããåŠç¿ãããã解決ããŠãããã®ã§ãã
4. ãããããåŠç¿SGD
å šããŒã¿ã䜿ã£ããããåŠç¿ã§ã¯ãŸãå šããŒã¿ã®åŸé ãèšç®ããã®ã§ãã®èšç®ã䞊ååããŸãããäžæ¹ã§SGDã¯åŸé èšç®ã®ããŒã¿æ°ã1ã€ã§ãã©ã¡ãŒã¿ã®æŽæ°ãçµãã£ããæ¬¡ã®ããŒã¿1ã€ã«å¯ŸããŠãã©ã¡ãŒã¿ãæŽæ°ããŸãããæŽæ°ã§äœ¿ãããŒã¿æ°ã1ã€ã ãããã䞊ååãã§ããªãã®ã ããããã®ããŒã¿æ°ãå¢ããã°ãããåŠç¿ã®ããã«äžŠååãã§ããŸãããã ããããåŠç¿ãšåãããã«å šããŒã¿ãŸã§å¢ãããŠããŸã£ãŠã¯ã©ã³ãã æ§ããªããªããŸãããšããããšã§ããããããåŠç¿SGDã§ã¯1åã®æŽæ°ã§äžå®æ°ã®ããŒã¿ã䜿ãåŠç¿ ããŸãã1åã®æŽæ°ã§çšããããŒã¿æ°ã¯16åã32åãªã©ããã䜿ããããã®ããŒã¿ã®å¡ãå°ããããããšããããšã§ããããã ãšèšããŸãããããããã®ãµã€ãºããã€ããŒãã©ã¡ãŒã¿ã§ãããã·ã³ã®ã¹ããã¯ãé«ãå Žåã¯å€§ãããµã€ãºãæå®ã§ããŸãã
4.1 ãããããåŠç¿SGDã®æ¬ ç¹
ãããŸã§ã§ææ¥éäžæ³ã®æ¬ ç¹ãããçšåºŠå
æããããšãã§ããã®ã§ããã
ããã«ã¯ããæå€±é¢æ°ã®æå°å€ã«ãã©ãçããããšããã¢ãããŒã·ã§ã³ããããŸãã
ããã§ åŠç¿ãé
ãããŠããŸãèŠå ãèããŸãããã®èŠå ã®1ã€ãšãªã£ãŠããã®ãPathological Curvatureã«ããã£ãŠããŸããšããããš ã§ãã
4.1.1 Pathological Curvatureãšã¯ïŒ
Pathological Curvatureãšã¯ä»¥äžã®å³ã®ãããªéãããŒã¿ãæã€åœ¢ç¶ã®ããšã§ãSGDã ãšãªãŒããŒã·ã¥ãŒã(1åºŠã®æŽæ°ã倧ããããããš)ããŠæ¯åãèµ·ãããŠããŸãããæå°å€ã«ãã©ãçãã®ãé ãã§ãã
ãã®æ¯åãæããŠä»¥äžã®å³ã®ããã«ãããããšãçæ³ã§ãã
æ¯åãæãããããã®ããéå»ã®åŸé
ã®å€åã䜿ã ãšãããã®ã§ãã
代衚çãªãã®ã« ã¢ãŒã¡ã³ã¿ã ãš RMSProp ããããŸãã
ããã§SGDã®æŽæ°åŒãããäžåºŠèŠãŠèŠãŸãããã
å®ã¯ãããã説æããæŽŸçç³»ã§ã¯ãã®åŒã®ç¬¬äºé
ã«ããã ã¢ãŒã¡ã³ã¿ã ã¯åŸé
(埮å)ããRMSPropã¯åŠç¿çãããã£ãŠããã ã ãªã®ã§ãã
ãã®ããã«ã©ã¡ããSGDã«äžå·¥å€«ãå ããŠããã ãã§ãã®ã§ãããŸã§çè§£ã§ããŠããã°ãããããã¯é£ãããšããã¯ç¹ã«ãããŸãããããããã¯åŸ®åãããæ£ç¢ºã« åŸé ãšåŒã¶ããšã«ããŸãããã ããããŸã§ã®ããã«åŸé ã¯é¢æ°ã®ããç¹ã§ã®åŸãã®æ¹åã衚ããŠãããšèããŠããŠå€§äžå€«ã§ãã
5. ã¢ãŒã¡ã³ã¿ã
ã¢ãŒã¡ã³ã¿ã (Momentum)ã¯ãæå€±é¢æ°äžã§ã®ä»ãŸã§ã®åããèæ ®ããããšã§SGDã®æ¯åãæãã ãšèšãèãã§å°å ¥ãããŸããã
ãã® ã¢ãŒã¡ã³ã¿ã ã¯ç§»åå¹³åãçè§£ããã°ç°¡åã«çè§£ãã§ãã ã®ã§ãã
5.1 ç§»åå¹³åãšã¯ïŒ
ç§»åå¹³åãšã¯çµæžã§ãã䜿ãããæ¥ãªå€åãããã°ã©ãã«å¯ŸããŠç§»åå¹³åãçšãããšãã®å€åãããããã«ãªã£ãã°ã©ããåŸãããåªããã®ã§ãã(æ£ç¢ºã«ã¯ææ°å¹³æ»ç§»åå¹³åãšããä»°ã ããååãã€ããŠããŸãããããã§ã¯ããããããç§»åå¹³åã«çµ±äžããŸãã)
ãªãªãžãã«ã®éç·ã«å¯ŸããŠç§»åå¹³åãçšããŠããã®ãèµ€ç·ã®ã°ã©ãã
æ¯åãæããç·©ããã«ãªã£ãŠããããšãããããŸãã
ã€ãŸããç§»åå¹³åã¯æ¥ãªå€åã«åããªãã°ã©ãã«ãªã£ãŠãã ããšãããããŸãã
ãã®èµ€ç·ã¯ä»¥äžã®åŒã§åç¹ãæ±ããããããããŠããŸãã
$$
\nu_t = \beta\nu_{t-1} + (1-\beta)G
$$
ããã§ $\nu_{t-1}, G$ ã¯ããããåæå»ã§ã®ç§»åå¹³åãããåŸã®å€ãGã¯çŸæå»ã®å€ã§ã$\beta$ ã¯0ãã1ã®å€ãåããã€ããŒãã©ã¡ãŒã¿ã§ããå³èŸºã«ãããŠã
- 第1é $\beta\nu_{t-1}$ : ä»ãŸã§ã® $G$ ãã¡ãèç©ããé
- 第2é $(1-\beta)G$ : çŸåšã®ç¹ã衚ãé
ãšããããšãããããŸãã$\beta$ ã倧ãããã°å€§ããã»ã©ç¬¬1é ã€ãŸãä»ãŸã§ã®å€ã®åœ±é¿ã倧ããã第2é ã€ãŸãçŸæå»ã®åœ±é¿ãå°ãããªããŸãã
5.2 ã¢ãŒã¡ã³ã¿ã ãšã¯ïŒ
æºãæããŠã®ã¢ãŒã¡ã³ã¿ã ã®ç»å Žã§ãããæ©éã¢ãŒã¡ã³ã¿ã ã䜿ã£ãSGDã®æŽæ°åŒãèŠããŸãã
\begin{align}
&\nu_t = \beta\nu_{t-1} + (1-\beta)\nabla_w\mathcal{L}(w) \\
&w_t = w_{t-1} - \alpha \nu_t
\end{align}
$\nu_{t-1}, \nabla_w\mathcal{L}(w), \alpha$ ã¯ããããååã®åŸé ãä»ã®åŸé ãåŠç¿çã§ãã$\beta$ ã¯0ãã1ã®å€ãåããã€ããŒãã©ã¡ãŒã¿ã§ããããã§ $\nabla_w\mathcal{L}(w)$ ãåçŽã« $G$ ãšè¡šããŠã¿ãŸãã
\begin{align}
&\nu_t = \beta\nu_{t-1} + (1-\beta)G \\
&w_t = w_{t-1} - \alpha \nu_t
\end{align}
SGDãšç°ãªããåŒã2ã€äœ¿ã£ãŠããŸãã
ãŸã第1åŒã§ãããäœãèŠãããšãããŸããã
ããã§ããç§»åå¹³åã®åŒã§ããå®ã¯ã¢ãŒã¡ã³ã¿ã ã¯åçŽã«åŸé
ãç§»åå¹³åããŠããã ã ãªã®ã§ãã
ç§»åå¹³åã¯æ¥ãªå€åã«åããªãããšèšããŸãããããã¯ã€ãŸããæ¯åã®ãããªæ¥æ¿ã«å€åã«åããªãããšããããšã§ããããããPathological Curvatureã§èµ·ããŠããæ¯åã¯æ£è² ã®ç¬Šå·ãéã«ãªãã»ã©ã®æ¯åã§ãããåæå»ã§æ£ã®æ¹åãåããŠããåŸé
ãçŸæå»ã§ã¯è² ã®æ¹åãåãåŸé
ã«ãªã£ãŠããŸãã»ã©ã®æ¯å ã§ãããã®ç§»åå¹³åã§ã¯åæå»ãšçŸæå»ãè¶³ãåãããã®ã§ãæ£è² ã§æ¯åããŠãããšæã¡æ¶ããã£ãŠæ¯åãå°ãããªã ã®ã§ãã
å®éã«æ£è² ãå€ããã»ã©ã®æ¯åã«å¯ŸããŠç§»åå¹³å(ã€ãŸãã¢ãŒã¡ã³ã¿ã )ãé©çšãããšã以äžã®ã°ã©ãã®ããã«ãªããŸãã
å ã æ¯åããŠããåŸé (éç·)ã¯å€§ãããŠãèŠäºã«æ¶å€± ãããŠããŸããã
第2åŒã¯ç§»åå¹³ååŸã®å€ã䜿ã£ãŠãã©ã¡ãŒã¿ãæŽæ°ããŠããã ãã§ãããšãŠãç°¡åã§ããã
5.3 ã¢ãŒã¡ã³ã¿ã ãå³ã§çè§£
å³(ãã¯ãã«å³)ãèŠããšããªãã¢ãŒã¡ã³ã¿ã (=æ £æ§)ãšèšãããã®ããçŽæçã«ãããããšæããŸããããã§äžè¿°ã®æŽæ°åŒããã¯ãã«å³ã§è¡šããšãäºé ã®åãªã®ã§ä»¥äžã®ããã«è¡šããŸããã
ãã®ããã«ãçŸåšã®åŸé ã®åãã«å¯ŸããŠãä»ãŸã§åããŠããæ¹åã€ãŸãæ £æ§(ã¢ãŒã¡ã³ã¿ã )ã®æ¹åãè¶³ãåãããæ¹åãæçµçãªæ¹å ãšããŠããã®ã§ãã
5.4 ã¢ãŒã¡ã³ã¿ã ã®æ¹è¯ç NAG
ããã§ ã¢ãŒã¡ã³ã¿ã ã®æ¹è¯çãšèšãããNesterov(ãã¹ããã)ã®å éåŸé æ³(=NAG) ã玹ä»ããŸããçµè«ããèšããšãNesterovã®å éåŸé æ³ã¯ã¢ãŒã¡ã³ã¿ã ã®æŽæ°åŒãšã»ãŒå€ãããŸããããã®èª¬æã®åã«ã¢ãŒã¡ã³ã¿ã ã«ã€ããŠããäžåºŠè§ŠããŸããã¢ãŒã¡ã³ã¿ã ã¯ãã°ããéå»ã®åŸé ãã¡ãèæ ®ããããšã§æ¥ãªå€åãæããããšãããã®ã§ãããåããŠãããã®ã¯ãã®æ¹åã«åãç¶ããããšããæ £æ§ã®ãããªãã®ãçŸåšã®åŸé ã«å¯ŸããŠä»ãå ããŠããŸããããã®æ £æ§ãããã¢ãŒã¡ã³ã¿ã ã®ããšã§ããããããããã®æ £æ§ã®æ¹åãæ£ããæ¹å(ã€ãŸãæå€±ãæžããæ¹å)ãåããŠãããšã¯éããŸããããã®æ £æ§ã®æ¹åãæ£ããæ¹åã§ããããšãå°ãã§ãä¿èšŒããããããšããã®ãNesterovã®å éåŸé æ³ ã§ããæ¹åãæ£ããã®ãã©ããã®ç¢ºèªæ¹æ³ã¯ä»ãŸã§éã埮åã§ã§ããŸããããšããããšã§ Nesterovã®å éåŸé æ³ã§ã¯ãã¢ãŒã¡ã³ã¿ã (æ £æ§)ã ãã§éã¿ãæŽæ°ããå Žåã®åŸé ãå ã«èŠãŠããã®ããšå®éã«æ £æ§ãå ããŠæŽæ° ããŸãããããåŒã§è¡šããšä»¥äžã«ãªããŸãã
\begin{align}
&\nu_t = \beta\nu_{t-1} + (1-\beta)\nabla_w\mathcal{L}(w-\beta\nu_{t-1}) \\
&w_t = w_{t-1} - \alpha \nu_t
\end{align}
ãããã¢ãŒã¡ã³ã¿ã ã®æŽæ°åŒãšããäžåºŠæ¯ã¹ãŠèŠãŸãããã
\begin{align}
&\nu_t = \beta\nu_{t-1} + (1-\beta)\nabla_w\mathcal{L}(w) \\
&w_t = w_{t-1} - \alpha \nu_t
\end{align}
ããèŠããšéãã1ã€ãããªãããšãããããŸãããå ã»ã©è¿°ã¹ãããã« å ã«æ £æ§é ã§éã¿ã倿ŽãããŠããåŸé ããšã£ãп޿° ããŠããŸããããããã¢ãŒã¡ã³ã¿ã åæ§ãã¯ãã«å³ã§æžããŠã¿ããšä»¥äžã®å³ã«ãªããŸãã
ã¢ãŒã¡ã³ã¿ã ã§ã¯çŸæç¹ããæ £æ§ããã³åŸé ãå¥ã ã«ãšã£ãŠããè¶³ãç®ãããŠããŸããããNesterovã§ã¯å ã«æ £æ§ã§éã¿ã倿ŽãããŠããåŸé ããšã£ãŠããŸããã
5.4.1 NAGã«é¢ããäœè«
ããããã¯äœè«ãšãããç§èªèº«ã®çè§£ã®ä»æ¹ãªã®ã§æ£ãããã¯ä¿èšŒã§ããªã(ç§ã¯æ£ãããšæã£ãŠãããã)ã®ã§ãããNAGãããå°ãåã¿ç ããèšèã§è¡šããšãæ £æ§é ã«ãã£ãŠç¢ºå®ã«æå€±ãèœã¡ãæ¹åãå ã«èŠã€ããŠãããå®éã«ãã®ç¹ã«åãããããã®ããªãšæã£ãŠããŸããäžã§è¡šããå³ã段éããã£ãŠæžããšããŸãæ £æ§é ã®ã¿ã§éã¿ãæŽæ°ããå Žåã«æå€±ãå°ããããåŸé ($w_{t+1}$)ãå ã«èŠã€ããŸãã
ããã㊠$w_{t+1}$ ãå ã«èŠã€ããŸãã$w_{t+1}$ ã¯åŸ®åã®ãããã§ç¢ºå®ã«æå€±ãäžãããã©ã¡ãŒã¿ã§ãã®ã§ãããšã¯ããã«åããããŸãããªã®ã§ãã®ããã«ã¯ç¹ç·ã®æ £æ§é ãå¿ èŠã§ãããã€ãŸããNesterovã®å éåŸé æ³ã®æŽæ°åŒç¬¬äžåŒã®ããã«æ £æ§é ãè¶³ããŠãããŸããããã§ãããããããŠããã®ã¯ãæ £æ§é ãæŽæ°åŒã®ç¬¬äžåŒå ã§2åã䜿ã£ãŠããŠãªããæ £æ§ãå¹ããããªããããªããããšæã£ãŠããŸãããã«ãªããŸãããã ããã§ç¬¬äžåŒã§æ £æ§é ã远å ããªãå ŽåãæŽæ°åŒã®ãã¯ãã«å³ã¯ä»¥äžã«ãªã£ãŠããŸããŸãã
ããã§ã¯ææã® $w_{t+1}$ ã«ãã©ãçããŠããªã ã§ããããªã®ã§ã¡ãããšèŸ¿ãçããããã«æ £æ§é ã远å ããŠãããŠæçµçã«ã¯ä»¥äžã®å³(åæ²)ã®ããã«ãªãã®ã§ããã
ç¹°ãè¿ãã«ãªããŸããããããŸã§ãç§èªèº«ã®çè§£ã§ã¯ Nesterovã®å éåŸé æ³ã¯æ £æ§é ã«ãã£ãп倱ãäžããæ¹åãå ã«èŠã€ããŠãããã®æ¹åã«åãããã ãã®ã ãšæã£ãŠããŸããééã£ãŠããã°ãææããã ãããšå€§å€å©ãããŸããNesterovã®å éåŸé æ³ã«é¢ããŠãã£ãšè©³çްãªèª¬æã¯ãã¡ãã®èšäºã§è§£èª¬ãããŠããŸããã®ã§ããã¡ãããåç §ãã ããã
6. RMSProp
RMSPropã¯MomentumãšåããSGDã®æ¯åãæãããšããç®çã§äœããããã®ã§ãããã ãã¡ã㯠åŠç¿çãèª¿æŽ ããŠããã®ã§ããã
Pathological Curvatureã§ã¯æ¯åæ¹åã¯æå°å€æ¹åã«æ¯ã¹ãŠåŸé
ã倧ãããªã£ãŠããŸãããããã解決ããã«ã¯ãæ¯åæ¹åã®åŠç¿çãäžããŠãããã°æ¯åãæãããããã§ãã
RMSPropã¯gradã®å€§ããã«å¿ããŠåŠç¿çã調æŽããããšãããã®ã§ãã
MomentumãŸã§çè§£ã§ããŠããã°ãšãŠãåçŽãªã®ã§æ©éæŽæ°åŒã§ãã
\begin{align}
&\nu_t = \beta\nu_{t-1} + (1-\beta)G^2 \\
&w_t = w_{t-1} - \frac{\alpha}{\sqrt{\nu_t + \epsilon}} G
\end{align}
äœãè€éããã«èŠããŸãã1åŒãã€ã¿ãŠãããšå®ã¯ãšãŠãåçŽãªã®ã§ãã $\epsilon$ ã¯ãŒãé€ç®ãé¿ããããã®æ¥µããŠå°ããå€($= 10^{-12}$ ãªã©)ã§ããæ¥µããŠå°ããã®ã§èšç®çµæã«ã¯åœ±é¿ãåãŒããŸããããã以å€ã®èšå·ã®æå³ã¯ã¢ãŒã¡ã³ã¿ã ã®ãšããšåãã§ãã
ãŸã第1åŒã¯ãç§»åå¹³åã®åŒã§ãã ããã§æ³šæããŠæ¬²ããã®ã¯ $G^2$ ã«ãªã£ãŠããããš ã§ããããã«ãããã¢ãŒã¡ã³ã¿ã ã®ããã«æ¯åãæ£è² ã§æã¡æ¶ãããã®ã§ã¯ãªããæ¯åããŠããæã®å€ã¯ããªã倧ãããªããŸãããªããè¯ããªãããã§ããããã第2åŒã«å¹ããŠããã®ã§ããã¢ãŒã¡ã³ã¿ã ã®ããã«ç¬¬1åŒãã°ã©ãã§è¡šããŸãã
ãã®ã°ã©ãã¯å ã ã®åŸé (éè²)ã«å¯ŸããRMSPropã®ç¬¬1åŒã®å€ãã¡ã§ããåŸé ãæ¥ã«å€åãããšããã§ã¯ç¬¬1åŒã®å€ã倧ãããªã£ãŠãã ããšãããããŸãã
第2åŒã¯ãã©ã¡ãŒã¿ãæŽæ°ããŠããŸãããã ããããã§ã¯å ã ã®åŠç¿çã調æŽããäžã§æŽæ°ããŠããŸããå ã ã®åŠç¿çã第1åŒã®å€ã§å²ã£ãŠãããã ãã§ããæ¯åããŠãããšãã¯ç¬¬1åŒã倧ãããªãã®ã§ãæ¯åããŠãããšãã®åæ¯ã倧ãããªãåŠç¿çãå°ãããããããšèšãç®çãéæã§ããŸããããã«ããæ¯åãæããããŸããã
7. Adam
ã©ã€ãã©ãªãªã©ã§ãã£ãŒãã©ãŒãã³ã°ãå®éã«äœã£ãããšããã人ãªãäœåºŠãèŠãããšãããã¯ãã®Adamãã³ã³ã«æ¥ãŠããããç»å Žã§ããAdamã¯ããŸãã©ã®ã¢ãã«ã«ãåºã䜿ãããŠããããã¡ã¯ãã¹ã¿ã³ããŒããªæé©åã¢ã«ãŽãªãºã ã§ãããã®Adamã®æ£äœã¯ã¢ãŒã¡ã³ã¿ã ãšRMSPropã®è¯ãã©ãã©ã ãšãããã®ã§ããç§»åå¹³åã§æ¯åãæå¶ããã¢ãŒã¡ã³ã¿ã ãš åŠç¿çã調æŽããŠæ¯åãæå¶ããRMSProp ãçµã¿åãããŠããã ããªã®ã§ããæ©éæŽæ°åŒãèŠãŠèŠãŸãããã
\begin{align}
&\nu_{t} = \beta_1\nu_{t-1} + (1-\beta_1)G \\
&s_{t} = \beta_2s_{t-1} + (1-\beta_2)G^2 \\
&w_t = w_{t-1} - \alpha\frac{\nu_{t}}{\sqrt{s_t + \epsilon}}
\end{align}
åŒã3ã€åºãŠæ¥ãŠããŸããŸããã
ãã ããããèŠããšAdamã®æ£äœãã¢ãŒã¡ã³ã¿ã ãšRMSPropãçµã¿åãããã ãããšããã®ãããããšæããŸãã
第1åŒãšç¬¬2åŒã¯ããããåŸé
ã®ç§»åå¹³åãšåŸé
ãäºä¹ãããã®ã®ç§»åå¹³åã«ãªã£ãŠããããã ã®ã¢ãŒã¡ã³ã¿ã åã³RMSPropã®ç¬¬1åŒã§ãã
第3åŒã¯ããããçµã¿åãããŠæŽæ°ããŠããã ãã§ããRMSPropã®ç¬¬2åŒã«ãšãŠã䌌ãŠããŸããã
ããã ãã§ããç°¡åã«Adamãçè§£ã§ããŠããŸããŸãããïŒ
8. ãçªå€ç·šãäºé埮åã¯äœ¿ããªãã®ãïŒ
ãããŸã§ã®æé©åã¢ã«ãŽãªãºã ã¯ææ¥éäžæ³ããŒã¹ã®ãã®ã§ããã
w_{t+1} = w_{t} - \alpha\nabla_w\mathcal{L}(w)
ææ¥éäžæ³ãšããã®ã¯ãæå€±é¢æ°ã埮åããŠæå€±ãå°ãããªãæ¹åã«å°ããã€ãã©ã¡ãŒã¿ãæŽæ°ããã®ã§ãããããæ£ç¢ºã«èšãã°ãæå€±é¢æ°ã1é埮åã㊠ãã©ã¡ãŒã¿ãæŽæ°ããŸããä»å玹ä»ããæé©åã¢ã«ãŽãªãºã ã¯å
šãŠææ¥éäžæ³ãå°ã倿Žããããã®ã§ãããå
šãŠ 1é埮åã䜿ã£ãæé©åã¢ã«ãŽãªãºã ãªã®ã§ãã
ããã§ã1é埮åã§ã¯ãªã2é埮åãŸã§äœ¿ã£ããã©ãã§ãããã ãã€ãŸããæŽæ°åŒã以äžã®ãããªå Žåã¯ã©ããªã®ã§ããããã
$$
w_{t+1} = w_{t} - \frac{\mathcal{L'}(w)}{\mathcal{L''}(w)}
$$
ãã®2é埮åãçšããæŽæ°åŒã®ææ³ããã¥ãŒãã³æ³(Newton's Method)ãšåŒã³ãŸãã詳现ã¯å¥èšäºã§è§£èª¬ããããšæã£ãŠãããŸãããããã§ã¯ãã®ãã¥ãŒãã³æ³ã®å©ç¹ã𿬠ç¹ãç°¡åã«ç޹ä»ããŸãã
å©ç¹
- ãã®ãããéãã§åŠç¿ãåæããã
- 2é埮åãŸã§èæ ®ããŠããããããæ£ç¢ºã
æ¬ ç¹
- èšç®éã倧ãããããææ¥éäžæ³ã®èšç®éã®2ä¹ã
æ¬ ç¹ã匷ãããŠå®çšã¯ã»ãŒãããŠããŸããã
ãã 䜿ããããã«ãªã£ãæã«ã¯å€§ããªãã¬ã€ã¯ã¹ã«ãŒã«ãªãã®ã§ã¯ããšæã£ãŠããŸããèšå€§ãªèšç®éã«å¯Ÿå¿ã§ããéåã³ã³ãã¥ãŒã¿ãå®å
šã«å®æããããã¥ãŒãã³æ³ã䜿ããããã«ãªãã®ãã©ãããšãŠãæ°ã«ãªããŸãã
9. ãŸãšããšææ
ãŸãšããŸãã
- ææ¥éäžæ³: å šããŒã¿ ã䜿ã£ãп倱颿°ãæå°å€ã«ãªãããã«ãåŸé ã䜿ã£ãŠãã©ã¡ãŒã¿æŽæ°ãããã
- SGD: ããŒã¿1ã€ã ã ããµã³ãã«ããŠäœ¿ãããšã§ãææ¥éäžæ³ã«ã©ã³ãã æ§ãå ¥ãããã
- ãããããSGD: ããŒã¿ã16åãšã32å ãšã䜿ãããšã§äžŠåèšç®ã§ããããã«ããSGDã ãã
- ã¢ãŒã¡ã³ã¿ã : SGDã« ç§»åå¹³å ãé©çšããŠãæ¯åãæå¶ãããã
- NAG: ã¢ãŒã¡ã³ã¿ã ã§æå€±ãèœã¡ãããã«ä¿èšŒããŠãããã
- RMSProp: åŸé ã®å€§ããã«å¿ã㊠åŠç¿çãèª¿æŽ ããããã«ããŠãæ¯åãæå¶ãããã
- Adam: ã¢ãŒã¡ã³ã¿ã + RMSProp ã ããä»ã§ã¯è³ãæã§äœ¿ãããŠãããã
- ãã¥ãŒãã³æ³: äºé埮å ã䜿ã£ãŠããã®ã§ããã®ãããéãã§åæãããããã èšç®éãèšå€§ãã㊠å®çšãããŠããªã ãã
ãã¥ãŒã©ã«ãããã¯ãŒã¯ãç¥ãã«ããã£ãŠãæé©åã¢ã«ãŽãªãºã ã®çè§£ã¯é¿ããŠéããŸããã
ãã ããããŸã§èªãã ãªãæé©åã¢ã«ãŽãªãºã ã®æ°æã¡ãããçšåºŠçè§£ã§ããããšæããŸãã
çè§£ããŠã¿ããšæé©åã¢ã«ãŽãªãºã ãšã¯æå€ãšåçŽãªãã®ã§ãããïŒãããŠãã¥ãŒãã³æ³ã«ã¯ãšãŠãããã³ã溢ããŠããæããããŸãããä»åŸã¯Adamã®çºå±åœ¢ããã¥ãŒãã³æ³ãªã©ã«ã€ããŠãããã«è§£èª¬ããèšäºãæžããã°ãšæã£ãŠããŸãããããŸã§ãèªã¿ããã ãããããšãããããŸããïŒ
èªãã§å°ãã§ãäœãåŠã¹ããšæããã ããããã ã ãã³ã¡ã³ãã ããããããšããããã®å±ã¿ã«ãªããŸãïŒãããããé¡ãããŸãïŒ
Twitterã§äººå·¥ç¥èœã®ããšãä»åªäœã§æžããŠããèšäºãªã©ã玹ä»ããŠããŸãã®ã§ãã²ãã©ããŒ@omiita_atiimoããŠãã ããïŒ
10. åè
-
Intro to Optimization in Deep Learning: Vanishing Gradients and Choosing the Right Activation Function
(è±èª) ç©åãåèã«ããŠããŸããåŸé éäžæ³ãããããããåŠç¿ãŸã§ã -
Intro to optimization in deep learning: Momentum, RMSProp and Adam
(è±èª) ç©åãåèã«ããŠããŸããäžã®ç¶ç·šãã¢ãŒã¡ã³ã¿ã ããAdamãŸã§ã -
Stochastic Gradient Descent with momentum
(è±èª) ã¢ãŒã¡ã³ã¿ã ã«ã€ããŠå³è§£ã§ãããããã解説ã -
Exponentially Weighted Averages (C2W2L03)
Understanding Exponentially Weighted Averages (C2W2L04)
Gradient Descent With Momentum (C2W2L06)
(è±èª) Andrew Ngæ°ã«ããç§»åå¹³åãšã¢ãŒã¡ã³ã¿ã ã®è§£èª¬åç»ã -
Understanding Nesterov Momentum (NAG)
(è±èª) ãã¹ãããã®å éåŸé æ³ã¯ã¢ãŒã¡ã³ã¿ã ã®æ £æ§é ãæ£ããæ¹åãåããŠããããåŸé ã«ãã£ãŠç¢ºãããŠããããšæžããŠããã -
æé©åè¶ å ¥é
16æç®ã®æé©åããŒãã·ãŒãã«ããããã®æé©ååé¡ã®å šäœåããããããããŸãšããããŠããŠæèœã§ãã -
æ©æ¢°åŠç¿ãããäžã§ç¥ã£ãŠããããé£ç¶æé©å
(Qiita) é£ç¶æé©ååé¡ã§ããææ¥éäžæ³ãã©ããã£ãŠæ±ããããŠããããã®åææ§ãªã©ãå埩æ³ããå§ããããããã解説ãããŠããã



















