
OpenAI o1ã¯ã©ãäœãã®ãïŒè©³çŽ°ç·šïŒ
1. ã¯ããã«
ååãOpenAI o1ãã©ãäœããã«ã€ããŠæŠèŠã¬ãã«ã§è°è«ããŸããã
OpenAI o1ãæ°ãã匷ååŠç¿ææ³ãçšããŠåŒ·åãããã¢ãã«ã§ããããšã¯ç¢ºå®ã§ãããå
·äœçã«ã©ãããããžãã¯ã§åŠç¿ãããŠãããèããŠãããããšæããŸãã
匷ååŠç¿ã«ã€ããŠééã£ãçè§£ãããã°ãææãã ãããïŒäžå¿åŠç¿ã¯ãããã®ã®æ®æ®µäœ¿ãããŠããããã§ã¯ãªãã®ã§å¿ããŠãããšããããããšæããŸããïŒ
2. 匷ååŠç¿ã®ææ³
äžè¬çã«åŒ·ååŠç¿ã«ãããŠãæçµçãªç®æšã¯æ¹çã¢ãã«ïŒæ¹ç颿°ïŒã®æ§èœãé«ããããšã«ãããšæããŸãã
æ¹çã¢ãã«ãšã¯ãããç°å¢ã«ãããŠãšãŒãžã§ã³ããã©ãåãã®ããæ±ºå®ããã¢ãã«ã»ã¢ã«ãŽãªãºã ã§ãã
è¿·è·¯æ¢çŽ¢ã¿ã¹ã¯ã«ãããŠã¯æ¬¡ã®ã¹ãããã®ç§»åå
ãäžäžå·Šå³ã®ãã¹ããæ±ºå®ããã¢ã«ãŽãªãºã ãæããŸãããèšèªçæã¿ã¹ã¯ã«ãããŠã¯æç« ãçæããäž»äœã§ããLLMèªäœãæ¹çã¢ãã«ã§ãã
æ¹çã¢ãã«ã¯ãç°å¢ãå ±é
¬ã¢ãã«ããè¡å(éžæ)ã«ããé·æçãªäŸ¡å€ã®æåŸ
å€ãäºæž¬ããè¡åãæ±ºå®ããŸãã
ïŒé·æçãªäŸ¡å€ãèšç®ããæ©æ§ã䟡å€é¢æ°ãšãããŸããïŒ
ãã®ãããªä»çµã¿ããã匷ååŠç¿ã§ã¯æ¹çæ¹åã®ã¢ãããŒããšããŠïŒã€ã®ãã¿ãŒã³ããããŸãã
䟡å€ããŒã¹ã®ææ³ïŒValue-Based MethodsïŒïŒäŸ¡å€é¢æ°ãåŠç¿ãã鿥çã«æ¹çã®ç²ŸåºŠãé«ãã
æ¹çããŒã¹ã®ææ³ïŒPolicy-Based MethodsïŒïŒæ¹çã¢ãã«èªäœãåŠç¿
Actor-Criticææ³ïŒActor-Critic MethodsïŒïŒæ¹çãšäŸ¡å€é¢æ°ã®ã©ã¡ããç¶ç¶çã«åŠç¿ïŒäŸ¡å€ããŒã¹+æ¹çããŒã¹ïŒ
ã§ã¯ãOpenAI o1ã®åŒ·ååŠç¿ã¯ã©ã®ã¢ãããŒããªã®ã§ãããããRLHFãããŒã¹ã«èããŠãããŸãã
2.1 RLHFã®ã¢ãããŒã
以äžã¯OpenAIã®ããŒãžããæã£ãŠããRLHFã®èª¬æå³ã§ãã
RLHFã§ã¯Step 2ã§å ±é
¬ã¢ãã«ãäœããStep 3ã§LLMïŒæ¹çã¢ãã«ïŒãåŠç¿ãããŸããStep 3ã§ã¯å ±é
¬ã¢ãã«ã¯åºæ¬çã«ã¯åºå®ãããå ±é
¬ãè¿ã颿°ãšããŠåäœããŠããã®ã§RLHFã®ã¢ãããŒãã¯æ¹çããŒã¹ã®åŒ·ååŠç¿ãšèšãããã§ãã
å ±é ¬ã¢ãã«ãåŠç¿ããŠããç¹ã§ã䟡å€+å ±é ¬ããŒã¹ã®Actor-Criticã®ããã«ãèŠããŸãããå ±é ¬ã¢ãã«ã¯ãããŸã§ç°å¢ã®ä»£æ¿ã§ããç¹ã»å ±é ¬ã¢ãã«ã¯ç¶ç¶çãªåŠç¿ãè¡ããªãç¹ã§ãActor-Criticãšã¯ç°ãªããŸããïŒå ±é ¬ã¢ãã«â 䟡å€é¢æ°ïŒ

Step 3ã®PPOã§ã¯ä»¥äžãç¹°ãè¿ããŸãã
æ¹çã¢ãã«(LLM)ã«ãã£ãŠInputã«å¯Ÿããè€æ°ã®OutputãåºåãããŸãã
Inputã«å¯ŸããåOutputã®è©äŸ¡ãå ±é ¬ã¢ãã«ãè¡ããŸãã
å ±é ¬ãé«ãOutputãåºåããããããã«æ¹çã¢ãã«(LLM)ã®éã¿ãæŽæ°ïŒåŠç¿ïŒãããŸãã
ãã®æããµã³ããªã³ã°ãããåŠç¿ããŒã¿(s, a, r, s')ã¯ã
ãs: å
¥åããã³ãã
ãa: çæãããããã¹ãå
šäœ
ãr: ããã¹ãå
šäœã«å¯Ÿããåäžã®å ±é
¬
ãs': å
¥åããã³ã+çæãããããã¹ãå
šäœ
ãšãªããŸããRLHFã¯æ¯åïŒã¹ãããã§ïŒæœè¡ãå®äºããã¿ã¹ã¯ã®åŒ·ååŠç¿ãšèŠåãããããs'ã¯äœ¿çšãããªãã¿ããã§ãã
* äžè¬çã«åŒ·ååŠç¿ã§ã¯s: çŸåšã®ç¶æ , a: åã£ãè¡å, r: åŸãããå ±é ¬, s': 次ã®ç¶æ ããµã³ããªã³ã°ããŸãã
å ·äœçãªPPOã®ã¯ãŒã¯ãããŒãå³ã«ãããšä»¥äžã«ãªããŸãã

2.2 Google DeepMindã®è«æã®ã¢ãããŒã
Google DeepMindã®è«æ "Scaling LLM Test-Time Compute Optimally can be More Effective than Scaling Model Parameters"ã®ã¢ãããŒãã¯ã©ãã§ãããããæ¯ãè¿ã£ãŠã¿ãŸãã
ããã¯åŒ·ååŠç¿ãšã¯èšããªããšæããŸãã
ãªããªãå ±é
¬ã¢ãã«ã®åŠç¿ã«ã¯äººéããã®è©äŸ¡ãçšããŠãããããã¯åãªãæåž«ããåŠç¿ã§ãããŸããæ¹çã¢ãã«èªäœã¯éã¿ãæŽæ°ãããŠããŸãããïŒåŠç¿ãããŠããªããïŒ
3. OpenAI o1ã®ã¢ãããŒã
3.1 æ¹çããŒã¹ã®ææ³
ã§ã¯äžèšã®RLHFãšDeepMindè«æã®ã¢ãããŒããããšã«OpenAI o1ã®åŒ·ååŠç¿æ¹æ³ãèããŠã¿ãããšæããŸãã
å°ãªããšãDeepMindè«æã®ã¢ãããŒãã¯åŒ·ååŠç¿ã§ã¯ãªããèãæ¹ã¯äŒŒã€ã€ãããããããã«çºå±ããããã®ã§ãããã®ã ãšæãããŸãã
ææ³ãšããŠã¯ããã¯ãRLHFãçºå±ãããæ¹æ³ãæãèªç¶ã«èããããŸããïŒæ¹çããŒã¹ïŒ
ã€ãŸããStep 2ã§æ¬æ¥äœãããå ±é
¬ã¢ãã«ïŒInputã«å¯ŸããŠOutputå
šäœã人éã®å奜ã«ãããã©ãããè©äŸ¡ããã¢ãã«ïŒãäœã£ãŠããã®ããDeepMindè«æã®PRMïŒããã»ã¹å ±é
¬ã¢ãã«ïŒInputã«å¯ŸããŠæèã¹ãããæ¯ãã€ãŸã1è¡æ¯ã®æ£ç¢ºæ§ã»æçšæ§ãè©äŸ¡ããã¢ãã«ïŒãå©çšããŸãã
PPOã«ãããOutputã®å ±é
¬èšç®ãPRMãçšããŠè¡ããŸãã
ãã®æ¹æ³ã¯ããã«2ã€ã«åããããŸãã
åã¹ãããã®å ±é ¬ã®çޝèš(+åçã®æ£ãã)ãå ±é ¬ãšããŠãïŒæœè¡æ¯ã«ïŒãµã³ããªã³ã°ãšãããïŒçŸè¡ã®RLHFã®PPOã«è¿ãïŒ
1ã¹ãããæ¯ã«å³æå ±é ¬ãèšç®ãããµã³ããªã³ã°ãããïŒéåžžã®PPOã«è¿ãïŒ
1ã®å Žåã¯ããµã³ããªã³ã°ãããåŠç¿ããŒã¿(s, a, r, s')ã¯ã
ãs: å
¥åããã³ãã
ãa: çæãããããã¹ãå
šäœ
ãr: åã¹ãããã®å ±é
¬ã®çޝèš(+åçã®æ£ãã)ãåäžã®å ±é
¬
ãs': å
¥åããã³ã+çæãããããã¹ãå
šäœ
ãšãªããŸãããã®(s, a, r)ã䜿ã£ãŠãRLHFã®PPOã®ããã«åŠç¿ããŸãã
2ã®å Žåã¯ããµã³ããªã³ã°ãããåŠç¿ããŒã¿(s, a, r, s')ã¯ã
ãs: å
¥åããã³ãã (+ãããŸã§ã«çæãããè¡)
ãa: æ°ããçæãããè¡
ãr: ãã®è¡ã«å¯Ÿããå ±é
¬
ãs': å
¥åããã³ãã (+ ãããŸã§ã«çæãããè¡) + æ°ããçæãããè¡
ãšãªããŸãããã®(s, a, r, s')ã䜿ã£ãŠãéåžžã®PPOã®ããã«åŠç¿ããŸãã
å®è£ ã®é£æåºŠçã«ã¯ 1 < 2 ã§ããã广çã«ã¯ 1 > 2ãªã®ã§ã¯ãªãããšæããŸããïŒ1ã¯æ¢åã®TRLãªã©ã®ã©ã€ãã©ãªãäžéšäœ¿ãåããããªããïŒ
2ãå
ã»ã©ã®PPOã®ã¯ãŒã¯ãããŒã«åœãŠã¯ãããšä»¥äžã®ããã«ãªããŸããïŒïŒã®ã¯ãŒã¯ãããŒèªäœã¯æ¢åRLHFã®PPOãšåæ§ã«ãªããšæããŸããïŒ
åã¹ãããæ¯ã«Policy Optimizationãè¡ãããã€ã¡ãŒãžã§ããïŒæåãå°ãããŠæçž®ã§ããïŒ


广çã«ãOpenAI o1ã¯ïŒã®ãã¿ãŒã³ã宿œããŠããã®ã§ã¯ãªãããšèããŸãã
3.2 䟡å€ããŒã¹ã®ææ³
ã§ã¯ã䟡å€ããŒã¹ã®åŒ·ååŠç¿ã®å¯èœæ§ã¯ããã®ã§ããããã
PRMã䜿ãããšãåæã«èãããšãPRM+å ±é
¬ãå
ã«ããæ¢çŽ¢ã¢ã«ãŽãªãºã ã䟡å€é¢æ°ãšæãããšå¯èœã§ã¯ãããšæããŸãã
ãã£ããèãããšã
ïŒDeepMindã®ææ³ã§ïŒPRMãäœã
LLMã«PRM+æ¢çŽ¢ã¢ã«ãŽãªãºã ã䜿ã£ãŠæšè«ããã
æ£è§£ã«èŸ¿ãçãããã©ããã§ãã®æšè«ã¹ãããå šäœãè©äŸ¡ãïŒPRMã®éã¿ãæŽæ°ïŒ
ãç¹°ãè¿ãããšã§PRMãåŠç¿ãããŠããããšã¯ã§ãããã§ãã
ãã ããæ£è§£ãããããŒã¿ã§ããåŠç¿ã§ããªãããšãè¯ãæšè«ã¹ãããããã£ãŠãåŸåã«æªãæšè«ã¹ããããå
¥ãããšã§æªãè©äŸ¡ãããããšãžã®æžå¿µããããŸãã
ïŒæ£è§£ããªãããŒã¿ã«å¯ŸããŠããæ£è§£äžæ£è§£ãéåžžã®RLHFã®å ±é
¬ã¢ãã«ã«ã€ããããã°ã§ããããïŒïŒïŒ
3.3 Actor-Criticææ³
Actor-Criticææ³ã«é¢ããŠãåŠç¿ã®äžå®å®æ§ã課é¡ã«ãªããšæããŸãã
詊ãã«ããŠãæ¹çããŒã¹ã§ã®ææãåŸãããŠããã«ãªãã®ã§ã¯ãªãã§ããããã
3.4 çåãšå¯èœæ§
äžèšã®éããæ¹çããŒã¹ã®åŒ·ååŠç¿ãè¡ãªã£ãŠããå¯èœæ§ãé«ããšæããŸãããã ããããã ãšäžèš2ç¹ã§çåãæ®ããŸãã
â æ¹çããŒã¹ã§åŠç¿ããå Žåãæšè«æã«ã¯ãããŸã§OpenAI o1ã®ããã«ããããã®ã¹ããããèžãã§æšè«ããããã«ãªãã®ãïŒ
â çè«ã®é£èºãPRMã«ãã£ãŠäœè©äŸ¡ãåããããããããã®ã¹ããããèžãããã«ãªãã®ã¯ãªããã
â¡ ããããã®ã¹ããããèžãã§æšè«ããããã«ãªããšããŠãããã§ã20-30ç§çšåºŠãåºåã«æéããããã®ãïŒ
â ããŒã¯ã³ã®æ¢çŽ¢ãè¡ãããã«ãïŒè¡æ¯ã®æšè«ã¹ãããã®æ¢çŽ¢ãæšè«æã«è¡ãªã£ãŠããã®ã§ã¯ãªããã
ããã«ã¯äžèšã®ããã«èããŸããã
4. ãŸãšã
ååã®æŠèŠã¬ãã«ã®è©±ããã匷ååŠç¿ã®ææ³ããµã³ããªã³ã°ããŒã¿ã«ã€ããŠèããŠã¿ãŸãããïŒåååæ§ãç§ã®ç¥èäžè¶³ããããééãããã¹ãªãŒããå«ãŸããå¯èœæ§ãããããšã«æ³šæããŠãã ãããïŒ
çµè«ãšããŠã¯ãïŒã€ã®æ¹çããŒã¹ã®åŒ·ååŠç¿ãæåã ãšèããŸãã
ãã®ãã¡ã®ç¹ã«ã1ã¹ãããæ¯ã«å³æå ±é
¬ãèšç®ãããµã³ããªã³ã°ããããæ¡çšãããŠãããšæããŸãã
ãã ããçåãæ®ãç¹ããããããAnthoropicãªã©ã«ãã£ãŠæ€èšŒãé²ãã§ãããšè¯ããªãšæããŸãã
æ¥æ¬å¢ãOpenAI o1ã«è¿œéããæå©ãã«ãªããšè¯ããšæããŸããè°è«ã®ããããšããŠå©çšããŠãã ããã°å¹žãã§ãã
äœãè°è«ãææ³ãããã°NoteãX(Twitter)ã§ã³ã¡ã³ãã»ãæç€ºããã ãããšå©ãããŸãã
ç§èªèº«æ°ããªæ°ã¥ããããã°è¿œèšããŠããããšæããŸãã
X: https://twitter.com/CurveWeb
ç®ãéããŠããã ãããããšãããããŸããã