
OpenAI o1ã¯ã©ãäœãã®ã(æŠèŠç·š)
1. ã¯ããã«
OpenAIããä¹ ãã¶ãã«æ°ããã¢ãã«ãåºãŸããã
OpenAI o1: 匷åãªæšè«èœåãšå¹ åºãäžçç¥è
OpenAI o1-mini: ã³ãŒãã£ã³ã°ã¿ã¹ã¯ã«ç¹å
ã¢ãã«ã«ã€ããŠã®çްãã説æã¯ããã§ã¯ããŸããããOpenAIã®å ¬åŒå ±åã«ããã°
ç«¶æããã°ã©ãã³ã°åé¡ (Codeforces) ã§ 89 ããŒã»ã³ã¿ã€ã«ã«ã©ã³ã¯ã€ã³ããç±³åœæ°åŠãªãªã³ãã㯠(AIME) ã®äºéžã§ç±³åœã®ããã 500 åŠçã«ã©ã³ã¯ã€ã³ããç©çåŠãçç©åŠãååŠã®åé¡ã®ãã³ãããŒã¯ (GPQA) ã§äººéã®å士ã¬ãã«ã®ç²ŸåºŠãè¶ ããŠããŸãã
ãšã®ããšã§ããå°ãªããšãæ¢åã®LLMããã¯äžæ©æããåºãèœåãæã£ãŠãããã§ãã
å ¬åŒããOpenAI o1ã®è©³çްãªäžèº«ã«ã€ããŠã¯èšåããããŸããããä¿¡ææ§ã®é«ãããªè©±ãšããŠãGoogle DeepMindã®è«æ "Scaling LLM Test-Time Compute Optimally can be More Effective than Scaling Model Parameters" ã§èšåãããŠããã¢ãããŒãã«è¿ãåããæ¡çšãããŠããã®ã§ã¯ãªãããšèšããã®ããããŸãã
OpenAI Strawberry (o1) is out! We are finally seeing the paradigm of inference-time scaling popularized and deployed in production. As Sutton said in the Bitter Lesson, there're only 2 techniques that scale indefinitely with compute: learning & search. It's time to shift focus to⊠pic.twitter.com/jTViQucwxr
â Jim Fan (@DrJimFan) September 12, 2024
åŸã£ãŠãä»åã¯ãã®è«æã®å 容ãão1ã®åããèããŠãããããšæããŸãã
2. è«æã®ã¢ãããŒã
Google DeepMindã®è«æ "Scaling LLM Test-Time Compute Optimally can be More Effective than Scaling Model Parameters"ã®ãã®ã¿ã€ãã«ã¯æ¥æ¬èªã«ãããšãLLMã®ãã¹ãæéèšç®(Test-Time Compute)ãæé©ã«ã¹ã±ãŒãªã³ã°ããããšã¯ãã¢ãã«ãã©ã¡ãŒã¿ãã¹ã±ãŒãªã³ã°ããããã广çã§ããããã§ãã
2.1 ææ³
ããã¯ãæãèªã¿ããæãã ãšä»¥äžã®æãã®ããã§ãã
éåžžãLLMåäœã§ã¯ã次ã®ããŒã¯ã³ã®çæç¢ºçãæ¢çŽ¢(Greedy/Beam Search/Samplingãªã©)ãæç« ãçæããã
LLMãšåãããŠæ°ããPRM(ããã»ã¹å ±é ¬ã¢ãã«)ãšããã¢ãã«ã䜵çšããŠäœ¿ãã
PRMã¯LLMãåºåããæšè«ã®ã¹ãããæ¯ã«æ£ç¢ºæ§ã»æçšæ§ãäºæž¬ããããã®äºæž¬ã«åºã¥ããŠã¹ãããã¬ãã«ã§æ¢çŽ¢ãè¡ãããšã§æçµçãªåºåæç« ãæ±ºå®ããã
é ã远ã£ãŠèª¬æããŠãããŸãã
éåžžãLLMã®çæã§ã¯ä»¥äžã®çµµã®ããã«ã次ã®ããŒã¯ã³ããã®çæç¢ºçããGreedy Search, Beam Search, Samplingã®ãããªææ³ãçšããŠãã確ããããæç« ãçæããŠãããŸãã
ãã®æ¢çŽ¢ã«äœ¿ããããããTemperatureãTop-kãªã©ã®ãã©ã¡ãŒã¿ãå¿
èŠãšãªã£ãŠããŸãã

è«æã®ã¢ãããŒãã§ã¯ãLLMãšåãããŠæ°ããPRM(ããã»ã¹å ±é ¬ã¢ãã«, Process Reward Model)ãšããã¢ãã«ã䜵çšããŠäœ¿ãããã§ãã
ãã®PRMã¯ãLLMãšç°ãªãäºæž¬æã«æšè«ã®ã¹ãããããšã«æ£ç¢ºæ§ã»æçšæ§ã0-1ã§äºæž¬ããŸãã

LLMãæ¬¡ã®ããŒã¯ã³ãæ¢çŽ¢ããŠæ±ºå®ããæç« ãçæããŠããã
æç« ãçæãããŠããéçšã§PRMãçæãããæ€èšã¹ããããè©äŸ¡ããæ£ããæšè«ã¹ããããæ¢çŽ¢ããŠããã
ãšèšãæµããèžãã§ãããšèšã話ã§ããã
ããã«ãã£ãŠãç°¡åãªåé¡ãäžçšåºŠã®é£ããã®åé¡ã§ã¯ããã¹ãæèšç®ã䜿çšããå°ããã¢ãã«ãã14å倧ããªã¢ãã«ãäžåãããã©ãŒãã³ã¹ã瀺ããŸããããã§ãã
ããã«ãã£ãŠãLLMã¯ã¢ãã«ãµã€ãºãããŒã¿éãã¹ã±ãŒã«ãããæä»£ããæšè«æéãã¹ã±ãŒã«ãããïŒã€ãŸããæ²¢å±±ã®æšè«ã¹ããããæ¢çŽ¢ããïŒæä»£ã«ç§»ã£ãŠããããã§ãã
2.2 PRM(Process Reward Model)ã®äœæ
PRMãLLMããäœæããŸããè«æã§ã¯ãPaLM 2-Sãšããã¢ãã«ãFine Tuningããããšã§äœæããŠããŸãã
Fine Tuningã«ã¯ãOpenAIãå
¬éããŠããPRM800KãšããããŒã¿ãçšããŠããŸããOpenAIãäœæããããŒã¿ãšããã®ããã®è©±ã®ä¿¡ææ§ãäžããŠããæ°ãããŸããã
PRM800Kãšã¯æ°åŠã«é¢é£ããããŒã¿ã»ããã§ä»¥äžã®ãããªåœ¢åŒã§ãã
MATHããŒã¿ã»ããã®åé¡ã«å¯ŸããŠãæšè«ã¹ãããæ¯ã«æ£èª€ã0,1ã®ãã€ããªåœ¢åŒã§ä»äžãããŠããŸãã
ãŸããã¹ãããã®åå²ã¯æ¹è¡æ¯ã«è¡ãããŠããŸãã

ãããçšããŠFine TuningãããããPRMã¯æšè«éçšã®æ£ç¢ºæ§ã»æçšæ§ãå€å®ã§ããããã«ãªãããã§ãã
ç§ã®ç¥ãéããã®åœ¢åŒã®ããŒã¿ã¯ãããããªãã®ã§ãäžè¬çãªæšè«ã«é¢ããŠããããããŒã¿ãçšæããã®ã¯ããªãé£ããã ãããªãšæããŸããã
3. OpenAI o1ã®äœãæ¹
3.1 匷ååŠç¿
å ¬åŒãGreg Brockmanæ°ã®æçš¿ãèŠããšã以äžã®ãããªæçš¿ããããŸãã
人éãé£ãã質åã«åçããåã«é·ãæéèããã®ãšåãããã«ão1 ã¯åé¡ã解決ããããšãããšãã«æèã®é£é(Chain of Thought)ã䜿çšããŸãã匷ååŠç¿ãéããŠão1 ã¯æèã®é£éã磚ãã䜿çšããæŠç¥ãæŽç·ŽããããšãåŠã³ãŸããééããèªèããŠä¿®æ£ããããšãåŠã³ãŸãã
OpenAI o1 â åçããåã«åé¡ã«ã€ããŠçå£ã«èããããã«åŒ·ååŠç¿ã§ãã¬ãŒãã³ã°ãããæåã®ã¢ãã«ã
ïŒäžç¥ïŒ
ãã°ããåã«ãã¢ãã«ã«ã段éçã«èãããããã«ä¿ããšããã©ãŒãã³ã¹ãåäžããããšãçºèŠãããŸããããããã詊è¡é¯èª€ããªããã¢ãã«ãæåããæåŸãŸã§ãã¬ãŒãã³ã°ããæ¹ãã¯ããã«ä¿¡é Œæ§ãé«ããGo ã Dota ãªã©ã®ã²ãŒã ã§ç¢ºèªããããã«ãéåžžã«å°è±¡çãªçµæãçã¿åºãããšãã§ããŸãã
ããŒã¯ãŒãã¯æèã®é£é(Chain of Thought; CoT)ãšåŒ·ååŠç¿ã®ããã§ãã
Google DeepMindãææ¡ããŠããææ³ã¯ãããŸã§LLMãšPRMãæšè«æã«äœµçšãããã®ã§ãLLMèªäœãåŠç¿ãããã®ã§ã¯ãããŸããã§ããã
äžæ¹ã§ãOpenAI o1ã¯åŒ·ååŠç¿ãè¡ãªã£ãŠãããšããããšãæèšããŠããŸãã
åŸã£ãŠãPRMãå ±é ¬ã¢ãã«ãšããŠLLMã匷ååŠç¿ããã¢ãã«ão1ãªã®ã§ã¯ãªãããšæããŸãã
ã€ãŸããAlphaGoãïŒæïŒæåã€ããã®éçãåŠç¿ããŠããããã«ããããããé害ç©ãé¿ããŠãŽãŒã«ãžã®èŸ¿ãçãæ¹ãåŠç¿ããããã«ãLLMãæ£ããæšè«çµæãžã®èŸ¿ãçãæ¹ãåŠã°ããŠããã®ã§ã¯ãªãã§ããããã
Reinforcement Learning for Chain of Thought(RLCoT)ãšã§ãåŒã¶ã®ã§ãããããå
ã
ã匷ååŠç¿ã匷ã¿ãšããŠããOpenAIãããææ³ã ãšæããŸãã
ïŒ2024/9/14 远èšïŒ
Tom Yehæ°ãXäžã§ãšãŠãããããããå³ãäœæããŠãããŠããŸããã
RLFHã¯Reward Model(å ±é
¬ã¢ãã«)ãçšããŠAnswerã«å¯ŸããŠå ±é
¬ãäºæž¬ããLLMãåŠç¿ãããŸãã
仿¹ãOpenAI o1ã®ææ³ã§ã¯PRM(ããã»ã¹å ±é
¬ã¢ãã«)ã¯ãAnswerã ãã§ãªãæšè«ããã»ã¹ã®ïŒã¹ãããæ¯ã«å ±é
¬ãäºæž¬ã§ããŸãããã®ãããæšè«éäžã®ã©ãã§ééããã®ããLLMãåŠãã§ããããšãã§ããã®ã ãšæããŸãã
How does OpenAI train the Strawberryð (o1) model to spend more time thinking?
â Tom Yeh (@ProfTomYeh) September 13, 2024
I read the report. The report is mostly about ðžð©ð¢ðµ impressive benchmark results they got. But in term of the ð©ð°ðž, the report only offers one sentence:
"Through reinforcement learning, o1⊠pic.twitter.com/OYpXpC2sZp
3.2 æšè«(Reasoning)ããŒã¯ã³
OpenAIã®ããã¥ã¡ã³ãã«ã¯ä»¥äžã®ãããªå³ã瀺ãããŠããŸãã

ãããã®æšè«ã¹ããããReasoningã«åœããå Žæã§è¡ãããã®ã ãšèããããŸãããŸãäžèšã®å³ããåããéããçŸç¶ã®LLMã¯äŸãã°
<user></user>
<assistant></assistant>ã®ããã«HumanMessageãšAIMessageãåºå¥ããŠããã®ãã
<user></user>
<reasoning></reasoning>
<assistant></assistant>ãšåºåããããã«Instruction TuningãããŠããããã§ãã
ãã®Reasoningå ã®åè¡ãæšè«ã®1 stepãšããŠPRMã«è©äŸ¡ãããã®ã ãšèããããŸãã"</reasoning>"ãåºåããããReasoningãå®äºãšãªãã®ã§ã¯ãªãã§ããããã
3.3 æšè«ã¹ãããã®åºåã
ã¹ãããã®æšè«ãã©ãåããŠãããã¯èª¬æããããŸãããäžèšã®PRM800Kã¯ãã¹ãããæ¯ã®åå²ã¯æ¹è¡ã§å€æããŠããããã§ãã
å
¬åŒã®ããã°ã確èªããŠã¿ããšã以äžã®ãããªæãã§ãã

å®å šã«åºåã®æããèŠãã ãã§ã®å€æã«ãªããŸãããäžèªç¶ã«ïŒè¡æ¯ã«æ¹è¡ãæãŸããŠãããPRM800Kãšåãããã«æ¹è¡æ¯ã«ã¹ãããåããããŠããããã«æããŸãã
äžæ¹ã§ãChatGPT UIã§æèå
容ãéããŠã¿ãŠã¿ããšãïŒè¡ã¥ã€äœããã¿ã€ãã«ã®ãããªãã®ãä»ããããŠããŸããïŒè§£èªã®ãã³ããæ¢ãããªã©ïŒ
ããã¯ãæ¹è¡æ¯ã«ã¹ãããåãããŠããŠãåŸãã軜éã¢ãã«ã§äœãèããŠãããã®ã¿ã€ãã«ã¥ããããŠããã®ã§ã¯ãªãããšæããŸãã

4. OpenAI o1ã®éçãšãã®å
ãããŸã§ã®æ³å®ãå
šãŠæ£ãããã°ãã®è©±ãããŸãã
ããäžèšã®æ¹æ³ããããã¯äŒŒããããªæ¹æ³ã§OpenAI o1ãäœãããŠããã®ãªãã°ãããéçãããã¯ãã§ãã
ããã¯PRMã®åŠç¿ããŒã¿ããæç¢ºã«çãã®ããåéã«éããããšããç¹ã§ãã
å²ç¢ãããããã®è¿·è·¯æ¢çŽ¢ã«ããã匷ååŠç¿ã«ã¯ãåã¡ããŽãŒã«å°ç¹ãžã®å°éãšãããŽãŒã«ããããŸãããäžè¬çãªæšè«ã«ã¯æ±ºå®çãªæ£è§£ããªãå Žåãå€ã
ãããšæãããŸãã
ããããæã«å®å®ããåŠç¿ãã§ããã®ã§ããããã
ã ããããOpenAI o1ã§ã¯ãæ°åŠãç§åŠçãªæšè«ãªã©æ¢åã®æç¢ºã«çãã®ããåé¡ãäžå¿ã«åŠç¿ãããã®ã ãšæããŸãã
ãã®å
ãŸã§ãã®ææ³ã§åé¡ãªããããããšã瀺ããããšãããªãšå人çã«ã¯æã£ãŠããŸããïŒAGIã®å°æ¥ïŒïŒ
ïŒãšã¯ããã匷å¶çã«Step by Stepã§æšè«ãããã®ã§ãã®åã®æšè«èœåäžæã¯ã©ã®åéã§ãããçšåºŠèµ·ãã£ãŠãããšæãããŸããïŒ
ãŸãšã
è«æãXã®æçš¿ãªã©ããè²ã ãšäºæž¬ã亀ããŠãµãã£ãšããOpenAI o1ã®äœãæ¹ãèããŠã¿ãŸããã
çµè«ãšããŠã¯ãã·ã³ãã«ã§ãã以äžã®æµããªã®ããªãšããçµè«ã§ãã
LLMãInput, Reasoning, Outputã®åºåãããããã«Instruction Tuning
PRMããŒã¿ãäœæããïŒæåïŒåæããŒã¿ã§äœæããæ¹æ³ãããïŒïŒ
LLMãFineTuningããŠPRMãäœæ
LLMã匷ååŠç¿
è²ã ãšç¥èäžè¶³ãªç¹ã»åŒ·ååŠç¿ã«ã€ããŠ(äžåºŠå匷ãããã®ã®)å¿ããŠããç¹ãããééããŠããç®æãããå¹ççãªããæ¹ãèããçãç¹çãããããããŸãããïŒæŽã«ã¯ããããŸã§ç§ã®äºæž¬ãªã®ã§ããã¹ãªãŒãã£ã³ã°ããããããããŸãããïŒ
äœãè°è«ãææ³ãããã°NoteãX(Twitter)ã§ã³ã¡ã³ãã»ãæç€ºããã ãããšå©ãããŸãã
ç§èªèº«æ°ããªæ°ã¥ããããã°è¿œèšããŠããããšæããŸãã
X: https://twitter.com/CurveWeb
ç®ãéããŠããã ãããããšãããããŸããã
ç¶ã
åç §
https://openai.com/index/learning-to-reason-with-llms/
https://platform.openai.com/docs/guides/reasoning
OpenAI o1-preview and o1-mini are rolling out today in the API for developers on tier 5.
â OpenAI Developers (@OpenAIDevs) September 12, 2024
o1-preview has strong reasoning capabilities and broad world knowledge.
o1-mini is faster, 80% cheaper, and competitive with o1-preview at coding tasks.
More in https://t.co/l6VkoUKFla. https://t.co/moQFsEZ2F6
OpenAI Strawberry (o1) is out! We are finally seeing the paradigm of inference-time scaling popularized and deployed in production. As Sutton said in the Bitter Lesson, there're only 2 techniques that scale indefinitely with compute: learning & search. It's time to shift focus to⊠pic.twitter.com/jTViQucwxr
â Jim Fan (@DrJimFan) September 12, 2024
o1 may be good at reasoning, but bombs on code completion
â Bindu Reddy (@bindureddy) September 13, 2024
However the models do quite well on code generation
Overall they underperform compared to Sonnet 3.5 pic.twitter.com/JA3VDyNy50
How does OpenAI train the Strawberryð (o1) model to spend more time thinking?
â Tom Yeh (@ProfTomYeh) September 13, 2024
I read the report. The report is mostly about ðžð©ð¢ðµ impressive benchmark results they got. But in term of the ð©ð°ðž, the report only offers one sentence:
"Through reinforcement learning, o1⊠pic.twitter.com/OYpXpC2sZp