ãã®èšäºã¯ MLSE Advent Calendar 2019 23æ¥ç®ã®èšäºã§ãã150 Successful Machine Learning Models: 6 Lessons Learned
at Booking.comãšããè«æããããŸããŠãäžçæå€§ã®ããã«äºçŽãµã€ã Booking.com ããæ©æ¢°åŠç¿ãå®ããžãã¹ã§æŽ»çšããŠããäžã§åŠãã æèšããŸãšããããŠããŸãã
ä»åã¯ãã®è«æãçºããªãããåèšåæã®äŒç€Ÿã«10幎身ããããŠããèªåã®çµéšãæ¯ãè¿ãããã¿ãã¿ããããšæããŸãããã¶ããããããèšäºã圹ã«ç«ã€äººãããããºãããªããè«æèªäœã®æ¥æ¬èªã§ã®è§£èª¬ã¯ãã¡ãã®ã¹ã©ã€ãã埡芧ãã ããã
All model families can provide value
Booking.com ã§ã¯ãæ§ã ãªå Žé¢ã§æ©æ¢°åŠç¿ã¢ãã«ã䜿ã£ãŠããŸãããã©ããã¡ãããšããžãã¹çãªäŸ¡å€ãåºããŠããŸããããšããã話ã
äžå³ã¯ãæ©æ¢°åŠç¿ãããžã§ã¯ãã6ã€ã®ã«ããŽãªã«åé¡ããååé¡ããšã«Booking.comãæ±ºããææšã«ã©ã®çšåºŠè²¢ç®ããã£ãã®ãã瀺ãããã®ã§ãã
- Traveller Preference Models: è¡ãå ãæ¥çšãªã©ã«ã€ããŠãã©ããããæè»ã«èããŠãããïŒãã ãã£ãŠãããïŒæšå®ããã¢ãã«
- Traveller Context Models: ã«ããã«ãªã®ããå®¶æãªã®ããè»ã§è¿ãã«è¡ãããã®ãé£è¡æ©ã§é ãã«è¡ãããããªã©ã®ã³ã³ãã¯ã¹ããæšå®ããã¢ãã«
- Item Space Navigation Models: ãŠãŒã¶ãŒãã¢ã€ãã ã
- User Interface Optimization Models: ãã©ã³ããµã€ãºãªã©ã®UIãæé©åããã¢ãã«
- Content Curation: ã¬ãã¥ãŒãªã©ã®æ å ±ãéçŽããŠæç€ºããã¢ãã«
- Content Augmentation:
ææšã¯è§£ãããå顿¯ã«ç°ãªãã®ã§ããã³ãããŒã¯ãçšæããŠãããããã®çžå¯Ÿçãªå€ïŒåã«ããŽãªæ¯ã«medianããšã£ãŠããïŒã«ãªã£ãŠããŸãããã³ãããŒã¯ã¯ãããäžå®æéã®éã«å®æœãããæ¹åããã»ã¹ïŒããã«ã¯æ©æ¢°åŠç¿ã«ãããã®ããããªããã®ãå«ãŸããïŒã«ãã广ã§ããã©ã®ã«ããŽãªãæ£ã®å€ãšãªã£ãŠããã®ã§ãåææšã«å¯ŸããŠãã©ã¹ã®å¹æããã£ãããšãããããŸãããContent Curation ãã®ãããŠããã³ãããŒã¯ãããè¯ãçµæãšãªã£ãŠããããã§ãããã ããã³ãããŒã¯ã®è©³çްãããããªããããã®çµæããã£ãŠãæ©æ¢°åŠç¿ããããããšã¯ãªããªãæ°ãããŸãã
äžå³ã¯ãç®çå°ã¬ã³ã¡ã³ãæ©èœã®æ¹åã®å±¥æŽïŒäžããæœçã®å®æœé ã«äžŠãã§ããïŒã衚ããŠãŸãã
ãã®ããã«ãåæœçã®ææšãå®çŸ©ãããŠããŠãã©ã®æœçã§ã©ãã ã广ããã£ãã®ããããããã«ãªã£ãŠãããšããã®ã¯çŽ æŽããããšæããŸããåºåç³»ã®æ¡ä»¶ã§ã¯ãBooking.comãšåãããã«å¹æãã¢ãã¿ãªã³ã°ã§ããããã«ãªã£ãŠããŠãæ¹åã®ã«ãŒãããŸãããããã«ãªã£ãŠããããšãå€ãã§ããããã以å€ã®åéã§ã¯ãããŸãå€ããªãããã«æããŸããçç±ãšããŠã¯ãããŒã¿åéã®å°é£ããªã©ãããããã§ãããŸããããããåèšåæã ãšã¹ãããã§å ¥ãããšãå€ãããã®å Žåã¯æ¹åã®ã«ãŒããåãç¶ãããšããããã¯ããã®äžã®1ã€ã宿œãããšãã圢ã«ãªãããããšããã®ããããããããŸããã
MODELING: OFFLINE MODEL PERFORMANCE IS JUST A HEALTH CHECK
ã¢ãã«ã®ç²ŸåºŠïŒãªãã©ã€ã³è©äŸ¡ïŒãšå®éã®ïŒããžãã¹äžã®ïŒäŸ¡å€ã¯å¥ç©ã ãããšããã話ã
äžå³ã¯ãåã¢ãã«æ¯ã«æšªè»žã«ãªãã©ã€ã³ã§ã®è©äŸ¡ææšã®æ¹åå¹
ã瞊軞ã«ã³ã³ããŒãžã§ã³ã¬ãŒãã®äžããå¹
ããããããããã®ã§ãã
ãªãã©ã€ã³ã§ã®è©äŸ¡ææšãšå®éã®ã³ã³ããŒãžã§ã³ã¬ãŒãã®äžããå¹ ã«ã¯ããŸãçžé¢ããªãã®ãããããŸãã
ããã¯åèšåæã§ãããèµ·ããããšã§ãããªã³ã©ã€ã³ã§è©äŸ¡ããã®ã¯æ¬åœã«å€§äºãã¬ã³ã¡ã³ããŒã·ã§ã³ã¯ããããããäŸã§ãåŒç€Ÿã®ããã°ã§ãåãäžããŠããŸããäºåã«å ¥æã§ããããŒã¿ã¯ãåºæ¬çã«æ©æ¢°åŠç¿ãçšããæœçã宿œããŠããªãç¶æ ã®ãã®ã§ãæœçã宿œããå Žåã®ããŒã¿ã¯èšç·Žã«äœ¿ããŸãããæ©æ¢°åŠç¿ã¯ããã£ãããã£ãŠããŒã¿ã«æœããã¿ãŒã³ãèŠã€ãããã®ãªã®ã§ãèšç·ŽããŒã¿ã«å«ãŸããªãäºè±¡ã¯äºæž¬ã§ããŸãããç»åèªèã«ããäžè¯åé€å»ã®ããã«ãããžãã¹äžã®äŸ¡å€ãã¢ãã«ã®ç²ŸåºŠããçŽæ¥çã«èšç®ã§ããå Žåã¯åé¡ã«ãªããŸããããã¬ã³ã¡ã³ããŒã·ã§ã³ã®å Žåãæ©æ¢°åŠç¿ã«ããäºæž¬çµæãããšã«ååãæç€ºãããšãã«äººãã©ã®ããã«è¡åãããããšããããŒã¿ããªãã®ã§ãå®éã«äœ¿ã£ãŠã¿ããŸã§ã¯é©åãªè©äŸ¡ãã§ããŸããã
ã¬ã³ã¡ã³ããŒã·ã§ã³ã®ããã«ããã¹ãŠããªã³ã©ã€ã³ã§ããå Žåã¯ãA/Bãã¹ãã®æ©èœãã€ãã£ãŠç¶ç¶çã«è©äŸ¡ã§ããŸããããã以å€ã®å Žåã¯ãPoCãšå®å°æ€èšŒã®ãã§ãŒãºãåããäºãå€ãã§ããPoCã§ã¯ãããŸã§ã¢ãã«ã®ç²ŸåºŠã確èªããå®å°æ€èšŒã§ã¯ãããžãã¹äžæ¬åœã«äŸ¡å€ãããããæ€èšŒããŸããäžè¯åé€å»ã®äŸã§èšãã°ãPoCã§ãäžè¯æ€ç¥ã®ã¢ãã«ã®ç²ŸåºŠãxx%ã§ãããäœæ¥å¡ã®æ¹ã®ç²ŸåºŠyy%ãšåçšåºŠä»¥äžã§ããããšããã®ã確èªããå®å°æ€èšŒã®æ®µéã§ã¯ãããã¿ã€ããã€ãã£ãŠå®éã«å·¥å Žã§äœ¿ã£ãŠããããŸãã
ããšãã¡ãã£ãšè©±ãããããããããŸããããPoCã®æ®µéã§å ¥æã§ããããŒã¿ããæ¬çªãšç°ãªããããªã±ãŒã¹ããããŸããäŸãã°ã¯ã©ã€ã¢ã³ããç ç©¶æãå è¡éçºã®éšçœ²ã§ã瀟å äºæ çã§å®ããŒã¿ã䜿ããªãå Žåã§ãããŸããæ©å¯æ§ã®é«ãæ å ±ã®ãããæ¬çªããŒã¿ã¯ç€Ÿå€ã«åºããªããããªå Žåããããåèšç¹æã®ãã©ããããæããããšããããŸãã
MODELING: BEFORE SOLVING A PROBLEM, DESIGN IT
åé¡èšå®ãã§ãŒãºã§ã¯ãããžãã¹ã±ãŒã¹ãã³ã³ã»ãããããæ©æ¢°åŠç¿ã®åé¡ãžã®èœãšã蟌ã¿ãè¡ããŸããã©ã®ãããªåé¡ã«èœãšã蟌ãããããªãéèŠã§ãè¯ããå¹ŽèŸŒã¿æ¹æ³ã¯ããã«æãã€ããããªãã®ã§ã¯ãªãããšãå€ãããšãã話ã
åèšåæã§ã¯ããŸãã«ãã®åé¡èšå®ãã§ãŒãºãéèŠã§ãããå®ã¯å®éã®åæãã§ãŒãºä»¥äžã«åæå®ãåãçºæ®ãããšããã§ããããããã®ã¯ãã¯ããã¯å®å
šãªèªååãç®æããŠããããå®ã¯äžéšåã®èªååã®ã»ããå¹ççãªã±ãŒã¹ã
äŸãã°åè³ªæ€æ»ãä»åããAIã«çœ®ãæããŠã人件費ãåããããšãã話ãèãããšããããŸããããã®å Žåã人éã®ç²ŸåºŠãè¶
ããïŒå®éã«ã¯äººéã®ç²ŸåºŠã¯èšæž¬ãããŠããªãããšãå€ãã100%ãããããããªãïŒãŸã§PoCãç¶ããŠããŸãããšã«ãªããŸããå®å
šãªèªååã¯æŸæ£ããŠäœæ¥å¡ããã®è² è·è»œæžãç®æããšãäŸãã°ç²ŸåºŠã50%ã§ãæå³ã®ãããã®ã«ãªãããšããããŸãã
ãããªãããããããã®ãã§ãŒãºã¯åºæ¬çã«åæå®ãšããžãã¹ãµã€ãã®äººéãã¯ã©ã€ã¢ã³ããäžç·ã«åãçµãããšã«ãªããŸãããåæå®ä»¥å€ã®äººéãæ©æ¢°åŠç¿ã§ã©ããããã®ããšãã§ããŠãã©ããããã®ããšãã§ããªãã®ããåæãæŽãã§ããããšãã¯ã©ã€ã¢ã³ãã¯å¿ èŠã«å¿ããŠãå®éã®ãŠãŒã¶ãŒãšãªãçŸå Žã®æ¹ãå·»ã蟌ããªã©ããŠãæ¬åœã®ããŒãºã»äŸ¡å€ã®ãããããªããšãåŒãåºãã®ãéèŠã«ãªããŸãããŸãããµã³ãã«ããŒã¿ãåé ããŠç°¡æçãªåæãããããšã§ãåæãã€ããããšããããããŸãã
Booking.com ã§ã¯ã以äžã®ãããªèгç¹ã»ãã¥ãŒãªã¹ãã£ã¯ã¹ãåé¡èšå®ã«å©çšããŠããããã§ãã
- Learning DifficultyïŒåŠç¿ã®é£æåºŠïŒ
- Data to Concept MatchïŒããžãã¹ã±ãŒã¹ã»ã³ã³ã»ããããã£ãããŒã¿ãåŸãããŠãããïŒ
- Selection BiasïŒéžæãã€ã¢ã¹ã®æç¡ïŒ
DEPLOYMENT: TIME IS MONEY
ã¬ã€ãã³ã·ã¯éèŠã ãããšããã話ã
ããèšãããããšã§ãããã¬ã€ãã³ã·ãå¢å ãããšã³ã³ããŒãžã§ã³ã¬ãŒããäžãããããé«ç²ŸåºŠã§éãã¢ãã«ããã軜éã§ã»ã©ã»ã©ã®ç²ŸåºŠã®ã¢ãã«ã®ã»ãããè¯ãçµæãšãªãããšããããŸãã
Booking.comã§ã¯ããã®åé¡ã«å¯Ÿå¿ããããã以äžã®ãããªåãçµã¿ãããŠããããã§ãã
- 忣åŠç
- é«éãªãç¬èªå®è£ ã®ç·åœ¢ã¢ãã«
- ãã©ã¡ãŒã¿ã®å°ãªãã¢ãã«ã®å©çš
- çµæã®ãã£ãã·ã¥ãšãäºååŠç¿
- ãã«ã¯åŠç
- ç¹åŸŽéã®å€æã®ç°¡ç¥å
å®éã®æ¡ä»¶ã ãšãããããè«ãšããŠéœåºŠæšè«ããå¿
èŠã®ãªãå ŽåããããŸããäŸãã°ã¬ã³ã¡ã³ããŒã·ã§ã³ã§ãçŽåã®è¡åã説æå€æ°ã«å
¥ã£ãŠããªãã±ãŒã¹ã ãšãåå€ã«ãããåŠçã§å
šã¢ã¯ãã£ããŠãŒã¶ãŒã«å¯Ÿããã¬ã³ã¡ã³ããªã¹ããçæããŠKVSã«ãããŠãããæšè«æã«ã¯åã«ãŠãŒã¶ãŒã®IDã§æ€çŽ¢ããã ãããšãããããªå®è£
ãããããšããããŸãïŒå®éã«ã¯ãåŸåŠçãšããŠç°¡åãªãã£ã«ã¿åŠçãããããšã§ãåçã«æšè«ããŠããããã«èŠããããšãã§ããŸãïŒã
å¥ã®ã±ãŒã¹ã§ã¯ãåã蟌ã¿ãäºåã«èšç·ŽããŠãããŠæšè«æã«ã¯å
ç©ãåãã ãïŒå®è³ªç·åœ¢ååž°ã«ãªã£ãŠããïŒããšãã£ãããã«ãã¢ãã«ã®äžéšåãäºåã«å®è¡ããŠãããæšè«æã«ã¯å¿
èŠãªåŠçã ããè¡ãïŒãããã¯è¿äŒŒåŠçããããªãïŒãããªããšããããããŸãã
MONITORING: UNSUPERVISED RED FLAGS
ãµãŒãã³ã°ããŠããã®ã§ããã°ãã¢ãã¿ãªã³ã°ã¯å¿ é ã ãã©ãã€ãªã«æ£è§£ããŒã¿ãå ¥æã§ãããããããªããããšããã話ã以äžã®2ã€ã®ç¶æ³ã«ã€ããŠè¿°ã¹ãResponse Distribution Chart (RDC)ãããŒã¹ã«ãããã¥ãŒãªã¹ãã£ã¯ã¹ã䜿ã£ãŠããããã§ãã
- Incomplete feedbackïŒäžå®å šãªãã£ãŒãããã¯ïŒ
- Delayed feedbackïŒãã£ãŒãããã¯ã®é å»¶ïŒ
äžå®å šãªãã£ãŒãããã¯ããã£ãŒãããã¯ã®é å»¶ã¯ãããããªãšããã§ãããããŸããåºåã¬ããŒããšãã ãšãã³ã³ããŒãžã§ã³ã®å€ãåŸã ã«åãŸã£ãŠãã£ãããã³ã³ããŒãžã§ã³ã®å®çŸ©ãåªäœã«ãã£ãŠéã£ãããé©åã«æ±ãã°ãããšããã°ããã®ã§ãããã€ãããããšã¯åçŽã«ç»ååé¡ãããããªãŠã§ãã¢ããªã§ããæç€ºçµæããŠãŒã¶ãŒãä¿®æ£ããŠããããããªä»çµã¿ã«ãªã£ãŠããªãéãã¯ãæ°ããªæ£è§£ã©ãã«ã¯èç©ãããªãã®ã§æ³šæãå¿ èŠã§ããã¢ãã«ãæ¹åããŠããããã®ã§ããã°æ£è§£ã©ãã«ãã©ããã£ãŠãããŠããããå§ãã«èããŠãããªããšãããïŒãã£ãŠãªããŸããæ£è§£ã©ãã«ã¯è²¯ãããããã®ã ãšæãããã§PoCããŠã¿ããã©ããæ£è§£ã©ãã«ã¯PoCã®ããã«äžçæžåœæã ãæã§äœããŸããããã¿ãããªã®ã¯æ®éã«çºçããããã©ãèããŠãæã ãèæ ®ããŠãããªããšãããªããã€ã³ããªã®ã§æ³šæã
EVALUATION: EXPERIMENT DESIGN SOPHISTICATION PAYS OFF
A/Bãã¹ãïŒRandomized Controlled TrialsïŒãçµæ§ãã£ããã§ããããšããã話ã
åçŽã«ã©ã³ãã ã«åããŠæžãæãããã°ãæžãŸãªãæããããŸãã
å°ãå€ãã£ãäŸãšããŠããªã³ã©ã€ã³ã ãšãŠãŒã¶ãŒåäœã§æœçãåããã°ããããšãå€ãã§ããããªãã©ã€ã³ã§A/Bãã¹ããããããšãããšé£ããå ŽåããããŸããäŸãã°ããªã³ã©ã€ã³ãšéã£ãŠãŠãŒã¶ãŒããšã«æœçããããã®ãé£ããã£ããããŠãŒã¶ãŒåå¥ã®ããŒã¿ãåéã§ããªãã£ãããããšããããæœçã®æ¯ãåãã®åäœãåºèãå¶æ¥æåäœãšãªãããšããããŸãããããšãå¶æ¥æã®æ°ã¯ãŠãŒã¶ãŒã»ã©å€ããªããããåçŽã«ä¹±æ°ã§æ¯ãåãããšåããçããŠããŸãã®ã§ãåŠäœã«åããæžããããšãã工倫ãå¿ èŠã«ãªã£ããããŸããããŒã¿ã®éãå ¥æã®ãããããæœçãã¡ããããªã©ããªã³ã©ã€ã³ã¯æµãŸããŠãããªãšæããŸãã
ãŸãšã
ãã¿ãã¿ããŸããã




