ãªããŒã¿ã§ãããã€ãã¿ãŒã§äººå·¥ç¥èœã®ããšãä»åªäœã§æžããŠããèšäºãªã© ã玹ä»ããŠããŸãã®ã§ã人工ç¥èœã®ããšããã£ãšç¥ãããæ¹ãªã©ã¯æ°è»œã«@omiita_atiimoããã©ããŒããŠãã ããïŒ
èªç¶èšèªåŠçã®çæ§ãBERTãã®è«æã培åºè§£èª¬
2018幎10æã«ç»å ŽããŠãèªç¶èšèªåŠçã§ããšããšã人éãè¶ ãã粟床ãå©ãåºãã ããšã§å€§ããªè©±é¡ãšãªã£ãBERTããã以éãXLNetãALBERTãDistillBERTãªã©BERTãããŒã¹ã«ããã¢ãã«ã次ã ãšç»å ŽããŠã¯SoTAãæŽæ°ãç¶ããŠããŸãããã®çµæãGLUEãã³ãããŒã¯ã§ã¯äººéã®èœåã12äœ(2020幎5æ4æ¥æç¹)ã«äœçœ®ããŠããŸããBERTã¯ç»å ŽããŠãŸã 1幎åçšåºŠã§ããã«ããããããã被åŒçšæ°ã¯2020幎5æ4æ¥çŸåšã§4809 ã«ãåã³ãŸããé©ç°çã§ãããã®èšäºã§ã¯ãããªBERTã®è«æã培åºçã«è§£èª¬ããŠãããããšæããŸããBERTã®çè§£ã«ã¯Transformer[Vaswani, A. (2017)]ãçè§£ããŠãããšãšãŠãç°¡åã§ããTransformerã«é¢ããŠã®èšäºã¯æè㮠解説èšäº ãã©ãããBERTã¯å ¬åŒã«ããTensorFlowã®å®è£ ãšPyTorchã䜿çšããŠããæ¹ã«ã¯HuggingFaceã«ããå®è£ ããããŸãã®ã§ãã¡ããåç §ããŠã¿ãŠãã ããã
èªãã§å°ãã§ãäœãåŠã¹ããšæããã ããããã ã ãã³ã¡ã³ãã ããããããšããããã®å±ã¿ã«ãªããŸãïŒãããããé¡ãããŸãïŒ
æµã:
- å¿ããæ¹ãž
- è«æè§£èª¬
- ãŸãšããšææ
- åè
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding, Devlin, J. et al. (2018)
0.å¿ããæ¹ãž
- BERTã¯Transformerã®Encoderã䜿ã£ãã¢ãã«ã ãã
- ããããNLPã¿ã¹ã¯ã«ãã¡ã€ã³ãã¥ãŒãã³ã°å¯èœãªã¢ãã«ã ãã話é¡ã«ãªã£ããã
- äºååŠç¿ãšããŠMLM(=Masked Language Modeling)ãšNSP(Next Sentence Prediction)ãåŠç¿ãããããšã§ççºçã«ç²ŸåºŠåäžãããã
- äºååŠç¿ã«ã¯é·ãæç« ãå«ãããŒã¿ã»ãããçšãããã
- 11åã®ã¿ã¹ã¯ã§å§åçSoTA ãåœæå©ãåºãããã
1. è«æBERT: Pre-training of Deep Bidirectional Transformers for Language Understanding解説
1.0 èŠçŽ
BERT㯠Bidirectional Encoder Representations from Transformers ã®ç¥ã§ãTransformerã®Encoderã䜿ã£ãŠããã¢ãã«ãBERTã¯ã©ãã«ã®ã€ããŠããªãæç« ãã衚çŸãäºååŠç¿ããããã«äœããããã®ã§ãåºåå±€ãä»ãå ããã ãã§ç°¡åã«ãã¡ã€ã³ãã¥ãŒãã³ã°ãå¯èœã
NLPã¿ã¹ã¯11åã§SoTA ãéæãã倧å¹
ã«ã¹ã³ã¢ãå¡ãæ¿ããã
1.1 å°å ¥
èªç¶èšèªåŠçã¿ã¹ã¯ã«ãããŠã粟床åäžã«ã¯èšèªã¢ãã«ã«ããäºååŠç¿ãæå¹ã§ããããã®èšèªã¢ãã«ã«ããäºååŠç¿ã«ã¯ãç¹åŸŽéããŒã¹ããšããã¡ã€ã³ãã¥ãŒãã³ã°ãã®2ã€ã®æ¹æ³ãããããŸãããç¹åŸŽéããŒã¹ããšã¯ äºååŠç¿ã§åŸããã衚çŸãã¯ãã«ãç¹åŸŽéã®1ã€ãšããŠçšãããã® ã§ãã¿ã¹ã¯ããšã«ã¢ãŒããã¯ãã£ãå®çŸ©ãããELMo[Peters, (2018)]ããã®äŸã§ããããŸããããã¡ã€ã³ãã¥ãŒãã³ã°ã㯠äºååŠç¿ã«ãã£ãŠåŸããããã©ã¡ãŒã¿ãéã¿ã®åæå€ãšããŠåŠç¿ããããã® ã§ãã¿ã¹ã¯ããšã§ãã©ã¡ãŒã¿ãå€ããå¿ èŠãããŸããªããäŸãšã㊠OpenAI GPT[Radford, (2018)]ãããããã ããããããããåé¡ããããããã¯äºååŠç¿ã«çšããèšèªã¢ãã«ã®æ¹åã1æ¹åã ããšããããšã ãäŸãã°ãGPTã¯å·Šããå³ã®æ¹åã«ããåŠç¿ãããæç« ã¿ã¹ã¯ãQ&Aãªã©ã®ååŸã®æèã倧äºãªãã®ã§ã¯æå¹ã§ã¯ãªãã
ããã§ããã®è«æã§ã¯ ããã¡ã€ã³ãã¥ãŒãã³ã°ã«ããäºååŠç¿ãã«æ³šå ãã粟床åäžãè¡ãªããå ·äœçã«ã¯äºååŠç¿ã«ä»¥äžã®2ã€ãçšããã
- Masked Language Model(= MLM)
- Next Sentence Prediction(= NSP)
ããããã
- MLM: è€æ°ç®æã穎ã«ãªã£ãŠããæç« ã®ããŒã¯ã³(åèª)äºæž¬
- NSP: 2æãæž¡ãããé£ç¶ããæãã©ããå€å®
ãã®è«æã®ã³ã³ããªãã¥ãŒã·ã§ã³ã¯ä»¥äžã§ããã
- äž¡æ¹åã®äºååŠç¿ã®éèŠæ§ã瀺ã
- äºååŠç¿ã«ããã¿ã¹ã¯ããšã«ã¢ãŒããã¯ãã£ãèããå¿ èŠãæžã
- BERTã11åã®NLPã¿ã¹ã¯ã«ãããŠSoTAãéæ
1.2 é¢é£ç ç©¶
ããã§ã¯èªç¶èšèªã«ãããäºååŠç¿ã«ã€ããŠè§ŠããŠããã
1.2.1 æåž«ãªãç¹åŸŽéããŒã¹ã®ææ³
äºååŠç¿ã§ããåèªã®åã蟌ã¿ã«ãã£ãŠã¢ãã«ã®ç²ŸåºŠã倧å¹
ã«äžããããšãã§ããçŸåšã®NLPã«ãšã£ãŠã¯å¿
èŠäžå¯æ¬ ãªååšãšãªã£ãŠããã
åèªã®åã蟌ã¿è¡šçŸãç²åŸããã«ã¯ãäž»ã«æ¬¡ã®2ã€ãããã
- æç« ã®å·Šããå³ã®æ¹åã§ã®èšèªã¢ãã«
- å·Šå³ã®æèããåèªãæ£ããã誀ã£ãŠããããèå¥ãããã®
ãŸããæã®åã蟌ã¿è¡šçŸã«ãããŠã¯æ¬¡ã®3ã€ãããã
- 次ã«ç¶ãæãã©ã³ãã³ã°åœ¢åŒã§äºæž¬ãããã®
- æ¬¡ã«æ¥ãæãçæãããã®
- denoisingãªãŒããšã³ã³ãŒããŒç±æ¥ã®ãã®
ããã«ãæèããã£ãããšãããŠåèªã®åã蟌ã¿è¡šçŸãç²åŸãããã®ã«ELMoãããã
ããã¯ãå·Šããå³ãããã³ãå³ããå·Šãã®äž¡æ¹åã§ã®åã蟌ã¿ãçšããããšã§ç²ŸåºŠã倧ããäžããã
1.2.2 æåž«ãªããã¡ã€ã³ãã¥ãŒãã³ã°ã®ææ³
ç¹åŸŽéããŒã¹ãšåãããåãã¯æäžã®åèªã®åã蟌ã¿ãè¡ãããšã§äºååŠç¿ã®éã¿ãç²åŸããŠããããè¿å¹Žã¯æèãèæ ®ããåã蟌ã¿ãè¡ãªã£ãããšã«æåž«ããã®äžæµã¿ã¹ã¯ã«ãã¡ã€ã³ãã¥ãŒãã³ã°ããŠãããã®ãå¢ããŠããããããã®äŸãšããŠæ¬¡ã®ãããªãã®ãããã
- æç« ã®å·Šããå³ã®æ¹åã§ã®èšèªã¢ãã«
- ãªãŒããšã³ã³ãŒããŒ
1.2.3 æåž«ããããŒã¿ã«ãã転移åŠç¿
ç»åèªèã®åéã§ã¯ImageNetãªã©ã®æåž«ããããŒã¿ãçšããäºååŠç¿ãæå¹ã§ã¯ããããèªç¶èšèªåŠçã«ãããŠãæå¹ãªäŸããããæåž«ããäºååŠç¿ãšããŠçšããããŠãããã®ã«ä»¥äžã®ãããªãã®ãããã
- æ©æ¢°ç¿»èš³
- èªç¶èšèªæšè«(= åæãšä»®èª¬ã®æã®ãã¢ãæž¡ãããããããæ£ãããççŸããŠãããå€å¥ããã¿ã¹ã¯)
1.3 BERT
ããã§ã¯BERTã®æŠèŠãè¿°ã¹ãã®ã¡ã«æ·±å ããããŠããã
1.3.1 BERTã®æŠèŠ
ãŸããBERTã®åŠç¿ã«ã¯ä»¥äžã®2段éãããã
- äºååŠç¿: ã©ãã«ãªãããŒã¿ãçšããŠãè€æ°ã®ã¿ã¹ã¯ã§äºååŠç¿ãè¡ã
- ãã¡ã€ã³ãã¥ãŒãã³ã°: äºååŠç¿ã®éã¿ãåæå€ãšããŠãã©ãã«ããããŒã¿ã§ãã¡ã€ã³ãã¥ãŒãã³ã°ãè¡ãªãã
äŸãšããŠQ&Aã¿ã¹ã¯ãå³ã§è¡šããšæ¬¡ã®ããã«ãªãã
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding, Devlin, J. et al. (2018)
ç°ãªãã¿ã¹ã¯ã«ãããŠãã¢ãŒããã¯ãã£ãçµ±äžãããŠãã ãšããã®ããBERTã®ç¹åŸŽã§ããã
-
ã¢ãŒããã¯ãã£: Transformerã®ãšã³ã³ãŒããŒã®ã¿ã
- $\mathrm{BERT_{BASE}}$ ($L=12, H=768, A=12$, ãã©ã¡ãŒã¿æ°:1.1å)
- $\mathrm{BERT_{LARGE}}$ ($L=24, H=1024, A=16$, ãã©ã¡ãŒã¿æ°:3.4å)
- $L$:Transformerãããã¯ã®æ°, $H$:é ãå±€ã®ãµã€ãº, $A$:self-attentionãããã®æ°
-
å ¥åºå: ã¿ã¹ã¯ã«ãã£ãŠ1ã€ã®æ(Ex.ææ åæ)ããŸãã¯2ã€ã®æãã€ãªãããã®(Ex.Q&A)
- BERTãžã®å ¥åã以äžãsentenceãšåŒã¶ã
- sentenceã®å é ã«[CLS]ããŒã¯ã³ãæãããã
- 2æããã£ã€ããæã¯ãéã«[SEP]ããŒã¯ã³ãå ¥ã ã〠ããããã«1æç®ã2æç®ãã衚ãåã蟌ã¿è¡šçŸãå ç® ããã
- æçµçã«å
¥åæã¯ä»¥äžã®ããã«ãªãã
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding, Devlin, J. et al. (2018)
- $E$:å ¥åã®åã蟌ã¿è¡šçŸ, $C$:[CLS]ããŒã¯ã³ã®é ããã¯ãã«, $T_i$:sentenceã®$i$çªç®ã®ããŒã¯ã³ã®é ããã¯ãã«
1.3.2 BERTã®äºååŠç¿
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding, Devlin, J. et al. (2018)
æãäž¡æ¹å(ãå·Šããå³ãããã³ãå³ããå·Šã)ã§åŠã¶ããã«BERTã§ã¯2ã€ã®äºååŠç¿ãè¡ãªãããããMLMãšNSPã§ãããããŒã¿ã»ããã«ã¯BooksCorpus(åèªæ°:8å)ãšEnglish Wikipedia(åèªæ°:25å)
-
Task1: Masked Language Modeling (MLM)
å ¥åã®15%ã®ããŒã¯ã³ã[Mask]ããŒã¯ã³ã§ãã¹ã¯ããå ã®ããŒã¯ã³ãåœãŠãã¿ã¹ã¯ãã€ãŸãã穎åãåé¡ããã®æãäžå³ã®$T_i$ã«Softmaxãé©çšããããšã§ããŒã¯ã³ãäºæž¬ããã
ãã ãããã¡ã€ã³ãã¥ãŒãã³ã°ã§ã¯åºãŠããªã[Mask]ããŒã¯ã³ãäºååŠç¿ã§äœ¿çšããŠããŸã£ãŠããããã«äºååŠç¿ãšãã¡ã€ã³ãã¥ãŒãã³ã°éã®å·®ç°ãçããŠããŸãããã®ããããã¹ã¯ããããŒã¯ã³ãåžžã«[Mask]ããŒã¯ã³ã«çœ®ãæããã®ã§ã¯ãªãããã¹ã¯ããããŒã¯ã³ã«å¯ŸããŠæ¬¡ã®ããã«ããããšã§åé¡ãç·©å ããã
| 確ç(%) | åŠç |
|---|---|
| 80 | [Mask]ããŒã¯ã³ã§çœ®ãæãã |
| 10 | ã©ã³ãã ã«éžãã ããŒã¯ã³ã§çœ®ãæãã |
| 10 | ãã®ãŸãŸã«ãã |
-
Task2: Next Sentence Prediction (NSP)
Q&Aãèªç¶èšèªæšè«ãªã©æå士ã®é¢ä¿ãèæ ®ããå¿ èŠãããåé¡ã«å¯ŸããŠãMLMã§ã¯å¯ŸåŠã§ããªãããã®ããã2æéžãã§ããããé£ç¶ããæãã©ãããåœãŠãã¿ã¹ã¯ ãè¡ãªãã
ãã®æãäžå³ã®$C$ãçšããŠäºæž¬ãè¡ãªãã
(ããã§ã$C$ã¯ãã®æç¹ã§ã¯NSPã«ç¹åãããã®ãªã®ã§ãæå šäœã衚çŸãããã¯ãã«ã«ã¯ãªã£ãŠããªãã)
1.3.3 BERTã®ãã¡ã€ã³ãã¥ãŒãã³ã°
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding, Devlin, J. et al. (2018)
åçŽã«ã¿ã¹ã¯ããšã«å
¥åããã ãã
åºåã®ãã¡ $C$ã¯èå¥ã¿ã¹ã¯(Ex.ææ
åæ) ã«äœ¿ããã$T_i$ã¯ããŒã¯ã³ã¬ãã«ã®ã¿ã¹ã¯(Ex.Q&A) ã«äœ¿ãããã
ãã¡ã€ã³ãã¥ãŒãã³ã°ã¯äºååŠç¿ãããåŠç¿ã軜ããã©ã®ã¿ã¹ã¯ãCloud TPUã1å䜿çšããã°1æé以å
ã§çµãã£ãã(GPU1åã§ã2~3æéçšåºŠ)
(ãã ããäºååŠç¿ã«ã¯TPU4ã€äœ¿çšã§ã4æ¥ããããã)
ä»ã®ãã¡ã€ã³ãã¥ãŒãã³ã°ã®äŸã¯ä»¥äžã®å³ã®ããã«ãªãã
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding, Devlin, J. et al. (2018)
1.4 å®éš
ããããã¯BERTãSoTAãå©ãåºãã11åã®NLPã¿ã¹ã¯ã«å¯ŸããŠã®çµæãèšãã
1.4.1 GLUE
GLUEãã³ãããŒã¯(General Language Understanding Evaluation)[Wang, A.(2019)]ãšã¯8ã€ã®èªç¶èšèªçè§£ã¿ã¹ã¯ã1ã€ã«ãŸãšãããã®ã§ãããæçµã¹ã³ã¢ã¯8ã€ã®å¹³åããšãããã¡ãã§çŸåšã®SoTAã¢ãã«ãªã©ã確èªã§ãããä»åçšããããŒã¿ã»ããã®å èš³ã¯ä»¥äžã
| ããŒã¿ã»ãã | ã¿ã€ã | æŠèŠ |
|---|---|---|
| MNLI | æšè« | åææãšä»®èª¬æã嫿/ççŸ/äžç«ã®ããããå€å® |
| QQP | é¡äŒŒå€å® | 2ã€ã®çåæãæå³çã«åããåŠããå€å¥ |
| QNLI | æšè« | æãšè³ªåã®ãã¢ãæž¡ãããæã«çããå«ãŸãããåŠããå€å® |
| SST-2 | 1æåé¡ | æã®ããž/ãã¬ã®ææ åæ |
| CoLA | 1æåé¡ | æãææ³çã«æ£ãããåŠããå€å¥ |
| STS-B | é¡äŒŒå€å® | 2æãæå³çã«ã©ãã ãé¡äŒŒããŠããããã¹ã³ã¢1~5ã§å€å¥ |
| MRPC | é¡äŒŒå€å® | 2æãæå³çã«åããåŠããå€å¥ |
| RTE | æšè« | 2æã嫿ããŠãããåŠããå€å® |
çµæã¯ä»¥äžã
$\mathrm{BERT_{BASE}}$ããã³$\mathrm{BERT_{LARGE}}$ãããããããŸã§ã®SoTAã¢ãã«ã§ããOpenAI GPTãã¯ããã«åé§ããŠãããå¹³åã§ $\mathrm{BERT_{BASE}}$ã¯4.5 %ã®ã²ã€ã³ã$\mathrm{BERT_{LARGE}}$ã¯7.0%ãã²ã€ã³ ãåŸãããã

BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding, Devlin, J. et al. (2018)
1.4.2 SQuAD v1.1
SQuAD(Stanford Question Answering Dataset) v1.1[Rajpurkar (2016)]ã¯Q&Aã¿ã¹ã¯ã§ã質åæãšçããå«ãæç« ãæž¡ãããçããã©ãã«ããããäºæž¬ãããã®ã
ãã®æãSQuADã®åã«TriviaQAããŒã¿ã»ããã§ãã¡ã€ã³ãã¥ãŒãã³ã°ããã®ã¡ã«SQuADã«ãã¡ã€ã³ãã¥ãŒãã³ã°ããã
çµæã¯ä»¥äžã
ã¢ã³ãµã³ãã«ã§F1ã¹ã³ã¢ã«ãŠ1.5ãã€ã³ãã®ã²ã€ã³ãã·ã³ã°ã«ã¢ãã«ã§ãF1ã¹ã³ã¢ã«ãŠ1.3ãã€ã³ãã®ã²ã€ã³ ãåŸããããç¹çãã¹ã㯠BERTã®ã·ã³ã°ã«ãã¢ã³ãµã³ãã«ã®SoTAãäžåã£ã ãšããããšã

BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding, Devlin, J. et al. (2018)
1.4.3 SQuAD v2.0
SQuAD v2.0ã¯SQuAD v1.1ã«ãçããååšããªãããšããéžæè¢ãå ãããã®ã
çããååšãããåŠãã¯[CLS]ããŒã¯ã³ãçšããŠå€å¥ã
ãã¡ãã§ã¯TriviaQAããŒã¿ã»ããã¯çšããªãã£ãã
çµæã¯ä»¥äžã
F1ã¹ã³ã¢ã«ãŠSoTAã¢ãã«ããã5.1ãã€ã³ãã®ã²ã€ã³ ãåŸãããã

BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding, Devlin, J. et al. (2018)
1.4.4 SWAG
SWAG(Situations With Adversarial Generations)[Zellers, R.(2018)]ã¯åžžèçãªæšè«ãè¡ãã¿ã¹ã¯ã§ãäžããããæã«ç¶ãæãšããŠãã£ãšãããããã®ã4ã€ã®éžæè¢ããéžã¶ãšãããã®ã
äžããããæãšéžæè¢ã®æããã¢ãšããŠã[CLS]ããŒã¯ã³ãçšããŠã¹ã³ã¢ãç®åºããã
çµæã¯ä»¥äžã
$\mathrm{BERT_{LARGE}}$ãSoTAã¢ãã«ããã8.3%ã粟床ãåäžããã

BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding, Devlin, J. et al. (2018)
1.5 ã¢ãã¬ãŒã·ã§ã³ã¹ã¿ãã£
BERTãæ§æãããã®ãã¡ã®çžé¢æ§ãªã©ãã¿ãããã«ããã€ãã¢ãã¬ãŒã·ã§ã³(éšåéšåã§èŠãŠãããããªå®éšã®ããšã)ãè¡ãªã£ãã
1.5.1 äºååŠç¿ã¿ã¹ã¯ã«ãã圱é¿
BERTãåŠãã æã®äž¡æ¹åæ§ãã©ãã ãéèŠãã確ãããããã«ãããã§ã¯æ¬¡ã®ãããªäºååŠç¿ã¿ã¹ã¯ã«ã€ããŠè©äŸ¡ããŠããã
- NSPãªã: MLMã®ã¿ã§äºååŠç¿
- LTR & NSPãªã: MLMã§ã¯ãªããé垞䜿ãããLeft-to-Right(å·Šããå³ã®æ¹å)ã®èšèªã¢ãã«ã§ã®ã¿äºååŠç¿
ãããã«ããçµæã¯ä»¥äžã

BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding, Devlin, J. et al. (2018)
ãããããããã®ã¯æ¬¡ã®3ã€ã
- NSPãç¡ããšQNLI, MNLIããã³SQuADã«ãŠããªãæªå($\mathrm{BERT_{BASE}}$ vs NoNSP)
- MLMã®äž¡æ¹åæ§ããªã(=éåžžã®LM)ã ãšãMRPCããã³SQuADã«ãŠããªãæªå(NoNSP vs LTR&NoNSP)
- BiLSTMã«ããäž¡æ¹åæ§ããããšSQuADã§ã¹ã³ã¢åäžãã§ããããGLUEã§ã¯äŒžã³ãªãã (LTR&NoNSP vs LTR&NoNSP+BiLSTM)
1.5.2 ã¢ãã«ãµã€ãºã«ãã圱é¿
BERTã¢ãã«ã®æ§é ã®ãã¡æ¬¡ã®3ã€ã«ã€ããŠèããã
- å±€ã®æ° $L$
- é ãå±€ã®ãµã€ãº $H$
- ã¢ãã³ã·ã§ã³ãããã®æ° $A$
ãããã®å€ãå€ããªãããèšèªã¢ãã«ã¿ã¹ã¯ãå«ã4ã€ã®ã¿ã¹ã¯ã§ç²ŸåºŠãèŠããšã以äžã®ããã«ãªã£ãã
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding, Devlin, J. et al. (2018)
ãã®çµæããèšããããšã¯äž»ã«æ¬¡ã®2ã€ã®ããšãèšããã
- æ©æ¢°ç¿»èš³ãšæ¯ã¹ãŠå°ããªã¿ã¹ã¯ã«ãããŠã倧ããã¢ãã«ã䜿ããšç²ŸåºŠãäžããã
- äžæµã¿ã¹ã¯ãå°ãããŠããã¡ã€ã³ãã¥ãŒãã³ã°ããããšã§äºååŠç¿ã倧ããããé«ã粟床 ãåºããã
1.5.3 BERTãçšããç¹åŸŽéããŒã¹ã®ææ³
ãã®è«æãéããŠç€ºããçµæã¯ãäºååŠç¿ããã¢ãã«ã«èå¥åšãã®ããŠåŠç¿ãçŽããã¡ã€ã³ãã¥ãŒãã³ã°ã«ãããã®ã§ãããããã§ã¯ãã¡ã€ã³ãã¥ãŒãã³ã°ã®ä»£ããã«BERTã«ç¹åŸŽéããŒã¹ã®ææ³ãé©çšããã
ããŒã¿ã»ããã«åºæè¡šçŸæœåºã¿ã¹ã¯ã§ããCoNLL-2003[Sang, T. (2003)]ãçšããã
çµæã¯ä»¥äžã
ç¹åŸŽéããŒã¹ã®$\mathrm{BERT_{BASE}}$ã¯ãã¡ã€ã³ãã¥ãŒãã³ã°ã®$\mathrm{BERT_{BASE}}$ãšæ¯ã¹F1ã¹ã³ã¢0.3ããå€ãããããã®ããšããBERTã¯ãã¡ã€ã³ãã¥ãŒãã³ã°ããã³ç¹åŸŽéããŒã¹ãããã®ææ³ã§ã广ãçºæ®ããããšããããã
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding, Devlin, J. et al. (2018)
1.6 çµè«
ãããŸã§ã«èšèªã¢ãã«ã«ãã転移åŠç¿ã䜿ãããšã§å±€ã®æµ ãã¢ãã«ã®ç²ŸåºŠãåäžããããšãããã£ãŠãããããã®è«æã§ã¯ããã«äž¡æ¹åæ§ãæã£ãããæ·±ãã¢ãã«(=BERT)ã«ãããŠã転移åŠç¿ã䜿ããããšã瀺ãããæ·±ãã¢ãã«ã䜿ãããæ ã«ãããã«å€ãã®èªç¶èšèªçè§£ã¿ã¹ã¯ã«å¯ŸããŠå¿çšãå¯èœã§ããã
2. ãŸãšããšææ
BERTã¯åºæ¬çã«ãTransformerã®Encoder + MLM&NSPäºååŠç¿ + é·æããŒã¿ã»ããããšããé¢šã«æããŸãããBERTããã£ããã«èªç¶èšèªåŠçã¯å é床ãå¢ããŠçºå±ããŠããŸããBERTã«ã€ããŠããã«çè§£ãæ·±ãããå Žåã¯ãã²è«æãããã£ãŠã¿ãŠãã ããïŒ
èªãã§å°ãã§ãäœãåŠã¹ããšæããã ããããã ã ãã³ã¡ã³ãã ããããããšããããã®å±ã¿ã«ãªããŸãïŒãããããé¡ãããŸãïŒ
ãã€ãã¿ãŒ@omiita_atiimoããã²ïŒ
3. åè
-
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding, Devlin, J. et al. (2018)
åè«æã -
GLUE: A MULTI-TASK BENCHMARK AND ANALYSIS
PLATFORM FOR NATURAL LANGUAGE UNDERSTANDING, Wang, A. (2019)
GLUEãã³ãããŒã¯ã®è«æã -
The feature of bidirection #83
[GitHub] BERTã®äž¡æ¹åæ§ã¯Transformersç±æ¥ã®ãã®ã§åçŽã«Self-Attentionã§å®çŸãããŠããããšããããšãæããŠãããŠããissueã -
BERT Explained!
[YouTube] BERTã®è§£èª¬åç»ãç°¡æœã«ãŸãšãŸã£ãŠããŠåãããããã -
[BERT] Pretranied Deep Bidirectional Transformers for Language Understanding (algorithm) | TDLS
[YouTube] BERTè«æã«ã€ããŠè©³è§£ããŠãããŠããåç»ã





