
ãSieve-Refereeãæ±ºå®è«çããã¹ãæ€èšŒãšãã§ãŒã³æ€åºãšã³ãžã³ã«ããæ§é çã¬ããã³ã¹
ã確çïŒAIïŒãå€å®ããããã«ãæ¬åœã«å¥ã®ç¢ºçïŒAIïŒã䜿ãã¹ããªã®ã ãããïŒã
çæAIãæç« ãäœãããã®æç« ãå¥ã®AIãè©äŸ¡ãããããããä»çµã¿ã¯äŸ¿å©ã§ããäžæ¹ãè©äŸ¡çµæãã©ã®ãããªæ ¹æ ããå°ãããã®ãããå³å¯ã«è¿œè·¡ããããšãé£ããå ŽåããããŸãã
ããšãã°ãããæç« ã«ãé¡äŒŒåºŠ 92%ããšããè©äŸ¡ãä»ãããšããŸãã
ãã®æ°åã¯ãäœãæ¯èŒããçµæãªã®ã§ãããããæç« ã®æ§é ãªã®ããèªåœãªã®ãããããšãæå³çãªé¡äŒŒæ§ãªã®ãããããŠããªã92%ãªã®ã§ããããã
æ°å€ãåºåãããããšãšããã®æ°å€ã®æ ¹æ ã説æã§ããããšã¯å¥ã®åé¡ã§ãã
AIçã®ç¢ºççã¢ãã«ã«ãã£ãŠçæãããããã¹ãã調ã¹ãéãå¥ã®ç¢ºççã¢ãã«ã§ã¹ã³ã¢ãç®åºãããšãããªããã®æ°å€ã«ãªã£ãã®ããã®æ°çç远跡ãé£ãããªãå ŽåããããŸãã
Sieve Referee ã¯ããæ°å€ã®è£ã«ããæç¢ºãªçç±ãã誰ã§ãåçŸã»ç¢ºèªã§ããããã«ããããšããåé¡æèµ·ããçãŸããŸããã
Sieve-Refereeã¯ãããã¹ãéã®æ§é çã»å
容çãªé¡äŒŒæ§ãè€æ°ã®ä»®èª¬ã«åè§£ãããããããå®çŸ©ããã«ãŒã«ã«åºã¥ããŠå€å®ãããæ±ºå®è«çãªããã¹ãæ€èšŒã»ã¹ã¯ãªãŒãã³ã°ãšã³ãžã³ã§ãã
å€éšã®æ©æ¢°åŠç¿ã¢ãã«ã«å€å®ãå§ããã®ã§ã¯ãªããæååã®æ£èŠåãããŒã¯ã³åãé¡äŒŒåºŠèšç®ãã«ãŒã«ããŒã¹ã®åé¡ãã¯ã©ã¹ã¿è§£æãçµã¿åãããããšã§ãå€å®ã®åçŸæ§ãšè¿œè·¡å¯èœæ§ã確ä¿ããããšãç®æããŠããŸãã
æ¬èšäºã§ã¯ãSieve-Refereeã®èšèšææ³ããå éšã¢ãŒããã¯ãã£ã4ã€ã®ä»®èª¬ã«ããå€å®ã¢ãã«ããããŠèª€å€å®ãæããããã®å®å šèšèšãŸã§ã解説ããŸãã
Sieve ãšã³ã·ã¹ãã ã«ããã sieve-referee ã¯ãçæAIã®åºåãå€éšããæµå ¥ããããã¥ã¡ã³ã矀ã«å¯Ÿããæ§é çæŽåæ§ãäŸåé¢ä¿ã®é£ç¶æ§ãããã³åè³ªåºæºã®æ€èšŒãé©çšããæ±ºå®è«çãªãŒã±ã¹ãã¬ãŒã·ã§ã³ãšã³ãžã³ã§ãã
æ¬èšäºã§ã¯ãå€éšã©ã€ãã©ãªãäžåæé€ãã Python æšæºã©ã€ãã©ãªã®ã¿ã®èšèšææ³ã®ããšãsieve-referee ãããã«ããŠäž»èгã確çè«ãæãã峿 Œãªããã¹ãæ€èšŒãšãã§ãŒã³æ€åºãå®çŸããŠãããããã®å éšã¢ãžã¥ãŒã«ãšã¢ãŒããã¯ãã£ã解説ããŸãã
1. ãšã³ã·ã¹ãã ã«ãããäœçœ®ã¥ããšãŒãäŸå
Sieve-Refereeã¯ãSieve-Coreã®èšèšææ³ããããã¹ãæ€èšŒãšæ§é çã¬ããã³ã¹ã®é åãžå±éãããããžã§ã¯ãã§ãã
Sieve-Coreãé¡äŒŒããããŒã¿ã®æ§é çãªéçŽãäž»ãªç®çãšããã®ã«å¯ŸããSieve-Refereeã¯ãè€æ°ã®ææžãæ¯èŒããã©ã®ãããªé¡äŒŒæ§ãååšããã®ããåé¡ããããšã«éç¹ã眮ããŠããŸãã
äž¡è ã«å ±éããã®ã¯ãæ å ±ã®é¢ä¿ãæé»çãªå€æã«ä»»ãããæç€ºçãªåŠçãšããŠèšè¿°ãããšããèãæ¹ã§ãã
åããã®èšèšææ³ãã篩ããšåŒãã§ããã®ã¯ãæ å ±ãç¡æ¡ä»¶ã«ä¿¡é Œãããããã¹ãŠãäžæ¬ããŠæé€ãããããã®ã§ã¯ãªããå®çŸ©ãããæ¡ä»¶ãäžã€ãã€é©çšããªãããæ å ±ãåé¡ããä»çµã¿ãç®æããŠããããã§ãã
Sieve-Refereeã§ã¯ãäž»ã«æ¬¡ã®3ã€ã®èŠçŽ ãçµã¿åãããŠããŸãã
æ§é ã®æ€èšŒïŒæç« ã®åœ¢åŒãããŒã¯ã³ã®äžŠã³æ¹ã«å ±éæ§ããããã
å å®¹ã®æ€èšŒïŒå 容èªãéšåæååã«ã©ã®çšåºŠã®å ±éæ§ããããã
é¢ä¿æ§ã®æ€èšŒïŒè€æ°ã®ææžãã©ã®ãããªé¡äŒŒé¢ä¿ã§çµã°ããŠãããã
ããããåé¢ããŠè©äŸ¡ããããšã§ãåçŽãªäžèŽå€å®ã§ã¯èŠåãã«ããã±ãŒã¹ããç°ãªããã¿ãŒã³ãšããŠæ±ããããã«ããŠããŸãã
sieve-core ãã°ã©ãæ§é ã®äŸåéçŽãšä¿¡é Œæ§è©äŸ¡ãæ ãäžæ¹ãsieve-referee ã¯ãå ¥åãããããã¹ããããŒã¿ã¹ããªãŒã ãã®ãã®ãããããããå®çŸ©ãããæ§é çã»è«ççå¶çŽãæºãããŠãããããå€å®ããæ€å®å®ïŒRefereeïŒãšããŠæ©èœããŸãã
Python æšæºã©ã€ãã©ãªã®ã¿: å€éšããŒãµãŒãéã NLP ã©ã€ãã©ãªã«äŸåããªãããããµã³ãããã¯ã¹ç°å¢ããšããžç°å¢ã«ãããŠã決å®è«çã§é«éãªå®è¡ãä¿èšŒããŸãã
observer-not-arbiter ã®ç¶æ¿: ç¬èªã®æšæž¬ãè£æ£ãè¡ãããå®çŸ©ãããã«ãŒã«ããã¯ã«åºã¥ã峿 Œãªãã¹/ãã§ã€ã«å€å®ã®ã¿ãåºåããŸãã
ãªã決å®è«çãªæ€èšŒãªã®ã
Sieve-Refereeã®åºæ¬æ¹éã¯ãObserver, not arbiter. â 芳枬è ã§ãã£ãŠã仲è£è ã§ã¯ãªãã ãšãããã®ã§ãã
ãã®æ¹éã¯ããšã³ãžã³ãäºå®ã®æ£ããã人éã®æå³ãç¬æã§ç¢ºå®ããã®ã§ã¯ãªããå®çŸ©ãããæ¡ä»¶ã«ç §ãããŠèŠ³æž¬å¯èœãªç¹åŸŽãè©äŸ¡ããããšããæå³ã§ãã
ããšãã°ã2ã€ã®æç« ãé«ãé¡äŒŒæ§ã瀺ããå Žåãããã ãã§ççšããã£ããšæå®ããããšã¯ã§ããŸããã
å ±éã®èª²é¡ãã³ãã¬ãŒãã䜿ã£ãŠããå¯èœæ§ãããã°ãæ³ä»€ãå¥çŽæžã®å®å衚çŸãäžèŽããŠããå¯èœæ§ããããŸãããããã¯ãåãäºå®ã説æããããã«ãäžè¬çãªçšèªãå¶ç¶äžèŽããŠããã ããããããŸããã
äžæ¹ã§ãæç« ã®è¡šé¢äžã®è¡šçŸãç°ãªã£ãŠããŠããå 容èªãéšåæååã®çµã¿åããã匷ãäžèŽããããšããããŸãã
éèŠãªã®ã¯ãäžã€ã®æ°å€ã ãã§çµè«ãåºãã®ã§ã¯ãªããäœãäžèŽããäœãäžèŽããŠããªãã®ããåºå¥ããããšã§ãã
Sieve-Refereeã§ã¯ãæ¬¡ã®æ§è³ªãéèŠããŠããŸãã
åçŸæ§
åãå ¥åãåãèšå®ãåãå®è£ ã§ããã°ãåãå€å®çµæãåŸãããããã«èšèšããŠããŸãã
説æå¯èœæ§
å€å®çµæã«ã¯ã4ãããã®ä»®èª¬ãã¹ã¯ããã¿ãŒã³åãå€å®çç±ãä»äžãããŸããå©çšè ã¯ãåãªãã¹ã³ã¢ã§ã¯ãªããã©ã®åé¡ã«ãŒã«ãé©çšãããã®ãã確èªã§ããŸãã
ããŒã«ã«å®è¡
ã³ã¢ã®æ¯èŒåŠçã«å€éšã®AIãµãŒãã¹ããµãŒãããŒãã£è£œã©ã€ãã©ãªãå¿ é ãšããŸãããææžãå€éšAPIãžéä¿¡ããã«è©äŸ¡ã§ãããããæ©å¯æ§ãæ±ããããç°å¢ã§ãçµã¿èŸŒã¿ãããæ§æã§ãã
ãã ããæ±ºå®è«çã§ããããšã¯ãå€å®ãåžžã«æ£ããããšãæå³ããŸãããè©äŸ¡é¢æ°ãéŸå€ãäžé©åã§ããã°ã誀ã£ãå€å®ãåçŸæ§ããã£ãŠçºçããŸãã
ãã®ãããSieve-Refereeãç®æãã®ã¯ã絶察ã«ééããªãAIãã§ã¯ãªããã©ã®æ¡ä»¶ã«åºã¥ããŠäœãå€å®ããã®ãã远跡ã§ããæ€èšŒãšã³ãžã³ã§ãã
2. ã³ã¢ã¢ãžã¥ãŒã«çŸ€ã®å éšæ§é
ãªããžããªå ã®èšèšã¯ã責åããšã«æç¢ºã«ã¢ãžã¥ãŒã«åãããŠããŸããäž»ãªæ§æèŠçŽ ãšãã®åœ¹å²ã¯ä»¥äžã®éãã§ãã
è©äŸ¡ãšãªãŒã±ã¹ãã¬ãŒã·ã§ã³ (referee.py, evaluator.py)
æ€èšŒãã€ãã©ã€ã³å šäœãçµ±æ¬ããè€æ°ã®æ€èšŒã«ãŒã«ãã¡ããªã¯ã¹ãé æ¬¡é©çšããŸããéäžã§ççŸãéåãæ€ç¥ãããå Žåãå³åº§ã«ãã©ã°ãç«ãŠãŠäŒæãããŸãã
äŸåãã§ãŒã³æ€åº (chain_detector.py)
ããã¥ã¡ã³ãéããããã¯éã®åç §é¢ä¿ãè«ççãªã€ãªããïŒãã§ãŒã³ïŒãèµ°æ»ããæçµ¶ã埪ç°çççŸããªãããæ€èšŒããŸããããã«ãããæèã®è±èœãäžæ£ãªåç §æ§é ãæ€åºããŸãã
ãã£ã³ãã³ã°ãšããããã³ã° (chunker.py, blocker.py)
巚倧ãªããã¹ããããã¥ã¡ã³ã矀ããæ€èšŒå¯èœãªæå°åäœïŒãã£ã³ã¯ïŒãžãšæ±ºå®è«çã«åå²ã»ãããã¯åããŸããåå²äœçœ®ã®æºãããæé€ããããããã€ãå¢çãæç¢ºãªã»ãã¬ãŒã¿ã«åºã¥ãäžæãªåå²ã¢ã«ãŽãªãºã ãæ¡çšããŠããŸãã
ããŒã¯ãã€ãŒãŒã·ã§ã³ãšã¡ããªã¯ã¹ (tokenizer.py, metrics.py)
å€éšã®éãããŒã¯ãã€ã¶ãŒã«äŸåãããæåçš®ãæ§é ã«åºã¥ãã軜éãªããŒã¯ã³åå²ããã³è©äŸ¡ã¡ããªã¯ã¹ïŒæåæ°ãå¯åºŠãæ§é è€é床ãªã©ïŒãç®åºããŸãã
3. å ç¢ãªäŸå€åŠçãš DLQ / Webhook 飿º
å®éã®éçšç°å¢ã«ãããŠãæ€èšŒã«å€±æããããŒã¿ãç°åžžå€ãæ€ç¥ããéã®ãã³ããªã³ã°ã¯æ¥µããŠéèŠã§ããsieve-referee ã§ã¯ããããã¬ã¿ãŒãã¥ãŒïŒDLQïŒããã³ Webhook 飿ºæ©æ§ãåããŠãããäžæ£ãªå ¥åãå®å šã«éé¢ã»éç¥ãããã€ãã©ã€ã³ãæ§ç¯ã§ããŸãïŒãã¹ãã³ãŒã test_e2e_dlq_webhook.py çã§ç¶²çŸ çã«æ€èšŒãããŠããŸãïŒã
DLQïŒãããã¬ã¿ãŒãã¥ãŒïŒ: æ§é æ€èšŒã«å€±æãããã€ããŒããæ¶å€±ãããããšãªããè§£æã»ãããã°çšã«éé¢ã»ä¿åããŸãã
Webhook 飿º: æ€èšŒçµæããšã©ãŒã€ãã³ããå€éšã·ã¹ãã ãžæ±ºå®è«çãªã¿ã€ãã³ã°ã§éç¥ããèªååãããã¬ããã³ã¹ã«ãŒãã圢æããŸãã
4. 決å®è«çã¢ãããŒãã®éçãšèª²é¡
峿 Œãªã«ãŒã«ããŒã¹ãšãã¬ãŒã¹ãã«ããŒæ€èšŒãéèŠããåé¢ãsieve-referee ã«ã¯ä»¥äžã®ãããªãã¬ãŒããªããååšããŸãã
衚çŸã®æºããã«å¯Ÿããè匱æ§
å³å¯ãªã¹ããŒãããã¿ãŒã³ãããã³ã°ã«äŸåãããããæèã¯æ£ããã衚çŸåœ¢åŒããããã«ç°ãªãå€ç°ã«å¯ŸããŠéå°ã«ãšã©ãŒå€å®ïŒåœéœæ§ïŒãäžãå¯èœæ§ããããŸãã
éçã«ãŒã«ã®äžçŸ©æ§
åçã«å€åããé«åºŠãªãã¥ã¢ã³ã¹ãæé»çãªæå³ããã¿åãããšã¯ããããããŸã§èšè¿°ãããæ§é çäºå®ã®ã¿ãè©äŸ¡ããŸãã
å éšã¢ãŒããã¯ãã£ â ææžãæ¯èŒå¯èœãªæ§é ãžå€æãã
Sieve-Refereeã®åŠçãã€ãã©ã€ã³ã¯ã倧ããæ¬¡ã®æ®µéã«åããããŸãã
Unicodeæ£èŠå
決å®è«çãªåè£ãã¢ã®æœåº
4ã€ã®ä»®èª¬ã«åºã¥ããã¢è©äŸ¡
é¡äŒŒé¢ä¿ã«ããã¯ã©ã¹ã¿æ§ç¯
å®å šæ¡ä»¶ãæºããå Žåã®ã¯ã©ã¹ã¿æ å ±ã®åæ
ã«ãã¯ã¢ããããŒãã«ã«ããæçµåé¡
çµæã®ã¬ããŒãåºå
ãã®æ§æã®ãã€ã³ãã¯ãæç« ãäžã€ã®ãã©ãã¯ããã¯ã¹ãžæž¡ããŠçµè«ãåŸãã®ã§ã¯ãªããåæ®µéã§äœãããŠããã®ããæç¢ºã«åããŠããããšã§ãã
Unicodeæ£èŠå
æåã«ãå ¥åããã¹ãã«Unicodeã®NFKCæ£èŠåãé©çšããŸãã
æåã«ã¯ãèŠãç®ãæå³ãè¿ããŠããUnicodeäžã§ã¯ç°ãªãã³ãŒããã€ã³ãã衚çŸãšããŠæ±ããããã®ããããŸããæ£èŠåã«ãã£ãŠäžéšã®è¡šèšäžã®å·®ç°ãçµ±äžããåŸç¶ã®æ¯èŒåŠçãå®å®ãããŸãã
ãã ããæ£èŠåã§ãã¹ãŠã®è¡šèšæºããè§£æ¶ãããããã§ã¯ãããŸãããæå³ãåãå¥ã®åèªããææ³çãªèšãæããŸã§åäžèŠãããã®ã§ã¯ãããŸããã
決å®è«çãªãã¬ã»ããããã³ã°
ææžæ°ãå¢ãããšããã¹ãŠã®ææžãã¢ãæ¯èŒããã³ã¹ãã倧ãããªããŸãã
ææžæ°ã (N) ãšãããšãããã¹ãŠã®ãã¢ãæ¯èŒããå Žåã®çµã¿åããæ°ã¯æ¬¡ã®ãšããã§ãã
[
P=\frac{N(N-1)}{2}
]
ææžæ°ã100ä»¶ãªã4,950ãã¢ã1,000ä»¶ãªã499,500ãã¢ã«ãªããŸãã
ããã§Sieve-Refereeã§ã¯ãæ¯èŒåã«3-gramã䜿ã£ã決å®è«çãªåè£æœåºãè¡ããŸãã
3-gramãšã¯ãããŒã¯ã³åããé£ç¶ãã3ã€ã®ããŒã¯ã³ãåãåºããéšååã§ããå ±éãã3-gramãæã€ææžã®çµã¿åãããåè£ãšããŠæœåºãããã®åŸã®è©³çŽ°ãªæ¯èŒãžæž¡ããŸãã
ãã®åŠçã«ã¯è»¢çœ®ã€ã³ããã¯ã¹ãå©çšããŸããå3-gramãã©ã®ææžã«åºçŸããããèšé²ããããšã§ããã¹ãŠã®ãã¢ãç¡æ¡ä»¶ã«è©³çްè©äŸ¡ããåŠçãæžãããŸãã
ã¢ãŒããã¯ãã£ä»æ§æžã«èšé²ããã52ææžã®ãã¹ãã»ããã§ã¯ãå š1,326ãã¢ã®ãã¡89ãã¢ãåè£ãšãªããçŽ93.3%ã®ãã¢åæžãå ±åãããŠããŸãã
ãã ããããã¯åœè©²ãã¹ãã»ããã§ã®çµæã§ãããããããããŒã¿ã»ããã«ãããæ§èœãä¿èšŒãããã®ã§ã¯ãããŸããã
ãŸããå ±é3-gramããªããã¢ã¯è©³çްè©äŸ¡ãçç¥ããŸãããã®ããã衚çŸã倧ããç°ãªãæå³çãªèšãæããèŠéãå¯èœæ§ããããŸããé«éåã®ããã®åè£æœåºã¯ãæ€åºå¯èœãªç¯å²ãšã®ãã¬ãŒããªãã䌎ããŸãã
4ã€ã®ä»®èª¬ â é¡äŒŒæ§ãäžã€ã®æ°åã«æŒã蟌ããªã
Sieve-Refereeã®äžæ žãšãªãã®ãã4ã€ã®ä»®èª¬ãçµã¿åãããå€å®ã¢ãã«ã§ãã
å仮説ã®çµæã0ãŸãã¯1ã§è¡šçŸãã4ãããã®ãã¹ã¯ãšããŠèšé²ããŸãã
仮説åç§°è©äŸ¡ãããã®H1èšè¿°å¯åºŠè©äŸ¡å¯Ÿè±¡ææžã®æåã®çš®é¡æ°ãšæåæ°ã®æ¯çH2æ§é é¡äŒŒæ§ããŒã¯ã³ã®æ§é çãªäžŠã³æ¹ã®äžèŽH3å 容é¡äŒŒæ§å 容èªãšãã®éšåæååã®äžèŽH4ã¯ã©ã¹ã¿æå±çŽæ¥çãªé¡äŒŒæ ¹æ ãæã€ãã¢ããäžå®èŠæš¡ä»¥äžã®ã¯ã©ã¹ã¿ã«å±ããã
ãã®ã¢ãã«ã§ã¯ãH1ããH4ãŸã§ãåãããšã枬ã£ãŠããããã§ã¯ãããŸãããããããç°ãªãç¹åŸŽã衚ãããšã§ãé¡äŒŒæ§ã®çš®é¡ãåé¢ããŸãã
1 H1ïŒèšè¿°å¯åºŠ
H1ã§ã¯ãæç« äžã«å«ãŸãããŠããŒã¯ãªæåã®çš®é¡æ°ããæååã®é·ãã§å²ã£ãå€ã䜿çšããŸãã
[
H_1^{score}(t)=
\frac{|\operatorname{unique}(t)|}{|t|}
]
ããã§ã(t)ã¯è©äŸ¡å¯Ÿè±¡ã®æç« ã§ãã
æåã®çš®é¡ãå°ãªããåãæåãç¹°ãè¿ãããæç« ã¯ãçžå¯Ÿçã«äœãå€ã«ãªããŸããäžæ¹ãæåã®çš®é¡ãå€ãæç« ã¯ãé«ãå€ã«ãªãããããªããŸãã
ããã©ã«ãã®éŸå€ã¯0.35ã§ãã
ãã ããããã¯æå³çãªæ å ±éãæç« ã®å質ãçŽæ¥æž¬å®ããææšã§ã¯ãããŸãããæååã®æ§è³ªãç°¡æçã«è©äŸ¡ããããã®ç¹åŸŽéã§ãã
ãªããçŸåšã®ãã¢è©äŸ¡å®è£ ã§ã¯ãH1ã¯äž»ã«è©äŸ¡å¯Ÿè±¡åŽã®ææžããç®åºãããŸããææžãã¢ã®åæ¹ã«å¯Ÿãã察称çãªå質è©äŸ¡ã§ã¯ãªãããããã®å€ãææžéã®é¡äŒŒåºŠãšè§£éããããšã¯ã§ããŸããã
2 H2ïŒæ§é é¡äŒŒæ§
H2ã§ã¯ãæç« ãããŒã¯ã³åããæåãã®ãã®ã ãã§ãªããæåçš®ãªã©ã«åºã¥ãæ§é çãªäžŠã³æ¹ãæ¯èŒããŸãã
å®è£ ã§ã¯ã挢åã»ã«ã¿ã«ãã»è±æ°åãªã©ã®äžéšãå ±éã®ã¿ã°ãžæœè±¡åããã²ãããªãèšå·ãªã©ã®ç¹åŸŽãæ®ããããŒã¯ã³åãäœããŸãã
ãã®åãã5-gramãš6-gramãçæããéåéã®Jaccardé¡äŒŒåºŠãèšç®ããŸãã
[
J(A,B)=\frac{|A\cap B|}{|A\cup B|}
]
ããã§ã(A)ãš(B)ã¯ãããããã®æç« ããçæãããn-graméåã§ãã
Jaccardé¡äŒŒåºŠã¯ã2ã€ã®éåã«å ±éããèŠçŽ ããå šäœã®èŠçŽ ã«å¯ŸããŠã©ã®çšåºŠååšãããã衚ããŸããå€ã¯0ãã1ã®ç¯å²ãåããŸãã
H2ã®ããã©ã«ãéŸå€ã¯0.80ã§ãã
ãã®ä»çµã¿ã§ã¯ãç°ãªãåèªã䜿ã£ãŠããŠããæç« ã®éªšæ Œã䌌ãŠããã°æ§é çãªäžèŽãšããŠæ€åºãããå¯èœæ§ããããŸãã
ãã ããåãææ³ãå®å衚çŸã䜿ãããšèªäœã¯äžæ£ã®èšŒæ ã§ã¯ãããŸãããæ§é ã®äžèŽã¯ããããŸã§äžã€ã®èŠ³æž¬çµæã§ãã
3 H3ïŒå 容é¡äŒŒæ§
H3ã§ã¯ãæç« ããæœåºããå 容èªãšããã®å éšã®2-gramãªã©ãç¹åŸŽéãšããŠæ¯èŒããŸãã
å®è£ ã§ã¯ãå 容èªãã®ãã®ã«å ããäžå®ä»¥äžã®é·ããæã€èªããéšåæååãçæããç¹åŸŽéåãäœããŸãããã®åŸãH2ãšåæ§ã«Jaccardé¡äŒŒåºŠãèšç®ããŸãã
H3ã®ããã©ã«ãéŸå€ã¯0.45ã§ãã
æ§é é¡äŒŒæ§ãšå 容é¡äŒŒæ§ãåããããšã§ãæç« ã®åœ¢åŒã䌌ãŠããã±ãŒã¹ãšãèªåœãå 容èªã䌌ãŠããã±ãŒã¹ãå¥ã ã«è©äŸ¡ã§ããŸãã
ããšãã°ãåã課é¡ãã³ãã¬ãŒãã䜿ã£ãŠããŠãæ¬æã®å 容ãç°ãªãå ŽåãH2ãé«ããH3ãäœããšããçµæã«ãªãå¯èœæ§ããããŸãã
éã«ãæç« ã®æ§é ã¯å€ãã£ãŠããŠããäž»èŠãªèªåœã匷ãäžèŽããŠããã°ãH3ãé«ããªãå¯èœæ§ããããŸãã
ãªããH3ã¯æå³ãçè§£ããã¢ãã«ã§ã¯ãããŸãããå ±éããå 容èªãéšåæååãç¹åŸŽéãšããŠæ±ããããæå³çã«åãæç« ãåžžã«æ€åºã§ããããã§ã¯ãªããäžè¬çãªå°éçšèªãäžèŽããã ãã§ãé¡äŒŒåºŠãäžããå ŽåããããŸãã
4 H4ïŒã¯ã©ã¹ã¿æå±
H4ã¯ãåã ã®ææžãã¢ã ãã§ã¯ãªããè€æ°ææžã®é¢ä¿æ§ãæ±ãããã®ä»®èª¬ã§ãã
ãã¢è©äŸ¡ã®çµæããREDãŸãã¯YELLOWã®é¢ä¿ãæœåºããææžéã®é£çµæåãæ§ç¯ããŸãããµã€ãº3以äžã®ã¯ã©ã¹ã¿ã«å±ããããã«ãã®ãã¢èªèº«ã«H2ãŸãã¯H3ã«ããçŽæ¥çãªé¡äŒŒæ ¹æ ãããå Žåã«éããH4ãç«ãŠãŸãã
ããã§éèŠãªã®ã¯ãã¯ã©ã¹ã¿ã«æå±ããŠãããšããäºå®ã ãã§ãçŽæ¥ã®æ¯èŒçµæãèªåçã«åŒ·ãäžèŽãžå€æŽããªãããšã§ãã
ãã®å¶çŽã¯ãSieve-Refereeã®å®å šèšèšã®äžæ žããªããŠããŸãã
4ããããã¹ã¯ãš16ç¶æ ã«ãã¯ã¢ããããŒãã«
4ã€ã®ä»®èª¬ãçµã¿åããããšãçè«äžã¯ (2^4=16) éãã®ç¶æ ãèããããŸãã
Sieve-Refereeã§ã¯ãããããã®ãã¹ã¯ãã«ãã¯ã¢ããããŒãã«ïŒLUTïŒã«å¯Ÿå¿ä»ããå€å®ã·ã°ãã«ããã¿ãŒã³åãçç±ã決å®ããŸãã
çŸåšã®å®è£ ã§ã¯ã12éãã®ç¶æ ãå®çŸ©ãããŠããŸããæ®ãã®4ç¶æ ã¯ãH4ã®å®å šæ¡ä»¶ã«ãã£ãŠæ§é çã«å°éããªãç¶æ ã§ãã
代衚çãªãã¿ãŒã³ãèŠãŠã¿ãŸãã
ãã¹ã¯ã·ã°ãã«ãã¿ãŒã³æå³0000GREENFULLY_UNIQUEæ§é ã»å 容ãšãã«é¡äŒŒæ ¹æ ããªã1000GREENSTANDARD_STRUCTURE_UNIQUEèšè¿°å¯åºŠã®åºæºãæºãããæ§é ã»å 容ã®é¡äŒŒæ ¹æ ããªã1100YELLOWCOINCIDENTAL_FORMAT_MATCHæ§é ã¯é¡äŒŒããããå 容ã®é¡äŒŒæ ¹æ ã¯åŒ±ã1010REDPARAPHRASE_DETECTEDæ§é ã¯ç°ãªãäžæ¹ãå 容ã®é¡äŒŒæ§ãéŸå€ãè¶ ãã1111REDCLUSTER_EXACT_COPYæ§é ã»å 容ã®é¡äŒŒæ ¹æ ãããã察象ã¯ã©ã¹ã¿ã®æ¡ä»¶ãæºãã
ãããã®åé¡ã¯ãã«ãŒã«ããã¯ã«æç€ºçã«å®çŸ©ãããŠããŸããå ¥åãšèšå®ãåãã§ããã°ãåããã¹ã¯ããåãåé¡ãå°ãããŸãã
ãã ãããã¿ãŒã³åã¯å®è£ äžã®åé¡ã©ãã«ã§ããããšãã°ãPARAPHRASE_DETECTEDãšããã©ãã«ãåºãããšã ãã§ãæå³çãªççšããã£ããšèšŒæãããããã§ã¯ãããŸããã
åæ§ã«ãFULLY_UNIQUEããããããæå³çãªéè€ãååšããªãããšã蚌æãããã®ã§ã¯ãããŸãããå®çŸ©ãããè©äŸ¡é¢æ°ãšéŸå€ã«ãããŠãé¡äŒŒæ§ã®æ ¹æ ãæ€åºãããªãã£ããšããæå³ã§è§£éããå¿ èŠããããŸãã
ãªã鿥é£éãå¥ã«æ±ãã®ã
Sieve-Refereeã®éèŠãªç¹åŸŽã®äžã€ãã鿥é£éããããããäŒèšã²ãŒã åãã®é¢ä¿ã奿 ã§æ€åºããä»çµã¿ã§ãã
次ã®3ã€ã®ææžããããšããŸãã
AïŒãå°åã®äº€ééã¯ãæé垯ã«ãã£ãŠå€åããã
BïŒã亀ééã¯æé垯ã«ãã£ãŠå€åãããããå°åããšã®å·®ã調ã¹ãã
CïŒãå°åã®äººå£ååžã¯ãé·æçãªéœåžèšç»ã«åœ±é¿ããã
AãšBã¯äŒŒãŠãããããããŸãããBãšCããäžéšã®èªåœãæ§é ãå ±éããå¯èœæ§ããããŸãã
ããããAãšCã®éã«çŽæ¥çãªé¡äŒŒæ ¹æ ããããšã¯éããŸããã
é¢ä¿ãåçŽãªé£çµã°ã©ããšããŠæ±ããšãAãšBãBãšCãçµã°ããŠãããšããçç±ã ãã§ãAãšCãŸã§åãã°ã«ãŒããšããŠåŒ·ãäžèŽå€å®ãåãããããªåé¡ãçãåŸãŸãã
ããã¯ãé¢ä¿ã®æšç§»æ§ãç¡æ¡ä»¶ã«ä»®å®ããŠããŸãããšã«ããåé¡ã§ãã
Sieve-Refereeã§ã¯ããã®åé¡ãé¿ããããã«ãæ¬¡ã®æ¡ä»¶ãèšããŠããŸãã
[
H_4=1
\iff
\operatorname{clusterSize}\geq3
\land
(H_2=1\lor H_3=1)
]
ã€ãŸããã¯ã©ã¹ã¿ãäžå®èŠæš¡ä»¥äžã§ãã£ãŠãã察象ãã¢èªèº«ã«çŽæ¥çãªæ§é é¡äŒŒæ§ãŸãã¯å 容é¡äŒŒæ§ã®æ ¹æ ããªããã°ãH4ãç«ãŠãŸããã
çŽæ¥ã®é¡äŒŒæ ¹æ ããªããã¢ã¯ã鿥é£éã®èŠåãšããŠå¥éå ±åããŸãã
ãã®ä»çµã¿ã¯ããã¹ãŠã®éæ¥é¢ä¿ãæé€ãããã®ã§ã¯ãããŸãããã¯ã©ã¹ã¿å šäœã®é¢ä¿ã芳枬ããªãããçŽæ¥æ¯èŒã®çµæãšãã°ã«ãŒãå ã§éæ¥çã«ã€ãªãã£ãŠãããšããäºå®ãåé¢ãããã®ã§ãã
ãåãã°ã«ãŒãã«å±ããããšããšãäºãã«çŽæ¥äŒŒãŠããããšãã¯éãã
ãã®åºå¥ãä¿ã€ããšããæ§é çã¬ããã³ã¹ã«ãããŠéèŠã ãšèããŠããŸãã
å®çšæ§ãæ¯ããã¬ããŒããšèªåå
Sieve-Refereeã¯ãPythonã©ã€ãã©ãªãšããŠã ãã§ãªããCLIããå©çšã§ããããŒã«ãšããŠãæ§æããŠããŸãã
å€å®çµæã«ã¯ãè©äŸ¡å¯Ÿè±¡ã®ææžIDãæ¯èŒçžæã®IDã仮説ãã¹ã¯ããã¿ãŒã³åãå€å®çç±ãªã©ãå«ãŸããŸãã
ãŸããHTMLããã·ã¥ããŒããææžéã®å·®åã¬ããŒããçæããããšã§ãã©ã®ãã¢ãã©ã®ãããªçç±ã§æ€åºãããã®ãã確èªã§ããŸãã
ããšãã°ã次ã®ãããªã³ãã³ãã§ãæå®ãããã£ã¬ã¯ããªå ã®ããã¹ããè©äŸ¡ã§ããŸãã
sieve-referee ./target_documentsåäžã®HTMLã¬ããŒããçæãããå Žåã¯ã次ã®ããã«å®è¡ããŸãã
sieve-referee ./target_documents \
--standalone \
--output-dir ./reportsCI/CDãªã©ã®èªååãã€ãã©ã€ã³ã«çµã¿èŸŒãå Žåã¯ãJSONåºåãšãªã¹ã¯å€å®ã«åºã¥ãçµäºã³ãŒããå©çšã§ããŸãã
sieve-referee ./target_documents \
--json \
--fail-on-riskãã®ããã«ãã³ã¢ã®è©äŸ¡åŠçãšãã¬ããŒãåºåãèªååã®ä»çµã¿ãåé¢ããããšã§ãããŒã«ã«ã§ã®ç¢ºèªããéçºã¯ãŒã¯ãããŒãžã®çµã¿èŸŒã¿ãŸã§ãçšéã«å¿ããå©çšãå¯èœã«ããŠããŸãã
ãªãããããã®æ©èœã¯ããã¹ãã®ã¹ã¯ãªãŒãã³ã°ãæ¯æŽããããã®ãã®ã§ãããæçµçãªå€æã人éããåãäžãããã®ã§ã¯ãããŸããã
決å®è«çãªæ€èšŒã®éç
ã«ãŒã«ããŒã¹ã®æ€èšŒã«ã¯æç¢ºãªå©ç¹ããããŸãããäžèœã§ã¯ãããŸããã
æå³çãªèšãæããå®å šã«ã¯æ€åºã§ããªã
Sieve-Refereeã¯ãæ§é çç¹åŸŽãèªåœãéšåæååãªã©ã䜿ã£ãŠæ¯èŒããŸããæå³ãçè§£ããèšèªã¢ãã«ãšã¯ç°ãªãããã衚çŸã倧ããå€ãã£ãæç« ããèªåœã®éè€ãå°ãªãæç« ãèŠéãå¯èœæ§ããããŸãã
å ±éãã³ãã¬ãŒãã¯äžæ£ã®èšŒæ ã§ã¯ãªã
åã課é¡ãå¥çŽæžãç³è«æžãå ±åæžãªã©ã§ã¯ãå ±éããæç« æ§é ãèªç¶ã«çãŸããŸããæ§é é¡äŒŒæ§ãé«ãããšã ãã§ãäžæ£ãªã³ããŒãæå³çãªæµçšãæå®ããããšã¯ã§ããŸããã
åè£æœåºã«ã¯èŠéãã®å¯èœæ§ããã
3-gramã«ãããã¬ã»ããããã³ã°ã¯ã詳现æ¯èŒãè¡ããã¢ãæžããããã®ä»çµã¿ã§ããå ±éãã3-gramããªãææžãã¢ã¯ã詳现è©äŸ¡ã®å¯Ÿè±¡ããå€ããå ŽåããããŸãããã®ããããã¹ãŠã®æå³çãªé¡äŒŒé¢ä¿ãçºèŠã§ããããšãä¿èšŒãããã®ã§ã¯ãããŸããã
GREENã¯ç¡æ¡ä»¶ã®å®å šèšŒæã§ã¯ãªã
åè£æœåºã«ãã£ãŠè©³çްè©äŸ¡ãçç¥ãããã¢ããçŸè¡å®è£ ã§ã¯GREENã®çµæãšããŠæ±ãããå ŽåããããŸãããã®å Žåãé¡äŒŒæ§ãååã«æ€èšŒãããããšãæå³ããŸããã
éçšäžã¯ãGREENããççšãååšããªãããšã®èšŒæããšè§£éããã®ã§ã¯ãªãããçŸåšã®æ€åºæ¡ä»¶ã§REDãŸãã¯YELLOWã®æ ¹æ ãåŸãããŠããªãç¶æ ããšçè§£ããå¿ èŠããããŸãã
éŸå€ã¯çšéã«ãã£ãŠæ€èšŒããå¿ èŠããã
H2ãH3ã®éŸå€ã倿Žãããšãæ€åºããããã¢ãã¯ã©ã¹ã¿ã®æ§é ãå€ãããŸããæè²ãæ³åãã³ã³ãã³ã管çãªã©ã誀æ€åºãšèŠéãã®ã³ã¹ããç°ãªãé åã§ã¯ã代衚çãªããŒã¿ã»ãããçšããè©äŸ¡ãšèª¿æŽãå¿ èŠã§ãã
ãããã®éçãé ãã®ã§ã¯ãªããæç€ºããŠããããšãéèŠã§ããã©ããŸã§ãæ©æ¢°çã«ç¢ºèªã§ããã©ãããå ã人éã«ãã倿ãå¿ èŠãšããã®ãããã®å¢çã瀺ãããšãã説æå¯èœæ§ã®äžéšã ããã§ãã
ä»åŸã®å±æ â å€å®çµæãæ€èšŒå¯èœãªèšŒæ ãž
ä»åŸã®Sieve-Refereeã§ã¯ãåã«å€å®ã®çš®é¡ãå¢ããã ãã§ãªããå€å®æ ¹æ ã®æ€èšŒå¯èœæ§ãããã«é«ããŠãããããšèããŠããŸãã
å ·äœçã«ã¯ã次ã®ãããªèª²é¡ããããŸãã
æ€åºæ§èœã®å®éè©äŸ¡ïŒæ£è§£ã©ãã«ä»ãã®ããŒã¿ã»ãããçšãã誀æ€åºçãèŠéãçãåŠçæéãæž¬å®ããã
æ¥æ¬èªããã¹ããžã®é©å¿ïŒåœ¢æ ãæäœã®ç¹åŸŽãèžãŸããæåçš®ã«äŸåããè©äŸ¡é¢æ°ã®åŠ¥åœæ§ãæ€èšŒããã
åè£æœåºã®æ¹åïŒé«éåãšæ€åºæŒãã®ãã¬ãŒããªããå®éçã«è©äŸ¡ããã
å€å®æ ¹æ ã®å¯èŠåïŒé¡äŒŒããããŒã¯ã³ãéšåæååãã¯ã©ã¹ã¿å ã®é¢ä¿ããå©çšè ã確èªããããåœ¢ã§æç€ºããã
ã¬ããã³ã¹çšéãžã®çµ±åïŒCI/CDãææžç®¡çã®ã¯ãŒã¯ãããŒãžçµã¿èŸŒã¿ãæ€åºçµæãã¬ãã¥ãŒãç£æ»ã«ã€ãªããã
ãããã¯ãçŸæç¹ã§ãã¹ãŠã宿ããŠãããšããæå³ã§ã¯ãããŸãããã³ã¢ã®æ€èšŒãšã³ãžã³ãåºç€ãšããŠãä»åŸè©äŸ¡ã»æ¹åããŠããæ¹åæ§ã§ãã
ãããã« â 芳枬è ã§ãã£ãŠã仲è£è ã§ã¯ãªã
AIãçæããæç« ãAIãè©äŸ¡ããããšèªäœããå¿ ãããééã£ãŠããããã§ã¯ãããŸããã
ããããè©äŸ¡çµæãéçšäžã®å€æã«äœ¿ãã®ã§ããã°ããã®çµæãã©ã®ãããªæ ¹æ ããå°ãããã®ãã確èªã§ããããšãéèŠã§ãã
Sieve-Refereeã¯ãæç« ã®æ§é ãå å®¹ãææžéã®é¢ä¿ãåè§£ããŠèŠ³æž¬ããããããæç€ºçãªã«ãŒã«ã«åºã¥ããŠåé¡ããããšã§ãå€å®ã®è¿œè·¡å¯èœæ§ã確ä¿ããããšãããããžã§ã¯ãã§ãã
4ã€ã®ä»®èª¬ã12ç¶æ ã®ã«ãã¯ã¢ããããŒãã«ãã¯ã©ã¹ã¿è§£æã鿥é£éã®æ€åºããããã¯åãªãæ©èœã®éåã§ã¯ãããŸããã
ããããã®åŠçãäœãæ ¹æ ã«ããŠããã®ããæç¢ºã«ããçŽæ¥çãªé¡äŒŒæ§ãšéæ¥çãªé¢ä¿æ§ãæ··åããªãããã®èšèšã§ãã
ã¹ã³ã¢ã¯çµè«ãã®ãã®ã§ã¯ãããŸãããèŠåã¯æçœªå€å®ã§ã¯ãããŸããããããŠãæ€åºãããªãã£ãããšã¯ãåé¡ãååšããªãããšã®èšŒæã§ã¯ãããŸããã
Sieve-Refereeãç®æãã®ã¯ãæ å ±ã«ä»£ãã£ãŠçµè«ãäžãããšã§ã¯ãªãã人éãçµè«ãæ€èšŒããããã®æ§é ãæäŸããããšã§ãã
Observer, not arbiter.
ãã®èšèšææ³ã軞ã«ãåçŸæ§ã説æå¯èœæ§ãå®çšæ§ã®ãã©ã³ã¹ãæ€èšŒããªãããSieveãšã³ã·ã¹ãã ãçºå±ãããŠãããŸãã
â»Sieve-Refereeã¯æ±ºå®è«çãªããã¹ãã¹ã¯ãªãŒãã³ã°ãšã³ãžã³ã§ããå€å®çµæã¯å®çŸ©ãããç¹åŸŽéãšã«ãŒã«ã«åºã¥ããã®ã§ãããççšã®æå³ãèäœæš©äŸµå®³ãäºå®ã®æ£ç¢ºæ§ãªã©ãåç¬ã§ç¢ºå®ãããã®ã§ã¯ãããŸããã
é¢é£ãªã³ã¯
Sieve ãšã³ã·ã¹ãã
Sieveã·ãªãŒãºå šäœã®èšèšææ³ãåãšã³ãžã³ã®åœ¹å²ã芳枬空éã®é¢ä¿ã«ã€ããŠã¯ãã¢ãŒããã¯ãã£ã€ã³ããã¯ã¹ãåç §ããŠãã ããã
Sieve ãšã³ã·ã¹ãã ã¢ãŒããã¯ãã£ã€ã³ããã¯ã¹ïŒnoteïŒ
PyPI â https://pypi.org/user/neguseatama/
GitHub ãªããžããª
Sieve-Core â æ±ºå®è«çãªæ§é éçŽã»ç¢ºä¿¡åºŠè©äŸ¡ã®åºç€ãšã³ãžã³
Sieve-Referee â ããã¹ãã®æ§é ã»æµçšã»èšãæãã芳枬ãããšã³ãžã³
Sieve-Scope â ãœãŒã¹ã³ãŒãã®æ§é çé¡äŒŒæ§ã芳枬ãããšã³ãžã³
Sieve-Lens â äžå¯èŠã³ã³ãã³ããé èœèŠçŽ ã芳枬ãããšã³ãžã³
Sieve-Redact â æå®é åã®æ±ºå®è«çãã¹ãã³ã°ãšå®å šæ§æ€èšŒ
Sieve-Order â ããã³ããã®è¡šå±€æ§é ã6-bit芳枬空éã§èŠ³æž¬ãããšã³ãžã³
Sieve-Tray â åãšã³ãžã³ãçµ±åããããŒã«ã«ããã³ããšã³ã
note æè¡èšäº
Sieve-ScopeïŒæ±ºå®è«çã¹ã³ãŒã管çãšã³ã³ããã¹ãå¢çã®å®çŸ©
Sieve-RedactïŒæ±ºå®è«çãã¹ãã³ã°ãšä»ãã€ãå®å šæ§æ€èšŒã«ããããŒã¿ä¿å š
Sieve-TrayïŒè€æ°ã®èŠ³æž¬ç©ºéãçµ±åããããŒã«ã«æ€èšŒããã³ããšã³ã
èšèšã®èšé²(ææèšäº)
Sieve ã·ãªãŒãºã®æ°çã¢ãã« â å°åºæ³ãšå®å®æ§
ã·ãªãŒãºã®ååŒããå£ãæ¹ã®å®æž¬ãããã©ãå°ãããããã
æèšç®ã§è¿œããç²åºŠã§è§£èª¬ãå°åºã® 3 段æé ãšèšèšãã§ãã¯ãªã¹ãä»ããSieve ã·ãªãŒãºã®æ§é èšèš â äŸåæ¹åãã¢ãŒããã¯ãã£ã決ãã
åäžãã¡ã€ã«ã»ãã©ã°ã€ã³ã»ã«ã¿ãã°çé¢ã»byte pinã5 ã€ã®æ§é 倿ã®
çç±ãšåäŸããããŠãã»ãŠã¹ã®è¹ããå§ãŸã£ãéçºçµç·¯ã