èæ¯
å¹³æã®æ¬¡ã®å å·ã¯äœã«ãªãã®ïŒ
ãšã³ãžãã¢ã«éãããå€ãã®äººãæ°ã«ãªã話é¡ã§ããã
ãšã©ã€åŠè
å
çãããã°ã£ãŠèª¿ã¹ããéžãã ãããŠæ±ºãããã§ããã
ããããæã¯AIãå²ç¢ãå°æ£ã§å人ãè² ããæä»£ã
ããã ïŒAIã«æ°å
å·ã決ããŠãããã°ãããããïŒïŒ
ãŸãã«æªéççºæ³ã»ã»ã»ïŒ
ãOKãGoogleãæ°ããå å·ã¯äœã«ãªãã®ïŒã
âãæããŠãããã¯ã±ããªããïŒå å·é¢é£èšäºãããã¯æããŠãããïŒ
ãããèªåã§äœã£ãŠã¿ããïŒ
**人ã®å€æãäžåå ¥ããã«ãAIã ãã§æ°å å·äœãããšãåºæ¥ãã®ãïŒ**ãä»åã®ããŒã
äœè«ïŒ ã¡ãªã¿ã«å€§åå©äººå·¥ç¥èœã®ã倧åå©Î²ããã«èãããšãã
ãã¢ã¡ãªã«ããšã®çããè¿ã£ãŠããã
ãã®ã¿ã€ãã³ã°ã§ãã¢ã¡ãªã«åè¡åœ51çªç®ã®å·ã¯ãæ¬å·ãã§ãããšãããšã®å瀺ãªã®ãïŒïŒ
æ°å å·ã®ã«ãŒã«ã¯ïŒ
以äžã®ããã«æ±ºããããŠãããããã
ã1. åœæ°ã®çæ³ãšããŠãµãããããããªããæå³ãæã€ãã®ã§ããããšã
ã2. 挢å2åã§ããããšã
ã3. æžããããããšã
ã4. èªã¿ãããããšã
ã5. ãããŸã§ã«å
å·åã¯ãããåãšããŠçšãããããã®ã§ãªãããšã
ã6. ä¿çšãããŠãããã®ã§ãªãããšã
ãã®ã«ãŒã«ããããAIãã ãã§æ±ºããããšãåºæ¥ãã®ã ãããïŒ
ãŸãã远å ã«ãŒã«ãšããŠãM(ææ²»)ãT(倧æ£)ãS(æå)ãH(å¹³æ)ãšã
é æåã®ã¢ã«ãã¡ããããéè€ããªãããã«ããããšãããã£ã«ã¿ãååšããã
ïŒâ»ãªããæ¬æçš¿ã§ã¯ãããããåãšããŠçšãããããã®éšåã¯ã
ãããªã¹ããäœãã®ãé¢åïŒåŸã§é€å€ããã°ããã ããªã®ã§ãç¡èŠããŠè©±ãé²ããïŒ
AIã«ãšã£ãŠã®æå€§ã®é£é¢ã¯ãè¯ãæå³ã®å€æã
- æ°å
å·ã¯ãèŠç¯ããçæããäœåœãã§ãã
- èŠç¯ïŒå¹ŽãèŠåŽãã«ãã¿åºãå³ããã£ãŠãããâª
- çæ10幎ããããããªã£ãŠããŸãâª
- äœåœïŒå¹Žãã ãã ã䌞ã³ãŠããâª
- 挢åïŒåã ãå°åŠæ ¡ã§ç¿ãåã ãããŒ
- ããããããªãã°ããïŒãã®ä¿çšãããŠããªããšããç¹ã§ãAIã§ããã¡ã ãšåããããïŒ
- æ°å
å·ã¯ãç³äžããå
å®ããçæ®ºãã§ãã
- ç³ã®äžã«ãïŒå¹Ž
- å å®âãã«ãã¥ãŠïŒ
- éªççæ®ºé»éŸæ³¢ïŒ
- ããã¯ã®ã£ã°ã«ãããªããªããšããããšãã©ããã£ãŠæ©æ¢°ã倿ããïŒ
- äžäºç ã®æåã¿ãããªååãåºãªãããã«ããªããšæ¥æ¬åœã®æ¥ã«ãªããã
ã€ãŸããã«ãŒã«ãïŒãã®ã
ãåœæ°ã®çæ³ãšããŠãµããããè¯ãæå³ãæã€ã
ãšããããšããã©ããã£ãŠæ©æ¢°ã倿ããã®ãïŒ
ãå§åçæå€§ã®é£é¢ã«ãªãããã§ããã
æ¬æçš¿ã®å 容
- ã¬ãã§ãAIã ãã«æŒ¢åã®ã»ã¬ã¯ããä»»ããæ°å å·ã«çžå¿ããèšèãèšç®ããŸãã
- 䜿ããINPUTæ
å ±ã¯ã以äžïŒç¹ãšããŸãã
- Wikipediaã®ããã¹ãããŒã¿
- ïŒâ»å å·é¢é£ã®èšäºãæ€çŽ¢ãããããªçšéã§ã¯ãªãããã ã®å€§éããã¹ããšããŠæ±ãïŒ
- æ¢ã«äœ¿ãããå å·ã®äžèЧïŒïŒïŒïŒåïŒïŒïŒããããææ¬ãšããŠåŠç¿ïŒ
- æè²çšæŒ¢åäžèЧïŒå°åŠæ ¡ã§èŠããæŒ¢åäžèЧïŒ1006åïŒïŒïŒåè£äžèЧïŒ
- èªã¿ãããïŒå°åŠæ ¡ã¬ãã«ã®æŒ¢åããšä»®å®ã
- ïŒâ»åžžçšæŒ¢åã®å Žåã¯1945åãããŒã¿ãå ¥ãæ¿ããã°ãã¡ãã§ãå¯ïŒ
- Wikipediaã®ããã¹ãããŒã¿
-
補äœè
ã調æŽåºæ¥ãããšã¯ãæ°å€ã®èšå®ã ããã§ã䞻芳ã§å€æã¯ãããªãããšããŸãã
- äŸãã°ããèŠãã®åã¯æå³ãæªããã䜿ãããªãããããšãã£ãŠé€å€ããã®ã¯çŠæ¢
- äœãã®åºæºã§ãäžäœNåã«çµããšããäœãã®åŸç¹å€ã10以äžããªã©ãšæ°å€ãåãã®ã¯OK
- ã€ãŸããåè£ã®çŽ1000åã«å¯ŸããŠã**ãå šãŠã®æåãå¹³çã«æ±ãã**ããšã«ãªãã
- æ°å€ãããã¹ãã®å€æŽã ãã«äŸåãã誰ããã£ãŠãåãçµæãåºãããšããšèšãæããŠãè¯ãã
å€èŠã§äººã倿ããŠã¯ãããªãããšããã®ã«äŒŒãŠãããçèšãã¹ãã®ç¹ã ãã§æŒ¢åã倿
- ãããŸããªå
容
- ãææ¬ããŒã¿ïŒå å·ã®äžèЧããšäŒŒããããªçµã¿åãããã
- 倧éã®æ¥æ¬èªã³ãŒãã¹ïŒwikipediaïŒã®åŠç¿çµæãããšã«ã
- 1006åÃ1006åã®çµã¿åããã®äžããèŠã€ãåºãããšããããš
- ã³ãŒãã®å®è¡ç°å¢ã¯å šãŠãWindows10 + Python3 +JupyterNotebook ãåæã
æªéççºæ³ã®èšç»
ããã ïŒChar2Vecã䜿ããïŒ
Char2Vecãšã¯ãWord2Vecã®ãïŒæåçãã§ããã
ïŒæçš¿æã«æ°ã¥ããããQiitaã«char2vecã®ã¿ã°ãç¡ãã£ãïŒ
Word2Vecã䜿ã£ãç§ã®å¥ã®èšäºã¯ä»¥äžïŒ
ããåç
§ïŒãèµ€ã®ä»äººãã®å¯ŸçŸ©èªã¯ãçœãæäººã ãããèªåçæãããç©èª
Word2Vecã®å Žåãåèªã®ãã¯ãã«åãè¡ãããšã§ã
ãçæ§ããŒãç·ãïŒã女ãïŒã女çã
ãªã©ã®æŒç®ãå¯èœã«ãããã
å
šãåãããšããä»åºŠã¯åèªåºåãã§ã¯ãªããïŒæååºåãã§è¡ãããšã§
æåïŒæŒ¢åïŒã«å¯ŸããŠããã¯ãã«ãå²ãåœãŠããšãããã®ã
çºæ³ãšããŠãè¯ãæå³ã®æŒ¢åïŒæ¢ã«å
å·ã§äœ¿ãããããšã®ããæŒ¢åããšããŠã
ããã«äŒŒããã¯ãã«ãæã€æŒ¢åããå
å·ã®åè£ãšãªã挢åãªã®ã§ã¯ãªããïŒ
ãŸãããã®æ¹éã§è¯ãæå³ãæã€ïŒå
å·ã«äœ¿ãããŠããããããªæŒ¢åãã®ãªã¹ããäœãã
次ã«ããã®çµã¿åããã®äžã§ãïŒåã®çµã¿åãããè¯ããã®ãæ¢ããšããæ¹éã
Char2Vecã®å®è¡äŸãšãã®èª²é¡
Char2Vecã¯ãã¢ãã«ã®äœææã®ãã©ã¡ãŒã¿ã«ããã®ãããããªããã
Word2Vecãããã人ãèŠãæã®çŽåŸæãç¡ãããšæãããã
以äžã¯ããæ°Žãã«è¿ãæåã衚瀺ããããšãã®å®è¡äŸã
from gensim.models.word2vec import Word2Vec
import gensim
JPmodel_path = 'ã¢ãã«ã®ãã¹ãèšèŒ'
JPmodel = Word2Vec.load(JPmodel_path)
JPmodel.most_similar(positive = u'æ°Ž', topn=10)
[('ç ', 0.689274251461029),
('å °', 0.6883330345153809),
('æ¹', 0.6873093843460083),
('å €', 0.6631444692611694),
('å¡©', 0.6606267690658569),
('æ°·', 0.6517341136932373),
('湿', 0.6269866228103638),
('浞', 0.6251579523086548),
('æµ', 0.6123377680778503),
('æ¹§', 0.6074004173278809)]
äžäœã¯ãªããšãç ãããµã©ãµã©ãšããæå³ã§è¿ã仲éãšæã£ãã®ã ãããïŒ
ã²ãŒã ã®å±æ§ã§ã¯å€©æµã«ãªããããªã»ã©çžæ§ãæªããã ãã
ä»ã«ãäŸãã°ãå³ããå
¥ãããšã䌌ãŠããäžäœã¯ãå·Šããªã®ã¯è¯ããã
ïŒïœïŒäœã¯ãè©ãè
ãè
°ãã§ãã£ãã
Word2Vecã®å Žåã¯ãäŸãã°ãã³ã¢ã©ãã¯ããã³ãããªã©ãšå
¥ãæ¿ããŠã
æç« ãšããŠæç«ããããã䌌ãèšèãéããããšãåºæ¥ããã
ãæŒ¢åäžåãã®å ŽåãWordãšéãã眮ãæããŠãæç«ãããããªåã¯å€ããªãããã
éåºŠã®æåŸ
ã¯åºæ¥ãªãã
æåã¯ã挢åã®æŒç®ïŒè¶³ãç®ãåŒãç®ïŒãé§äœ¿ããŠã
æ°å
å·ãäœããããšèãããããããŸã§æåŸ
ããã«ã
æå³çã«äŒŒãŠãããã©ããïŒãæœåºããç®çã«éã£ãŠäœ¿ãã
äœè«ïŒ ãè±èªåã§ã¯ãAïœZããã¯ãã«åããŠãå šãé¢çœããªãããšãããã
char2vecã¯æ³šç®ãããŠããªãæ°ããããã
æ¥æ¬èªãäžåœèªãªã©ã挢åãæ±ãæåã«ãããŠã¯ã
ãã£ãšé¢çœãããšãåºæ¥ããããããªãã
æ¬æçš¿ããæ¥æ¬èªãšäžåœèªã®æŒ¢åã®æå³ã®éããchar2vecã§æ¯èŒããŠããæã«ã
幎å·ã®è©±é¡ãèããããã«æãã€ããå 容ã
æ¥æ¬èªäžåœèªã®æ¯èŒã«ã€ããŠãããŸãšãŸã£ããã©ããã§çºè¡šãããããããªãã
Char2Vecã®ã¢ãã«ã®äœæ
ååã®ããèµ€ã®ä»äººãã®å¯ŸçŸ©èªã¯ãçœãæäººã ãããèªåçæãããç©èªãã§ã¯ã
åŠç¿æžã¿ã®Word2Vecã¢ãã«ã䜿ã£ãããä»åã¯ã
Char2Vecã®åŠç¿æžã¿ã®ã¢ãã«ã¯è¯ããããªãã®ãç¡ãããã«ãèªåã§äœãããšã«ããã
ãããæ¬æçš¿ã®æã圹ç«ã€éšåã¯ãã®ç« ãããããªãã
ããããæ¬æçš¿ã«ãã£ãŠãAIãæ°å
å·ãäœã£ããšããŠãäœãäžã®äžã«åœ¹ç«ã€ã®ãïŒ
ãšãã誰ããæåã«æãçåã¯æ¢ã«åé€ãããŠãã
Char2Vecã®ã¢ãã«ãwikipediaããçæãã
â Wikipediaãã³ãããŒã¿ã®å ¥æ
Wikipediaã®ãã³ã眮ãå Žããã
ãjawiki-20180501-pages-articles-multistream.xml.bz2ããããŠã³ããŒããã
â¡ããŒã¹ããŠæŽç
WikiExtractorãšãããWikipediaã®ããŒãµãŒãããŠã³ããŒãããã
以äžã®ã³ãã³ãã§ãå å·¥ããã
ã¿ã€ãã«ïŒèšäºã®ã¿ã°åœ¢åŒã«ããŠããããïŒåéãããã®ã§æéã¯ããããïŒ
python WikiExtractor.py -o extracted jawiki-20180501-pages-articles-multistream.xml.bz2
â¢ïŒè¡ïŒïŒèšäºã«ãªãããã«å å·¥
Hironsanæ§ã®ãja.text8ã®ãªããžããªããã
ãprocess.pyãã䜿ã£ãŠãã¿ã°ãæ¹è¡ã®é€å»ããïŒè¡ïŒèšäºåã宿œããã
â¡ã®æé ã§åºæ¥ããextractedãã©ã«ããã®äžã§å®è¡ããã ãã
Hironssanæ§ããããšãããããŸãã
ãªããäžèšã®process.pyã¯python2åãã®ã³ãŒãã§ãããã
ãã¡ã€ã«æäœã®å Žæãªã©ãã¡ãã£ãšå€ããã ãã§åäœãããïŒäŸãã°ä»¥äžïŒ
with open(filename) as f:
import codecs
with codecs.open(filename,"r","utf-8") as f:
â£ïŒæååäœã«åºåã
Word2Vecã®å Žåã¯ãããã§mecabãåºãŠããŠã
åèªåäœã«ãªãããã«åœ¢æ
çŽ è§£æãè¡ãã®ã ãã
Char2Vecã®å Žåãå
šãŠïŒæåã§åºåãã ãã
%%time
# âã¯ãjupyternotebookäžã§çµéæéãèŠããã
# 倧ãã¡ã€ã«ãæ±ãã®ã§ãã¡ã¢ãªæº¢ã察çã«ãïŒè¡ãã€èªãã§çµæãã¡ã€ã«ã«è¿œèšããŠããã
with codecs.open('ItiGyouGotoKiji.txt',"r", "utf-8") as f:
for line in f.readlines():
chars = [c for c in line if c != u' ']
with codecs.open('allWiki_1kugiri.txt',"a", "utf-8") as newf:
newf.write(u' '.join(chars))
â€æ©æ¢°åŠç¿ã§Char2Vecã¢ãã«ãäœã
ãã©ã¡ãŒã¿ã¯äœåã詊è¡é¯èª€ããã»ããè¯ãã
以äžã«é¢é£ãã©ã¡ãŒã¿ã®æå³ãã³ã¡ã³ãã«èšèŒã
äžåäœãã®ã«æ°æéãããã
%%time
import logging
from gensim.models import word2vec
logging.basicConfig(format='%(asctime)s : %(levelname)s : %(message)s', level=logging.INFO)
sentences = word2vec.Text8Corpus('allWiki_1kugiri.txt')
model = word2vec.Word2Vec(sentences, size=60, window=60, min_count=50, hs=0, negative=15, iter=6)
# size:空éãµã€ãº=ãã¯ãã«ã®æ¬¡å
æ°(ããã©ã«ãã¯ïŒïŒïŒã ããä»åã¯ïŒåãªã®ã§æžããïŒ
# hs:åŠç¿ã«éå±€åãœããããã¯ã¹ã䜿çšãããã©ãã =1ã0
# ããéå±€åãœããããã¯ã¹ãšã¯ãåèªã®åºçŸã®ä»£ããã«æå³ã¯ã©ã¹ã®åºçŸã䜿ãããšã§
# ããèªåœã®å€§ãããæžãããåèªãéå±€çã«è¡šçŸããŠåèªã®æ°ãæžããææ³ã
# negative ã©ã³ãã ã«ãåºçŸããŠããªãåèªãåŽãåŠç¿ãããã®åèªæ° =0
# sample: ã©ã³ãã ã«é »åºåèªãæ¶å»ããã(ããã©ã«ãã¯1e-3)éåžžã«äœã
# iter:ãã¬ãŒãã³ã°ååŸ©åæ° ã¡ãªã¿ã«ããã©ã«ãã¯ïŒã
# min-count: nåæªæºç»å Žããèªå¥ã¯ç Žæ£ãããå°ãå€ãã«50ã«èšå®ãã¬ã¢ãªåã¯æ¶ããã
# window:ååŸã«ããåèªããäœåãŸã§æèãšããŠå€æãããããããã©ã«ãã¯ïŒã
# ããåèªã®å Žåã¯10以äžã§ãè¯ãããä»åã¯ïŒåããšãªã®ã§å€ãã«èšå®ã
model.save("mychar2vec_fromWikiALL.model")
â¥ãããããããŠã¿ãã
from gensim.models.word2vec import Word2Vec
import gensim
JPChmodel_path = 'mychar2vec_fromWikiALL.model'
JPmodel = Word2Vec.load(JPChmodel_path)
JPmodel.most_similar(positive = [u'æž'], topn=10)
[('å', 0.7298516035079956),
('æ', 0.7217180728912354),
('èš³', 0.6349352598190308),
('èš', 0.6302940845489502),
('ç', 0.6280542612075806),
('å³', 0.6250900626182556),
('çš¿', 0.619011402130127),
('çŽ', 0.6081395149230957),
('å®', 0.6069372296333313),
('è', 0.5928876996040344)]
ãŸããçåã«æãç¹ã¯ããããã©ãã
ãè¿ããžã£ã³ã«ãã®æåã¯äŒŒãŠããæåãšããŠè¿ããŠããæ°ã¯ããã
ã€ãŸããå
å·ã®èŠæ¬ïŒè¯ãæå³ã®æåããå
¥ããã°ã
è¯ãæå³ã®æŒ¢åãè¿ããŠãããã®ã§ã¯ãªãã ãããïŒïŒ
AIã«ã䜿ãããæŒ¢åããæ±ºãããã
ãŸããïŒæåã®çµã¿åãããèããåã«ã
ã©ã®æŒ¢åã䜿ã£ããè¯ãã®ããå
å·ã«çžå¿ããæŒ¢åãæœåºããããšã«ããã
æ¹éãšããŠã¯ãéå»ã«äœ¿ãããæåãšãæå³ã®æ¹åã䌌ãæåãæ¢ãããšã«ããã
éå»ã®å å·ã®ãªã¹ããå å·¥ãã
ããææ¬ããšããæŒ¢åããŒã¿ã®å 容ã«ã€ããŠèšåãããã
ã倧åããããå¹³æããŸã§ãçŽïŒïŒïŒåã®å
å·ãããã
æ¥µå°æ°ã®äŸå€ãé€ããŠãã»ãšãã©ã¯ãïŒæåãã§æ§æãããŠããã
å
å·ã®ãªã¹ãã¯ç°¡åã«èŠã€ããã®ã§èªåã§ããã£ãŠã
ããããå
å·ã«äœ¿ãããŠããæŒ¢åã®çš®é¡ã¯æå€ãšå°ãªãã
ãªããšãïŒïŒæåã
å€ãé ãããªãã¹ããšã
æ°žå å€©æ²»å¿æ£é·æåå®å»¶æŠå¯åŸ³ä¿æ¿ä»å平康å®ã»ã»ã»
äžèšã®æåã¯ã©ããïŒïŒå以äžäœ¿ãããŠããããšãŒã¹æåãæ¥æ¬ã®çæ³ã
äžçªå€ããæ°žãã¯ãã€ã«ïŒïŒå
äžåããåºãŠããªãæåã¯ãäœãã®ééãã§å
¥ã£ããããããªãã®ã§é€å€ãããšã
ïŒïŒæåã«ãªã£ãã
ãããã®æåããããã«å¯ŸããŠã䌌ãŠããäžäœïŒïŒäœã衚瀺ããŠã¿ããã
# å
å·ã«äœ¿ãããŠããæŒ¢åããå€ãé ãã䞊ã¹ãŠããããïŒåããåºãŠããªãåã¯æ¶ããŠãããã
str = "æ°žå
å€©æ²»å¿æ£é·æåå®å»¶æŠå¯åŸ³ä¿æ¿ä»å平康å®ä¹
æ
¶å»ºäº«åŒè²æçŠå€§äºå¯¿äžå芳åç¥æ¿äžé€é²è·"
nnn=20
for idx in range(0, len(str)):
out = JPmodel.most_similar(positive = str[idx], topn=nnn)
resulttext=""
resulttext+=str[idx]+": "
for var in range(0, nnn):
resulttext+=out[var][0]
print(resulttext)
æ°ž: è²çŠå¯å¿ æ
¶åŸ³åŒä¹
åä»äº«å
åéåæ³°æž
è©äºè³¢
å
: æ°žè²å§å¯ä»çŠå¿ åä¹
è³¢åŒè©åæ
¶è€æ³°åŸ³è«±äº«æ°
倩: ä»çæŠå
åé²å®®éœåŽæè²æ®¿æ°žç¥ç¥ å玫埳倪é
æ²»: åºçæ°ç±é»çç¬å·±æ²å¯å
àœæžæ°àžé¢Ú身àžá
å¿: åªæå察äºxæé+è§Šæé
žçž®qä¿¡ãèä¿å€è±¡
æ£: æ
¶åæ°žè²å¿ æåŸ³è£åŒå¯å®æ²å»¶æ¹å
çŠçæŠå¯ºæ¡
é·: å¯å
Œåä»»äœå°±å¡è·åå®åŒèŒä¿æ¿å§å€åŽå¯¿åžé
æ: æžå
žèå³è©©èª¬æŒ¢åœçºèš³åæŠèšå·ååèçåæ
å: éšåŠåŒå¡æç§æåå髊èŽéåŒå åœå®è»æ«ç£éµ
å®: ä»åæž
氞康æè²åŸ³æ³°å®äžçä¹
å°å¿ åè£äº«å¯¿åŒ
å»¶: é
çŽè³æ£å
æ
¶è²æ°žç·äºäœ5é3âé·4çŠ6æ¿
æŠ: é倩氞çå¯ä»å
è²é·äº«æå®çŠå
没æåææ£å
å¯: çŠæ°žäº«è²å¿ æ
¶è©åŸ³ååŒæ±å¥å£å
äºèä¹
ä¹éæž
埳: æ°žå¿ è²åæ
¶å¯ååŒä»ä¹
æž
çŠè³¢å£åæ³°æ±é乿
ä¿: äŸå±æ¢åšåº·åºç¢æç¶ç害èªç åå®ç¥è²¡åŒå®å µ
æ¿: å«ç¥å£æŒå©¿äŒ¯åžå¿çµççµ±é¢äŸ¯åŒç¶ç¥å»è€ä¿éº
ä»: ååè²æ³°æž
å°å€©åŒæ°žæåŸ³äºè³¢ææŸé亮æ
¶æèŒ
å: 仿ž
åæ³°å°åŸ³åŒè²ææ°žæ
¶ä¹è³¢å¯åçŠäº®å¿ æèŒ
å¹³: æé£œåç·©è¹æ§å®çŠæµŠç瞟蟯衡ä»å¶ºç¥äœŒïŒéœç¬
康: åŸ³æ°žå¿ ä¿éå®ååŒåæž
é€è²å£å¯å¥åä¹
æ
¶ä¿æ
å®: èµå¡çèéäºäŒœæ®¿èå
ç
å®®å¯å
«äž¹å竹åžçŽæŠ
ä¹
: åæ°žåŒç¥ç°äºè€å¿ äŒèåŸ³ä¿æŸå¹žæŽ¥éä¹äœè²è±
æ
¶: æ°žçŠåŒå¿ è²å¯åŸ³éååæž
æ³°è©ä»æåä¹
ææ£äº
建: æ·åµæœç«çœ®èšè·¡æ®¿å£æ£é³éžè€åèçŠå¡æå ç
享: 坿°žçŠè²æ²¡å¿ 埳è©åæ
¶å£å«¡è奿»ä¹
éä»äºç¯€
åŒ: éåä¿åæ³°æ
¶è²åæ°žè³¢ç¥ä¹
圊ä¹èŒäºéè€ä»å¿
è²: æ°žå¿ åŸ³ä»åŒå¯æ
¶ååæ³°çŠéè³¢è€äº«å£å
å°æž
ä¹
æ: æ£æŒè³¢æ¿æ°žè²å
埳è
æŠæ åŒæ
¶è£ä»åæ®å¯å²è
çŠ: 坿°žæ
¶å¿ è²è©äº«å埳è³åŒæ±å£äºå¥å
æ°éæ³°å
倧: äžé·æ ¡æ©çš²é«èª²æ¯å£«åºåŠäº¬éç§èé¢äžå€âåŸ
äº: åŒåè
äºç«¹ä»åé壿Ÿå¥¥æŸä¹
ç¥å
«å氞䞹接è²
寿: å仿¿æ©éåéæž
åŒè²åæ¡èŒæ¶Œæ·³æ³°æµæ°žå€«å€ª
äž: ååé¡$çŽ0,åïŒ
åªè²»çŸçޝé£俞éçšæäºº
å: 質é
žçæ žåªçŽ èèå€å¡©é
µäœåèéβçä¿é埮
芳: å
æž¬å®¢çºæŽé¡æ¯é€šå±ç¯æž©åªæå€©å¥æ
§è颚æ¶å£¢
å: 幞èŒåé竹é
è³¢åä¹åæ·³ç¥åŒæŸæŽæºæŠäºæŸåœŠ
ç¥: ç¥ééç¥å®®ç¥ ç¥åŽç¥è橿殿祷巫幡çç¥åçŠå¥
æ¿: çµ±æ°éŠæ æé©æŽŸæç¶åæ¯é é¥é£åšéŽåŒŒåœèª¥æŸ
äž: å
åå€åºåè¿å€§ç£æ¡æ£åé«è¥¿ééœçæŒè¡ç®åŸ
é€: è²ç¶ä¹³å¹Œåº·èæ¯æ³é
ªèŠªç§åž«èçæåŒåä»å享
é²: é³³çéŸéœæ·µèæ²æèŒéçŽ«ç«ºäž¹é§æŽå€©å°å¶ºåéµ
è·: å¥éºé²å®æŽéæçè¡ç¥è
æè¢«é Œåºå¢å®ç¯å
æ
ããã¯ãã¡ãã£ãšãã£ãããšãåŸæããã¬ãã«
æ³å以äžã®ã«ãªã¹ãçãŸããŠããŸã£ã
å°å·ããã°å°çå
çã¬ïœã¹ïœã®çµæãšããŠã³ã¹ãã¬ã«äœ¿ãããããããªã
ã«ãªã¹ã®äžããå ãæãåºã
ãŸãã䌌ãŠããTOP20ãå
šãŠè¡šç€ºããšããã®ã¯ããããã ã£ãã
ã©ãŒã¿ãŠãé¢ä¿ã®ç¡ããšããããæŒ¢åã§ãããªãã¬ãã«ã®æåãŸã§åºãŠããã
ããã§ãã³ãµã€ã³é¡äŒŒåºŠãäžå®å€ä»¥äžãšããéŸå€ãèšå®ããããšã§ã
åå䌌ãŠããæåã ããæœåºããããšã«ããã
ãŸããè¶
é£è§£ãªæŒ¢åãŸã§æœåºãããŠããããã
ããã§ãåžžçšæŒ¢åã®ãªã¹ããšãæè²æŒ¢åã®ãªã¹ãã®ãã£ã«ã¿ãé©çšããŠã¿ããã
str = "æ°žå
å€©æ²»å¿æ£é·æåå®å»¶æŠå¯åŸ³ä¿æ¿ä»å平康å®ä¹
æ
¶å»ºäº«åŒè²æçŠå€§äºå¯¿äžå芳åç¥æ¿äžé€é²è·"
nnn=30
ruijido=0.6
jyouyoulist = "äºåææªæ¡å§æ±ã»ã»ã»ã»ä»¥äžçç¥"
kyouikulist = "ææªå§å®ææ¡ä»¥ã»ã»ã»ã»ä»¥äžçç¥"
allresult=""
for idx in range(0, len(str)):
out = JPmodel.most_similar(positive = str[idx], topn=nnn)
resulttext=""
resulttext+=str[idx]+": "
#å
ã®å
å·ã«äœ¿ãããæŒ¢åããæè²ãªã¹ãã«å«ãŸããŠãããªãã°ãåè£ã«å
¥ããã
if kyouikulist.count(str[idx]):
allresult+=str[idx]
for var in range(0, nnn):
#ã³ãµã€ã³è¿äŒŒåºŠãruijido以äžã®ãã®ã察象ãšããã
if(out[var][1]>ruijido):
if str.count(out[var][0]):
resulttext+="("+out[var][0]+")"
else:
if kyouikulist.count(out[var][0]):
resulttext+="ã"+out[var][0]+"ã"
allresult+=out[var][0]
else:
if jyouyoulist.count(out[var][0]):
resulttext+="{"+out[var][0]+"}"
print(resulttext)
lst = list(set(allresult))
print(','.join(lst))
print(len(lst))
æ¬æçš¿ã®ãã€ã©ã€ãã®äžã€ã以äžã®è¡š
æ°ž: (è²)(çŠ)(å¯)ãå¿ ã(æ
¶)(埳)(åŒ)(ä¹
)(å)(ä»)(享)(å
){å}{é}ãåã{æ³°}ãæž
ã{è©}(äº){è³¢}ãæ°ã{æ}(康)(æŠ)(æ£)
å
: (æ°ž)(è²){å§}(å¯)(ä»)(çŠ)ãå¿ ã
倩: (ä»)ãçã(æŠ)ãå
ããåã(é²)ãå®®ã
æ²»: ãåºã
å¿:
æ£: (æ
¶)ãåã(æ°ž)
é·: ãå¯ã{å
Œ}ãåããä»»ã{äœ}
æ: ãæžããå
žããèã
å:
å®: (ä»)
å»¶:
æŠ: (倩)(æ°ž)ãçã
å¯: (çŠ)(æ°ž)(享)(è²)ãå¿ ã(æ
¶){è©}(埳)(å)(åŒ){æ±}{å¥}{å£}(å
)(äº)ãèã(ä¹
){é}ãæžã
埳: (æ°ž)ãå¿ ã(è²)(å)(æ
¶)(å¯)ãåã(åŒ)(ä»)(ä¹
)ãæž
ã(çŠ){è³¢}{å£}{å}{æ³°}{æ±}{é}{æ}{è©}(康)(享){æ}{å°}{接}
ä¿: {äŸ}ãå±ã
æ¿:
ä»: ãåã(å)(è²){æ³°}ãæž
ã{å°}(倩)(åŒ)(æ°ž)(埳)(äº){è³¢}{æ}{æŸ}{é}(æ
¶){æ}ãåã{äž¹}ãå®®ããå¿ ã(å®){å}(å
)
å: (ä»)ãæž
ããåã{æ³°}{å°}(埳)(åŒ)(è²)(æ°ž)(æ
¶){è³¢}(å¯){å}(çŠ)ãå¿ ã{æ}{å}{é}(äº){å}(ä¹
){æ}ãäºã(寿){äž¹}{æŸ}
å¹³: ãæã
康: (埳)(氞)
å®: ãèµã
ä¹
: {å}(æ°ž)(åŒ)ãç°ã{äº}ãå¿ ã{è}(埳){ä¿}ãæŸãã幞ã{接}{é}{äœ}(è²)ãè±ããåã(äº)ãè³ã(å)ãåã(æ
¶){è©}
æ
¶: (æ°ž)(çŠ)(åŒ)ãå¿ ã(è²)(å¯)(埳){é}(å)ãåããæž
ã{æ³°}{è©}(ä»){å}(ä¹
){æ}(æ£)(äº){è³¢}{å°}{æ±}{å£}
建: {æ·}ãåµã{æœ}ãç«ã
享: (å¯)(æ°ž)(çŠ)(è²){没}ãå¿ ã(埳){è©}(å)
åŒ: {é}ãåã{ä¿}{å}{æ³°}(æ
¶)(è²)(å)(æ°ž){è³¢}(ä¹
){äº}{é}(ä»)ãå¿ ããç°ã(埳)(äº)ãæž
ãã竹ã(å¯){é
}ãæŸã
è²: (æ°ž)ãå¿ ã(埳)(ä»)(åŒ)(å¯)(æ
¶)ãåã(å){æ³°}(çŠ){é}{è³¢}(享){å£}(å
){å°}ãæž
ã(ä¹
){æ}ãæ°ã{å«¡}{å}(äº){æŸ}
æ:
çŠ: (å¯)(æ°ž)(æ
¶)ãå¿ ã(è²){è©}(享)(å)(埳){è³}(åŒ){æ±}{å£}(äº){å¥}(å
)ãæ°ã{é}{æ³°}
倧:
äº: (åŒ){å}{äº}ã竹ã(ä»)ãåã{é}{å£}ãæŸã{奥}{æŸ}(ä¹
)ãå
«ã(å)(æ°ž){äž¹}{接}(è²){æ³°}ãç°ã(æ
¶)ãåã{å }{å°Ÿ}
寿: (å)(ä»)
äž: ãåããåããé¡ããçŽã
å: ã質ã
芳: ãå
ã
å: ã幞ããåã{é}
ç¥: {é}{é}{ç¥}ãå®®ã{åŽ}
æ¿: ãçµ±ã
äž:
é€:
é²: ãéœã{玫}{äž¹}{é§}{æŽ}(倩)
è·:
äžèšã®è¡šã®å¡äŸã¯ä»¥äžã®éãã
- äžçªå·Šã®åã¯ãå å·ã«äœ¿ãããŠããæŒ¢åãå€ãé ã«äžŠã¹ããã®
- å³åŽã«ãåæŒ¢åã«å¯ŸããŠã³ãµã€ã³é¡äŒŒåºŠã0.6以äžã®æåããé¡äŒŒåºŠé ã«èšèŒ
- åžžçšæŒ¢åã«ç¡ã挢åã¯åé€
- ããã¯ãå å·ã®æŒ¢åãªã¹ãã«ååšããŠããæŒ¢å
- {}ãã¯ãåžžçšæŒ¢å(1945å)ã«å«ãŸããŠããæŒ¢å
- () ã¯ãæè²çšæŒ¢å (1006å)ã«å«ãŸããŠããæŒ¢å
ãã®çµæãèŠããšããªããšãªãçžèµ·ã®è¯ããããªæŒ¢åãå€ãã
ãã®äžã§å€å°é£ããåã¯ïœïœã§å²ãããŠããããšããé°å²æ°ã«ãªãã
ãã®ææ³ã¯ãååãä»ããéãªã©ã®æ¡åºããªã©ãšããŠããå€å°äœ¿ãããã ã
äŸãã°ãäžããèŠãŠãããšããå¿ãããåããäŒŒãæŒ¢åãç¡ããšããã§ãããã
ããã¯ã䌌ãæå³ã®èšèãèŠã€ãããªããç¬èªæ§ãé«ã挢åã§ããããšãæå³ããŠããã
ãã®ãªã¹ãã®äžã§ããæè²çšæŒ¢åãã«å«ãŸããŠããæŒ¢åã ãã«éå®ããŠåºåãããšã
以äžã®ïŒïŒæåã«ãªãã
ïŒâ»äžè¿°ã®ããã°ã©ã ãå®è¡ãããšãã¡ãã®çµæãåæã«åºåããïŒ
å,å¿ ,è,å,建,è·,康,åµ,å,æž
,幞,é¡,åº,äž,æ²»,å,倩,æ,å®,èµ,ç°,å
,æ¿,芳,ç,æž,å¹³,æ,æ¿,ä»,å
,ä»»,å¯,äž,æ£,ç«,æ,äº,å¿,è,ä¹
,å,æ°,倧,çŽ,æž,ä¿,å,å
ž,å®,埳,çµ±,é€,ç¥,å,å»¶,å,å
«,è±,é²,æ°ž,å,éœ,質,è³,é·,å,å±,å®®,æŸ,竹
71
ã©ããäžäœãªã®ïŒé çªãã€ããã
䜿ãããåè£ãšãªãæåã71æåãŸã§æœåºãããã
çµå±ã©ãã䜿ããããã ãïŒã£ãŠããšã§ã©ã³ãã³ã°ãã€ããã
ãããŸã§ã®èãæ¹ã«åãããŠãããææ¬ã®æŒ¢åããšã®
è·é¢ã®è¿ããè¿ããã®ãäžäœãšããã
ãã ããäŸãã°ãæ°žããåããªã©ãèªåèªèº«ããææ¬ã«å
¥ã£ãŠããæŒ¢åã«ã€ããŠã¯ã
èªåèªèº«ãšã®è·é¢ã¯1.0ïŒæå€§ïŒã«ãªã£ãŠããŸããºã«ã€ã®ã§ã
èªèº«ãšã®æ¯èŒãé€ããŠãå¹³åå€ã§æ¯èŒããããšã«ããã
ãªãããã®ã©ã³ãã³ã°ã«ãã£ãŠã
ãªãããŸã 71åã«å
¥ã£ãŠãããå±ããªã©ã¯ãã»ãŒããªã®é äœã«ãªã£ãŠé€å€ãããã
ããã以äžã®ããã°ã©ã ã§ããããTOP20ã®çºè¡šã ïŒ
str = "æ°žå
å€©æ²»å¿æ£é·æåå®å»¶æŠå¯åŸ³ä¿æ¿ä»å平康å®ä¹
æ
¶å»ºäº«åŒè²æçŠå€§äºå¯¿äžå芳åç¥æ¿äžé€é²è·"
resultlist=[]
for lstidx in range(0, len(lst)):
distance=0
cnt=0
for stridx in range(0, len(str)):
#å¥ãªæåã§ããå Žåã®ã¿å®æœ
if lst[lstidx] != str[stridx]:
distance += JPmodel.similarity(lst[lstidx], str[stridx])
cnt+=1
resultlist.append([lst[lstidx],distance/cnt])
ranking = sorted(resultlist, key=lambda x:x[1], reverse=True)
finalkouho=ranking[:20]
import pprint
pprint.pprint(finalkouho)
[['æ°ž', 0.4432615391959312],
['埳', 0.42958927220753584],
['ä»', 0.42234036442990081],
['å¿ ', 0.39814997500397165],
['æž
', 0.39167382076459972],
['å', 0.38281166761176649],
['ä¹
', 0.35123899427760841],
['倩', 0.33663049257426525],
['å
', 0.33296518845281042],
['æ£', 0.32448461789657612],
['å®', 0.31749253943023814],
['å®®', 0.29887811652228685],
['èµ', 0.29585180662184218],
['è', 0.29422004885350239],
['äº', 0.29177360054646811],
['康', 0.29155911853392569],
['æ°', 0.28970518143898977],
['è±', 0.28275205021451155],
['æ', 0.27604375636628525],
['æŸ', 0.26546272673926241]]
ä»åã®çµæã§ãããšã
ãæ°žãã埳ããªã©ã¯ããšããšãææ¬ã«ãå
¥ã£ãŠããæŒ¢åã§ãããã
ãå¿ ããåããæž
ããå®®ããèµããèããäºããæ°ããè±ããæŸã
ã®åã¯ããææ¬ã«ã¯ãªãAIãå°ãåºããæŒ¢åã§ããã
ãèµããèããäºããæ°ããããã¯äœ¿ãããå¯èœæ§ã¯ã»ãŒç¡ããããªãã®ã®ã
ä»ã®åã¯ãããªã«æªãã»ã³ã¹ã§ã¯ç¡ãæ°ãããã
ïŒãšæã£ãŠãããããèãã¯å®ã¯æ¢ã«éå»ã«äžå䜿ãããŠããã
ãäžã®ãææ¬ã¯ãäžåã ãã®æåã¯é€å€ããŠãããããå«ãã§ããªãã£ããïŒ
ããã«ããŠããä»åã®çµæããããäžåã®ãšãŒã¹**ãæ°žã**ã®æ¯æçã®é«ãã¯ãããã
äžäœé£ã®é¡ã¶ãã¯ããææ¬ã®äžäœé£ãšããªãè¿ãç¶æ³ã§ããã
å
å·ã§è¯ã䜿ãããæŒ¢åã¯ãAIããèŠãŠããããªãã®çç±ããã£ããšããããšã ã
æ¹ããŠä»åã®æ±ºåæŠã¡ã³ããŒãçºè¡šïŒïŒ
ãããŸã§ã§ãAIã䜿ãããæŒ¢åã®ãªã¹ãããé äœä»ãã§äœæããããšãåºæ¥ãã
äžèšã§ã¯ããåèãŸã§ã«TOP20ãŸã§è¡šç€ºããŠãããã
çµã¿åããã ãšããã400ãã¿ãŒã³è¿ãã«ãªã£ãŠããŸãããŸã å€ãããã
以éã§ã¯äžèšã®äžã®TOP10ã®æŒ¢åã䜿ãããšããã
ã€ãŸããåã¡äžãã£ãã¡ã³ããŒã¯ãAIã©ã³ãã³ã°é ã«ä»¥äžã®ïŒïŒåã
ãæ°žãïŒé廿¡çšïŒïŒäœïŒïŒïŒåã®äžåã®ãšãŒã¹ãAIã©ã³ãã³ã°ã§ãå ã
ïŒäœ
ã埳ãïŒé廿¡çšïŒïŒäœïŒïŒïŒåãAIçã«ã¯ãããæ¬åœ
ãä»ãïŒé廿¡çšïŒïŒäœïŒïŒïŒåãå
«ç¬äŒããè«èªãªã©ã«ãç»å Žããããæ¬åœ
ãå¿ ãïŒAIãåºããæ°é¡ããå
«ç¬äŒããè«èªãªã©ã«ãç»å Žãã¡ãšå€ãæŠå¿µãïŒ
ãæž
ãïŒAIãåºããæ°é¡ããè¯ãæå³ã ããäžåœã®åœåã ã£ãã®ããŸããïŒ
ãåãïŒAIãåºããæ°é¡ããå
«ç¬äŒãè«èªãªã©ã«ãç»å Žãæ°é¡ã®äžã®ãšãŒã¹æ Œã
ãä¹
ãïŒé廿¡çšïŒïŒäœïŒïŒïŒåãæžãããããããããã
ã倩ãïŒé廿¡çšïŒïŒäœïŒïŒïŒåããã¯ãäºäœã匷ãã
ãå
ãïŒé廿¡çšïŒïŒäœïŒïŒïŒåãã倩ããšåæ§ã«åŒ·ãã
ãæ£ãïŒé廿¡çšïŒïŒäœïŒïŒïŒåããããæžããããã
ãåèïŒåç·éèŠå «ç¬äŒ ã®ïŒã€ã®å
ä»ã»çŸ©ã»ç€Œã»æºã»å¿ ã»ä¿¡ã»åã»æ
éäžèå¯
äžèŠãéå»ã«äœ¿ãããæŒ¢åã倿°åºããŠãããAIãåããŠããªãããã§ããããã
ãããããå
å·ã«äœ¿ãããŠããæŒ¢åã¯ãã©ããè¯ãæå³ã§ããã
çžäºã«æå³ã®é¡äŒŒåºŠãé«ããããé廿¡çšã®æŒ¢åãå€ããªãã®ã¯åŠ¥åœã§ããã
ãå
å·ã«äœ¿ãããæŒ¢åãšåããããªæ¹åæ§ã®åãã®ã©ã³ãã³ã°ã§ããããã
åããããªæ¹åãæã¡ãªããä»ãŸã§äœ¿ãããŠããªãã£ããå¿ ããæž
ããåãã
æ¢ãåºãããšãã§ããããšèšãã¹ãã§ãããã
æåã®ãæããå¹³æã®ãæãã¯ããããåããŠäœ¿ãããåã§ããã
ä»åãïŒååã¯æ°ããæŒ¢åã䜿ãå¯èœæ§ãããããéã«èšãã°ã
ããäžæ¹ã¯éå»ã«äœ¿ãããåã®ã©ããã䜿ãããå¯èœæ§ãé«ããã
ïŒæåãšãéå»ã«äœ¿ãããåã䜿ããããããããªãã
æ°æŒ¢åïŒïŒæåãæ§æŒ¢åïŒïŒæåãTOP10ã§ãã£ããšãããã©ã³ã¹ã¯é¢çœãã
AIã«ãå å·ãšããŠé©åãªçµã¿åããããäœããã
ããŠãäœ¿ãæŒ¢åã®æŽçã¯åºæ¥ããã
ã©ãçµã¿åãããã°ãå
å·ããããã®ãïŒ
ãŸããéå»ã®å
å·ã§äœ¿çšæžã¿ã®ãã¿ãŒã³ã¯é€å€ããå¿
èŠãããã
ãŸãããæ°žä¹
ããšããã倩å
ããšããæ¢ã«å¥ã®æå³ãããåèªãé€å€ããå¿
èŠãããã
ä»ã«ãäŸãã°ãä¹
æ°žããšããèšèã¯ç¡ãã£ããšããŠãã
ãä¹
ãããæ°žããéåžžã«è¿ãããæå³ã§ãããããã«ãã³ããç¡ãã
仮説ãšããŠããããŸã§ã®å
å·ããããããæåã®ãã©ã³ã¹ãèæ
®ããŠããã®ã§ã¯ãªããïŒ
å
šéšã®çµã¿åãããã¿ãŒã³ã詊ããŠã¿ããã ãšãAIãåºããŠããæãæžã£ãŠããŸãããã
AIã«ãéå»ã®å
å·ã®æåã®ãè·é¢æããäžããããšã«ããã
å å·ãšããŠã®çµã¿åããã®ãã©ã³ã¹ããèšç®ããã
ãŸããä»®èª¬ãæ£ããã®ã確èªããããã
éå»ã®çŽïŒïŒïŒåã®å
å·ã®ãïŒæåç®ãšïŒæåç®ã®ãè·é¢ãïŒã³ãµã€ã³é¡äŒŒåºŠïŒãèŠãã
ïŒâ»ïŒæåã§åºæ¥ãŠããäžéšã®å
å·ã¯é€å€ïŒ
%matplotlib inline
import numpy as np
import matplotlib.pyplot as plt
gengoulist = ["倧å","çœé","æ±é³¥",ã»ã»ã»çç¥ã»ã»ã»"æ
¶å¿","ææ²»","倧æ£","æå","å¹³æ"]
distancelist=[]
for idx in range(0, len(gengoulist)):
distance = JPmodel.similarity(gengoulist[idx][0], gengoulist[idx][1])
distancelist.append(distance)
# æãç·ã°ã©ããåºå
left = np.array(range(0, len(distancelist)))
height = np.array(distancelist)
plt.plot(left, height)
å³ã®å·Šç«¯ãã倧åããå³ç«¯ããå¹³æãã§ãããããäºæåã®æŒ¢åã®é¡äŒŒåºŠãã°ã©ãåãããã®ã
ã°ã©ããèŠããšãïŒæåŸã®æ¹ã®ãæåãã-0.43ã§æ¥µããŠç°äŸã§ã¯ãããïŒ
ããçšåºŠäžå®ã®è·é¢ãä¿ã£ãŠããæ°ãããã
ã€ãŸãã仮説ãšããŠãéå»å
å·ã®è·é¢æã«è¿ã¥ãçµã¿åããã«ããããšããã®ã¯ã
è¯ãæ¡ãªã®ã§ã¯ãªãã ãããïŒ
ã§ã¯å®éã«è·é¢ãããã€ãããã«ããã¹ããªã®ãèšç®ããŠã¿ãã
from statistics import mean, median,variance,stdev
m = mean(distancelist)
median = median(distancelist)
variance = variance(distancelist)
stdev = stdev(distancelist)
print('å¹³å: {0:.2f}'.format(m))
print('äžå€®å€: {0:.2f}'.format(median))
print('忣: {0:.2f}'.format(variance))
print('æšæºåå·®: {0:.2f}'.format(stdev))
å¹³å: 0.30
äžå€®å€: 0.28
忣: 0.06
æšæºåå·®: 0.24
äžèšã®çµæããã0.3ããäžå¿ãšããŠããã©ã¹ãã€ãã¹ã0.24ãã«å
¥ãç¯å²ã®ã
挢åã®çµã¿åããæ¹ãæ¢ãããšã«ããã
é©åãªè·é¢æã®æŒ¢åã®çµã¿åãããæ¢ã
åã¡äžãã£ã挢åïŒïŒæåÃïŒïŒæåã®å
šïŒïŒïŒãã¿ãŒã³ã詊ããŠã
ãã®æŒ¢åã®è·é¢æãäžèšã®æšæºåå·®ã®ç¯å²å
ã«å
¥ããããªçµã¿åãããæ¢ãã
ãŸãããã®æ©äŒã«ãæ¢ã«éå»ã®å
å·ãªã¹ãã«å
šãåããã®ãããå Žåã¯é€å€ããã
ïŒãŸãããæ°žæ°žããªã©ã®ããã«ãåãåãéããå Žåãè·é¢ïŒ1.0ã«ãªããããèªåçã«é€å€ïŒ
ãããŠããã®çµæããæãé©åãªè·é¢ïŒå¹³åå€ïŒã§ããã0.3ãã«è¿ãé ã«äžŠã¹ãã
finalkouho="æ°žåŸ³å¿ ä»åæž
ä¹
å
æ£å€©"
# â»åå宿œã®ããŒã¿ã§TOP20â10ã«ããŠããã°åå®çŸ©ããå¿
èŠã¯ç¡ãã
hodoyoikyorilist=[]
for idxone in range(0, len(finalkouho)):
for idxtwo in range(0, len(finalkouho)):
distance = JPmodel.similarity(finalkouho[idxone][0], finalkouho[idxtwo][0])
if(0.30-0.24<distance and distance <0.30+0.24):
#0.3ããã®è·é¢ã®çµ¶å¯Ÿå€ãèšç®ããŠããã
absval = abs(distance-0.30)
kumiawase=finalkouho[idxone][0]+finalkouho[idxtwo][0]
#æ¢åã®gengoulistã®äžã«å«ãŸãããã¿ãŒã³ã¯ããã®æç¹ã§é€å€ããã
if not(kumiawase in gengoulist):
hodoyoikyorilist.append([kumiawase, distance, absval])
# ã0.3ããšããçæ³çãªè·é¢æïŒå¹³åå€ïŒããã®ä¹é¢ãå°ãªãé ã«äžŠã¹ãã
finalkouho_ranking = sorted(hodoyoikyorilist, key=lambda x:x[2], reverse=False)
import pprint
pprint.pprint(finalkouho_ranking)
[['å¿ å€©', 0.34512571556537058, 0.045125715565370594],
['å€©å¿ ', 0.34512571556537058, 0.045125715565370594],
['æž
æ£', 0.3541912868235782, 0.054191286823578211],
['æ£æž
', 0.3541912868235782, 0.054191286823578211],
['ä¹
倩', 0.35828565796442502, 0.058285657964425031],
['倩ä¹
', 0.35828565796442502, 0.058285657964425031],
['å
æ£', 0.38558117860512242, 0.085581178605122432],
['ä¹
æ£', 0.39211709853284332, 0.092117098532843333],
['æ£ä¹
', 0.39211709853284332, 0.092117098532843333],
['忣', 0.39522379769497534, 0.095223797694975354],
['æ£å', 0.39522379769497534, 0.095223797694975354],
['æ£å€©', 0.41378394594834822, 0.11378394594834823],
['æž
å
', 0.47026353686028266, 0.17026353686028267],
['å
æž
', 0.47026353686028266, 0.17026353686028267],
['æž
倩', 0.47101191032529927, 0.17101191032529928],
['倩æž
', 0.47101191032529927, 0.17101191032529928],
['å倩', 0.47803022604673118, 0.17803022604673119],
['倩å', 0.47803022604673118, 0.17803022604673119],
['仿£', 0.48142248276590699, 0.181422482765907],
['æ£ä»', 0.48142248276590699, 0.181422482765907],
['å
倩', 0.50123434913804443, 0.20123434913804444],
['埳倩', 0.51178692951545024, 0.21178692951545025]]
æšæºåå·®ã®ç¯å²å
ãåæ Œãšãããããçµæ§å¹
ãåºããšã£ãã€ããã ãã
ãããïŒïŒãã¿ãŒã³ã®ãã¡ã§ãäžèšïŒïŒãã¿ãŒã³ããçãæ®ããªãã£ãã
æåŸã®ä»äžããã«ãŒã«ã«æ²¿ã£ãé€å€
ãããŸã§ã§ã以äžïŒç¹ã®ã¡ã€ã³éšåãçµäºããã
- è¯ãæå³ã®æŒ¢åãæœåºãã
- ãã®æŒ¢åãçµã¿åãããŠãéå»ã®å å·ãšäŒŒãè·é¢æã®çµã¿åãããäœã
æåŸã«ãã³ã¬ãŸã§æ©æ¢°ã«ããããã®ã¯ã¡ãã£ãšé¢åã§ã¯ããã
以äžïŒç¹ã®å®è£
ãè¡ããæ±ºå®ãŸã§æã£ãŠããããšæãã
- æ¢ã«ä»ã®æå³ã§äœ¿ãããŠããåèªãé€å€
- MïŒææ²»ïŒãTïŒå€§æ£ïŒãSïŒæåïŒãHïŒå¹³æïŒãšéè€ããªã
æ¢ã«ä»ã®æå³ã§äœ¿ãããŠãããïŒMecabã§åè§£
Mecabã«ãã圢æ
çŽ è§£æã«ãã£ãŠã
åããããããããããïŒãä»ã®æå³ã¯ç¡ãã
åãããã«äžèªãšèªèãïŒãä»ã®æå³ã§èªèãããã
ãããå©çšããŠãæ¢åã§å¥ã®æå³ããããã®å€æåºæºãšããã
ã€ãã§ã«ãèªã¿ä»®åãæ¯ã£ãŠããããšã§ãããšã§MTSHé€å€ã«äœ¿ããããã«ããã
åèªãåè©åè§£ãã颿°ã«ã€ããŠã¯ã
ååã®ããèµ€ã®ä»äººãã®å¯ŸçŸ©èªã¯ãçœãæäººã ãããèªåçæãããç©èªãã§ã
ã倿©åç©å
¬åããåè§£ããã®ã«äœã£ã颿°ãã¡ãã£ãšã ãå€ããŠåå©çšããã
import MeCab
def check_hinsi(taisyou_word):
mecab = MeCab.Tagger("-Ochasen")
#mecab = MeCab.Tagger(r"-Ochasen -d .\mecab-ipadic-neologd")
WordHinsi = []
#ååŠçïŒããªã ãªã©ãäŸã\tãã¯ããªã ããŠãããå
šè§ã¹ããŒã¹å
trline = taisyou_word.replace(u'\t', u'ã')
#####ã¡ã€ã³åŠçïŒ
#Mecabã§ã®è§£æã宿œ
parsed_line = mecab.parse(trline)
#åæçµæããïŒè¡ïŒåèªïŒããšã«åå²ãããæ¹è¡ã³ãŒãã§åå²ããã
wordsinfo_list = parsed_line.split('\n')
#ç¶æ³ãèŠã
#print(taisyou_word+"ã¯"+str(len(wordsinfo_list))+"åèªã«åãããŸãã")
#åèªããšã«ãè§£æåŠçã«å
¥ããŠããã
for wordsinfo in wordsinfo_list:
#print("wordsinfo= " + wordsinfo)
# ååèªã®æ
å ±ãã¿ãã§åå²ããã
# ããç«,ã¿ãã,ããç«,åè©-äžè¬ ãªã©ã®ãããªïŒè¡ããlist圢åŒã«ãªãã
info_list = wordsinfo.split('\t')
#print(info_list)
#空çœãªã©ã®å Žåãäžçªç®ã®èŠçŽ ãç¡ãå Žåãããããã
#æåã«ãã®ãªã¹ãã®é·ãããã§ãã¯
#以äžã®ããã«ç¡æå³ã«ïŒåå²ããããããåè©æ
å ±ãååšããªããã®ã¯ç¡èŠããã
#['ã³ã¢ã©', 'ã³ã¢ã©', 'ã³ã¢ã©', 'åè©-äžè¬', '', '']
#['EOS']
#['']
if(len(info_list)>2):
#info_list[1]ãèªã¿ä»®åãå
¥ã
WordHinsi.append((info_list[0], info_list[1], info_list[3]))
return WordHinsi
# check_hinsi("倿©åç©å
Œ")
# å
å·ã®åè£ã«ããããäžèšé¢æ°ãé©çšããŠã¿ãã
ffor idx in range(0, len(finalkouho_ranking)):
print(check_hinsi(finalkouho_ranking[idx][0]))
[('å¿ ', 'ãã¥ãŠ', 'åè©-äžè¬'), ('倩', 'ãã³', 'åè©-äžè¬')]
[('倩', 'ãã³', 'åè©-äžè¬'), ('å¿ ', 'ã¿ãã·', 'åè©-åºæåè©-人å-å')]
[('æž
æ£', 'ããšããµ', 'åè©-åºæåè©-人å-å')]
[('æ£æž
', 'ã·ã§ãŠã»ã€', 'åè©-åºæåè©-å°å-äžè¬')]
[('ä¹
', 'ããµ', 'åè©-äžè¬'), ('倩', 'ãã³', 'åè©-äžè¬')]
[('倩ä¹
', 'ã¢ã¡ã¯', 'åè©-åºæåè©-人å-å§')]
[('å
æ£', 'ã¬ã³ã·ã§ãŠ', 'åè©-äžè¬')]
[('ä¹
', 'ããµ', 'åè©-äžè¬'), ('æ£', 'ã¿ãã·', 'åè©-åºæåè©-人å-å')]
[('æ£ä¹
', 'ããµããµ', 'åè©-åºæåè©-人å-å')]
[('忣', 'ã¿ã«ããµ', 'åè©-åºæåè©-人å-å')]
[('æ£', 'ã»ã€', 'æ¥é è©-åè©æ¥ç¶'), ('å', 'ã³ãŠ', 'åè©-äžè¬')]
[('æ£', 'ã»ã€', 'æ¥é è©-åè©æ¥ç¶'), ('倩', 'ãã³', 'åè©-äžè¬')]
[('æž
å
', 'ããšã¢ã', 'åè©-äžè¬')]
[('å
', 'ã²ã³', 'åè©-åºæåè©-人å-å§'), ('æž
', 'ããšã·', 'åè©-åºæåè©-人å-å')]
[('æž
', 'ã»ã€', 'åè©-åºæåè©-人å-å§'), ('倩', 'ã¿ã«ã·', 'åè©-åºæåè©-人å-å')]
[('倩', 'ãã³', 'åè©-äžè¬'), ('æž
', 'ããšã·', 'åè©-åºæåè©-人å-å')]
[('å', 'ã³ãŠ', 'åè©-äžè¬'), ('倩', 'ãã³', 'åè©-äžè¬')]
[('倩', 'ãã³', 'åè©-äžè¬'), ('å', 'ã³ãŠ', 'åè©-äžè¬')]
[('ä»', 'ããã·', 'åè©-åºæåè©-人å-å'), ('æ£', 'ã¿ãã·', 'åè©-åºæåè©-人å-å')]
[('æ£', 'ã»ã€', 'æ¥é è©-åè©æ¥ç¶'), ('ä»', 'ããã·', 'åè©-åºæåè©-人å-å')]
[('å
', 'ã¢ã', 'æ¥é è©-åè©æ¥ç¶'), ('倩', 'ãã³', 'åè©-äžè¬')]
[('埳', 'ãã¯', 'åè©-äžè¬'), ('倩', 'ãã³', 'åè©-äžè¬')]
äŸãã°ããæž
æ£ãã¯æ¢ã«äººåãšããŠåºãŸã£ãŠããã®ã§ã¢ãŠãïŒ
ã倩ä¹
ããšããã忣ããªããã¯äžèŠãŸã ã»ãŒããªæ°ããããã
人åãªã©ã§ãè¯ã䜿ããããããªãã¿ãŒã³ãšããŠãMecabã«èªèãããŠãããšããããšã§ã
ãããã£ãèªå¥ã¯èœãšããããšããèãæ¹ã§è¯ããšæãã
MïŒææ²»ïŒãTïŒå€§æ£ïŒãSïŒæåïŒãHïŒå¹³æïŒã®é€å€
äžèšã§æ©æ¢°ãäžããèªã¿æ¹ããã§ãã¯ããŠãMTSHã§å§ãŸã£ãŠããªããã®ããæœåºããã
ïŒã€ãã§ã«ãåè§£çµæãïŒèªã®ãã®ïŒæ¢åæå³ã®ãããã®ãã¯é€å€ïŒ
for idx in range(0, len(finalkouho_ranking)):
bunkaigo=check_hinsi(finalkouho_ranking[idx][0])
if not(bunkaigo[0][1][0] in "ããã ã¡ã¢ã¿ãããããµã·ã¹ã»ãœããããã"):
if len(bunkaigo)>1:
print(bunkaigo)
[('å
', 'ã²ã³', 'åè©-åºæåè©-人å-å§'), ('æž
', 'ããšã·', 'åè©-åºæåè©-人å-å')]
[('å', 'ã³ãŠ', 'åè©-äžè¬'), ('倩', 'ãã³', 'åè©-äžè¬')]
ãå æž ããšãå倩ããåºãã
â»ããããã®æç¹ã§å
šéšåè£ãæ¶ããŠããŸã£ãããããå°ãçµãæ¡ä»¶ãç·©ããŠ
ãå宿œããã°ããããŸãããã£ãšåè£ãããããåºãããå Žåãåæ§ã
æåŸã®ãã®ïŒç¹ã®åªå
床ãã©ããããã§ãããã
ïŒç¹ã ãåŸä»ãã«ãŒã«ã§ç³ãèš³ãªããããå§ãããåãã§äžåã§ã䜿ãããŠããåã¯ã
å®åšã®äººç©ã«ãªã£ãŠããŸãå¯èœæ§ãããããã
ãåè©äžè¬ãã®çµã¿åããã®æ¹ããããã€ã³ããé«ããšããããšã«ããã
ïŒãŸãåè£ãäºã€ã§ãããã§ãè¯ããã
ãæ©æ¢°ãåºããããŒã¿ã ãã«äŸåããäžäœã決ãããããã
ãåè©å€å®çµæïŒãã€ã³ãããšããïŒ
ããã§ããå倩ã ïŒ ãå æž ã ãšãªã£ãã
çµæïŒ AIã«æ°å å·ãäœããããšãããªã
çµæ
è¶
é·ãæŠãã ã£ããããšããšãæåŸã®æ±ºå®ãŸã§ãã©ãçããã
AIãå°ãã ããçµæã¯ã以äžã§ããã
çµæïŒãå倩ã
ïŒæ¬¡ç¹ïŒãå æž ãïŒ
ãŸããå床æçµåè£ã«æãã£ã挢åïŒïŒæåãå床åæããŠããã
ãæ°žãïŒé廿¡çšïŒïŒäœïŒïŒïŒå
ã埳ãïŒé廿¡çšïŒïŒäœïŒïŒïŒå
ãä»ãïŒé廿¡çšïŒïŒäœïŒïŒïŒå
ãå¿ ãïŒAIãåºããæ°é¡
ãæž
ãïŒAIãåºããæ°é¡
ãåãïŒAIãåºããæ°é¡
ãä¹
ãïŒé廿¡çšïŒïŒäœïŒïŒïŒå
ã倩ãïŒé廿¡çšïŒïŒäœïŒïŒïŒå
ãå
ãïŒé廿¡çšïŒïŒäœïŒïŒïŒå
ãæ£ãïŒé廿¡çšïŒïŒäœïŒïŒïŒå
ããšã§ããäºæ³ãåœãã£ããïœãã£ãŠèšãããã«ã¯ã
ãã£ãšå€ãã«åºããŠãããã»ããè¯ãã®ã ãã
ãïŒ11äœã®ãå®ããå¯èœæ§ããããïŒïŒ
誰ããæ¢ã«äºæ³ããŠããå
å·ã«ã¯ããªãããšãã説ãããããã
ãªããŠã²ããããæ±ºãæ¹ã ããã
ãããã£ãèšäºãæçš¿ããæç¹ã§çµ¶å¯Ÿå€ãã«ãªãäžæ¹ã§ã
å
å·èãããšã©ã€äººã¯qiitaã¯èŠãªãããããïŒ
èŠãŠããã®è¶
é·ãé§æïŒããã°ã©ã ã¯åãããªãããããïŒãšããæ°ãããã
å®éã«æ°å
å·ãçºè¡šããããã
ãã²ãã®æŒ¢åãæ¬èšäºã®æ¹æ³ã§äœäœã ã£ãã®ããåŸã§ç¢ºãããŠã¿ããã
ãŸãããã®ïŒã€ã®æåã®è·é¢ãã0.3ã«è¿ãã£ãã®ããèšç®ããŠã¿ããã
çµæãåºããããã®çµæãèšäºã«ããã®ã§èŠéããªãã§ã
䜿ã£ãæ°å€çèŠçŽ ãŸãšãïŒãã¥ãŒãã³ã°ãã€ã³ãïŒ
- ãïŒå以äžéå»å
å·ã«äœ¿çšãããããšã®ããæŒ¢åãããææ¬ãšããã
- åœç¶ïŒåãšãããšããã¯å€æŽå¯èœã§ããããå å·ãšããã®ã¯ã
- æ¡çšãŸã§ãããã«åè£ã«ãªã£ããã®ããããããã§ã
- ããããã¿ãŒã²ããã«å«ãããã©ããã§ã調æŽãå€ãã
- éžãã§ãã察象ããæè²æŒ¢åãã«ããç¹
- ãåžžçšæŒ¢åããŸã§åºããæ¹æ³ãããããåªå 床ãšããŠã¯ãæè²ãïŒãåžžçšãã§ããã
- ä»åã¯å€æ°ã®åè£ãåºãããšããŠããªãã£ããããæ©ãã«ãæè²ãã«çµã£ã
- é¡äŒŒåºŠã®è¶³åãã®ç¹æ° ïŒ 0.6
- 0.7ãããã®é«ãã«ããŠãã£ãšçµã£ãŠãè¯ãããã§ã¯ãã£ãã
- ãææ¬ã«è¿ã床åã§ãããäœäœãŸã§ãå©çšå¯Ÿè±¡ãšãããïŒ ïŒ ïŒïŒ
- äºæ³ãšç§°ããŠæ²¢å±±ã®åè£ããããããã®ãé¢å/é¢çœããªãããã
- ããªãçµã£ãå€ã ãšæãããã
- äºã€ã®æŒ¢åã®è·é¢æã®é©åå€ïŒ0.3
- ä»åã¯ãå¹³åå€ããæ¡çšããããã äžå€®å€ããæ¡çšããæ¹æ³ããããã
- ãããããã倧åãã®æä»£ããå šãŠã®ããŒã¿ãã¿ããããªããšã¯ããã«ã
- çŽè¿ã®äœå¹Žãã®ããŒã¿ã ãã§ã¿ããªã©ã®æ¹æ³ãããã
- åè©å€å®çµæã®èŸæžåã³ããã®çµæã©ã®ãããªåè©ãåªå
/é€å€ãããïŒ
- åè©äžè¬ ïŒ å§åããšããç¹ããäºã€ã«åããããé€å€ãªã®ãã©ããã
- Mecabã®åè§£èŸæžãå€ããã°çµæãå€ãã
- é€å€ã®æ¹æ³/åºæº
- ä»åã¯ããããåãã¯èª¿æ»ããŠããªã
- çŸä»£ã§ããã°ãGoogleæ€çŽ¢ããŠã¿ãŠèªå¥ã®æ®å床ãèŠãæ¹æ³ããã
- Char2Vecã®ã¢ãã«ã®çæ
- ä»åã¯WikipediaãããŒã¹ã«ããããå å·ã®æ§è³ªãèãããšã
- è«èªãªã©ã®ãåæžããäºçµããåŠç¿ãããçµæãè¯ããšæãã
ããšãã
挢åãã®ãã®ãçµã¿åãããã®ãã®ãã«ã€ããŠã
å
šã人ã®è¯è/åžžèã«ãã倿ãå
¥ããªããŠãã
ãã©ã¡ãŒã¿èšå®ã𿩿¢°åŠç¿ã ãã§ã
ããçšåºŠããã£ãœãæ°å
å·ã®æ¡ãäœãããšæãã
æ°å
å·ã¯èª°ã決ããã®ãè¯ãããããªããã
å¹³æã決ãŸã£ãæä»£ããã¯ããã«ã€ã³ã¿ãŒããããæ®åããŠããçŸåšã§ã¯ã
ã©ããªå
å·ã«ããããšããGoogleæ€çŽ¢ã§äœèšãªæå³ä»ããç¡ããã
å
¥å¿µãªèª¿æ»ãèŠããã ããã
ãããŠã©ãçºè¡šããããšãããããããªäººãSNSãªã©ã§ãã®è©äŸ¡ã奜ãåæå«ã¶ã
ãã€ãŠã¯å
šãèããããªãã£ããé廿倧ã«é£ããå
å·æ±ºå®ãªã®ã§ã¯ãªããïŒ
ååŠè
å
çããšã©ã€äººã®ãããããªèª¿æŽãææãå
¥ãããã¯ã
äºãå®ãããããåºæºãïŒä»åã§ãããã©ã¡ãŒã¿ããINPUTããã¹ãïŒãã¿ãªã§åæããŠã
åæåŸã«åããŠäžçºåè² ã§ããã°ã©ã ãèµ°ãããŠãTOP10ãããã®åè£ãåºãããŠã
ãã®åè£ã®äžã§æç¥šãããããªä»çµã¿ã®ã»ãããå
¬ææ£å€§ã
æ±ºãæ¹ã«éææ§ããã£ãŠã¿ããªãçŽåŸãããããªæ°ãããã
Word2VecãChar2Vecã¯ããã®æŒç®ã ãã§ãé¢çœããããããŒã«ã§ããã
ãã ãã¡ãã£ãšèšèã®æŒç®ãããŠçµãããã§ã¯ãªãã
ãã²ã©ããã£ããéã³ããèããããã®ãã
人éã®ç¥æµãçµããšæ°ããªäžçãåºããããã§ããã
人éãšããŠã¯ãAIã«è² ããªããæªéççºæ³ããå§åçéãããç®æããããã®ã ã
人é¡ã®é²åã¯ãéã³ãããã¯ããŸãã
ãããªãéã³ããåºæ¥ããªãã°ããšããã¢ã€ãã¢ã«è§Šçºããã人ãã§ãŠã
çæŽ»ã«ã圹ã«ç«ã€ãããªãçºæããçãŸããã®ã ã
ããããããããããïœ ãChar Fuitter (1847ïœ1912 ãªã©ã³ã)ãïœ
2018/06/03 åçäœæãïŒå°æ¥ã®çãåããã®ãããæ¥ä»ãæèšïŒ
ãã®ç©èªã¯ãã£ã¯ã·ã§ã³ã§ãã
ç»å Žãã人ç©ã»å£äœã»åç§°çã¯æ¶ç©ºã§ããã
å®åšã®ãã®ãšã¯é¢ä¿ãããŸããã
Char Fuitter ïŒãã£ãŒã»ãã€ã¿ãŒïŒã¯æ¶ç©ºã®äººç©ã§ãã
