
ãSieve-Lensã7-bit芳枬空éãšäžå¯èŠã³ã³ãã³ãæ€åºã«ããæ±ºå®è«çããã³ããã¯ã¬ã³ãžã³ã°
人éã«èŠããææžãšãæ©æ¢°ãè§£æããææžã¯åãã§ã¯ãªã
ç§ãã¡ã¯ãææžãç®ã§èŠãŠèªã¿ãŸãã
äžæ¹ãAIè©äŸ¡ã·ã¹ãã ãèªå審æ»ããã°ã©ã ã¯ãå ¥åãããã¹ãã»æ§é ã»ã¡ã¿ããŒã¿ãªã©ãšããŠåŠçããŸãã人éãç®ã§ç¢ºèªããææžãšãæ©æ¢°ãå®éã«åãåãæ å ±ã¯ãå¿ ãããäžèŽããŸããã
ãã®å·®ã¯ãæå³çã«å©çšãããå¯èœæ§ããããŸãã
ããšãã°ã次ã®ãããªã±ãŒã¹ã§ãã
ãŒãå¹ æåïŒ èŠãç®ã«ã¯ã»ãšãã©å€åããªãæåããããã¹ãå éšã«åã蟌ãŸããŠããã
åæ¹åå¶åŸ¡æåïŒ æååã®è¡šç€ºé åºãšãå éšã®æååé åºã«å·®ãçããã
HTMLã»CSSã«ããé èœïŒ ææžã®ãœãŒã¹ã«ã¯ååšããããã¹ãããéåžžã®è¡šç€ºã§ã¯èŠããªãã
ã³ã¡ã³ããã¡ã¿ããŒã¿ïŒ æ¬æãšã¯å¥ã®å Žæã«ã远å ã®æç€ºã説æãæ ŒçŽãããŠããã
ç°æåäœç³»ã®æ··åšïŒ èŠãç®ã®äŒŒãã©ãã³æåã»ããªã«æåã»ã®ãªã·ã£æåããåãåèªã®äžã§æ··åšããŠããã
ãããã¯ãã¹ãŠåãåé¡ã§ã¯ãããŸãããæååã®å éšã«ååšããäžå¯èŠæåãšãææžæ§é ã«ãã£ãŠé ãããããã¹ãã§ã¯ãæ€æ»æ¹æ³ã蚌æ ã®ç€ºãæ¹ãç°ãªããŸãã
Sieve-Lensã¯ããã®ãããªäººéã®èŠèªæ§ã𿩿¢°ã®å¯èªæ§ã®ä¹é¢ã芳枬ããããã«éçºãããæ±ºå®è«çãªããã³ãã芳枬ãšã³ãžã³ã§ãã
GitHub: https://github.com/neguseatama/sieve-lens
ãSieve Lensããšããåç§°ã¯ãã¬ã³ãºã®ããã«ãèçŒã§ã¯èŠããªããã®ã å¯èŠåãããšããèšèšææ³ã«ç±æ¥ããŸãã
çŸä»£ã® AI 審æ»ã·ã¹ãã ã¯ãææžãããã¹ããšããŠè§£æããŸãã
人éã¯ãææžãã¬ã³ããªã³ã°çµæãšããŠèªã¿ãŸãã
ãã®2ã€ã®èŠç¹ã¯åãã§ã¯ãªãããã®å·®ã¯æªçšããåŸãŸãã
ãŒãå¹ æå (U+200B ãªã©)ïŒã¢ãã«ã¯èªãããã人éã«ã¯èŠããªãã
CSS é èœ (display:none)ïŒäººéã«ã¯èŠããªãããããŒãµã¯èªããã
垯åå€ãã£ãã«ïŒã³ã¡ã³ãã»ã¡ã¿ããŒã¿ã»ããããŒã»è泚ïŒïŒããŒãµã¯èªãããã人éã¯èŠèœãšãã
Sieve Lens ã¯ããã®ä¹é¢ã芳枬ããŸãã
æªæã®æç¡ã¯å€å®ããŸãããäœçœ®æ
å ±ãšãã³ãŒãçµæãæç€ºãã æçµå€æã人éã«å§ããŸãã
Sieve-Lensã®åœ¹å²ã¯ãæ»æã®æå®ãã§ã¯ãªãã蚌æ ã®æç€ºã
Sieve-LensãéèŠããŠããã®ã¯ãææžã«æªæããããã©ãããæšæž¬ããããšã§ã¯ãããŸããã
ããæåãååšããããšãäžå®ã®å¯åºŠã§åºçŸããããšãéè¡šç€ºç¶æ ã®èŠçŽ ã«ããã¹ããå«ãŸããããšãã³ã¡ã³ããã¡ã¿ããŒã¿ã«äžå®ä»¥äžã®é·ãã®å 容ãããããšããããã芳枬å¯èœãªäºå®ããæç€ºçãªã«ãŒã«ã«åŸã£ãŠèšé²ããŸãã
ãã®èšèšã§ã¯ã次ã®3ã€ãåºå¥ããŸãã
èŠ³æž¬ïŒ å ¥åã«äœãååšããŠãããã調ã¹ãã
è§£éïŒ èŠ³æž¬ãããæ å ±ããã©ã®ãããªæå³ããªã¹ã¯ãæã€ããèããã
åŠçœ®ïŒ å¿ èŠã«å¿ããŠã察象ã®é€å»ã»çœ®æã»éé¢ãªã©ãå®è¡ããã
Sieve-LensãçŽæ¥æ ãã®ã¯ãäž»ãšããŠæåã®æ®µéã§ãã
äžå¯èŠæåãå«ãŸããŠããŠããæ£åœãªçšéã§ããå¯èœæ§ã¯ãããŸããHTMLã®ã³ã¡ã³ãããã¡ã¿ããŒã¿ããããèªäœãæ»æãšã¯éããŸããã
ãããã£ãŠãèŠ³æž¬çµæããã®ãŸãŸãæ»ææåããæªæããããšèªã¿æ¿ããããšã¯ããŸãããäœçœ®æ å ±ãæ€åºå 容ã確èªããåŸç¶ã®åŠçã人éã®å€æã«ã€ãªããŸãã
ãã®å¢çãå®ãããšãã説æå¯èœæ§ãšèª€å€å®ãžã®å¯ŸåŠãäž¡ç«ããããã®åºæ¬æ¹éã§ãã
ð§© Sieve ã·ãªãŒãº
Sieve ã¯ã決å®çã»ãŒãäŸåã»å®å
šããŒã«ã«ãšããå
±éã®èšèšååãå
±æãã
ãšã³ãžã³çŸ€ã§ããåãšã³ãžã³ã 1 ã€ã®ãªã¹ã¯ã®èŠ³æž¬ãŸãã¯æé€ãæ
ãã
çžäºã«æ¥ç¶ããŸã:
| sieve-core-engine | ãšã³ã·ã¹ãã ã®åºç€ããžã㯠/ ãã©ã°ã€ã³ãã¹ã | â |
| sieve-lens | ææžå
ã®äžå¯èŠã³ã³ãã³ãã»é ãããã³ããã芳枬 | â `--from-lens` çµç±ã§ redact ãž |
| sieve-scope | æ§é çã«é¡äŒŒããã³ãŒããã¢ãæ€åº | â |
| sieve-referee | èšãæãã»æµçšãããããã¹ããã¢ãæ€åº | â |
| sieve-redact | æ©å¯é åãã¬ã·ãŒãã€ãã§ãã¹ãã³ã° (ããã¹ã & PNG) | â lensãâ tray |
| sieve-tray | ãã¡ã€ã«ãé©åãªãšã³ãžã³ãžæ¯ãåãããã¹ã¯ãããããã³ããšã³ã | â å
šãšã³ãžã³ |
芳枬ããæé€ãŸã§ 1 ãã¹ã§: Lens ã人éã«èŠããªããã®ãèŠã€ãã
Redact ããä»ã¯äœãå€ãã£ãŠããªããããšã蚌æããã¬ã·ãŒããšãšãã«
é€å»ããŸãã
ð 7-bit芳枬空éïŒç°ãªã蚌æ ãå ±éã®åœ¢åŒã§è¡šçŸãã
Sieve-Lensã¯ãææžã®ç¶æ ã7ã€ã®ä»®èª¬ãã©ã°ãH1ãH7ãšããŠè¡šçŸããŸãã
ããã§ããã7-bitãã¯ãææžå šäœã7ãããã«å§çž®ããŠåŸ©å ãããšããæå³ã§ã¯ãããŸãããç°ãªãçš®é¡ã®èŠ³æž¬çµæãã7ã€ã®0/1ãã©ã°ã«æŽçããããã®è¡šçŸã§ãã
åãããã¯ç¬ç«ãã芳枬æ¡ä»¶ãæã¡ãŸãããã¹ãŠã®ããããåãçš®é¡ã®ãªã¹ã¯ã衚ãããã§ã¯ãªããèªã¿èŸŒã¿å¯èœæ§ãäžå¯èŠæåãææžæ§é ãæåäœç³»ã®æ··åšãªã©ãç°ãªãåŽé¢ãæ±ããŸãã
ããã仮説芳枬察象H1Parseabilityææžã®èªã¿èŸŒã¿ã»æœåºãæåãããH2Zero-Width DensityãŒãå¹ æåã®å¯åºŠãèšå®éŸå€ä»¥äžãH3Bidi Controlsåæ¹åå¶åŸ¡æåãååšãããH4Format ConcealmentHTML/CSSãç»åã¬ã€ã€ãŒãªã©ã«é èœã®èšŒæ ããããH5Out-of-Band Channelã³ã¡ã³ããã¡ã¿ããŒã¿ãªã©ãæ¬æä»¥å€ã®é åã«äžå®ä»¥äžã®å 容ããããH6Script MixingåãããŒã¯ã³å ã«ç°ãªãæåäœç³»ãæ··åšãããH7Contiguous Payloadäžå¯èŠæåãèšå®é·ä»¥äžã«é£ç¶ããŠããã
芳枬ãã¹ã¯ã¯ã次ã®åœ¢åŒã§åºåãããŸãã
H1H2H3H4-H5H6H7
ããšãã°ã1101-101 ã¯ãH1ã»H2ã»H4ã»H5ã»H7ã1ã§ãããH3ã»H6ã0ã§ããããšã衚ããŸãã
ãã ããããã¯7ã€ã®èŠ³æž¬æ¡ä»¶ã®çµæã䞊ã¹ããã®ã§ãããæ»æç¢ºçãç·åçãªå±éºåºŠãçŽæ¥è¡šãã¹ã³ã¢ã§ã¯ãããŸããããããã®æ°ãæ°ããã ãã§ãææžã®å®å šæ§ãé äœä»ãããããšãæ³å®ããŠããŸããã
H1ïŒParseability â ãããã芳枬ã§ããã
H1ã¯ãå¯Ÿè±¡ææžã®æœåºåŠçãæ£åžžã«å®äºãããã衚ããŸãã
èªã¿èŸŒã¿ã»æœåºã«å€±æããå ŽåãSieve-Lens㯠0000-000 ãè¿ããŸãããã®ãã¹ã¯ã¯ããã¹ãŠã®æ€æ»ã§åé¡ããªãã£ãããšããæå³ã§ã¯ãããŸããã芳枬ãã®ãã®ãæç«ããªãã£ãããšã瀺ãçµæã§ãã
ãã®ããããã¹ã¯ã ãã§ãªãèšºææ å ±ã確èªããããšãéèŠã§ããè§£æäžèœãªææžããåé¡ã®ãªãææžãšåãæ±ãã«ããŠã¯ãããŸããã
H2ïŒZero-Width Density â èŠããªãæåã®å¯åºŠ
ãŒãå¹ æåã¯ãéåžžã®è¡šç€ºã§ã¯èŠèªãã«ããäžæ¹ãæååããŒã¿ã«ã¯ååšããŸãã
Sieve-Lensã®æ¢å®èšå®ã§ã¯ã察象ãšãªããŒãå¹ æåã®æ°ãæ¬ææåæ°ã§å²ãããã®å¯åºŠã1%以äžã®å Žåã«H2ã1ãšããŸãã
[
D_{\mathrm{ZW}} =
\frac{N_{\mathrm{zero\text{-}width}}}
{N_{\mathrm{body}}}
]
æ¢å®ã®éŸå€ã¯ 0.01 ã§ãã
å¯åºŠãå©çšããçç±ã¯ãåçŽãªååšç¢ºèªãšãææžå ã§ã©ã®çšåºŠäœ¿ãããŠããããåºå¥ããããã§ãããã ãã1%ãšããéŸå€ã¯èšèšäžã®èšå®å€ã§ãããããããææžã«ãããŠæªæã®æç¡ãåããæ®éçãªå¢çã§ã¯ãããŸããã
H3ïŒBidi Controls â 衚瀺é åºã«åœ±é¿ããå¶åŸ¡æå
åæ¹åããã¹ãã®å¶åŸ¡æåã¯ãæåã®è¡šç€ºé åºã«é¢ãããŸãã
ãããã¯ãå³ããå·Šãžèšè¿°ããèšèªãªã©ãæ£åœãªããã¹ãåŠçã«ãå©çšãããŸããäžæ¹ãæååã®å éšé åºãšè¡šç€ºçµæã®å·®ãæªçšããäœå°ããããŸãã
Sieve-Lensã¯ã察象ãšãªãåæ¹åå¶åŸ¡æåãæ¬æã«ååšãããã確èªããŸããååšãæ€åºããããšãšããã®å©çšãäžæ£ã§ãããšå€æããããšã¯å¥ã§ãã
H4ïŒFormat Concealment â ææžæ§é ã衚瀺圢åŒã«ããé èœ
ããã¹ããã®ãã®ã«ç°åžžããªããŠããææžã®æ§é ã«ãã£ãŠå 容ãé ãããŠããå ŽåããããŸãã
HTML/CSSã§ã¯ãããšãã°æ¬¡ã®ãããªæå®ãæ€æ»å¯Ÿè±¡ã«ãªããŸãã
<div style="display:none">
éåžžã®è¡šç€ºã§ã¯èŠããªãããã¹ã
</div>Sieve-Lensã¯ãHTMLã®éè¡šç€ºå±æ§ãã¹ã¿ã€ã«ãªã©ã調ã¹ãé èœã®èšŒæ ã芳枬ããŸããæ¡åŒµæ©èœãå°å ¥ããå Žåã¯ãCSSã»ã¬ã¯ã¿ã®è§£æ±ºãç»åã®ééã¬ã€ã€ãŒã»äœã³ã³ãã©ã¹ããªã©ã察象ã«ã§ããŸãã
éèŠãªã®ã¯ãH4ããé ãããå 容ã¯å¿ ãæªæããæç€ºã§ããããšæå³ããããã§ã¯ãªãç¹ã§ãããã¶ã€ã³äžã®é衚瀺èŠçŽ ããæ£åœãªç»ååŠçãããåŸãŸããæ€åºçµæã¯ã察象èŠçŽ ãäœçœ®æ å ±ãšãšãã«ç¢ºèªããŸãã
H5ïŒOut-of-Band Channel â æ¬æä»¥å€ã®æ å ±ãèŠèœãšããªã
ææžã®å 容ã¯ãç»é¢äžã«è¡šç€ºãããæ¬æã ãã«ååšãããšã¯éããŸããã
Sieve-Lensã¯ã察å¿ããæœåºåšãéããŠãHTMLã³ã¡ã³ããDOCXã³ã¡ã³ããææžããããã£ãHTMLã® altã»titleã»aria-label ãªã©ã芳枬察象ã«ããŸãã
æ¢å®ã®èšå®ã§ã¯ã察象é åã®ããã¹ãã10æå以äžã®å Žåã«H5ã1ãšããŸãã
ããã¯çãååãã¿ã€ãã«ãªã©ãæ©æ¢°çã«æŸãäžããããªãããã®åºæºã§ãããã ããçãå 容ãç¡å®³ã ãšããæå³ã§ãã10æåæªæºã«ãªã¹ã¯ãååšããªããšããæå³ã§ããããŸããã
ãã®ä»®èª¬ã®çãã¯ãæ¬æã ããæ€æ»ããŠãããšèŠèœãšããããæ å ±æºããæç€ºçã«æ€æ»å¯Ÿè±¡ãžå«ããããšã§ãã
H6ïŒScript Mixing â èŠãç®ã®äŒŒãæåãåºå¥ãã
ç°ãªãæåäœç³»ã«ã¯ãèŠãç®ããã䌌ãæåãååšããŸãã
Sieve-Lensã®H6ã¯ãåã空çœåºåãããŒã¯ã³ã®äžã«ã©ãã³æåãšããªã«æåããŸãã¯ã©ãã³æåãšã®ãªã·ã£æåãæ··åšããŠããå Žåã芳枬ããŸãã
ããšãã°ãèŠãç®ã䌌ãŠããŠããæåã³ãŒãäžã¯ç°ãªãæåãäžã€ã®èå¥åãåèªã«æ··ãã£ãŠããå¯èœæ§ããããŸãã
ãã ããè€æ°ã®æåäœç³»ã䜿ãããšèªäœã¯ç°åžžã§ã¯ãããŸããã察象èšèªãåºæåè©ã«ãã£ãŠã¯èªç¶ã«çºçããŸããH6ã¯æåäœç³»ã®æ··åšã瀺ããã®ã§ããããªãããŸããæ»æã確å®ãããã®ã§ã¯ãããŸããã
H7ïŒContiguous Payload â äžå¯èŠæåãé£ç¶ããŠããã
äžå¯èŠæåã1æåååšããã±ãŒã¹ãšãäžå®ã®é·ãã§é£ç¶ããŠããã±ãŒã¹ã¯ãåãã§ã¯ãããŸããã
Sieve-Lensã¯ã察象ãšãªãäžå¯èŠæå矀ã®é£ç¶é·ã調ã¹ãæ¢å®ã§ã¯16æå以äžã®é£ç¶ãæ€åºããå Žåã«H7ã1ãšããŸãã
察象ã«ã¯ããŒãå¹ æåãåæ¹åå¶åŸ¡æåã«å ããã¿ã°æåãããªãšãŒã·ã§ã³ã»ã¬ã¯ã¿ãªã©ãå«ãŸããŸãã
ããã«ãç¹å®ã®ãŒãå¹ æåã®äžŠã³ã«ã€ããŠã¯ãU+200Bã0ãU+200Cã1ãšãããããåãšããŠè§£éããé·ãã8ã®åæ°ã§ããå Žåã«ASCIIãžã®ãã³ãŒãã詊ã¿ãŸãã
ããã¯äžå¯èŠæåã®é£ç¶ããåãªãæåæ°ãšããŠæ±ãã ãã§ãªãã笊å·åãããå 容ãååšããå¯èœæ§ã調ã¹ãããã®ä»çµã¿ã§ãããã ãããã³ãŒãã§ããããšã ãã§ããã®å å®¹ãæ»æã§ãããšæå®ããããã§ã¯ãããŸããã
| H1 | Parseability | ææžãæ£åžžã«èªã¿èŸŒããã |
| H2 | Zero-Width Density | ãŒãå¹
æåã®å¯åºŠãéŸå€è¶
é |
| H3 | Bidi Controls | åæ¹åå¶åŸ¡æåã®ååš |
| H4 | Format Concealment | CSS ã«ããé èœïŒdisplay:none çïŒ |
| H5 | Out-of-Band Channel | ã³ã¡ã³ãã»ã¡ã¿ããŒã¿ã»ããããŒã»ããã¿ãŒã»è泚ã«å®è³ªçå
容 |
| H6 | Script Mixing | åäžããŒã¯ã³å
ã®ç°æåäœç³»æ··åš |
| H7 | Contiguous Payload | éŸå€é·ä»¥äžã®é£ç¶ããäžå¯èŠæå |
ðŠ å¯Ÿå¿åœ¢åŒ
| `.txt` / `.md` | ãã¬ãŒã³ããã¹ããå
šæåã¹ãã£ã³ | ãªãïŒCoreïŒ |
| `.html` / `.htm` | ã€ã³ã©ã€ã³ style 屿§ããã³ `<style>` ãããã¯ã®é èœæ€åºããªãã·ã§ã³ã§å€éš CSS 解決 | ãªãïŒCoreïŒïŒ`tinycss2`ïŒãªãã·ã§ã³ïŒ |
| `.docx` | æ¬æã»ã³ã¡ã³ãã»core ããããã£ã»ããããŒã»ããã¿ãŒã»èæ³šã»æ«å°Ÿè泚 | ãªãïŒCoreïŒ |
| `.pdf` | æ¬æã»ã¡ã¿ããŒã¿ã»æ³šéã»äžå¯èŠããã¹ãïŒTr 3ïŒ | `pypdf`ïŒãªãã·ã§ã³ïŒ |
| `.png` / `.jpg` / `.jpeg` / `.gif` / `.bmp` / `.tif` / `.tiff` / `.webp` | ã¡ã¿ããŒã¿ïŒEXIF / PNG tEXt / XMPïŒïŒ OCR ããã¹ã | `Pillow`ã»`pytesseract` ïŒ tesseract ãã€ããªïŒèšèªããã¯ïŒãªãã·ã§ã³ïŒ |
é察å¿ïŒé³å£°ãåç»
ð¥ äž»ãªç¹åŸŽ
7-bit 芳枬空éïŒH1ãH7ïŒ
äžå¯èŠã³ã³ãã³ãã®çš®é¡ããšã«ç¬ç«ã»æ±ºå®è«çã«èŠ³æž¬ããŒãäŸå
Python æšæºã©ã€ãã©ãªã®ã¿ããªãã©ã€ã³åäœããã¬ã¡ããªãªããæ±ºå®è«
åäžå ¥åã¯ãã©ã®ç°å¢ã§ãåžžã«åäžã®èŠ³æž¬çµæãè¿ãã説æå¯èœæ§
ãã¹ãŠã®èŠ³æž¬ã«äœçœ®æ å ±ãšãã³ãŒãçµæãä»äžã芳枬ã§ãã£ãŠå€å®ã§ã¯ãªã
ãšã³ãžã³ã¯èšŒæ ãæç€ºããã®ã¿ããæ»æã§ããããšã¯å®£èšããªããèªå·±å®çµå HTML ã¬ããŒãïŒv0.1 以éïŒ
PDF 察å¿ïŒ`pypdf` çµç±ãv0.2 以éïŒ
ç»å OCR 察å¿ïŒ`Pillow` + `pytesseract` çµç±ãv0.3 以éïŒ
OCR èšèªã¯ `install(engine, ocr_lang="eng+jpn")` ã§èšå®å¯èœïŒæ¢å®ã¯ `eng`ïŒã€ã³ã¿ã©ã¯ãã£ã HTML ããã·ã¥ããŒãïŒv0.5 以éïŒ
ãã£ã¬ã¯ããªåäœã§äžæ¬èŠ³æž¬ãããµããªãŒã«ãŒãã»ãã¹ã¯ååžã»ä»®èª¬çºåãã£ãŒãã»
ãœãŒãå¯èœãªãã¡ã€ã«äžèЧãå«ãåäžã®èªå·±å®çµå HTML ãçæããŸãã
`Jinja2` + `Plotly` ãå¿ èŠã§ããå°å·å¯èœ PDF ã¬ããŒãïŒv0.6 以éïŒ
ãã£ã¬ã¯ããªåäœã§ A4 PDF ã¬ããŒããçæããŸãããµããªãŒã»ä»®èª¬ããŒãã«ã»
ãã¹ã¯ååžã»è²åãããããã¡ã€ã«äžèЧãå«ã¿ãŸãã`weasyprint` ãå¿ èŠã§ã
ïŒãã£ãŒãã«ã¯ãªãã·ã§ã³ã§ `kaleido`ïŒãç»åã¬ã€ã€ãŒè§£æïŒv0.7 以éïŒ
ç»åå ã®ãã¯ã»ã«ã¬ãã«ã®é èœãæ€åºããŸããéé RGBA ã¬ã€ã€ãŒïŒéãŒã RGBïŒã
ããã³ WCAG ã³ã³ãã©ã¹ãæ¯ 2:1 æªæºã®äœã³ã³ãã©ã¹ãããã¹ãã察象ãšããŸãã
`Pillow` ãå¿ èŠã§ããCSS ã»ã¬ã¯ã¿å®å šè§£æ±ºïŒv0.8 以éïŒ
`cssselect2` ã«ãããåå«ã»åã»å åŒã»å±æ§ã»ç䌌ã¯ã©ã¹ã»ã¬ã¯ã¿ãå«ã
å®å šãª CSS ãããã³ã°ã HTML æœåºåšã«è¿œå ããŸããé èœã«ãŒã«ã¯ã
å®éã«èŠçŽ ãžãããããå Žåã®ã¿å ±åãããŸããã¬ããŒãå€èšèªåïŒi18nïŒïŒv0.9 以éïŒ
ããã·ã¥ããŒããš PDF ã¬ããŒããè±èªã»æ¥æ¬èªã§åºåã§ããŸãã
å ¥åææžã®å 容ã«åºã¥ããèšèªãèªåæ€åºããŸããPDF åã蟌ã¿ç»åè§£æïŒv0.10 以éïŒ
PDF ããŒãžã«åã蟌ãŸããç»åãæœåºããåç»åã« OCRã»ééã¬ã€ã€ãŒæ€åºã»
äœã³ã³ãã©ã¹ãæ€åºã»ã¡ã¿ããŒã¿è§£æãé©çšããŸãã
`pypdf` + `Pillow` ãå¿ èŠã§ãã
決å®è«çã§ããããšã®æå³
Sieve-Lensã®èšèšã§ã¯ã確ççãªåé¡ã¢ãã«ã«å€å®ãå§ããã®ã§ã¯ãªããæç€ºçãªã«ãŒã«ãšéŸå€ãçšããŠèŠ³æž¬ããŸãã
åãå ¥åãåãã«ãŒã«ãåãæœåºæ¡ä»¶ã§ããã°ãèŠ³æž¬çµæãåçŸãããããªããŸããåºåããããã¹ã¯ã蚌æ ããå ¥åææžãšç §åããŠæ€èšŒã§ããããšãéèŠããŠããŸãã
ãã®æ§è³ªã¯ã次ã®ãããªå Žé¢ã§åœ¹ç«ã¡ãŸãã
ç£æ»ïŒ äœãæ ¹æ ã«èŠ³æž¬çµæãåºãã®ãã远跡ããã
ååž°ãã¹ãïŒ ã³ãŒã倿Žã«ãã£ãŠèŠ³æž¬çµæãæå³ããå€åããŠããªããã確èªããã
ååŠçïŒ åãã«ãŒã«ãé©çšããŠçµæãæ¯èŒããã
ãšã³ãžã³é£æºïŒ åŸç¶åŠçã«ãèªç±åœ¢åŒã®èª¬æã ãã§ãªãæ§é åãããèŠ³æž¬çµæãæž¡ãã
ãã ããæ±ºå®è«çã§ããããšã¯ãã©ã®ãããªç°å¢ã»å ¥åã»æ¡åŒµæ©èœã§ãå¿ ãåãçµæã«ãªããšããç¡æ¡ä»¶ã®ä¿èšŒã§ã¯ãããŸãããæœåºåšãå€éšã©ã€ãã©ãªãOCRãšã³ãžã³ãèšèªããã¯ãªã©ã®éãã¯ãæœåºå 容ã«åœ±é¿ããå¯èœæ§ããããŸãã
ãã®ãããåçŸæ§ãè©äŸ¡ãããšãã¯ãå ¥åã ãã§ãªãã䜿çšããããŒãžã§ã³ãæ¡åŒµæ©èœãèšå®æ¡ä»¶ãåãããŠç®¡çããããšãéèŠã§ãã
ãŒãäŸåã®ã³ã¢ãšãæ¡åŒµå¯èœãªæœåºã¢ãŒããã¯ãã£
Sieve-Lensã®ã³ã¢ã¯ãPythonæšæºã©ã€ãã©ãªã®ã¿ã§åäœããããã«èšèšãããŠããŸããåºæ¬çãªå¯Ÿå¿åœ¢åŒã¯ããã¬ãŒã³ããã¹ããMarkdownãHTMLãDOCXã§ãã
äžæ¹ãå®éã®ææžã«ã¯PDFãç»åãå«ãŸããŸãããã®ãããSieve-Lensã§ã¯æœåºåšãæ¡åŒµããæ§é ãæ¡çšããŠããŸãã
察象䞻ãªèŠ³æž¬å 容远å äŸåã®äŸTXT / Markdownæ¬æã®æåã»äžå¯èŠæåã³ã¢ã®ã¿HTMLæ¬æã屿§ãã³ã¡ã³ããé衚瀺ã¹ã¿ã€ã«é«åºŠãªCSSè§£æã¯ä»»æDOCXæ¬æãã³ã¡ã³ããããããã£ãããããŒã»ããã¿ãŒãªã©åºæ¬æœåºã¯ã³ã¢ã®ã¿PDFæ¬æãã¡ã¿ããŒã¿ã泚éãäžå¯èŠããã¹ãpypdfç»åOCRããã¹ããã¡ã¿ããŒã¿ãç»åã¬ã€ã€ãŒPillowãpytesseractãTesseractãªã©
æ¡åŒµæ©èœã®æç¡ã«ãã£ãŠã察å¿ç¯å²ãååŸã§ãã蚌æ ã¯ç°ãªããŸããããŒãäŸåããšããååã¯ã³ã¢ã«å¯Ÿãããã®ã§ããããã¹ãŠã®æ¡åŒµæ©èœãæšæºã©ã€ãã©ãªã ãã§å®è¡ã§ãããšããæå³ã§ã¯ãããŸããã
ãŸããææžåœ¢åŒã«ãã£ãŠæœåºå¯èœãªæ å ±ãç°ãªããŸããããšãã°OCRã¯ç»åå ã®æåãèªã¿åãããã®ææ®µã§ãããç»åã«å«ãŸãããã¹ãŠã®æ å ±ãå®å šã«åŸ©å ã§ããããã§ã¯ãããŸããã
ãã®èšèšã«ãããåºæ¬æ©èœã®è»œéããç¶æããªãããå¿ èŠã«å¿ããŠèŠ³æž¬ç¯å²ãæ¡åŒµã§ããŸãã
Sieve-Redactãšã®é£æºïŒèŠ³æž¬çµæãåŠçœ®ã«ã€ãªãã
芳枬ã ãã§ã¯ãææžãã¯ã¬ã³ãžã³ã°ããããšã«ã¯ãªããŸããã
Sieve-Lensã¯ãæ€åºããäžå¯èŠæåãé¡äŒŒæåãªã©ã®æ å ±ããäœçœ®ã»çš®é¡ã»åºçŸåæ°ãªã©ã®æ§é åãããããŒã¿ãšããŠåºåã§ããŸããsieve_lens_ext.redact_bridge ã¯ããã®èŠ³æž¬æ å ±ãSieve-Redactãå©çšããã«ã¿ãã°åœ¢åŒãžå€æããããã®é£æºæ©èœã§ãã
åŠçã®æµãã¯ãæŠå¿µçã«ã¯æ¬¡ã®ããã«ãªããŸãã
å
¥åææž
â
âŒ
Sieve-Lens
â
ââ äžå¯èŠæåã»é èœã»æåäœç³»ã®æ··åšã芳枬
ââ äœçœ®æ
å ±ããã³ãŒãçµæãèšé²
ââ 芳枬ã«ã¿ãã°ãåºå
â
âŒ
Sieve-Redact
â
ââ 察象æåã®åé€ã»çœ®æ
ââ 倿Žå
容ã®ã¬ã·ãŒããåºåãã®åé¢ã«ã¯æå³ããããŸãã
芳枬ãšã³ãžã³ããäœãèŠã€ããããèšé²ããããšãåŠçœ®ãšã³ãžã³ãããã®èšé²ãããšã«äœã倿Žããããæããã«ããããšã圹å²ãåããããšã§ãæ€åºãšå€æŽãæ··åããã«æ±ããŸãã
Sieve-Redactã¯ã察象ã«å¿ããŠäžå¯èŠæåãåé€ãã察å¿ããé¡äŒŒæåã眮æãããªã©ã®åŠçœ®ãè¡ããŸããåŠçœ®åŸã«ã¯å芳枬ããããšã§ãå¯Ÿè±¡ã®æ€åºæ¡ä»¶ãè§£æ¶ããããã確èªããããšãã§ããŸãã
ãã ããå芳枬ã§ç¹å®ã®ãã©ã°ã0ã«ãªã£ããšããŠããããã¯ãã®æ€æ»æ¡ä»¶ã«è©²åœããªããªã£ãããšã瀺ãã®ã§ãã£ãŠãææžå šäœã®å®å šæ§ã蚌æãããããšãæå³ããŸããããŸããåŠçœ®ã«ãã£ãŠå ã®æå³ãæžåŒãå€ããå¯èœæ§ãããããã倿Žèšé²ãšåæãšã®æ¯èŒãéèŠã§ãã
Sieve-Lensã®èŠ³æž¬ãšãSieve-Redactã®åŠçœ®ãåããããšã§ãåãªãæååã®åé€ã§ã¯ãªããäœãæ ¹æ ã«å€æŽãããã远跡ã§ããã¯ã¬ã³ãžã³ã°å·¥çšãç®æããŠããŸãã
ãã¹ããšå質ä¿èšŒïŒå®è£ ã®æ£ãããç¶ç¶çã«æ€èšŒãã
決å®è«çãªãšã³ãžã³ã«ãšã£ãŠããã¹ãã¯è£å©çãªãã®ã§ã¯ãããŸããã芳枬ã«ãŒã«ãæå³ããéãã«åãããšã確èªããå°æ¥ã®å€æŽã«ããäºæããªãæåãæããããã®åºç€ã§ãã
Sieve-Lensã§ã¯ãã³ã¢ã®ä»®èª¬å€å®ã«å ããHTMLæœåºãPDFãç»åOCRãç»åã¬ã€ã€ãŒãCSSã»ã¬ã¯ã¿ãRedactãšã®é£æºãªã©ãæ©èœããšã«ãã¹ããç©ã¿éããŠããŸãã
ãŸããéå»ã®å®è£ ã§ã¯ãæœåºåŠçã®äžå ·åã«ãã£ãŠHTMLæ¬æã®èŠ³æž¬ãåããªããPDFå ã®åã蟌ã¿ç»åè§£æãæ©èœããªããã¬ããŒãåºåã«å¯ŸããHTMLãšã¹ã±ãŒããäžååã§ããããšãã£ãåé¡ãä¿®æ£ããŠããŸããã
ããããä¿®æ£å±¥æŽã瀺ããŠããã®ã¯ãæ€åºã«ãŒã«ãå®çŸ©ããã ãã§ã¯äžååã ãšããããšã§ããå ¥åã®æœåºã蚌æ ã®çæãã¬ããŒãã®è¡šç€ºãåŸç¶ãšã³ãžã³ãšã®åãæž¡ããŸã§å«ããŠãå®éã«åäœããããšãæ€èšŒããå¿ èŠããããŸãã
ãªãããã¹ããééããŠããããšã¯ããã¹ãã§æ±ã£ãæ¡ä»¶ã«å¯Ÿããåè³ªã®æ ¹æ ã«ã¯ãªããŸãããæªç¥ã®æ»æææ³ãå«ããã¹ãŠã®å ¥åã«å¯Ÿããå®å šãªå®å šæ§ãä¿èšŒãããã®ã§ã¯ãããŸããã
æ¢ç¥ã®éçãšãä»åŸã®èª²é¡
Sieve-Lensã¯ãäžå¯èŠã³ã³ãã³ãã®èŠ³æž¬ç¯å²ãåºããããã®ãšã³ãžã³ã§ããããã¹ãŠã®åé¡ã解決ãããã®ã§ã¯ãããŸããã
1 æå³ãæå³ãçŽæ¥å€å®ããªã
å¯èŠã®æç« ã ãã§æ§æãããæªæããæç€ºããæå³äžã®æ¬ºçã¯ãäžå¯èŠã³ã³ãã³ãã®æ€åºã ãã§ã¯å€æã§ããŸãããéã«ãäžå¯èŠæåãé èœèŠçŽ ãèŠã€ãã£ãŠããå¿ ãããäžæ£ãªæå³ããããšã¯éããŸããã
2 æœåºã§ããæ å ±ã«ã¯éçããã
ãã¡ã€ã«åœ¢åŒãæœåºåšãå€éšã©ã€ãã©ãªãOCRã®ç²ŸåºŠãªã©ã«ãã£ãŠã芳枬ã§ããç¯å²ã¯ç°ãªããŸããæå·åãããææžãæªå¯Ÿå¿ã®æ§é ãè€éãªã¬ã³ããªã³ã°åŠçãªã©ã¯ã远å ã®æ€æ»ãå¥ã®ææ®µãå¿ èŠã«ãªãå ŽåããããŸãã
3 éŸå€ã¯èšèšäžã®å€æã§ãã
ãŒãå¹ æåã®å¯åºŠãé£ç¶ããäžå¯èŠæåã®é·ããæ¬æå€ããã¹ãã®æå°é·ãªã©ã¯ãæ€åºæ¡ä»¶ãæç¢ºã«ããããã®èšå®ã§ãããããã¯ããããææžã«å ±éãã絶察çãªå®å šåºæºã§ã¯ãªããçšéã«å¿ããŠæ€èšŒã»èª¿æŽããäœå°ããããŸãã
4 æªç¥ã®ææ³ãèªåçã«ç¶²çŸ ã§ããããã§ã¯ãªã
æ°ããUnicodeæåãç°ãªãé èœææ³ãæªç¥ã®ç¬Šå·åãã¿ãŒã³ãç»å Žããå Žåãæ¢åã®ã«ãŒã«ã ãã§ã¯ååã«èŠ³æž¬ã§ããªãå¯èœæ§ããããŸããæ€åºå¯Ÿè±¡ã®è¿œå ããã¹ãã±ãŒã¹ã®æ¡å ã察å¿åœ¢åŒã®æ€èšŒãç¶ç¶ããå¿ èŠããããŸãã
ä»åŸã¯ãå®éã®ææžãçšããè©äŸ¡ã誀æ€åºãšèŠéãã®åæãæœåºçµæã®å¯èŠåãæ¡åŒµæ©èœéã®äžè²«æ§ãç°ãªãç°å¢ã§ã®åçŸæ§ãªã©ãéããŠã芳枬ãšã³ãžã³ãšããŠã®ä¿¡é Œæ§ãé«ããŠãããããšèããŠããŸãã
Sieveãšã³ã·ã¹ãã ã®äžã§
Sieve-Lensã¯ãSieveã·ãªãŒãºã®äžã§ãææžã®å ¥å段éã«ãããäžå¯èŠã³ã³ãã³ãã®èŠ³æž¬ãæ ããŸãã
Sieve-LensïŒ äººéã«ã¯èŠãã«ããæ å ±ããæ¬æå€ã®æ å ±ã芳枬ããã
Sieve-RedactïŒ èŠ³æž¬çµæãå©çšããŠã察象ã®åé€ã»çœ®æãšå€æŽèšé²ãæ ãã
Sieve-CoreïŒ å ±éã®åºç€ããžãã¯ããã©ã°ã€ã³ãã¹ããæäŸããã
Sieve-ScopeïŒ ã³ãŒãã®æ§é çãªé¡äŒŒæ§ã芳枬ããã
Sieve-RefereeïŒ ããã¹ãã®èšãæããæµçšãªã©ã芳枬ããã
Sieve-TrayïŒ ãã¡ã€ã«ãé©åãªãšã³ãžã³ãžæ¯ãåããããã®ããã³ããšã³ããæ ãã
åãšã³ãžã³ã¯ãåãåé¡ãå¥ã®æ¹æ³ã§è§£ãããšããã®ã§ã¯ãªããããããç°ãªã芳枬ã»åŠçœ®ã®è²¬åãæã€ããšãç®æããŠããŸãã
å ¥åã«äœãå«ãŸããŠãããã芳枬ããããšãèŠ³æž¬çµæãæ ¹æ ãšããŠåŠçœ®ããããšãåŠçœ®ã«ãã£ãŠäœãå€ãã£ãããèšé²ããããšã
ãã®æµããåé¢ããçžäºã«æ¥ç¶ã§ãã圢ã«ããããšããSieveã·ãªãŒãºå šäœã®èšèšæ¹éã§ãã
ãããã«ïŒèŠããªããã®ãã倿å¯èœãªèšŒæ ãž
Sieve-Lensã®åºçºç¹ã¯ãAIã®å€æãçãããšã§ã¯ãããŸããã
人éã確èªããã€ããã®å ¥åãšãæ©æ¢°ãå®éã«è§£æããå ¥åãšã®éã«å·®ãååšãããªãããã®å·®ã芳枬å¯èœã«ããå¿ èŠããããŸãã
7-bit芳枬空éã¯ãç°ãªãçš®é¡ã®èšŒæ ãå ±éã®åœ¢åŒã«æŽçããããã®æ çµã¿ã§ããæ±ºå®è«çãªã«ãŒã«ã¯ãçµæã远跡ããåæ€èšŒããããã®åºç€ã§ããSieve-Redactãšã®é£æºã¯ã芳枬ãå®éã®åŠçœ®ã«ã€ãªããããã®ä»çµã¿ã§ãã
ãã¡ããã芳枬ã§ããããšãšããã¹ãŠãçè§£ã§ããããšã¯åãã§ã¯ãããŸãããæ€åºã§ããããšãšãå®å šãä¿èšŒã§ããããšãåãã§ã¯ãããŸããã
ã ãããããSieve-Lensã¯æçµå€æã眮ãæããã®ã§ã¯ãªãããã®å€æã«å¿ èŠãªèšŒæ ãæç€ºããããšãç®æããŸãã
Sieve-Lensã¯ãææžãè£ãããã®ãšã³ãžã³ã§ã¯ãªããææžã®äžã«äœãããã®ããã倿ã§ããåœ¢ã§æããã«ããããã®ã¬ã³ãºã§ãã
é¢é£ãªã³ã¯
Sieve ãšã³ã·ã¹ãã
Sieveã·ãªãŒãºå šäœã®èšèšææ³ãåãšã³ãžã³ã®åœ¹å²ã芳枬空éã®é¢ä¿ã«ã€ããŠã¯ãã¢ãŒããã¯ãã£ã€ã³ããã¯ã¹ãåç §ããŠãã ããã
Sieve ãšã³ã·ã¹ãã ã¢ãŒããã¯ãã£ã€ã³ããã¯ã¹ïŒnoteïŒ
PyPI â https://pypi.org/user/neguseatama/
GitHub ãªããžããª
Sieve-Core â æ±ºå®è«çãªæ§é éçŽã»ç¢ºä¿¡åºŠè©äŸ¡ã®åºç€ãšã³ãžã³
Sieve-Referee â ããã¹ãã®æ§é ã»æµçšã»èšãæãã芳枬ãããšã³ãžã³
Sieve-Scope â ãœãŒã¹ã³ãŒãã®æ§é çé¡äŒŒæ§ã芳枬ãããšã³ãžã³
Sieve-Lens â äžå¯èŠã³ã³ãã³ããé èœèŠçŽ ã芳枬ãããšã³ãžã³
Sieve-Redact â æå®é åã®æ±ºå®è«çãã¹ãã³ã°ãšå®å šæ§æ€èšŒ
Sieve-Order â ããã³ããã®è¡šå±€æ§é ã6-bit芳枬空éã§èŠ³æž¬ãããšã³ãžã³
Sieve-Tray â åãšã³ãžã³ãçµ±åããããŒã«ã«ããã³ããšã³ã
note æè¡èšäº
Sieve-ScopeïŒæ±ºå®è«çã¹ã³ãŒã管çãšã³ã³ããã¹ãå¢çã®å®çŸ©
Sieve-RedactïŒæ±ºå®è«çãã¹ãã³ã°ãšä»ãã€ãå®å šæ§æ€èšŒã«ããããŒã¿ä¿å š
Sieve-TrayïŒè€æ°ã®èŠ³æž¬ç©ºéãçµ±åããããŒã«ã«æ€èšŒããã³ããšã³ã
èšèšã®èšé²(ææèšäº)
Sieve ã·ãªãŒãºã®æ°çã¢ãã« â å°åºæ³ãšå®å®æ§
ã·ãªãŒãºã®ååŒããå£ãæ¹ã®å®æž¬ãããã©ãå°ãããããã
æèšç®ã§è¿œããç²åºŠã§è§£èª¬ãå°åºã® 3 段æé ãšèšèšãã§ãã¯ãªã¹ãä»ããSieve ã·ãªãŒãºã®æ§é èšèš â äŸåæ¹åãã¢ãŒããã¯ãã£ã決ãã
åäžãã¡ã€ã«ã»ãã©ã°ã€ã³ã»ã«ã¿ãã°çé¢ã»byte pinã5 ã€ã®æ§é 倿ã®
çç±ãšåäŸããããŠãã»ãŠã¹ã®è¹ããå§ãŸã£ãéçºçµç·¯ã