1. ã¯ããã«
ãã®èšäºã§ã¯ãåå®ä»®æ³æ©æ¢°åŠç¿ã«ããããªãæ¹çè©äŸ¡ã«ã€ããŠãçè«çãªåŽé¢ãäžå¿ã«ç޹ä»ããŸãã
2. åå®ä»®æ³æ©æ¢°åŠç¿
åå®ä»®æ³æ©æ¢°åŠç¿ïŒCounterfactual Machine Learning: CFMLïŒãšã¯ãæ©æ¢°åŠç¿ãå¿çšããèµ·ããåŸããå®éã«ã¯èµ·ãããªãã£ãç¶æ³ïŒåå®ä»®æ³ïŒCounterfactualïŒã®æšè«ãªã©ãããæè¡ã§ãããã®æè¡ã¯ãããžãã¹ã«ãããæ¹çïŒè²©ä¿ãã£ã³ããŒã³ã®å®æœãæ°æ©èœã®è¿œå ãªã©ïŒã®å¹æãè©äŸ¡ããã±ãŒã¹ãªã©ã®å¿çšãå«ããæè¿çãã«ç ç©¶ãããŠããæè¡ã§ããäŸãã°ãããåºèãWebããŒãžã§è²©ä¿ãã£ã³ããŒã³ã宿œããå Žåãããä»®ã«ãã£ã³ããŒã³ã宿œããªãã£ãå Žåããå¥ã®ãã£ã³ããŒã³ãè¡ã£ãå Žåã«ãã©ã®çšåºŠã®å£²äžã«ãªã£ãã®ãã®ããŒã¿ãåŸãããšã¯ã§ãã1ãããã®ãã£ã³ããŒã³ã«ãã£ãŠã©ãã ã売äžãå¢ããã®ãïŒããšããåãã«çããããšã¯é£ããã§ããããããåå®ä»®æ³ãèããããšã§ãå®éã«ã¯ãšããªãã£ãæ¹çããšã£ãå Žåã®çµæãäºæž¬ã»æšå®ãããããããšã«ãã£ã³ããŒã³ã®å¹æãæ€èšŒããããšã§ããã£ã³ããŒã³ã®ç¶ç¶ã»äžæ¢ã®å€æãå 容倿Žãªã©ã®æææ±ºå®ã«æŽ»çšããããšãã§ããŸãã
ãã®åå®ä»®æ³ã«ããšã¥ãæšè«èªäœã¯ãåŸæ¥ããçµ±èšçå ææšè«ã®åéã«ãããŠãåäºå®ã¢ãã«ïŒCounterfactual ModelïŒãïŒå²©åŽ(2015)ïŒããæœåšçãªçµæå€æ°ïŒPotential OutcomesïŒãïŒæé(2009)ïŒãªã©ãšãã£ãåœ¢ã§æ±ãããŠãããã®ãšå€§å·®ãããŸãããããããäŸãã°Webãµã€ãäžã§ã®ååã®æšèŠãå人ã屿§ããšã«æé©åããé«åºŠãªããŒãœãã©ã€ãŒãŒã·ã§ã³ãªã©ãããžãã¹ã«ãããŠè€éãªæ¹çãæ±ãããããšãšãã«ããããæ¯ããæè¡ã®çºå±ã«äŒŽã£ãŠãè€éãªéçšãçµãŠåŸãããããŒã¿ãåæããææ®µãšããŠæ©æ¢°åŠç¿ã掻çšãããã±ãŒã¹ãå¢ããŠããŸããåå®ä»®æ³æ©æ¢°åŠç¿ã§æ±ãããããŒãã¯æ§ã ã§ãããæè¿ã§ã¯ç¹ã«ãæ¹çã®æææ±ºå®ã®è©äŸ¡ããã°ããŒã¿ã§è¡ããªãæ¹çè©äŸ¡ïŒOPE: Off-Policy EvaluationïŒãåå®ä»®æ³ã«ããæ©æ¢°åŠç¿ã¢ãã«ã®èª¬æææ³ïŒCounterfactual ExplanationïŒ ã«æ³šç®ãéãŸã£ãŠããå°è±¡ããããŸãããã®èšäºã§ã¯ããã®äžã§ãããªãæ¹çè©äŸ¡ã®ç޹ä»ãããŸãã
3. ãªãæ¹çè©äŸ¡
3-1. ãªãæ¹çè©äŸ¡ãšã¯
ãªãæ¹çè©äŸ¡ãšã¯ãå çšè¿°ã¹ãéããæ¹çã®æææ±ºå®ã®è©äŸ¡ããã°ããŒã¿ã§è¡ãããšãæããŸããæ¹çã®æææ±ºå®ã®è©äŸ¡ã¯ãéåžžA/Bãã¹ãã®ããã«å®éã®ããžãã¹ç°å¢ã§è¡ãããšãå€ãã§ããããããå®éã®ããžãã¹ã«æ¹çãé©çšããã«ã¯ãæ©äŒæå€±ãªã©ã®ãªã¹ã¯ãããäžãã³ã¹ãããããããæ éã«è¡ãå¿ èŠããããŸãããªãæ¹çè©äŸ¡ã¯ãã°ããŒã¿ã掻çšãããããå®éã®ããžãã¹ã«æ¹çãé©çšããéã«çãããªã¹ã¯ãã³ã¹ããåé¿ã»è»œæžããããšãã§ããŸãããŸãããªãæ¹çè©äŸ¡ã®çµæãããšã«æ¹çãéžå®ããããã§å®éã®ããžãã¹ç°å¢ã§è©Šãããšã§ãããå¹ççã«æ¹çãæ±ºå®ããããšãã§ããŸãããªãæ¹çè©äŸ¡ã¯ã匷ååŠç¿ïŒReinforcement LearningïŒãšããŠæããããšãã§ããUehara et al.(2022) ã®ã¬ãã¥ãŒè«æã«ããŸãšããããŠããŸãããªããããã§ããæææ±ºå®ã«çšããã¢ãã«ã¯ãåè¡åãéžæãã確çãç®åºã§ãããã®ãæ³å®ããŠããŸã2ã
次ç¯ä»¥éã§ã¯ã以äžã®è¡šèšã䜿ã£ãŠ3çš®é¡ã®è©äŸ¡æ¹æ³ïŒDirect Methodã»IPSæšå®éã»DRæšå®éïŒã®ç޹ä»ãããŸãããã®è¡šèšã¯ãUehara et al.(2022)ã®ã¬ãã¥ãŒè«æãåèã«ããŠããŸãã
- æ¹çïŒpolicyïŒ
- $\pi$ïŒæ¹ç
- $\pi_0$ïŒçŸè¡ã®æ¹ç
- $V_{\pi}$ïŒæ¹ç $\pi$ ã®äŸ¡å€
- è¡åïŒactionïŒ
- $\mathscr{A}$ïŒè¡åã®éå
- $a (\in \mathscr{A})$ïŒè¡åã®èŠçŽ
- $A$ïŒè¡åã®ç¢ºç倿°
- $a_i$ïŒè¡åã®å®çŸå€
- ç¹åŸŽéïŒcontextïŒ
- $\mathscr{X}$ïŒç¹åŸŽéã®ãã¯ãã«ç©ºé
- $X\sim q(x)$ïŒç¢ºçå¯åºŠé¢æ° $q(x)$ ããåŸãããç¹åŸŽéãã¯ãã«
- $x_i$ïŒç¹åŸŽéãã¯ãã«ã®å®çŸå€
- å ±é
¬ïŒrewardïŒ
- $R$ïŒå ±é ¬ã®ç¢ºç倿°
- $r_i$ïŒå ±é ¬ã®å®çŸå€
- $r(a,x)$ïŒç¹åŸŽé $x$ ãšè¡å $a$ ãããããå ±é ¬
- ããŒã¿
- $D (=\{a_i,x_i,r_i\}_{i=1}^n)$ïŒãã°ããŒã¿ïŒ$n$ ã¯ããŒã¿æ°ïŒ
äžèšã®è¡šèšãçšãããšããªãæ¹çè©äŸ¡ã¯ãçŸè¡ã®æ¹ç $\pi_0$ ã®ããšã§åŸããããã°ããŒã¿ $D$ ãçšããŠãæ¹ç $\pi$ ã®äŸ¡å€ $V_{\pi}$ ãæšå®ããããšããšè¡šçŸã§ããŸããæ¹çã®äŸ¡å€ãšã¯ãå ±é ¬ã®æåŸ å€ $r(a, x) = E[r|x, a]$ ãæããŸãããŸããæ¹ç $\pi$ ã®äŸ¡å€ã®æšå®ã«éããããã§ã¯ä»¥äžã®3ã€ã®ä»®å®ã眮ããŠããŸãã
- ä»®å®1. ç¹åŸŽéãæ¡ä»¶ä»ãããšããè¡åãšå ±é ¬ã¯ç¬ç«ã§ãã3
- ä»®å®2. ç¹åŸŽéãæ¡ä»¶ä»ãããšããè¡åãéžæãã確çã¯0ãã倧ããïŒ$\pi(a |x)>0$
- ä»®å®3. ConsistencyïŒ$R=R(A)$
ä»®å®1.ã¯ãè©äŸ¡èšç®ãæ±ããããããããã«çœ®ãä»®å®ã§ããä»®å®2. ã¯ããªãæ¹çè©äŸ¡ã®èšç®ã®äžã§ç¢ºçã®éæ°ãæ±ãããããã®ç¢ºçã¯0ã«ãªããªããšããä»®å®ã§ããä»®å®3. ã¯ãããè¡åã®å®çŸ©ãææ§ã ãšãåŸãããçµæïŒå ±é ¬ïŒãæ³å®ããŠããè¡åã«ç±æ¥ãããšã¯éããªããªããæç¢ºã«çµæãåºå¥ã§ããªããªããããå ±é ¬ã¯ããè¡åã®å ±é ¬ãšäžè²«ããŠããããšãæ³å®ããŠããŸãã
3-2. Direct Method (DM)
Direct Method (DM) ã¯ãå ±é ¬ãçŽæ¥æšå®ããæ¹æ³ã§ããDMã¯ãç¹åŸŽé $x_i$ ãšè¡å $a$ ãããããå ±é ¬ $r(a, x)$ ãæšå®ããã¢ãã«ãäœæããããã§ããã®æšå®å€ $\hat r(a, x_i)$ ãããšã«æ¹ç $\pi$ ã®äŸ¡å€ãæšå®ããŸããDMã§æšå®ãããæ¹ç $\pi$ ã®äŸ¡å€ $\hat V_\pi^{DM}$ ã¯ä»¥äžã®åŒã§è¡šãããŸãã
\hat V_\pi^{DM}=\frac{1}{n} \sum_{i=1}^n \bigg\{\sum_{a \in \mathscr{A}}~\hat r (a,x_i) \pi(a|x_i)\bigg\}
ããã§ã$\pi(a_i |x_i)$ã¯ãæ¹ç $\pi$ ã«ãããŠç¹åŸŽé $x_i$ ãæäžãšããå Žåã«è¡å $a_i$ ãéžæãã確çã衚ããŸããã€ãŸããäžåŒã® $\sum_{a \in \mathscr{A}}~\hat r (a,x_i) \pi(a|x_i)$ ã¯ã$i$ çªç®ã®ãã°ããŒã¿ã«ãããå ±é ¬ã®æåŸ å€ã衚ããŸãããããŠããã®æåŸ å€ã®å¹³åãæ¹çã®äŸ¡å€ã®æšå®éãšããŠããŸãã
DMã¯å ±é ¬ãæšå®ããã¢ãã«èªäœã¯ç¹ã«æå®ããŠããªããããéåžžã®éååž°ãLightGBMãªã©æ§ã ãªã¢ãã«ã掻çšã§ããŸãããã ããå®éã«éžæãããã芳枬ã§ããªãã£ãè¡åã«å¯Ÿããå ±é ¬ãäºæž¬å€ã§ä»£çšãããããå ±é ¬ã®æšå®ã¢ãã«ã«å¯ŸããŠé«ãæ±åæ§èœãèŠæ±ãããŸãããã®ãããããããéåŠç¿ãèµ·ããªãããã«ãããšãã°Chernozhukov et al.(2018)ã®Cross-fittingãçšããæ¹æ³ããããŸããCross-fittingã®æé ã¯ä»¥äžã®éãã§ãã
- ããŒã¿ $D$ ãåçã« $K$ïŒ2以äžã®æŽæ°ïŒåã«åå²ãïŒK-foldïŒãåå²ãããããŒã¿ã $D_k (k=1,...,K)$ã$D_k$ 以å€ã®ããŒã¿ã $D_{(k)}$ ãšè¡šèšããã
- $D_{(k)}$ ã䜿ã£ãŠã¢ãã« $m(D_{(k)})$ãåŠç¿ããã
- ã¢ãã« $m(D_{(k)})$ ã«ããŒã¿ $D_k$ ãé©çšããæšå®å€ãåŸãã
3-3. IPSæšå®é
IPSïŒInverse Propensity ScoreïŒæšå®é4ã¯ãå ±é ¬ã®å®çŸå€ $r_i$ ã®éã¿ä»ãå¹³åå€ã§å ±é ¬ãæšå®ããæ¹æ³ã§ããIPSæšå®éã¯ä»¥äžã®åŒã§è¡šãããŸãã
\hat V_\pi^{IPS}=\frac{1}{n} \sum_{i=1}^n w(a_i,x_i)r_i
ããã§ãéã¿ãšããŠéèŠåºŠïŒimportanceïŒ $w(a_i,x_i)=\frac{\pi(a_i |x_i)}{\pi_0 (a_i |x_i)}$ ã䜿ã£ãŠããŸãããã®éã¿ã¯ãè©äŸ¡å¯Ÿè±¡ã§ããæ¹ç $\pi$ ã«ãããŠè¡å $a_i$ ãéžæãã確çãããã°ããŒã¿ãååŸã§ããŠããçŸè¡ã®æ¹ç $\pi_0$ ã«ãããŠè¡å $a_i$ ãéžæãã確çã§å²ã£ãå€ã§ãããçŸè¡ã®æ¹çã«ãããç¹å®ã®è¡åã®éžæãããããã«èµ·å ãããã€ã¢ã¹ãé€å»ãã圹å²ãæãããŸãããŸããIPSæšå®éã¯ãDMãšç°ãªãå ±é ¬ãæšå®ããã¢ãã«ã䜿çšããªãã®ã§ãã¢ãã«ã«ç±æ¥ãããã€ã¢ã¹ããªãã®ãç¹åŸŽã§ãã
ã¢ãã«ã«ç±æ¥ãããã€ã¢ã¹ããªãããšãæ°åŒã§ç¢ºèªãããæ¹ã¯ãã¡ãããåèãã ããã
æšå®ãããæ¹çã®äŸ¡å€ã«ã€ããŠæŽçãããšã
\begin{align}
V_{\pi}&=E_{(A,X)\sim \pi(a|x)q(x)}[r(A,X)]\\
&=E_{X\sim q(x)}\big[E_{A|X=x~\sim \pi(a|x)}[r(A,X)|X=x]\big]\\
&=\int_\mathscr{X} \bigg\{\sum_{a\in \mathscr{A}} \thinspace r(a,x)\pi(a|x)\bigg\}q(x)dx\\
&=\int_\mathscr{X} \bigg\{\sum_{a\in \mathscr{A}} \frac{\pi(a |x)}{\pi_0 (a |x)}r(a,x)\pi_0(a|x)\bigg\}q(x)dx\\
&=\int_\mathscr{X} \bigg\{\sum_{a\in \mathscr{A}} w(a,x)r(a,x)\pi_0(a|x)\bigg\}q(x)dx\\
&=E_{(A,X)\sim \pi_0(a|x)q(x)}[w(A,X)r(A,X)]
\end{align}
ãæç«ããŸããIPSæšå®éã¯ã$r(a_i,x_i)=r_i$ã§ããããšã«æ³šæãããšãæåŸã®åŒããã®ãŸãŸå¹³åå€ã«çœ®ãæããæšå®éãšèããããšã§ãIPSæšå®éããã€ã¢ã¹ãªã $V_{\pi}$ ãæšå®å¯èœã§ããããšãããããŸãããŸããIPSæšå®éã§ã¯ $a\in \mathscr{A}$ ã§ã®åãçŽæ¥èšç®ããå¿ èŠããªããæç€ºçã« $r(a,x)$ ã«å¯Ÿããä»®å®ã眮ããŠããªãããšãããããŸãã
äžæ¹ãDM㯠$\int_\mathscr{X} {\sum_{a\in \mathscr{A}} \thinspace r(a,x)\pi(a|x)q(x)}dx$ ããã®ãŸãŸå¹³åå€ã«çœ®ãæããæšå®éã§ããã$r(a,x)$ ã $\hat r(a,x)$ ã§ä»£çšããäžã§ $r(a,x)\pi(a|x)$ ãçŽæ¥èšç®ããŠãããšèããããŸããã€ãŸããDMã§ã¯ $r(a,x)$ ãæ£ããæšå®ã§ããŠããªããã€ãŸã $\hat r(a,x)\neq r(a,x)$ã§ããå Žåã¯ã$V_{\pi}$ ã®æšå®å€ã«ãã€ã¢ã¹ãå ¥ãããšã«ãªããŸãã
ãã ããIPSæšå®éã¯ç¢ºçãéã¿ãšããŠäœ¿çšããŠãããããéã¿ã極端ã«å€§ãããªãããšããããæšå®å€ãå®å®ããªãïŒæšå®å€ã®åæ£ã倧ãããªãïŒããšããããŸããå¯ŸåŠæ³ã¯æ§ã ãããŸãããäŸãã°ãã宿° $\lambda$ 以äžã®å€ãæã€éã¿ã¯ $\lambda$ ã§äžžããŠããŸãclipped-IPSæšå®éããããŸã5ã
\hat V_\pi^{clippedIPS}=\frac{1}{n} \sum_{i=1}^n \min\{w(a_i,x_i ),\lambda\}r_i
3-4. DRæšå®é
DRïŒDoubly RobustïŒæšå®éã¯ãDMãšIPSæšå®éãçµã¿åãããŠå ±é ¬ãæšå®ããæ¹æ³ã§ããDRæšå®éã¯ä»¥äžã®åŒã§è¡šãããŸãã
\hat V_\pi^{DR}=\hat V_\pi^{DM}+\frac{1}{n} \sum_{i=1}^n w(a_i,x_i ) (r_i-\hat r(a_i,x_i))
ãªãDMãšIPSæšå®éãçµã¿åãããã®ããã説æããã«ããã£ãŠãããããæ¹çã®äŸ¡å€ã®æšå®å€ã¯ã©ãããå€ã§ããã°é©åãªã®ããèããŸããæ¹çã®äŸ¡å€ã®æšå®å€ã¯ãçã®äŸ¡å€ãšè¿ãå€ããšãããšãæãŸããã§ããã€ãŸããæ¹çã®äŸ¡å€ã®æšå®å€ã®èª€å·® $MSE = E[(\hat V_\pi - V_\pi)^2]$ ãå°ããããšãæãŸããã§ããããã§ã誀差ã¯ä»¥äžã®ããã«Biasé ãšVarianceé ã«åè§£ããããšãã§ããBiasãšVarianceã®åæ¹ãå°ããããšãæãŸãããšèšããŸãã
\begin{align}
MSE = E[(\hat V_\pi - V_\pi)^2] &= (E[\hat V_\pi - V_\pi])^2 +\{E[(\hat V_\pi - V_\pi)^2]-(E[\hat V_\pi - V_\pi])^2 \}\\
&=Bias^2 +Variance
\end{align}
ãã ããäžè¬çã«BiasãšVarianceã«ã¯ãã¬ãŒããªãã®é¢ä¿ããããŸããå®ã¯ãå çšç޹ä»ããDMã¯Biasã倧ããäžæ¹ã§Varianceãå°ãããIPSæšå®éã¯Varianceã倧ããäžæ¹ã§Biasãå°ããåŸåã«ãããŸããããã§ãDRæšå®éã¯DMãšIPSæšå®éãé©åã«çµã¿åãããããšã§ãBiasãšVarianceã®åæ¹ãå°ããããããšãçã£ãŠããŸããDRæšå®éã¯ãå ±é ¬ãæšå®ããã¢ãã«ã誀ã£ãŠããŠããIPSåæ§ã«ã¢ãã«ã«ç±æ¥ãããã€ã¢ã¹ããªããšããç¹åŸŽããããŸãã
ã¢ãã«ã«ç±æ¥ãããã€ã¢ã¹ããªãããšãæ°åŒã§ç¢ºèªãããæ¹ã¯ãã¡ãããåèãã ããã
æšå®ãããæ¹çã®äŸ¡å€ã«ã€ããŠæŽçãããšã
\begin{align}
V_{\pi}&=\int_\mathscr{X} \bigg\{\sum_{a\in \mathscr{A}} \thinspace r(a,x)\pi(a|x)q(x)\bigg\}dx\\
&=\int_\mathscr{X} \bigg\{\sum_{a\in \mathscr{A}} \thinspace \big[r(a,x) - \hat r(a,x) + \hat r(a,x)\big]\pi(a|x)q(x)\bigg\}dx\\
&=\int_\mathscr{X} \bigg\{\sum_{a\in \mathscr{A}} \thinspace \hat r(a,x)\pi(a|x)q(x)\bigg\}dx\\
&~~~~~~~+\int_\mathscr{X} \bigg\{\sum_{a\in \mathscr{A}} \thinspace [r(a,x) - \hat r(a,x) ]\pi(a|x)q(x)\bigg\}dx\\
&=\int_\mathscr{X} \bigg\{\sum_{a\in \mathscr{A}} \thinspace \hat r(a,x)\pi(a|x)q(x)\bigg\}dx\\
&~~~~~~~+\int_\mathscr{X} \bigg\{\sum_{a\in \mathscr{A}} \frac{\pi(a |x)}{\pi_0 (a |x)}\{r(a,x) - \hat r(a,x)\}\pi_0(a|x)q(x)\bigg\}dx\\
&=\int_\mathscr{X} \{\sum_{a\in \mathscr{A}} \thinspace \hat r(a,x)\pi(a|x)q(x)\}dx\\
&~~~~~~~+E_{(A,X)\sim \pi_0(a|x)q(x)}[w(A,X)\{r(A,X) - \hat r(A,X)\}]
\end{align}
ãæç«ããŸããDRæšå®éã¯ã$r(a_i,x_i)=r_i$ ã§ããããšã«æ³šæãããšãäžçªäžã®åŒã®æšæ¬å¯Ÿå¿ãšããŠèããããšãã§ãã$r(a,x)$ ãæ£ããæšå®ã§ããŠããªããã€ãŸã $\hat r(a,x)\neq r(a,x)$ã§ããå Žåã§ããã€ã¢ã¹ãªãã$V_{\pi}$ ãæšå®å¯èœã§ããããšãããããŸãã
3-5. DRæšå®éã®æŽŸç
DRæšå®éã¯IPSæšå®éãšåãéã¿ãçšããŠãããããéã¿ã極端ã«å€§ãããšãã«æšå®ãå®å®ããªãïŒåæ£ã倧ãããªãïŒããšããããŸãããã®å Žåã«å¯ŸåŠããçºå±åœ¢ã®ææ³ãšããŠãSwitchDRãDRpsïŒDR with pessimistic shrinkageïŒãDRosïŒDR with optimistic shrinkageïŒã玹ä»ããŸãã
3-5-1. SwitchDR
SwitchDRïŒWang et al.(2017)ïŒã¯ãéã¿ã«äœ¿çšããŠããéèŠåºŠã $\lambda$ 以äžãªãDRãããã§ãªããã°DMãçšããŠèšç®ããæ¹æ³ã§ããSwitchDRã¯ä»¥äžã®åŒã§è¡šãããŸãã
\begin{align}
\hat V_\pi^{SwitchDR}&= \frac{1}{n}\sum_{i=1}^n \bigg[I\{w(a_i,x_i )>\lambda\} \sum_{a \in A}\hat r(a,x_i) \pi(a|x_i)\\
&~~~~~~~~~~~~~~~+I\{w(a_i,x_i )\leq \lambda\}\big\{w(a_i,x_i)\big(r_i-\hat r(a_i,x_i)\big) \\
&~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~+\sum_{a\in A}\hat r(a,x_i)\pi(a|x_i)\big\}\bigg]\\
&= \frac{1}{n} \sum_{i=1}^n \bigg[\sum_{a\in A}\hat r(a,x_i)\pi(a|x_i)\\
&~~~~~~~~~~~~~~~~~+I\{w(a_i,x_i )\leq \lambda\}w(a_i,x_i)\big(r_i-\hat r(a_i,x_i)\big)\bigg]\\
\end{align}
ããã§ã$I\{\cdot\}$ã¯ãæ¡ä»¶åŒãæãç«ã€ãšãã«1ãæãç«ããªããšãã«0ããšãæç€ºé¢æ°ã§ããSwitchDRã¯ã倧ããªå€ããšãéèŠåºŠã«å¯ŸããŠé å¥ïŒrobustïŒã§ããã$\lambda\to 0$ãšããå Žåã«DMã$\lambda \to \infty$ãšããå Žåã«DRã«äžèŽããŸãã
3-5-2. DRps
DRpsïŒDR with pessimistic shrinkageïŒïŒSu et al.(2020a)ïŒã¯ãclippedIPSãšåæ§ã®ã¢ã€ãã¢ãçšããŠãDRã«ãããŠéèŠåºŠã $\lambda$ 以äžã®å€ãåãå Žåã« $\lambda$ ã«çœ®ãæããææ³ã§ããDRpsã¯ä»¥äžã®åŒã§è¡šãããŸãã
\hat V_\pi^{DRps}=\hat V_\pi^{DM}+\frac{1}{n} \sum_{i=1}^n \min\{w(a_i,x_i ),\lambda\} (r_i-\hat r(a_i,x_i))
$\lambda\to 0$ãšããå Žåã«DMã$\lambda \to \infty$ãšããå Žåã«DRã«äžèŽããŸãããŸããéã¿ã®çœ®ãæãæ¹ã¯Su et al.(2019)ã®CAB-DRïŒContinuous Adaptive Blending-DRïŒãšåæ§ã§ããã¢ã€ãã¢èªäœã¯ã·ã³ãã«ã§ãããMSEãæå¶ããããã«äžçãèæ ®ããéã¿ã«çç®ãã颿°ã«å¯Ÿãããã¬ãŒãããã³ããè§£ã圢ã§å°åºãããæšå®éã§ããã€ãŸããæ°ãã«å°å ¥ããéã¿ã $\hat w(a_i,x_i)$ã$\hat r(a,x)$ ãæšå®ããéã«äœ¿çšããããŒã¿ã«å¯Ÿããéã¿ã $z(a,x)$ ãšãããšãã$\hat w(a_i,x_i)$ ã«ããšã¥ãMSEã®äžçã
E\big[\{\hat w(a_i,x_i)-w(a,x)\}^2 / z(a,x)\big]L(\hat r)+\sqrt{L(\hat r)E\big[\hat w(a_i,x_i)^2w(a,x)^2/ z(a,x)\big]}+\frac{1}{n}\\
(ãã ãL(\hat r)=E[z(a,x)\{r-\hat r(a,x)\}^2])
ãšãªãããšã«çç®ããããã$\hat w(a_i,x_i)$ãããšã«æå°åããå€ç®çæé©ååé¡ãšèŠåããäžã§ã
2\lambda E\big[|\hat w(a_i,x_i)-w(a,x)|\big]+E\big[\hat w(a_i,x_i)^2\big]
ãæå°ã«ããè§£ãšããŠå°åºãããŠããŸãïŒSu et al.(2020a)ïŒãçŽæ¥äžçãæå°ã«ãªãããã«è§£ããŠããããã§ã¯ãªããããpessimisticãšããåç§°ãä»ããŠããŸãã
3-5-3. DRos
DRosïŒDR with optimistic shrinkageïŒïŒSu et al.(2020a)ïŒã¯ãDRpsãšåæ§ã«MSEã®äžçãèæ ®ããŸãããçŽæ¥MSEã®äžçã®æå°åãè¡ãæ¹æ³ã§éã¿ãå°åºããææ³ã§ããDRpsã¯ä»¥äžã®åŒã§è¡šãããŸãã
\hat V_\pi^{DRos}=\hat V_\pi^{DM}+\frac{1}{n} \sum_{i=1}^n \frac{\lambda}{w^2 (a_i,x_i )+\lambda} w(a_i,x_i) (r_i-\hat r(a_i,x_i))
$\lambda\to 0$ãšããå Žåã«DMã$\lambda \to \infty$ãšããå Žåã«DRãšäžèŽããŸãããŸããDRosã§çšããéã¿ã«ã€ããŠã$w(a_i,x_i)>0$ ã§ããããã$\frac{\lambda}{w^2 (a_i,x_i )+\lambda} w(a_i,x_i) < w(a_i,x_i)$ ãæãç«ã¡ãŸããDRosã§ã¯ã
\lambda E\big[\{\hat w(a_i,x_i)-w(a,x)\}^2 / z(a,x)\big]+E\big[\hat w(a_i,x_i)^2w(a,x)^2/ z(a,x)\big]
ã®æå°åãèãã$\hat w(a_i,x_i)$ã«ããäžé埮åã0ã«ãªããšãã®è§£ãå°åºããŠãããçŽæ¥MSEã®äžçã®æå°åãèããŠãããããoptimisticãšããåç§°ãä»ããŠããŸãã
4. ãŸãšã
ãã®èšäºã§ã¯ãåå®ä»®æ³æ©æ¢°åŠç¿ã«ããããªãæ¹çè©äŸ¡ã玹ä»ããŸãããä»åã¯3-1. ã§è¿°ã¹ã3ã€ã®ä»®å®ãæç«ããŠããå Žåã«çŠç¹ãåœãŠãŸããããä»®å®ãæãç«ããªããšã6ã¯ã©ããã¹ãããªã©ãå®åã§é©çšããå Žåã«èµ·ããåŸãåé¡ã«å¯Ÿå¿ããåæææ³ãç ç©¶ãããŠããŸãããŸãã玹ä»ããããªãã£ããã®ãšããŠãããçºå±çãªæšå®ææ³ïŒäŸãã°Metelli et al.(2021)ã®IS-$\lambda$ã»DR-$\lambda$ïŒããåå²ããããŒã¿ãããšã«æé©ãªæšå®éãéžæããæ¹æ³ïŒUdagawa et al.(2022)ïŒãªã©ããããŸãã
次åã¯ãä»å玹ä»ããææ³ãçšããå®è£ äŸã玹ä»ããäºå®ã§ãã
5. åèæç®
ïŒè±èªïŒã¢ã«ãã¡ãããé ïŒâæ¥æ¬èªïŒãããããé ïŒã§èšèŒïŒ
Cross-fittingã«é¢ããè«æïŒChernozhukov, V., Chetverikov, D., Demirer, M., Duflo, E., Hansen, C., Newey, W. and Robins, J. (2018), Double/debiased machine learning for treatment and structural parameters. The Econometrics Journal, 21: C1-C68.
æ ¹æ¬åé¡ã«é¢ããè«æïŒHolland, P. W. (1986). Statistics and causal inference. Journal of the American statistical Association, 81(396), 945-960.
IS-$\lambda$ã»DR-$\lambda$ã«é¢ããæç®ïŒMetelli, A. M., Russo, A., & Restelli, M. (2021). Subgaussian and differentiable importance sampling for off-policy evaluation and learning. Advances in Neural Information Processing Systems, 34, 8119-8132.
clipped-IPSã«é¢ããæç®ïŒStrehl, A., Langford, J., Li, L., & Kakade, S. M. (2010). Learning from logged implicit exploration data. Advances in neural information processing systems, 23.
DRosãDRpsã«é¢ããæç®(Su et al.(2020a))ïŒSu, Y., Dimakopoulou, M., Krishnamurthy, A., & Dudik, M. (2020). Doubly robust off-policy evaluation with shrinkage. In International Conference on Machine Learning (pp. 9167-9176). PMLR.
CAB-DRã«é¢ããæç®ïŒSu, Y., Wang, L., Santacatterina, M., & Joachims, T. (2019). CAB: Continuous adaptive blending for policy evaluation and learning. In International Conference on Machine Learning (pp. 6005-6014). PMLR.
åå²ããããŒã¿ãããšã«æé©ãªæšå®éãéžæããæ¹æ³ã«é¢ããæç®ïŒUdagawa, T., Kiyohara, H., Narita, Y., Saito, Y., & Tateno, K. (2022). Policy-Adaptive Estimator Selection for Off-Policy Evaluation. arXiv preprint arXiv:2211.13904.
匷ååŠç¿ã«ããããªãæ¹çè©äŸ¡ã«é¢ããã¬ãã¥ãŒè«æïŒUehara, M., Shi, C., & Kallus, N. (2022). A Review of Off-Policy Evaluation in Reinforcement Learning. arXiv preprint arXiv:2212.06355.
SwitchDRã«é¢ããè«æïŒWang, Y. X., Agarwal, A., & Dudık, M. (2017). Optimal and adaptive off-policy evaluation in contextual bandits. In International Conference on Machine Learning (pp. 3589-3597). PMLR.
åå®ä»®æ³æ©æ¢°åŠç¿ã«é¢ããæžç±ïŒéœè€åªå€ª, å®äºç¿å€ª. (2021). æœçãã¶ã€ã³ã®ããã®æ©æ¢°åŠç¿å ¥é-ããŒã¿åææè¡ã®ããžãã¹æŽ»çšã«ãããæ£ããèãæ¹. æè¡è©è«ç€Ÿ
芳å¯ããŒã¿ã«å¯Ÿããçµ±èšçå ææšè«ã«é¢ããæ¬ïŒæéåŽå®. (2009). 調æ»èгå¯ããŒã¿ã®çµ±èšç§åŠ-å ææšè«ã»éžæãã€ã¢ã¹ã»ããŒã¿èå. 確çãšæ å ±ã®ç§åŠ, 岩波æžåº
çµ±èšçå ææšè«ã«é¢ããæ¬ïŒå²©åŽåŠ. (2015). çµ±èšçå ææšè«. çµ±èšã¹ã¿ã³ããŒã, æåæžåº
-
ããã¯å ææšè«ã«ãããæ ¹æ¬åé¡ïŒthe Fundamental Problem of Causal InferenceïŒãšåŒã°ãããã®ïŒHolland(1986)ïŒã«èµ·å ããŸããããªãã¡ããããåäœã«å¯ŸããåŠçœ®ãé©çšããçµæãšé©çšããªãã£ãçµæãåæã«èŠ³æž¬ããããšã¯ã§ããªããããããåäœãžã®åŠçœ®ã®å¹æãã®ãã®ã芳枬ããããšã¯äžå¯èœã§ããããšããåé¡ã«ããããã£ã³ããŒã³ã宿œããå Žåãšå®æœããªãã£ãå Žåã®çµæã¯åæã«ã¯èŠ³æž¬ã§ããŸããããã®ãããå®éã®åæã§ã¯A/Bãã¹ãã®ããã«ããŒã¿ã®åééçšã工倫ããããCausalImpactã®ããã«ããŒã¿ã«é¢ããŠä»®å®ã眮ããäžã§è§£æããææ³ãçšããããšã§ã芳枬ã§ããªãçµæãæšå®ããã¢ãããŒãããšãããšãäžè¬çã§ãã â©
-
äŸãã°ã«ãŒã«ããŒã¹ã§ã®æææ±ºå®ã§ãã確ççãªæ¡ä»¶ã«ãããã£ãŠè¡åãæ±ºãŸãã®ã§ããã°åè¡åã®ç¢ºçãèšç®å¯èœãªã®ã§åé¡ãããŸããããã ãããã°ãæ®ããŠããªããªã©ã®çç±ã§ãçŸè¡ã®æ¹çã«ãããåè¡åã®ç¢ºçãããããªãå Žåã¯ãå¥éæšå®ããå¿ èŠããããŸãããã®å Žåã«é¢ããŠã¯æ¬èšäºã§ã¯æ±ããŸããããäŸãã°ãåè¡åã®ç¢ºçã®æšå®ã¢ãã«ãçã®ã¢ãã«ãšäžèŽããªãå Žåãããã«èµ·å ãããã€ã¢ã¹ãè©äŸ¡çµæã«å«ãŸããŠããŸãããšãã£ã課é¡ããããŸãã â©
-
確ç倿°ãçšãã衚çŸããããšãä»»æã® $a$ ã«ã€ããŠã$R(a)$ ãš $A$ ã $X$ ãæ¡ä»¶ã¥ããå Žåã«ç¬ç«ãšãªãããšãæå³ããŸããèšãæãããšã$X$ ãæ¡ä»¶ã¥ããå Žå $A=a'(a\neq a')$ ã§ãããšãã«æœåšçãªçµæ $R(a)$ ã«ã¯åœ±é¿ãäžããªãïŒæœåšçã«èããŠããè¡å $a$ ã®çµæã¯ã»ãã®è¡å $a'(a\neq a')$ ã«äŸåããªãïŒããšã衚ããŸãã â©
-
IPWïŒInverse Probability WeightingïŒæšå®éãšåŒã°ããããšããããŸãããçµ±èšçå ææšè«ã®æèã§ææ¡ãããŠããIPWæšå®éãšã®æ··ä¹±ãé¿ãããããããã§ã¯IPSæšå®éãšåŒã¶ããšã«ããŠããŸãã â©
-
ããã§ã¯Su et al.(2020a)ã§ã®å®çŸ©ã«åãããŠããŸãã â©
-
äŸãã°ãååã®ã¬ã³ã¡ã³ããWebæ€çŽ¢ãèãããšãäžäœã®ååãæ€çŽ¢çµæããæåã®ç»é¢ã«è¡šç€ºãããªããããã¹ã¯ããŒã«ããªãéãäžäœã®ãã®ã¯é²èЧããã確çã0ã«ãªãåŸãŸãã â©