(ITEVXQIRXJSV)ZEPYEXMSR
ERH-RXIVREP%YHMX
7MHE7XYHMIWMR)ZEPYEXMSR
8SSI|OMDQGHVWXGLHDY
6LGDVUHVXOWDWDQDO\VHU
+¸VER7GLMPP
6LGD6WXGLHVLQ(YDOXDWLRQ 'HSDUWPHQWIRU(YDOXDWLRQ DQG,QWHUQDO$XGLW
8SSI|OMDQGHVWXGLHDY6LGDV
UHVXOWDWDQDO\VHU
*|UDQ6FKLOO6:(',6+,17(51$7,21$/'(9(/230(17&223(5$7,21$*(1&< $GGUHVV66WRFNKROP6ZHGHQ2IILFH6YHDYlJHQ6WRFNKROP 7HOHSKRQH7HOHID[ 7HOHJUDPVLGDVWRFNKROP3RVWJLUR² (PDLOLQIR#VLGDVH+RPHSDJHKWWSZZZVLGDVH 6LGD6WXGLHVLQ(YDOXDWLRQLVDVHULHVFRQFHUQHGZLWKFRQFHSWLDODQGPHWKRGRORJLFDO LVVXHVLQWKHHYDOXDWLRQRIGHYHORSPHQWFRRSHUDWLRQ,WLVSXEOLVKHGE\6LGD·V 'HSDUWPHQWIRU(YDOXDWLRQDQG,QWHUQDO$XGLW5HSRUWVPD\EHRUGHUHGIURP %LVWnQGVIRUXP6LGD 66WRFNKROP 3KRQH )D[ (PDLOLQIR#VLGDVH+RPHSDJHKWWSZZZVLGDVH $XWKRU*|UDQ6FKLOO 7KHYLHZVDQGLQWHUSUHWDWLRQVH[SUHVVHGLQWKLVUHSRUWDUHWKHDXWKRU·VDQGGRQRWQHFHVVDULO\UHIOHFW WKRVHRIWKH6ZHGLVK,QWHUQDWLRQDO'HYHORSPHQW&RRSHUDWLRQ$JHQF\6LGD 6LGD6WXGLHVLQ(YDOXDWLRQ &RPPLVVLRQHGE\6LGD'HSDUWPHQWIRU(YDOXDWLRQDQG,QWHUQDO$XGLW &RS\ULJKW6LGDDQGWKHDXWKRU 5HJLVWUDWLRQ1R 'DWHRI)LQDO5HSRUW'HFHPEHU 3ULQWHGLQ6WRFNKROP6ZHGHQ ,661³;
,QQHKnOOVI|UWHFNQLQJ
,QWURGXNWLRQ cWJlUGHUI|UI|UElWWUDGNYDOLWHW %HG|PQLQJDY6LGDVUHVXOWDWDQDO\VHU +XUP\FNHWELVWnQGDQDO\VHUDV" 9LONDUHVXOWDWDQDO\VHUDV" +XUDQDO\VHUDVUHVXOWDWHQ" bUUHVXOWDWDQDO\VHUQDWLOOI|UOLWOLJD" bUUHVXOWDWDQDO\VHUQDDQYlQGEDUD" 5HNRPPHQGDWLRQHU %LODJD$879VNODVVLILFHULQJDYUHVXOWDWDQDO\VHUQDVXSSJLIWHU1
8SSI|OMDQGHVWXGLHDY6LGDVUHVXOWDWDQDO\VHU
,QWURGXNWLRQ
8QGHUK|VWHQJMRUGH6HNUHWDULDWHWI|U8WYlUGHULQJRFK,QWHUQ5HYLVLRQ 879SnXSSGUDJDY6LGDVJHQHUDOGLUHNW|U%R*|UDQVVRQHQJUDQVNQLQJDYGH UHVXOWDWDQDO\VHUVRPGLWWLOOVSURGXFHUDWVLVDPEDQGPHG6LGDVODQGVWUDWHJLDUEHWH 879VJUDQVNQLQJSHNDGHSnHWWDQWDOEULVWHUDYVHHQGHUHVXOWDWDQDO\VHUQDVNYDOLWHW PHQRFNVnSnGHJHQHUHOODSUREOHPVRPI|UNQLSSDVHOOHUERUGHI|UNQLSSDVPHG VW\UQLQJDYRIIHQWOLJYHUNVDPKHWSnJUXQGYDODYWLGLJDUHUHVXOWDWRFKHUIDUHQKHWHU ,VDPEDQGPHGnUVJUDQVNQLQJEHVWlPGHVDWW879VNXOOHJ|UDHQ XSSI|OMDQGHVWXGLHDYGHUHVXOWDWDQDO\VHUVRPIUDPVWlOOWVDY6LGDXQGHU )|UHOLJJDQGHUDSSRUWlUUHVXOWDWHWDYGHQQDDQGUDVWXGLH*UDQVNQLQJDUQDlUHWW OHGL879VDUEHWHXQGHUWHPDW´)|UlQGULQJRFK/lUDQGHLQRP6LGDµ 5DSSRUWHQLQOHGVPHGHQVDPPDQIDWWQLQJDYGHnWJlUGHUVRP6LGDYLGWDJLWXQGHU SHULRGHQLV\IWHDWWI|UElWWUDUHVXOWDWDQDO\VHUQDVNYDOLWHWGHQIRUWVlWWHU PHGHQEHG|PQLQJDYGHUHVXOWDWDQDO\VHUVRPIUDPVWlOOWVXQGHUVDPPDSHULRG RFKGHQDYVOXWDVPHGWUHVDPPDQIDWWDQGHUHNRPPHQGDWLRQHU5DSSRUWHQlU JHQRPJnHQGHGLVSRQHUDGSnHWWVlWWVRPP|MOLJJ|UHQNODMlPI|UHOVHUPHG879V I|UVWDJUDQVNQLQJcWJlUGHUI|UI|UElWWUDGNYDOLWHW
8QGHUXWDUEHWDGHUHJLRQDYGHOQLQJDUQDVPHWRGJUXSSDQYLVQLQJDUI|U UHVXOWDWDQDO\VHUQDVIUDPVWlOOQLQJIRUPDWRFKLQQHKnOO%HVOXWRPDQYLVQLQJDUQD IDWWDGHVDY$VLHQDYGHOQLQJHQ(QOLJWEHVOXWHWVNDHQUHVXOWDWDQDO\VEHVWnDY • HQUHVXOWDWUHGRYLVQLQJLPDWULVIRUPPHGNRVWQDGVLQIRUPDWLRQRFKXSSJLIWHURP SODQHUDWRFKIDNWLVNWYHUNVDPKHWVXWIDOOVDPWUHVXOWDW • HQlPQHVPlVVLJUHVXOWDWDQDO\VGYVHQlPQHVPlVVLJWLQGHODGRFKSn UHVXOWDWUHGRYLVQLQJHQJUXQGDGDQDO\VDYXWYHFNOLQJVVDPDUEHWHWVUHVXOWDWRFK 6FKLOO*|UDQ*UDQVNQLQJDYUHVXOWDWDQDO\VHUQDL6LGDVODQGVWUDWHJLDUEHWH6LGD6WXGLHVLQ(YDOXDWLRQ 1LFDUDJXD6UL/DQND9LHWQDP9lVWEDQNHQ*D]D)LOLSSLQHUQD/DRV6\GDIULNDRFK1DPLELD 5HVXOWDWDQDO\VHUQDI|U(VWODQG/HWWODQGRFK/LWDXHQKDUH[NOXGHUDWVIUnQ879VVWXGLHHIWHUVRP GHLQWHNRQFHQWUHUDVWLOOHQJLYHQWLGVSHULRG'lUPHGVDNQDVHQDYXWJnQJVSXQNWHUQDI|U879V VWXGLHQlPOLJHQIDNWLVNDXWEHWDOQLQJDUXQGHUHQYLVVWLGVSHULRG 5HJLRQDYGHOQLQJDUQDV0HWRGJUXSS$QYLVQLQJDUI|UUHVXOWDWDQDO\V2 • HQVDPPDQIDWWDQGHUHVXOWDWDQDO\VVRPEDVHUDVSnGHQlPQHVPlVVLJD UHVXOWDWDQDO\VHQRFKVRPKHOWHOOHULVLQDYlVHQWOLJDGHODULQJnUL ODQGVWUDWHJLGRNXPHQWHW (QOLJWEHVOXWHWVNDGHWYnVLVWQlPQGDDQDO\VGHODUQDNRQFHQWUHUDVWLOOVOXWVDWVHURP ELVWnQGHWVVDPPDQWDJQDUHVXOWDWSUHVWDWLRQHURFKHIIHNWHUVDPWRPGHWVlWWSn YLONHWJMRUGDHUIDUHQKHWHUE|USnYHUNDLQULNWQLQJSnRFKLQQHKnOOLQlVWNRPPDQGH ODQGVWUDWHJL (QWLOOI|UOLWOLJRFKXWW|PPDQGHUHVXOWDWUHGRYLVQLQJEHWUDNWDVVnOHGHVVRPHQ JUXQGOlJJDQGHI|UXWVlWWQLQJI|UEnGHlPQHVPlVVLJRFKVDPPDQIDWWDQGH UHVXOWDWDQDO\V'HWlUGRFNYLNWLJWDWWQRWHUDDWWUHGRYLVQLQJHQLQWHHIWHUIUnJDVDY UHJHULQJHQ'HWlUVQDUDUH6LGDVHJHQWRONQLQJDYUHJHULQJHQVLQVWUXNWLRQDWW UHVXOWDWUHGRYLVQLQJLPDWULVIRUPDWlUHQYLNWLJI|UXWVlWWQLQJI|UWLOOI|UOLWOLJ UHVXOWDWDQDO\VLODQGVWUDWHJLSURFHVVHQ $QYLVQLQJDUQDI|UUHVXOWDWUHGRYLVQLQJLPDWULVIRUPDWlUHQNODRFKW\GOLJD )RUPDWHWPHGNROXPQHUI|UNRUWIDWWDGHNRPPHQWDUHUNRVWQDGVXWIDOORFK XSSJLIWHURPPnOVlWWQLQJDURFKUHVXOWDWSnSUHVWDWLRQVRFKHIIHNWQLYnlUULPOLJW PHGWDQNHSnDWWUHGRYLVQLQJHQLI|UVWDKDQGVNDDQYlQGDVVRPHQV\VWHPDWLVN JUXQGI|UGHSnVWnHQGHQVRPJ|UVLGHlPQHVPlVVLJDRFKVDPPDQIDWWDQGH DQDO\VGHODUQD 0HGWDQNHSnUHVXOWDWDQDO\VHUQDVUROOL6LGDVODQGVWUDWHJLSURFHVVHUlUGHW DQJHOlJHWDWWPHWRGJUXSSHQVDQYLVQLQJDUJHQRP*'EHVOXWJHVVWDWXVVRP YHUNVJHPHQVDPPDULNWOLQMHUI|UUHVXOWDWDQDO\VHUQDVIUDPVWlOOQLQJIRUPRFKLQQHKnOO 'HWlULGDJWLOOH[HPSHORNODUWnWPLQVWRQHI|U879KXUXYLGD6LGDgVWVND RPIDWWDVDY$VLHQDYGHOQLQJHQVEHVOXW 0LQVWOLNDYLNWLJWlUDWWDQYLVQLQJDUQDJHVHQP|MOLJKHWDWWEOLOnQJVLNWLJWKnOOEDUD /LYVOlQJGHQI|UWLGLJDUHULNWOLQMHU6$03%$%6RFK5(6$VLQVWUXNWLRQIUnQ KDULQWHYDULWPHUlQHWWSDUnU'HQQDRPE\WOLJKHWKDUNQDSSDVWYDULW J\QQVDPI|UP\QGLJKHWHQVDUEHWVUXWLQHURFKSURGXNWLYLWHWLODQGVWUDWHJLSURFHVVHQ 'HWNDQJLYHWYLVILQQDVVNlODWWILQVOLSDRFKYLGDUHXWYHFNODP\QGLJKHWHQVULNWOLQMHU I|UUHVXOWDWDQDO\V879NDQGRFNNRQVWDWHUDDWWSUREOHPHQPHGGH UHVXOWDWDQDO\VHUVRPXWIRUPDWVPHGKMlOSDYPHWRGJUXSSHQVDQYLVQLQJDUVQDUDUH KDUDWWJ|UDPHGGHDQVYDULJDDPEDVVDGHUQDVRFKDYGHOQLQJDUQDVWLOOlPSQLQJlQ PHGDQYLVQLQJDUQDVRPVnGDQD
%HG|PQLQJDY6LGDVUHVXOWDWDQDO\VHU
,OLNKHWPHG879VI|UVWDJUDQVNQLQJJ|UVNYDOLWHWVEHG|PQLQJHQDY6LGDV UHVXOWDWDQDO\VHUXWLIUnQIHPIUnJHVWlOOQLQJDU • +XUP\FNHWELVWnQGDQDO\VHUDV" 7HOHIRQVDPWDOPHG)UHGULN6YHGlQJ3 • 9LONDUHVXOWDWDQDO\VHUDV" • +XUDQDO\VHUDVUHVXOWDWHQ" • bUUHVXOWDWDQDO\VHUQDWLOOI|UOLWOLJD" • bUUHVXOWDWDQDO\VHUQDDQYlQGEDUD" +XUP\FNHWELVWnQGDQDO\VHUDV" )UnJDQNDQEHVYDUDVSnROLNDVlWW879KDUGHODWLQUHVXOWDWDQDO\VHUQDVXSSJLIWHUL WUHNDWHJRULHUH[HPSHOnWHUILQQVL%LODJD$ • XSSJLIWHUVRPVSHFLILFHUDUXWEHWDOQLQJDUQDVVWRUOHNNDWHJRUL • XSSJLIWHURPXWEHWDOQLQJDUVRPlYHQLGHQWLILHUDUGHQ6LGDILQDQVLHUDGH YHUNVDPKHWHQNDWHJRULRFK • XSSJLIWHURPLGHQWLILHUDGHXWEHWDOQLQJDUVRPGHVVXWRPJ|UHQEHG|PQLQJDY YHUNVDPKHWHQVUHVXOWDWNDWHJRUL 5HVXOWDWDQDO\VHUQDVLQIRUPDWLRQJUXSSHUDGHIWHU879VWUHXSSJLIWVNDWHJRULHU UHGRYLVDVLI|OMDQGHWDEHOO 7DEHOO.ODVVLILFHULQJDYUHVXOWDWDQDO\VHUQDVXSSJLIWHURP6LGDVXWEHWDOQLQJDUWNU 7LGV
SHULRG 8WEHWDODWHQOLJW($ NDWHJRUL8WEHWDODW ,GHQWLILHUDWNDWHJRUL NDWHJRUL%HG|PW EHG|PW$QGHO EHG|PQLQJDU$QWDO 1LFDUDJXD 6UL/DQND /DRV 1DPLELD 9LHWQDP 6\GDIULND 9lVWE*D]D )LOLSSLQHUQD 7RWDOW 7RWDOW 'HWlUYLNWLJWDWWXSSPlUNVDPPDDWWWDEHOOHQVEDNRPOLJJDQGHEHUlNQLQJDUlU SUREOHPDWLVNDSnROLNDVlWW)OHUDXSSVNDWWQLQJDUKDUJMRUWVSnHWWRIWDRNODUW XQGHUODJ'HWlUWLOOH[HPSHOLEODQGRNODUWYLONHQWLGVSHULRGVRPHQVNLOGDXSSJLIWHU RPXWEHWDOQLQJDUHJHQWOLJHQDYVHULQVDWVDYWDOHWVWLGVSHULRGHOOHUGHQWLGVSHULRG VRPUHVXOWDWDQDO\VHQNRQFHQWUHUDWVWLOO"bQPHURVlNHUWlUKXUVWRUGHODYGH
4 DQJLYQDNRVWQDGHUQDVRPUHVXOWDWXSSJLIWHUQDIDNWLVNWRPIDWWDUVNDHQJLYHQHIIHNW EHWUDNWDVVRPHQHIIHNWDYKHODVW|GHWHOOHUHQGDVWDYGHQGHODYVW|GHWVRP UHVXOWDWDQDO\VHQEHKDQGODU"6YnULJKHWHUQDDWWEHVYDUDGHVVDRFKOLNQDQGHIUnJRU LQQHElUDWWWDEHOOHQVXSSJLIWHUIUDPI|UDOOWYDGJlOOHUDQGHOHQEHG|PWELVWnQG GYVNDWHJRULNQDSSDVWNDQEHWUDNWDVVRPH[DNWD (IWHUVRPVDPPDEHUlNQLQJVSUREOHPJlOOHUI|UVDPWOLJDUHVXOWDWDQDO\VHUlUGHW GRFNLQWUHVVDQWDWWMlPI|UDGHWVDPPDQODJGDXWIDOOHWPHGXWIDOOHWLWLGLJDUHnUV UHVXOWDWDQDO\VHU'HWNDQNRQVWDWHUDVDWWDQGHOHQEHG|PWELVWnQGKDU|NDWWLOO MlPI|UWPHGL879VI|UVWDJUDQVNQLQJ7URWVDWWGHQI|UVWD JUDQVNQLQJHQRPIDWWDGHUHVXOWDWDQDO\VHUGYVQlVWDQGXEEHOWVnPnQJDVRP I|UHOLJJDQGHVWXGLHKDUGHWWRWDODDQWDOHWEHG|PQLQJDUWUHGXEEODWVIUnQWLOO EHG|PQLQJDURP6LGDVELVWnQGXQGHUGHDNWXHOODWLGVSHULRGHUQD (QRUVDNWLOOGHW|NDGHEHG|PDQGHLQVODJHWLGHVHQDVWHnUHQVUHVXOWDWDQDO\VHUlU GHQUHVXOWDWUHGRYLVQLQJLPDWULVIRUPVRPXWDUEHWDWVDYUHJLRQDYGHOQLQJDUQDV PHWRGJUXSS0DWULVHQP|MOLJJ|UHWWNODUWVW|UUHDQWDONRUWIDWWDGHEHG|PQLQJDU RPELVWnQGHWVUHVXOWDWlQYDGVRPlUP|MOLJWHOOHUWLOOUnGOLJWLO|SDQGHWH[WIRUP 'HQlUGHVVXWRPHWWLSULQFLSXWPlUNWXQGHUODJI|UUHVXOWDWDQDO\VHUQDV lPQHVPlVVLJDRFKVDPPDQIDWWDQGHGHODU 'HWPnVWHGRFNSnSHNDVDWWGHWlURP|MOLJWDWWIUnQ7DEHOOGUDQnJUDVOXWVDWVHU RPUHVXOWDWDQDO\VHUQDVNYDOLWHW)|UGHWI|UVWDE\JJHUWDEHOOHQXWHVOXWDQGHSn NYDQWLWDWLYDEHUlNQLQJDU(QOLJW879VVlWWDWWUlNQDLQQHKnOOHUUHVXOWDWDQDO\VHUQD EHG|PQLQJDURP6LGDVELVWnQGPHQKXUXYLGDGHVVDXSSJLIWHUlUWLOOI|UOLWOLJD RFKDQYlQGEDUDLODQGVWUDWHJLSURFHVVHQlUIRUWIDUDQGHRNODUW )|UGHWDQGUDlUGHWRFNVnRNODUWPHGYLONHQPnWWVWRFNUHVXOWDWDQDO\VHUQDE|U EHG|PDVbUEHG|PWELVWnQGGYVNDWHJRULP\FNHWHOOHUOLWH"lUHQ RP|MOLJKHWHIWHUVRPP\QGLJKHWHQKHOWHQNHOWLQWHKDUXWYlUGHULQJVLQIRUPDWLRQRP DOOWELVWnQGRFKHIWHUVRPSURGXNWLRQVNRVWQDGHUQDI|UHQIXOOVWlQGLJUHVXOWDWDQDO\V VNXOOHEOLRULPOLJWK|JD2PIXOOVWlQGLJKHWlUHQRP|MOLJKHWYLONDNDWHJRULHUDY ELVWnQGHWE|UGnH[NOXGHUDV"879VI|UVODJlUDWWUHVXOWDWDQDO\VHUQDV lPQHVPlVVLJDRFKVDPPDQIDWWDQGHGHODUE|UNRQFHQWUHUDVWLOOGH VWUDWHJL|YHUYlJDQGHQVRPIDNWLVNWlUI|UHPnOI|UGLVNXVVLRQLGHQDNWXHOOD ODQGVWUDWHJLSURFHVVHQVHDYVQLWW 'HQI|UHVODJQDDYJUlQVQLQJHQWLOODNWXHOODVWUDWHJL|YHUYlJDQGHQE|UGRFNHMJlOOD UHVXOWDWUHGRYLVQLQJHQLPDWULVIRUP'HQQDUHGRYLVQLQJERUGHLVWlOOHWYDUDI|UHPnO I|UHQDQQDQVRUWVDYJUlQVQLQJ'HWYRUHQlPOLJHQKHOWRULPOLJWDWWDY -lPI|UHOVHUPHOODQGHVVDXSSJLIWHUlUlYHQGHYDQVNOLJD(QDQOHGQLQJWLOODWWnUVJUDQVNQLQJ GURJVOXWVDWVHQDWW´HQGDVWµDYGHWDNWXHOODELVWnQGHWJLYLWVQnJRQIRUPDYEHG|PQLQJYDUDWW UHVXOWDWDQDO\VHUQDI|U0R]DPELTXH7DQ]DQLD=DPELDRFK=LPEDEZHLQWHJMRUGHQnJRQNRSSOLQJ PHOODQXWEHWDODGHPHGHORFKUHGRYLVQLQJRFKDQDO\VDYIDNWLVNDUHVXOWDW2PGHVVDUHVXOWDWDQDO\VHU KDGHH[NOXGHUDWVIUnQJUDQVNQLQJHQVNXOOHDQGHOHQEHG|PWELVWnQGL879VI|UVWDJUDQVNQLQJYDUD LQWH
5 ELVWnQGVDQVODJHWVNDUHGRYLVDVLPDWULVIRUP8QGHUXWJMRUGHXWEHWDOQLQJDUSn PLQGUHlQHQPLOMRQNURQRUDYGHWWRWDODDQWDOHWLQVDWVHUPHQHQGDVWDY WRWDOWXWEHWDODGHPHGHOnWPLQVWRQHRPPDQInUWURGHWJDPOD($V\VWHPHWV XSSJLIWHU'HWILQQVGlUI|UVNlOI|U6LGDDWWDQJHULNWOLQMHUlYHQI|U UHVXOWDWUHGRYLVQLQJDUQDVLQQHKnOOWLOOH[HPSHODWWUHGRYLVQLQJDUQDHQGDVWVNDDYVH LQVDWVHU|YHUHQPLOMRQNURQRU6YDUSnGHVVDRFKDQGUDYLNWLJDIUnJRUE|UHQOLJW 879VXSSIDWWQLQJLQJnLRYDQQlPQGD*'EHVOXWVH$YVQLWWU|UDQGH P\QGLJKHWHQVYHUNVJHPHQVDPPDQRUPHURFKULNWOLQMHUI|UUHVXOWDWDQDO\VHUQDV IUDPVWlOOQLQJIRUPRFKLQQHKnOO 9LONDUHVXOWDWDQDO\VHUDV" )|UDWWEHVYDUDIUnJDQKDU879IRNXVHUDWSnXSSJLIWHUVRPLQQHKnOOHUHQ EHG|PQLQJDYELVWnQGHWVUHVXOWDWGYVNDWHJRUL8SSJLIWHUQDKDUVHGDQ NODVVLILFHUDWVXWLIUnQ/)$PRGHOOHQVWUHUHVXOWDWQLYnHUSUHVWDWLRQHUNRUWVLNWLJD HIIHNWHURFKOnQJVLNWLJDHIIHNWHU(WWVXEMHNWLYWLQVODJLNODVVLILFHULQJHQKDUYDULW RXQGYLNOLJW'HWUHVXOWDWVRPSUHVHQWHUDVL7DEHOOlUGlUI|UXQJHIlUOLJW 7DEHOO$QGHOEHG|PGDXSSJLIWHUSHUUHVXOWDWQLYn
5HVXOWDWDQDO\VHU 3UHVWDWLRQHU .RUWVLNWLJDHIIHNWHU /nQJVLNWLJDHIIHNWHU 6RPHQGLUHNWI|OMGDYPHWRGJUXSSHQVDQYLVQLQJDUKDUDQGHOHQLQIRUPDWLRQRP NRUWVLNWLJDHIIHNWHU|NDWPHGQlVWDQHQWUHGMHGHOLMlPI|UHOVHPHGnUV JUDQVNQLQJ$QGHOHQLQIRUPDWLRQRPSUHVWDWLRQHUKDUVMXQNLWXQJHIlUOLNDP\FNHW 'HQQDWUHQGOLJJHUDQWDJOLJHQKHOWLOLQMHPHGUHJHULQJHQV|QVNHPnO +XUDQDO\VHUDVUHVXOWDWHQ" )|UDWWEHVYDUDIUnJDQKDU879NODVVLILFHUDWUHVXOWDWDQDO\VHUQDV EHG|PQLQVXSSJLIWHUHIWHUXWYlUGHULQJVNULWHULHUQDUHOHYDQVNRVWQDGVHIIHNWLYLWHW PnOXSSI\OOHOVHHIIHNWHUSnNRUWRFKOnQJVLNWRFKElUNUDIWLJKHW(WWPLQGUHDQWDO EHG|PQLQJVXSSJLIWHUlUGRFNLQWHDQQDWlQDOOPlQQDRPG|PHQVRPVSHFLILFHUDU DWWPHQLQWHYDUI|UHOOHUSnYLONHWVlWWUHVXOWDWHQlUEUDHOOHUGnOLJD 7DEHOO$QGHOEHG|PGDXSSJLIWHUSHUDQDO\VYHUNW\J
$OOPlQQDRPG|PHQ 5HOHYDQV .RVWQDGVHIIHNWLYLWH
W 0nOXSSI\OOHOVH (IIHNWHU %lUNUDIWLJKHW
,MlPI|UHOVHPHGnUVJUDQVNQLQJKDUDQGHOHQDOOPlQQDRPG|PHQVMXQNLW JDQVNDGUDVWLVNWYLONHWlUEUDHIWHUVRPVnGDQDXSSJLIWHUNQDSSDVWJDJQDU UHVXOWDWDQDO\VHUQDVIRUPDWLYDV\IWHDWWOLJJDWLOOJUXQGI|UHQQ\ODQGVWUDWHJL
6 'HQVWDUNDVWHWUHQGHQlUGRFNDWWDQGHOHQXSSJLIWHURPELVWnQGHWVHIIHNWHUXWJ|U WUHIMlUGHGHODUDYVDPWOLJDEHG|PQLQJVXSSJLIWHUMlPI|UWPHGNQDSSWKlOIWHQL nUVJUDQVNQLQJbYHQGHQQDWUHQGWRUGHOLJJDLOLQMHPHGUHJHULQJHQV |QVNHPnO bUUHVXOWDWDQDO\VHUQDWLOOI|UOLWOLJD" (QGLVNXVVLRQRPWLOOI|UOLWOLJKHWNDQVHXWSnROLNDVlWW,JUDQVNQLQJHQIUnQ YDOGH879DWWIRNXVHUDSnNlOOKlQYLVQLQJRFKVDNOLJKHW .lOOKlQYLVQLQJ (QQRJJUDQQEHG|PQLQJDYWLOOI|UOLWOLJKHWI|UXWVlWWHUHQJHQRPJnQJDYGHNlOORU VRPUHVXOWDWDQDO\VHUQDIDNWLVNWE\JJHUSn(QVnGDQEHG|PQLQJKDULQWHYDULW P|MOLJDWWJ|UDLQI|UGHQQDVWXGLH879KDULVWlOOHWI|UV|NWNDUWOlJJD NlOOKlQYLVQLQJDUQDI|UGHEHG|PQLQJVXSSJLIWHUVRPGLVNXWHUDWVLI|UHJnHQGH DYVQLWW/LNVRPLJUDQVNQLQJHQIUnQKDUWYnIUnJRUVWlOOWV • ,QQHKnOOHUUHVXOWDWDQDO\VHUQDW\GOLJDKlQYLVQLQJDUWLOOXSSJLIWHUQDVNlOORU" • ,YLONHQXWVWUlFNQLQJlUXSSJLIWHUQDVNlOORUXWYlUGHULQJVUDSSRUWHU" 2PPDQPHG´W\GOLJµNlOOKlQYLVQLQJPHQDUDWWGHWSnHWWHQNHOWVlWWJnUDWW NRQWUROOHUDHOOHUInPHULQIRUPDWLRQRPHQJLYHQXSSJLIWVnlUVYDUHWDWWGHWWD JHQHUHOOWVHWWLQWHlUP|MOLJWIUnQGHVHQDVWHWYnnUHQVUHVXOWDWDQDO\VHU 5HVXOWDWUHGRYLVQLQJDUQDLYLONDPHUSDUWHQDYGHXSSJLIWHUQDnWHUILQQVVDNQDU LUHJHOW\GOLJDNlOOKlQYLVQLQJDUbYHQRPGHWLEODQGILQQVHQNlOOI|UWHFNQLQJL VOXWHWDYUHGRYLVQLQJHQlUGHWVYnUWI|UHQXWRPVWnHQGHDWWLGHQWLILHUDYLOND XSSJLIWHUVRPIDNWLVNWK|UWLOOHQJLYHQNlOODEODQGDQQDWHIWHUVRP NlOOKlQYLVQLQJHQRIWDlURSUHFLVWLOOH[HPSHO´.lOORUPRWWDJDUUDSSRUWHULQJµ 5HVXOWDWDQDO\VHUQDVEULVWHUDYVHHQGHNlOOKlQYLVQLQJLQQHElUDWWGHWLQWHlUP|MOLJW DWWEHVYDUDI|OMGIUnJDQRPLYLONHQXWVWUlFNQLQJNlOORUQDXWJ|UVDYXWYlUGHULQJDU 6DNOLJKHW 5HVXOWDWDQDO\VHUQDVXSSJLIWHUlURIWDVnDOOPlQWKnOOQDDWWHQEHG|PQLQJDY VDNOLJKHWlUVYnUDWWJ|UD+XUXYLGDSnVWnHQGHQRPWLOOH[HPSHO´|NDG LQVWLWXWLRQHOONDSDFLWHWµHOOHU´|NDGPHGYHWHQKHWKRVEHIRONQLQJHQµNDQEHWUDNWDV VRPNRUUHNWDEHURUJLYHWYLVSnYDGVRPLGHWDNWXHOODIDOOHWPHQDVPHG|NDG NDSDFLWHWRFK|NDGPHGYHWHQKHW8WDQQlUPDUHLQIRUPDWLRQRPGHWWDlUGHW NQDSSDVWP|MOLJWDWWEHG|PDXSSJLIWHUQDVVDNOLJKHW(QXSSJLIWRP´|NDG LQVWLWXWLRQHOONDSDFLWHWµNDQVNHLQWHLQQHElUPHUlQDWWHWWNRQWRUKDUXWUXVWDWV PHGWLRGDWRUHURFKDWWGHODUDYSHUVRQDOHQYDULWSnNXUVL6YHULJH,VnIDOOILQQV GHWNDQVNHPLQGUHDQOHGQLQJDWWEHWYLYODSnVWnHQGHWVVDNOLJKHWlQRPGHWlUHQ PHUGMXSJnHQGHRFKRPIDWWDQGHI|UlQGULQJVRPDYVHV ,QnJUDIDOOILQQVGHWLUHVXOWDWDQDO\VHUQDXSSJLIWHUVRPI|UHQQRUPDONULWLVNOlVDUH I|UHIDOOHUWlPOLJHQRVDQQROLND'HWlUWLOOH[HPSHOLQWHVDQQROLNWDWWHWW IRUVNQLQJVELGUDJSnPLOMRQHUNURQRUVRPDQYlQWVI|UDWWIUDPVWlOODROLND
7 SXEOLNDWLRQHURFKI|UDWWILQDQVLHUDHWWEHJUlQVDWDQWDOGRNWRUDQGRFK PDVWHUVXWELOGQLQJDUKDUUHVXOWHUDWLHWWLQIRUPDWLRQVFHQWHURFKHQJnUGI|U XWELOGQLQJDYE|QGHUSn6UL/DQND7URWVDWWXSSJLIWHUVRPSnGHWWDVlWWIDOOHUSn VLQHJHQRULPOLJKHWlUUHODWLYWVlOOV\QWDSnYHUNDUGHUHVXOWDWUHGRYLVQLQJDUQDV WURYlUGLJKHWLVWRUW bUUHVXOWDWDQDO\VHUQDDQYlQGEDUD" 5HVXOWDWDQDO\VHUQDKDUHWWKXYXGVDNOLJWDQYlQGQLQJVRPUnGHQlPOLJHQVRP XQGHUODJI|UEHVOXWRPQ\ODQGVWUDWHJL7DQNHQlUDWW6LGDJHQRPDWWGUDOlUGRP DYWLGLJDUHUHVXOWDWRFKJHQRPDWWDQYlQGDGHVVDOlUGRPDULGHWQ\D ODQGVWUDWHJLI|UVODJHWVNDnVWDGNRPPDHWWElWWUHIUDPWLGDELVWnQG0HQWURWVDWW ´OlUDQGHµRFK´UHVXOWDWVW\UQLQJµlUIOLWLJWDQYlQGDEHJUHSSLQRPP\QGLJKHWHQlU GHWLQWHVMlOYNODUWnWPLQVWRQHLQWHI|U879YDGVRPHJHQWOLJHQNlQQHWHFNQDUHWW UHVXOWDWEDVHUDWODQGVWUDWHJLI|UVODJ'lUPHGlUGHWLQWHKHOOHUVMlOYNODUWYDGVRPL SUDNWLNHQNDUDNWlULVHUDUHQDQYlQGEDUUHVXOWDWDQDO\V 8WDQHQNODUXSSIDWWQLQJRPYDGVRPlUHQDQYlQGEDUUHVXOWDWDQDO\VlUGHW JLYHWYLVLQWHKHOOHUP|MOLJWDWWEHVYDUDIUnJDQRPGHVHQDVWHnUHQVUHVXOWDWDQDO\VHU lUDQYlQGEDUD(QOlPSOLJXWJnQJVSXQNWI|UDWWUHGDXWIUnJDQRPYDGVRP NDUDNWlULVHUDUHQDQYlQGEDUUHVXOWDWDQDO\VlUGHVWUDWHJLVND|YHUYlJDQGHQVRP J|UVLODQGVWUDWHJLI|UVODJHW'HWlUQlPOLJHQI|UVWPHGHQNODUXSSIDWWQLQJRP NDUDNWlUHQSnGHVVD|YHUYlJDQGHQVRPGHWlUP|MOLJWDWWGLVNXWHUDYLONHQW\SDY UHVXOWDWLQIRUPDWLRQVRPlUUHOHYDQWRFKGlUPHGDQYlQGEDUI|UROLNDW\SHUDY |YHUYlJDQGHQ (QVQDEEJHQRPJnQJDY6LGDVODQGVWUDWHJLHUYLVDUDWWROLNDVHNWRUYDOlUGHQNODUW YDQOLJDVWHW\SHQDY|YHUYlJDQGHQVRPJ|UVLODQGVWUDWHJLSURFHVVHQEHVO|W UHJHULQJHQWLOOH[HPSHODWWDYVOXWDVW|GHWWLOO6UL/DQNDLQRPXQGHUYLVQLQJVVHNWRUQ RFKLVWlOOHWVDWVDSnQlULQJVOLYVXWYHFNOLQJ (QP|MOLJKHWDWWJ|UDUHVXOWDWDQDO\VHUQDPHUDQYlQGEDUDWRUGHYDUDDWW NRQFHQWUHUDGHlPQHVPlVVLJDRFKVDPPDQIDWWDQGHDQDO\VGHODUQDWLOOGH |YHUYlJDQGHQVRPIDNWLVNWJ|UVNRPPHUDWWJ|UDVLGHQDNWXHOOD ODQGVWUDWHJLSURFHVVHQ2PHWW|YHUYlJDQGHJnUXWSnDWWDYVOXWDHWW XQGHUYLVQLQJVVW|GWLOOI|UPnQI|UQlULQJVOLYVXWYHFNOLQJVnE|UUHVXOWDWDQDO\VHQ IRNXVHUDSnRFKVNUlGGDUV\VI|UMXVWGHWWDVDPPDQKDQJ1RUPDOWE|UHQ ´K\SRWHVµRPHWWVnGDQWVHNWRUYDONXQQDJ|UDVUHGDQLGHQI|UHJnHQGH ODQGVWUDWHJLSURFHVVHQGYVWUHnULQQDQ6LGDVVWUDWHJLI|UVODJIDNWLVNWVNLFNDVLQWLOO UHJHULQJHQ'lUPHGILQQVGHWRFNVnWLOOUlFNOLJWPHGWLGI|UDWWSODQHUDRFK JHQRPI|UDXWYlUGHULQJDUVRPNDQOLJJDWLOOJUXQGI|UHQVNUlGGDUV\GG UHVXOWDWDQDO\V (WWODQGVWUDWHJLI|UVODJNDQJLYHWYLVLQQHKnOODROLNDW\SHUDY|YHUYlJDQGHQ0HQ GHWVSHODUPLQGUHUROO3RlQJHQlUDWWUHVXOWDWDQDO\VHQI|UDWWYDUDDQYlQGEDU RFKNRVWQDGVHIIHNWLYHQGDVWE|UDQDO\VHUDVnGDQDUHVXOWDWVRPlUUHOHYDQWDI|U DNWXHOODVWUDWHJLVND|YHUYlJDQGHQ
8 'HQFHQWUDODIUnJDQlUGlUI|UYDGVRPXWJ|UUHOHYDQWDUHVXOWDWJLYHWROLNDW\SHUDY |YHUYlJDQGHQ")UnJDQlUVYnUDWWEHVYDUDPHQQnJUDULNWOLQMHUERUGHNXQQDJHV JHQRPDWWQlUPDUHGLVNXWHUDNDUDNWlUHQSnW\SLVNDVWUDWHJL|YHUYlJDQGHQ DQDO\VHUDYDGVRPNDQYDUDUHOHYDQWUHVXOWDWLQIRUPDWLRQI|UGHVVD|YHUYlJDQGHQ RFKDWWXQGHUV|ND6LGDVP|MOLJDWLOOJnQJSnUHVXOWDWLQIRUPDWLRQVRPLROLND VDPPDQKDQJEHG|PVVRPUHOHYDQW
5HNRPPHQGDWLRQHU
,MlPI|UHOVHPHGGHUHVXOWDWDQDO\VHUVRPSURGXFHUDGHVDY6LGDXQGHUSHULRGHQ LQQHKnOOHUGHVHQDVWHnUHQVUHVXOWDWDQDO\VHUNYDQWLWDWLYWVHWWPnQJGXEEHOW IOHUEHG|PDQGHXSSJLIWHURPELVWnQGHWVUHVXOWDW(QNYDOLWDWLYMlPI|UHOVHlUVYnU DWWJ|UDPHQGHWlU879VLQWU\FNDWW6LGDKDUWDJLWIOHUDYLNWLJDVWHJIUDPnWL DUEHWHWDWW|NDUHVXOWDWDQDO\VHUQDVNYDOLWHW 'HWDUEHWHVRPnWHUVWnUI|U6LGDDWWJ|UDLWHUPHUDYDQYLVQLQJDUKDQGODUHQOLJW 879VXSSIDWWQLQJIUDPI|UDOOWRPDWW|NDDQDO\VGHODUQDVDQYlQGEDUKHWRFKDWW DYJUlQVDUHVXOWDWUHGRYLVQLQJDUQDVRPIDWWQLQJ6LGDUHNRPPHQGHUDVGlUI|U • DWW|YHUYlJDRFKE\JJDYLGDUHSn879VI|UVODJWLOODWW|NDUHVXOWDWDQDO\VHUQDV DQYlQGEDUKHWVHDYVQLWW • DWWDQJHYHUNVJHPHQVDPPDULNWOLQMHUI|UKXUP\FNHWRFKYLONDGHODUDY6LGDV ELVWnQGVRPVNDUHGRYLVDVLPDWULVIRUPVHDYVQLWW • DWWJHQRP*'EHVOXWJHGHDQYLVQLQJDUVRPEHVOXWDWVDY$VLHQDYGHOQLQJHQ VWDWXVVRPYHUNVJHPHQVDPPDDQYLVQLQJDU9
%LODJD$879VNODVVLILFHULQJDYUHVXOWDWDQDO\VHUQDVXSSJLIWHU
.DWHJRUL
,QVDWVEHVOXW
RFKWLGVSHULRG 6HNWRUPnO 3URMHNWPnO 8SSQnGGDHIIHNWHU 3ODQHUDGHSUHVWDWLRQHU 8SSQnGGDSUHVWDWLRQHU .RVWQDG06(. 6NDWWHGDWDH[S HUWJHQRP ,0)81'3 1\WW GDWDEDVHUDW V\VWHPI|U PRPV XWYHFNODWRFK WHVWDW ,QIR VDNQDV 5nGJLYQLQJI|UXWYRFKWHVWDYGDWDSURJUDPI|U WLOOlPSQLQJLGQUI|U VNDWWVN\OGLJD%LVWnPHG NUDYVSHFI|U SLORWSURMHNWHQVDPWI|U XSSKDQGOLQJRFK LQVWDOODWLRQDYXWUXVWQ %LVWnPHGXWEI|U UHOHYDQWSHUVRQDO ,QIRVDNQDV )DNWLVNW 3ODQHUDW 7DEHOO$LQQHKnOOHUXSSJLIWHURP6LGDVXWEHWDOQLQJDUWLOOGHWDNWXHOODSURMHNWHW PHQLQWHXSSJLIWHUVRPLGHQWLILHUDUSURMHNWHWVIDNWLVNDSUHVWDWLRQHURFKLQWHKHOOHU XSSJLIWHUVRPJ|UHQEHG|PQLQJDYSURMHNWHWVXSSQnGGDHIIHNWHU .DWHJRUL ,QVDWVEHVOXW
RFKWLGVSHULRG 6HNWRUPnO 3URMHNWPnO 8SSQnGGDHIIHNWHU 3ODQHUDGHSUHVWDWLRQHU 8SSQnGGDSUHVWDWLRQHU .RVWQDG06(. %HO\VQLQJ
,1(&8UEDQ gNDGVlNHUKHWLIDWWLJD RPUnGHQ 6WDWLVWLN VDNQDV 8SSI|UDQGHDYEHO\VQLQJV PDVWHUJDWXEH O\VQLQJXWPHG KXYXGVWUnN EHO\VQLQJV PDVWHUXSSI|UGD JDWXEHO\VQLQJ XWPHGKXYXG VWUnNKDUVOXWI|UWV 7RWDOW 7DEHOO%LQQHKnOOHUXSSJLIWHURP6LGDVXWEHWDOQLQJDUWLOOGHWDNWXHOODSURMHNWHW VDPWXSSJLIWHUVRPLGHQWLILHUDUSURMHNWHWVIDNWLVNDDNWLYLWHWHWHURFKSUHVWDWLRQHU PHQLQWHXSSJLIWHUVRPJ|UHQEHG|PQLQJDYSURMHNWHWVXSSQnGGDHIIHNWHU .DWHJRUL ,QVDWVEHVOXW
RFKWLGVSHULRG 6HNWRUPnO 3URMHNWPnO 8SSQnGGDHIIHNWHU 3ODQHUDGHSUHVWDWLRQHU 8SSQnGGDSUHVWDWLRQHU .RVWQDG06(. 2PVWUXNWXUH ULQJDY WUDQVSRUW PLQLVWHULXP 6lNUDRFK HIIHNWLYD WUDQVSRUW WMlQVWHU /lJUHSULVHU |NDWDQVYDUV WDJDQGHHIIHN WLYLWHWRFK NDSDFLWHWSn PLQLVWHULHW 5HJHULQJVEHVOXW RPSROLF\I|U NRPPHUVLDOLVH ULQJRFKSULYDWL VHULQJK|MG PHG YHWDQGHJUDGRFK VDWVQLQJSn +5' (IIHNWLYDUHRFK PLQGUHRUJDQL VDWLRQ HWDEOHUDGH IO\JSODWVERODJ YlJYHUN YlJIRQGRFK YlJHQWUH SUHQ|UHU 2UJDQLVDWLRQV I|UVODJEHVOXW RPNRPPHU VLDOLVHULQJDY IO\JSODWVHU HWDEOHUDGHYlJ YHUNRFKYlJ IRQGXWELOG QLQJVLQVDWVHU 7RWDOW 7DEHOO&LQQHKnOOHUXSSJLIWHURP6LGDVXWEHWDOQLQJDUWLOOGHWDNWXHOODSURMHNWHW XSSJLIWHUVRPLGHQWLILHUDUSURMHNWHWVIDNWLVNDSUHVWDWLRQHUVDPWGHVVXWRPXSSJLIWHU VRPJ|UHQEHG|PQLQJDYSURMHNWHWVXSSQnGGDHIIHNWHU
6LGD6WXGLHVLQ(YDOXDWLRQ
96/1 Evaluation and Participation - some lessons. Anders Rudqvist, Prudence Woodford-Berger
Department for Evaluations and Internal Audit
96/2 Granskning av resultatanalyserna i Sidas landstrategiarbete. Göran Schill
Department for Evaluations and Internal Audit
96/3 Developmental Relief? An Issues Paper and an Annotated
Bibliography on Linking Relief and Development. Claes Lindahl Department for Evaluations and Internal Audit
96/4 The Environment and Sidas Evaluations. Tom Alberts, Jessica
Andersson
Department for Evaluations and Internal Audit
97/1 Using the Evaluation Tool. A survey of conventional wisdom and common practice at Sida. Jerker Carlsson, Kim Forss, Karin Metell, Lisa Segnestam, Tove Strömberg
Department for Evaluations and Internal Audit
97/2 Poverty Reduction and Gender Equality. An Assessment of Sida’s
Country Reports and Evaluations in 1995-96. Eva Tobisson, Stefan de Vylder
Secretariat for Policy and Corporate Development.
98/1 The Management of Disaster Relief Evaluations. Lessons from a Sida evaluation of the complex emergency in Cambodia. Claes Lindahl Department for Evaluation and Internal Audit
7;)(-7,-28)62%8-32%0():)0341)28'334)6%8-32%+)2'= 77XSGOLSPQ7[IHIR
8IP*E\ 8IPIKVEQWMHEWXSGOLSPQ4SWXKMVSr )QEMPMRJS$WMHEWI,SQITEKILXXT[[[WMHEWI