Assessing the development of performance audit methodology in the supreme audit institution: The case of Indonesia

Badan Pemeriksa Keuangan (BPK), or The Audit Board of the Republic of Indonesia, has introduced performance audits intensively since public sector audit reforms were initiated in 2001. However, the implementation of performance audits still faces several obstacles. This study examines the development of the performance audit methodology in BPK, which can be divided into three stages: before the revitalization of BPK in 2006, the initial development of performance audits during 2006 -2010, and the maturity of the implementation of performance audits since 2011. This study concludes that several key factors have affected the implementation of performance audits at BPK, including the development of audit guidelines, the dynamics of medium-term government planning, the focus of the BPK’s Management, and assistance provided by other supreme audit institutions (SAI). To address new realities and challenges in the future, more sophisticated performance audit methods need to be adopted by BPK, such as big data analytics, the adoption of governance audits, and employing public policy analysis. Without adopting such methods, the institutionalization of effective performance audits might not be achieved in the future.


INTRODUCTION
Performance audit is also referred to as value for money (VFM) audit in the UK (Parker, Jacobs, & Schmitz, 2018) or comprehensive auditing in Canada (Everett, 2003). It was introduced by several supreme audit institutions (SAIs) from developed countries in the late 1970s. During that time, developed countries such as the UK, Canada, and Australia embarked on extensive public management reform that emphasized the performance of the public sector (Nicoll, 2017). However, it is believed that the management of the public sector should be similar to the private sector because both public and private sectors have principle-agent problems (Lapsley & Miller, 2019).
In the new public management era, the demand for accountability and transparency goes beyond financial and legal accountability. Because resources are limited, the public demands their government spend public resources wisely, deliver results (effectiveness), attain efficiency, and continuously seek to reduce the cost of public services (economy). SAI is expected to verify the performance of the government in meeting these three aspects: economy, efficiency, and effectiveness (three "E's"), and this expectation creates demand for performance audit (Grönlund, Svärdsten, & Öhman, 2011). Power (2003) labels the sudden increase of performance audit demand in developed countries as an "audit explosion." Furthermore, several authors argue that the scope of performance audit should be expanded into other "E's" such as ethics (Bringselius, 2018), environment (and sustainability) (Irawan & McIntyre-Mills, 2016), and equity (Everett, 2003;Johnsen et al., 2019;Reichborn-Kjennerud, 2013). In several countries, the scope of performance audit also includes compliance and policy audits (Grönlund et al., 2011;Pollitt, et al., 1999).
International Standards of Supreme Audit Institutions (ISSAI), published by the International Organization of Supreme Audit Institutions (INTOSAI), give the definition of a performance audit. The performance audit is an independent, objective, and reliable examination of whether government undertakings, systems, programs, activities, or organizations perform under the principles of economy, efficiency, and effectiveness and whether there is room for improvement (INTOSAI, 2019). Based on these definitions, there are two underlying characteristics of performance audit, which are (1) assessing performance on the economy, efficiency, and effectiveness aspects; and (2) providing recommendations for improving government performance.
Performance audit aims to fix existing weaknesses and not just find mistakes or determine who should be responsible for errors. As an independent institution, SAI is expected to see a problem objectively and provide recommendations to improve the performance of audited entities (Loke, Ismail, & Hamid, 2016;Pierre & de Fine Licht, 2019). The quality of public financial management and government services is expected to improve after the government implements performance audit recommendations. Thus, a performance audit is not only a tool to discharge accountability (Desmedt, Morin, Pattyn, & Brans, 2017;Reichborn-Kjennerud, 2013) but also a tool for learning, change, and improvement (Desmedt et al., 2017;Raudla, Taro, Agu, & Douglas, 2016;Van Loocke & Put, 2011).
Previous research has addressed Performance Audit (PA) from several perspectives, such as the impact of performance audit on public administration (Bringselius, 2010;Raudla et al., 2016;Reichborn-Kjennerud & Vabo, 2017;Torres, Yetano, & Pina, 2016;Van Loocke & Put, 2011), key stakeholders of a SAI and their role in shaping performance audit (Parker et al., 2018), the scope of performance audit (Bringselius, 2018;Grönlund et al., 2011), the development of performance audit in developed countries (Lonsdale, 2000(Lonsdale, , 2011Pollitt, 2003) and the selection of performance audit themes (Nath, Othman, & Laswad, 2020). However, these studies focus on the context of developed countries, and there are limited studies on the development of performance audits in developing countries; thus, authors try to fill the gap in the literature by investigating the development of performance audits in developing countries. Specifically, the authors address following research questions: 1. What are the main factors that influence the development of performance audits in BPK? 2. Performance audit has a different objective, methodology, and criteria in comparison with financial or compliance audits. What types of performance audit methodology conducted by BPK? 3. What evolutionary methods are used in BPK's performance audit?

RESEARCH METHOD
This research uses a qualitative method to explain the development of performance audits in BPK. The empirical data were gathered from 1.291 performance audit reports published by BPK from 2016 to 2020, Summary of Semester Audit Results (Ikhtisar Hasil Pemeriksaan Semester, IHPS) from 2004 to 2020, four peer review reports, and BPK's performance audit manuals and guidelines. The IHPS contains a summary of all BPK audit reports in one semester and audit follow-up monitoring. Critical content analysis is used to elaborate performance audit reports, IHPS reports, peer review reports, and performance audit guidelines. The authors also interviewed several senior BPK's performance auditors to get their perspective and experiences based on peer re-view conducted by other SAI and the development of performance audit in BPK to pursue data triangulation.

Performance Audit Development and Factors that Influence Performance Audit Development in BPK
While performance audit relatively matures in developed countries by 2006, it is a relatively new practice in BPK (Andrianto, 2015). The development of performance audit in BPK can be divided into three stages: before the revitalization of BPK (before 2006), initial development (2006)(2007)(2008)(2009)(2010), and the maturity stage (2011 onwards). Every year, the number of performance audit increases steadily. Figure 1 presents the number of performance audits conducted by the BPK from 2004 to 2020. Figure 1 shows that from 2006 the number of performance audits continued to increase and reached the highest number in the last ten years in 2016. Furthermore, although the number of performance audits began to decline in 2017, the number still shows a significant number until 2020. This data shows BPK's relatively strong commitment to conducting performance audits.

Before the revitalization of BPK (before 2006)
Before 2006 The performance audit was known with many terminologies, such as operational audit (the 1980s) and management audit (1990s). As shown in Figure 1, before 2006, the number of performance audits is low.
BPK was peer-reviewed for the first time in 2004. The peer review was conducted by the Office of the Auditor-General of New Zealand (OAG-NZ). The OAG-NZ reported that performance audit in BPK was not fully developed, and BPK was still learning to conduct performance audit correctly, the conclusion that be agreed by BPK's senior performance auditor interviewed. BPK did not have a performance audit manual; therefore, BPK used international standards or other SAI guidance such as ISSAI and US Government Accountability Office (GAO) Yellow Book as performance audit guidance. BPK did not have a special unit to conduct a performance audit, so its financial auditors conduct its operational and management audits.
Moreover, BPK did not have auditors who have good capability to conduct the perfor-mance audit. According to the OAG-NZ, most of the performance audits conducted did not satisfy 3 E's audit methodology. The OAG-NZ recommended BPK to improve its capability on performance audit and provide intensive performance audit training to its auditors. As a follow-up peer review recommendation, BPK sent several auditors to learn performance audits to SAIs from developed countries such as GAO, Australian National Audit Office (ANAO), and OAG-NZ.

Initial development of performance audit (2006-2010)
Indonesian public financial management was undergoing major reform in the 2000s.   Rai (2008) investigative audit). After Law Number 15 of 2006 was introduced, the number of performance audits increased. As shown in Figure  1, the number of performance audits is 36 in 2006 and increased to 157 in 2010. Start from 2005, all of BPK's audit results, including performance audit reports was published on BPK's official website as part of transparency and accountability (Andrianto, 2007).
The Netherlands Court of Audit conducted the second peer review in 2009. The Netherlands Court of Audit found that BPK still did not have a performance audit manual until 2008 and used performance audit manual draft as audit guidance. Finally, this performance audit manual was approved by BPK's Board on 5 June 2008, so BPK had a performance audit manual for the first time. The Netherlands Court of Audit concluded that BPK's performance audits complied with this performance audit manual. However, the reviewer concluded that BPK's performance audit manual needs to be improved. In addition, the Netherlands Court of Audit also stated that there were several deficiencies in the scope, methodology, and other aspects of BPK's performance audit. This conclusion was also agreed upon by BPK's senior performance auditor interviewed. These review findings were not surprising because performance audit was relatively new for the BPK's board members and auditors. The Netherlands Court of Audit also concluded that the performance audit methods in BPK were influenced by compliance and financial audit methodology. Moreover, it still needs more supervision through quality control and quality assurance to ensure the next performance audit focuses on the government's performance.
In this period, BPK received assistance from ANAO through the Government Partnership Fund (BPK, 2013), bilateral cooperation that ANAO deployed its performance audit experts to BPK to assist the development of performance audit methodology in BPK. This bilateral cooperation continues until 2019. During this period, ANAO also assists with training for BPK auditors and consultations in conducting performance audits. The assistance of ANAO has a significant impact on BPK's audit methodology. It can be shown how BPK develops audit criteria similar to ANAO methods to develop audit criteria. BPK also continued receiving assistance from other SAI such as the GAO Fellowship program (2003-2009), NAO Secondment (2006, ANAO Secondment (2007-2010, and Jabatan Audit Negara (JAN) Malaysia Secondment (2009) to enhance its performance audit capacity. However, the influence of this assistance is not as strong as the impact of ANAO assistance. Nevertheless, these assistances improve the quality of BPK's performance audit.

The maturity of performance audit (2011 onwards)
Since 2011, the practice of performance audit in BPK can be considered mature. The number of performance audits conducted by BPK is more than 100 annually. BPK revised its performance audit manual in 2011. This new audit manual refers to INTOSAI performance audit guidelines, GAO's Yellow Book, and other performance audit manuals. In addition, BPK also produces several technical bulletins on performance audits based on international best practices. BPK has also continued its bilateral cooperation with the ANAO that deployed its Subject Matter Expert (SME) to support the revision of BPK's performance audit methodology. BPK also send its auditors to Australasian Council of Auditors-General (ACAG) Performance Audit Methodology Workshop in Canberra in 2017 and 2018. BPK has also got Tailor-Made Training on the public policy aspect of performance audit from Algemene Rekenkamer, the Netherlands, in 2013 and did a comparative study to South Africa in 2014.
The third peer-review was conducted by the SAI of Poland in 2014 and concluded significant improvements on the BPK's performance audit quality. The performance audit conducted by BPK already focused on identifying problems and providing recommendations for improvement of public services. However, several challenges were identified by the peer reviewer. First, there was no time flexibility to conduct a performance audit since the performance audit was only conducted in the second semester. In the first semester, BPK uses all resources to conduct financial audits. As a consequence, the SAI of Poland recommended BPK to start conducting performance audits in the first semester, the conclusion that also agreed by BPK's senior performance auditor interviewed.
In this peer review, Poland's SAI also highlighted the selection of performance audit themes, the absence of a special performance audit unit, and BPK's auditors' understanding of performance audit methodology. The SAI of Poland recommended that BPK selected essential issues and on the larger scale, not only on simple topics of an individual entity at the local government level, that the senior performance auditor interviewed called high impact performance audit. Furthermore, BPK needed to create a performance audit unit since BPK's auditors usually conducted three audits. The SAI of Poland believed that specialization would increase the expertise of BPK's auditors. However, this process needs to be managed smoothly to prevent BPK losing its current knowledge and capability. It also needs to be supported by a good process of recruitment, training, and knowledge management.
However, BPK still has no specialized unit for the performance audit unit until now, even though the peer reviewer from SAI of Poland (2019) recommends BPK to create a special performance audit unit. The auditors in this unit must only conduct the perfor-mance audit. Regarding the selection of audit themes, BPK conducts a thematic performance audit. The audit topic for thematic performance audit is selected from significant national issues such as public financial management and education management during the COVID-19 Pandemic. In this period, the BPK started to share its knowledge of performance auditing within ASEAN Supreme Audit Institutions (ASEANSAI) members such as SAI of Vietnam, Cambodia, and Laos PDR (Jefriando, 2015;BPK berbagi ilmu, 2019;BPK bantu SAI, 2019). The knowledge sharing themes include how to develop performance audit guidelines and a training center.
At the maturity stage, the performance audit development in BPK is in line with the development plan prepared by the government, and it is indicated by the content of BPK Strategic Plan 2011-2015. BPK Strategic Planning stated that every audit unit has to conduct a performance audit. The focus is on government entities that provide services to the general public, such as Education Department and Health Department. However, the BPK Strategic Planning did not specifically state the performance audit topic. Thus, the audit topic selected by the audit unit was very diverse even though there was a general audit theme in Strategic Plan. The topic was chosen by Board Members and senior management, usually related to government programs, prioritizing several important topics such as education, health, public infrastructure, and social security system. In this period, BPK tries to formulate a report that summarizes the audit report results from 2009 to 2014. However, audit conclusions at the national level could not be formulated due to limited audit scope. Therefore, the government's performance in executing the Government's Medium-Term National Development Plan (Rencana Pembangunan Jangka Menengah Nasional, RPJMN) cannot be evaluated.
From 2016 to 2020, performance audits were directed to monitor the Government's Medium-Term National Development Plan 2016-2020. BPK Strategic Plan 2016-2020 contains focus or themes of performance audit for the next five years. Thus BPK started to conduct the thematic performance audit, which audit was focused on a particular theme. BPK created a technical team, a centralized coordination unit, and a Working Group to assist thematic performance audit teams that usually including auditors from the head office and regional offices. The thematic audit aims to formulate a conclusion on the implementation of government programs for five years period. In addition, the thematic audit is expected to provide insight into public financial management (Pramono, 2016). Hence, performance audit coverage should be sufficient, and BPK used a statistical method to generalize the audit results at the national level.

Performance Audit Development: New Experiences and Trend Ahead
In recent years, BPK has participated in several international performance audit events. Another new development is related to the audit of the COVID-19 Pandemic. It is an extensive audit scale because BPK employs financial, performance, and compliance audits simultaneously, and the audit's scope covers both central and local governments' responses to the COVID-19 Pandemic. In this audit, the concept of combine audit, integrated audit, and comprehensive audit is introduced. Furthermore, the BPK is also introducing recommendations at the public policy level derived from performance audit results, such as managing the National Health Insurance Programme (2020) and Special Autonomy Fund for Papua and West Papua Province. BPK also tries to develop its capability by introducing the usage of governance audit, big data analytics to support the insight and foresight function of BPK. These new developments require BPK to respond appropriately by upgrading its performance audit methodology and guidance.

The development of Performance Audit Guidelines and Manuals
As part of the capacity development, BPK develops performance audit manual and performance audit guidelines. These performance audit guidelines and manuals are revised regularly to adapt to the new development in the performance audit. The first Performance Audit Guidance was introduced in 2008. This performance audit guideline is revised several times since performance audit topics are getting more various and complex.  Figure 2 describes the structure of audit standards and performance audit guidelines in BPK.

Performance Audit Guidelines (2008)
The first performance audit guideline developed by BPK is Performance Audit Guidelines 2008. Before these performance audit guidelines were developed, the performance auditors used international performance audit guidelines to conduct the audit. The 2008 Performance Audit Guideline substance in-cludes an overview of Performance Audit, guidelines for planning, conducting, and reporting audits. The audit methods introduced in this guidance are data finding and data testing. The overview of the 2008 Performance Audit Guideline is shown in Appendix 1. Based on peer review in 2009, the Netherlands Court of Audit concluded that BPK Performance Audit Guideline had complied with INTOSAI and ASOSAI guidelines. This guidance has also provided auditors with a clear concept of performance audit (3E's with examples) and its objectives, and basic competency of auditor needed. This guidance also guides the auditor to step the performance audit process from planning to reporting stage, explaining its relationship between many activities, and documenting the process to get the finding, conclusions, and recommendations. This guidance was used as a quality control measurement. BPK revised this guidance in 2011 to make the performance audit guidelines comply with international audit standards.

Performance audit guidelines/technical audit guideline (2011)
The 2011 Performance Audit Guidelines were introduced to revise several weaknesses of the 2008 Performance Audit Guidelines. This guideline referred to international audit standards, with several adjustments to comply with national regulations. The 2011 Performance Audit Guideline substance still includes an overview of Performance Audit, guidelines for planning, conducting, and reporting audits but in a more detailed and comprehensive manner. There are two approaches introduced in these new guidelines, namely process and result performance audit approach. However, the process approach using Good Management Model was more popular than the results approach. In addition to these audit guidelines, there are also two technical guidance, namely Technical Guidance for Key Area Selection and Technical Guidance for Developing Performance Audit Criteria. The overview of the 2011 Performance Audit Guideline is shown in Appendix 2.

Supplementary guideline or technical guidelines (2015 -2019)
During amples of responses from the audited entity/ es.
The Technical Guideline in Developing Performance Audit Report explained the minimum content of Performance Audit Report in BPK. The guidelines give more flexibility in reporting performance audit results -different with financial audit report format, more alternatives of format, more flowing and avoid blocking system, presentation of table and graphic, also not providing "template" but minimum elements of the report. The guidelines also give many examples of the part of performance audit reports to broaden the auditor's perspective. While Supplementary Guideline of Performance Audit Design introduces examples of performance audit designs, namely the process/ system approach, the problem approach, the results approach, and the mixed approach. The approaches presented are combined with aspects of the economy, efficiency, and effectiveness with many examples.

Performance audit guidance (2020)
In 2020, the Indonesian style of the fully adapted international standard introduced the new revision of performance audit guidance. The overview of the 2020 Performance Audit Guideline is shown in Appendix 3. This guidelines substance includes an overview of performance audit, defining performance audit topic, guidelines for planning, conducting, and reporting audit also follows up monitoring audit. The 2020 Performance Audit Guideline introduced direct reporting engagement, strategic planning, the concept of significance/materiality, and sampling techniques using probability or nonprobability sampling. It is also introduced data analysis using quantitative and qualitative methods.
The revision of the previous performance audit guidance is also introduced problem approach, the audit design matrix, audit finding matrix, risk aspect in the performance audit, auditor competence, communication in audit, and follow-up audit. In response to the audit of the COVID-19 pandemic, BPK has also been released Technical guidance on performance audit in emergency period (2020) and revision of Supplementary Guideline in Formulating Audit Conclusion (2020) with many implementable examples.

The Performance Audit Methods
Before 2010, all of the performance audits have been conducted are individual audits. The topic of the individual audit varies from local government, state-owned enterprises, local hospitals, water producers until ministerial program and project at the national level. However, in 2010-2015 and 2016-2020, there is a tendency of introducing "thematic" performance audit in BPK. This process is alongside the process of assisting the implementation of the RPJMN. As a consequence, the percentage of thematic audit is increasing dramatically, reaching 72,66% in 2018. Furthermore, from the data found in 2018, 97% audit objective conducted by BPK is effectiveness, and the rest is related to economy and efficiency.
In terms of methods used, the evidencegathering technique employed by BPK slightly changes in response to the changing audit process and the problem the auditors face. Appendix 4 summarizes the methods used by BPK during 2016-2020. Appendix 4 depicts the performance audit methods used, alongside the data analysis methods. Interviews, observation, survey/questionnaires, and document examination are the most common data gathering methods. This finding is similar to previous studies that found interviews, observation, survey/questionnaires, and document examination as the core of performance audit methods (Lonsdale, 2000(Lonsdale, , 2011. It is interesting to examine the development in performance audit methods during this period. Review of the electronic database increases from 10 audits in 2016 to 61 audits in 2018, then declines to 8 audits in 2020. A similar pattern was also observed for the walkthrough. Pandemic COVID-19 might explain the decrease of the walkthrough, but further investigation is needed to find the cause of the decrease in the usage of review of the electronic database. Overall, root cause analysis (RCA) is the most popular qualitative data analysis, while the analytical hierarchy process (AHP) is the most popular quantitative data analysis. It is not surprising that RCA is popular because performance auditors need to identify the root causes of a problem. Based on this root cause identification, the performance auditors can formulate audit recommendations. AHP is used to draw a conclusion. This method is unique for BPK because it is not used by others SAIs. AHP is introduced by several auditors that just completed master's and doctoral educations.
The historical review reveals that the emergence of the performance audit is mainly due to the intense attention to the public sector, the restructuration of the major public sector, SAI-led initiatives, similar developments in other countries, the initiative of the ruling government, and the presence of mandates to conduct this type of audit (Manaf & Athirah, 2010). Furthermore, a performance audit is an essential tool for achieving accountability and transparency and promoting efficiency and effectiveness (Leeuw, 2011, in Lonsdale, 2011. While performance audit was implemented in developed countries since the 1970s following new public management, performance audit is not formally recognized in Indonesia until 2006. The performance audit was formally introduced in Indonesia after Indonesian public financial management was reformed. Indonesia made major reforms and implemented new public financial management by introducing decentralization, result-based management, performance -based payment (Brinkerhoff & Wetterberg, 2013), and improving financial transparency and accountability by introducing accrual accounting (Mir & Sutiyono, 2013). The development of performance audits in BPK is driven by the adoption of public financial management and the introduction of the new law that strengthening the role of BPK as a supreme audit institution (Parker et al., 2018;Pollitt et al., 1999). Thus, the development of performance audits in BPK is similar to developing performance audits in developed countries.
Initially, BPK did not have sufficient personnel with good knowledge and skills to conduct performance audits. However, this study finds that the current audit methods employed by BPK are relatively similar to performance audit methods employed by other SAIs. BPK takes several steps to develop performance audit capacity by entering into bilateral cooperation with other SAIs and sending its auditors to study abroad. BPK receives assistance from ANAO to develop performance audit capacity. BPK also sends its auditors to the USA, Australia, New Zealand, The Netherland, Malaysia, and South Africa to study performance audit practice through secondment and short courses. In addition, the peer review process also has a significant contribution to the development of performance audits. Peer reviewers identify the weaknesses of performance audits in BPK. The BPK's Board Members usually give a high priority to peer review findings. These findings indicate that the role of SAIs from developed countries influences the development of performance audits in BPK.
The secondment programs conducted in several SAIs can improve auditors' knowledge and skills. Because BPK's auditors are mostly accountants, BPK provides scholarships to its auditors to take master's degrees in other disciplines such as public policy, public administration, environmental studies, and others. BPK also recruits many nonaccountant auditors to bring diversity to the educational background of its auditors. Performance audit methodology is relatively similar to scientific research. Therefore BPK provides research methods training to most of its auditors. The existing auditors are also encouraged to acquire knowledge through short courses or training from the international community. Thus, the decision was taken by BPK's senior management to send its auditors to take post-graduate degrees also has an impact on the development of performance audit in BPK.

CONCLUSION
From the discussion above, it can be concluded that the development of the performance audit in BPK is influenced by the implementation of new state finance law and the implementation of new public financial management in Indonesia. The assistance by several SAIs such as ANAO, GAO, and AOG New Zealand also drive the development of BPK's performance audit. At the same time, the peer-review process also accelerates the development of performance audits in BPK. Finally, secondment programs and short courses also impact the development of BPK's performance audit capability.
The results of this study also indicate that other SAIs and International Audit Standards influence the development of performance audit methodology in BPK. Another aspect that influences BPK's performance audit methodology is the dynamics of the government's medium-term planning and the BPK's board member's focus. SMEs also have an essential role in the development of the BPK's performance audit guidelines. In addition, it is also supported by secondary aspects such as seminars, workshops, and training accepted by BPK's auditor. BPK has a plan to provide the government with foresight about several scenarios and policy alternatives. In order to give meaningful foresight, BPK needs to improve its audit methodology. Authors recommend BPK obtain other knowledge and approaches such as governance auditing, big data analytics, and applying aspects of public policy to performance audits.