A case study on the improvement of institution of “High-Risk High-Return R&D” in Korea

Abstract

Despite the increase of government spending on R&D in South Korea, there have been limits in enhancing the challenging trait and creativity of research outcomes. A new approach to the current mode of R&D is considered necessary to tackle this problem. In 2015, South Korea operated fifteen programs, namely “High-risk High-return R&D,” from seven government ministries. The purpose of this study is to examine the actual conditions for adoption and to further promote early establishment and wide implementation of the new “High-risk High-return policy” in National R&D, and to suggest ways to improve it.

In this study, we have approached the case with a life-cycle perspective of plan-management-evaluation of R&D by carrying out a survey and unstandardized interviews with key staff from R&D management agencies. Based on the results of the analysis, we suggest improvements in three aspects: 1) flexible system operation, 2) government ministries’ autonomy and accountability, 3) effective incentives.

Finally, we discuss possible improvements, future directions, and the limits of this study.

Introduction

It has been pointed out that the success rate of government R&D in Korea was 98.5% in 2009 and 88.0% in 2012 (Jin Seong-gi, 2013), which is higher than that of advanced countries, while the actual performance is lower.

The main reason for this phenomenon is that government support is mainly focused on completing safe projects that do not have a risk of failure, rather than selecting and supporting projects that are innovative and powerful but difficult to achieve. The presidential directive at the “Science-Technology Strategy Council” in May 2013, calling for an increase in the performance of R&D investment by activating challenging R&D programs through the establishment of a research management system that accepts honorable failure, can be seen as evidence of this issue.

Another reason is that the current research environment is not suitable for innovative research, and there is a lack of a research system that supports challenging and creative research from a long-term perspective, beyond a simple quantitative evaluation.

Since 2013, Korea has been implementing a High-risk High-return R&D system to improve these problems. However, unlike the original intention of the implementation process, it is being implemented only in a limited number of projects, and the system is being operated and managed without any distinction from existing projects.

The purpose of this study is to examine the current status for adoption, and to further promote early establishment and wide implementation of the new “High-risk High-return policy” on National R&D, and to identify the ways to improve it.

Research background & methodology

Overview of High-Risk High-Return R&D Institution

As for the progress of the High-risk High-return R&D institution, in February 2013, regulations related to the management of national R&D programs laid the foundation for High-risk High-return R&D programs.¹ Based on these, in August 2013, the guideline for promoting High-risk High-return R&D was established by the 2^nd regular session, NSTC (2013) and in June 2016, the guideline for promoting High-risk High-return R&D was reestablished by the 22th expert committee, NSTC (2016). 

As for the main contents, each ministry promoted the development and operation of 2014–2015 High-risk High-return R&D pilot programs and presented criteria for the selection of the projects, the mid-term/final evaluation, and Honorable Failure. It was made clear at the time of the announcement that a project was a “High-risk High-return R&D project.” As an evaluation criteria during the selection evaluation, a rating ratio was applied of at least 50% of the indicators related to the challenging trait of the research goals and the creativity of the research contents, and 20% or more of the items related to the research capacity and ethics level of the research supervisor. In the case of annual and phase evaluations, it was suggested that it is not necessary to conduct an interim evaluation, and that, if conducted, the evaluation should be conducted for the purpose of adjusting and supplementing the research contents. The final evaluation focuses on the R&D performance such as the achievement of research objectives.

In addition, if the final evaluation of the Honorable Failure shows that the research goal was not achieved but the research was conducted faithfully, researchers will be exempted from penalties and given an opportunity to retry. The content of the evaluation criteria for honorable failure is shown in Table 1 below.

Table 1 Guidelines on the Evaluation Standards for Honorable Failure

After that, the results of the pilot program and the plan for the next year were reviewed twice, in April 2015 and February 2016. In the inspection, the department performance was checked and eligibility for additional points was surveyed during the department interim evaluation. In the 2016 Government R&D Innovation Plan (National Science and Technology Strategy Council, 5.12.2016), the acceleration of R&D innovation through strengthening support for challenging, creative, and strategic research was reported in order to shift the paradigm in national R&D programs.

Looking at the results of promotion of the High-risk High-return R&D program, in 2014, ten departments and agencies, including the Ministry of Science, ICT and Future Planning and the Ministry of Trade, Industry and Energy, established a plan to designate and implement 24 programs (budget KRW 581.9 billion). Of these programs, only eight programs (KRW 200.6 billion) in six departments (34.5% of the original plan) were actually implemented. In 2015, ten departments and agencies, including the Ministry of Science, ICT and Future Planning and the Rural Development Administration, planned 27 programs (budget KRW 469.1 billion). Of these, 72.8% of the programs were implemented (seven departments and agencies and KRW 342.1 billion).

In addition, in the National R&D plans proposed to the National Science and Technology Council from 2013 to 2015 by eighteen ministries in total, 40 out of the 162 plans (24.7%) refer to the improvement of the challenging trait and creativity in the main R&D plans, as shown in Table 2 However, it was not enough to implement the High-risk High-return R&D institution.

Table 2 2013–2015 High-risk High-return status in government R&D plans

Policy trends: Domestic and overseas

In the case of Korea, the precedent studies of high-risk high-return R&D research, which covers a broader range of R&D systems including challenging and creative R&D, are as follows: First, there are studies by Lee Min-Hyung (2013, 2014) and Kim Sang-Il (2013) with a research approach from the perspective of an R&D paradigm shift. The studies made the criticism that, from the 1990s, Korea’s R&D system has not been successfully transformed from a “fast-follower type” into a “first-mover type” to lead the world. As an alternative, these studies suggested development of the national science and technology policy direction and improvement of the efficiency of national R&D programs.

Second, there is a study by Lee Ju-Ho et al. (2014) on the method of supporting national R&D programs from the perspective of bureaucratic control. This study tried to investigate the vulnerable causes of High-risk High-return research in the transition of Korea’s R&D system and problems of bureaucratic control in the process. In addition, Cho Hyun-Dae et al. (2014) pointed out that in the national R&D programs in Korea, individual ministries have strong control over the subordinate planning/management agencies, and projects are created according to the situation and needs of individual departments, hampering efficiency.

Third, through a research approach from the perspective of investment efficiency, Song Chi-Woong et al. (2013) raised concerns that researchers from universities and institutes in Korea are aiming for overly safe research and that it is not contributing to solutions for urgent practical problems or creating new opportunities. Park Ki-Bum et al. (2013) presented the direction of the R&D program to enhance the investment efficiency of the national R&D program and the competitiveness of the university. In addition, Cho Hyun-Dae et al. (2014) suggested the improvement of basic research investment allocation and the basic research fund support system.

Finally, Kim Hyun-Min et al. (2014) can be cited for a research approach from the perspective of deregulation. This study argued for the promotion of performance for realizing a creative economy, deregulation to create a researcher-friendly environment, and establishment of a rational national science and technology innovation system.

The following are examples of High-risk High-return research conducted by some advanced countries referred to a study by Lee Il-Hwan (2015). First, the United States has been expanding its support for challenging and adventurous programs to strengthen the leading research. The Defense Advanced Research Projects Agency (DARPA) is attempting a competitive R&D strategy different from the existing one in its strategy and operation method in order to conduct leading and challenging R&D (DOE, 2014). In addition, since 2007, the National Science Foundation (NSF) has emphasized the potential innovation of research in selecting projects in order to strengthen the challenging trait and adventurousness of R&D, and has expanded the scope of research innovation from 2013 on the recommendation of the National Science Board (NSB) (NSF, 2015).

Second, in Japan, strategic innovation is pursued through the flexibility of R&D program management. Since 2004, the Advanced Technology Exploration Research (ERATO)² http://www.jst.go.jp/erato/ http://www.jst.go.jp/impact/ program of the Japan Science and Technology Agency (JST) has been reinforcing the flexible management of R&D by providing an appropriate amount of research funds and granting researchers full authority to conduct R&D, including revising research objectives.

In addition, the Impulsing Paradigm Change through Disruptive Technologies program (ImPACT) of the Council for Science, Technology and Innovation (CSTI), which has been in operation since 2013, has been supporting challenging projects that have a high probability of failure but will bring about changes that transform the social and industrial paradigm if they succeed (Yuko Harayama, 2014).

In the case of the UK, a research environment is being established to maximize creativity and encourage challenging research. For example, in 2004, the “IDEAS Factory” approach was introduced to connect research with innovative ideas. And the IDEAS Factory has been holding workshops called “sandpits” at which 20 to 30 experts from various fields collaborate to solve challenging problems.³

Europe is focusing on creating jobs through supporting technology commercialization. “Horizon 2020⁴ http://ec.europa.eu/programmes/horizon2020/” is a program for promoting challenging R&D projects in Europe. Horizon 2020 is meant to preemptively carry out projects centering on challenging R&D projects that reflect high interest in smart growth, responding to climate change, energy efficiency, and public welfare. In addition, Horizon 2020 focuses on maximizing R&D achievements such as creativity through streamlining of administrative procedures and simplification of the structure of program operations, and emphasizes international cooperation and utilization of intellectual property rights to create performance.

Research methods

As the role of science and technology becomes significant in securing the nation’s competitiveness and as the R&D budget increases accordingly, the need for a life-cycle system for the efficient management of national R&D is growing. Hwang et al. (2016) reviewed the issues on the national R&D Life-Cycle System as follows:

The arguments on the life-cycle system of the National R&D programs have addressed the need for developing a coordination system, aiming to increase the effectiveness and efficiency of R&D investment by closely linking the analysis and evaluation of the program with budget allocation (Jeong et al. 2005; Oh 2006; Lee 2006; Kim et al. 2008; Park and Lee 2009; Ahn et al. 2014).

The research analyzes the current status of adoption of the High-risk High-return R&D policy from a macro perspective concerning the life-cycle aspect of R&D, involving the process of planning (PLAN) – R&D (DO) – evaluation (SEE). Such a life-cycle perspective enables the research to contribute to enhancing the efficiency of national R&D investment by fulfilling the initial purpose of introducing the High-risk High-return R&D policy.

In this research, we carried out a survey and unstandardized interviews with key staff from R&D management agencies. The survey and interviews began on April 18, 2016 and ended on April 30, 2016. Analysis was carried out after survey results were collected from twelve organizations and interviews were conducted with twelve interviewees. In addition, we also referred to unstandardized interviews with experts in the R&D management agencies in order to bring out possible methods of improvement from the research field perspective.

The survey was analyzed on the basis of each measurement result (7-point Likert scale) of each category of the questionnaire. To prove the statistical validity of the measurement results, a one-sample t-test was carried out (Kang & Kim, 2005). 

Analysis of current status of High-Risk High-Return Type R&D

Usefulness of High-Risk High-Return R&D institution

First, the positive response as shown in Table 3 was 50.0% for the questionnaire item of "The current system was improved when comparing the national R&D system before and after introduction of the High-risk High-return R&D institution." On the other hand, the negative response was 8.3% and the difference from the test value was also significant. In other words, the perceived usefulness of the introduction of the High-risk High-return R&D institution was positive.

Table 3 Usefulness of High-risk High-return R&D

Second, the positive response to the questionnaire item of “I favor the High-risk High-return R&D institution which was implemented twice (2014, 2015)” was 41.7% and the negative response was 25.0%. The difference between the test values was significant, indicating that the preference for adopting the High-risk High-return R&D institution was positive.

Third, the positive response to the questionnaire item of "The High-risk High-return R&D institution contributes to the improvement of investment efficiency and the improvement of the performance of national R&D programs" was 66.6% and the negative response was 25.0%. The difference from the test value was also significant, and it can be seen that the introduction of the High-risk High-return R&D institution contributes to the enhancement of investment efficiency and performance of national R&D programs.

Overall satisfaction with High-Risk High-Return R&D Institution

As shown in Table 4, the positive response was 41.7% and the negative response was 25.0% for the question "Are you satisfied with the High-risk High-return R&D institution that is currently being implemented?" The difference from the test value was significant, indicating that the overall satisfaction of the respondents was positive.

Table 4 Overall satisfaction with High-risk High-return type R&D

Analysis of current status from the R&D life-cycle perspective

Planning of High-Risk High-Return R&D

Regarding the feasibility of the program objective, the question "Do you think that the project you are promoting corresponds with the breakthrough or innovative goal defined by the High-risk High-return R&D program?" showed results of 50.0% for “Yes, it corresponds” and 50.0% for “No, it does not correspond.” These results prove that there is a slight lack of distinctive planning factors of High-risk High-return R&D that differentiate it from existing R&D.

Regarding this issue, the representatives of the ministries related to R&D program promotion point out the following points (interview with the R&D management agency in April 2016):

• ☆☆☆Agency, Team Leader, △△△△△: “There is no distinction between the case of giving a high score on challenging trait and creativity in the existing R&D program and the case of promoting a High-risk High-return R&D program. It is necessary to present guidelines based on clear definitions of the concepts of creativity and challenging trait so that it can be easily carried out by the ministries and R&D management agencies.”

The above non-standardized interview results can be regarded as a case in which the High-risk High-return R&D concept is divided into challenging and creative R&D, and supplementation to the existing guidelines reflecting the characteristics of the program are needed.

Management of High-Risk High-Return R&D

First, in relation to the program announcement, as shown in Table 5, the positive response to the survey item of “It is identified as a High-risk High-return R&D program at the time of program announcement” was 58.3%, and the negative response was 25.0%. The difference from the test value was significant, and it can be seen that identification as a High-risk High-return R&D program was being made positively at the time of program announcement.

Table 5 Management stage of High-risk High-return R&D

Second, the question “Does your organization have its own regulations or guidelines including the contents of the High-risk High-return R&D program guidelines?” showed results of 50.0% for “No, regulations/guidelines are not provided” and 33.3% for “Yes, regulations and amendments are provided.” This can be seen as a limit to the full development of the researchers’ challenging trait and creativity by using the management and evaluation systems of existing projects.

Evaluation of High-Risk High-Return R&D

First, in relation to the questionnaire item of “At the time of the project selection evaluation, the evaluation is conducted based on ⓐ Challenging trait of research objective and creativity of research contents (50% or more), ⓑ Capability and ethics level of research director (20% or more), and ⓒ Other (30%),” the positive response was 58.3% and the negative response was 16.7%. As shown in Table 6, the difference from the test value was significant, and the overall response was statistically positive.

Table 6 Evaluation of High-risk High-return R&D

However, due to the mixed concept of the existing High-risk High-return R&D, the evaluating field simultaneously applies the evaluation factors of “Challenging R&D (challenging trait of research goal)” and “Creative R&D (research content’s creativity),” and it is necessary to separate the standards.

Second, 25.0% of those surveyed responded positively to the questionnaire item “During the project, mid-term evaluation is not carried out for the immersion of research” while 66.7% responded negatively. This showed that there are many institutions that conduct interim evaluation that may interfere with researchers’ engagement in the research.

Third, the positive response to the questionnaire item of “When the final evaluation is made for the High-risk High-return R&D program, the focus of the evaluation is centered on the R&D achievement” was 91.7% and the negative response was 8.3% As shown in Table 6, the difference from the test value was significant, indicating that the High-risk High-return R&D program focused on R&D achievements in the final evaluation.

Measures after evaluation

First, the positive response to the questionnaire item “A prize is awarded for outstanding research results” was rather low (16.7%), while the negative response was 58.4%. As shown in Table 7, the difference from the test value was significant, indicating that no award was implemented for outstanding research achievements. These results suggest that, in order to attract the transition from risk-averse R&D to risk-taking R&D, a significant level of rewards and incentives for risk-taking, challenging goals is necessary.

Table 7 Measures after Evaluation

Second, 41.7% of those surveyed responded positively and 33.3% responded negatively to the questionnaire item “In the case of Honorable Failure, the researchers are exempted from penalties and given an opportunity to retry.” As shown in Table 7, the difference from the test value was significant, indicating that Honorable Failure was exempted from penalties and an opportunity to retry was given.

Third, the positive response to the questionnaire item "Cases of excellent and valuable failure are published" was very low (8.3%), while the negative response was high (83.3%). As shown in Table 7, the difference from the test value was significant, indicating that the publication of cases of excellent and valuable failure is not systematically conducted.

Discussion: Methods for improving the performance of High-Risk High-Return R&D

In the following section, we propose improvements to complement the overall weaknesses of the High-risk High-return R&D system and activate the system.

Securing flexibility of system operation

As a result of the survey on current status, it was found that the ambiguity of the High-risk High-return R&D concept and insufficient operation of the system limited the intended drive to pursue challenging and creative R&D. As an alternative, we propose securing flexibility in the operation of the system by dividing and classifying High-risk High-return R&D into Creative R&D and Challenging R&D, while maintaining the existing exception of the common management regulations.⁵

Specifically, creative R&D is to be limited to projects that can contribute to the creation of future value, such as academic development and creation of new knowledge, by introducing and applying creative research themes and methods. In addition, challenging R&D is a project with high industrial and economic effects and high utilization of public welfare, and is to be developed as a program with specific challenging goals and a clear purpose of utilization of performance.

Table 8 summarizes the characteristics and operation of High-risk High-return R&D divided and classified into the above sub-concepts.

Table 8 Characteristics and Operation of High-risk High-return R&D

Regarding the above suggestion, as shown in Tables 9, 91.7% of those surveyed responded positively to the questionnaire item of “A High-risk High-return R&D operation method differentiated from general R&D is established,” while the negative response was 8.3%. The difference from the test value was significant, indicating that the response to the questionnaire item was statistically positive.

Table 9 High-risk High-return R&D operation method differentiated from general R&D

Improvement of autonomy and accountability of ministries

The survey above pointed out that the system is not expanded well due to the lack of willingness of the ministries and the insufficient provision of regulations and guidelines by the R&D management agencies.

As for improvements, first of all, improvement of the evaluation and operation management system of the High-risk High-return R&D program shall be proposed. Specifically, in this method, the National Science & Technology Council reviews each department’s High-risk High-return R&D program eligibility and performance annually, checks performance targets and performance indicators so that items related to the challenging trait and creativity can be reflected in the evaluation of the program and the budget review, and returns it to the budget review.

In addition, the project should not be promoted as part of a research project or sub-project; rather, it must establish a unified system of budget allocation-execution-management by promoting the program in “phased program” units in order to use it as basic data for program evaluation and budget allocation/adjustment.

Second, securing the flexibility of program management are proposed in order to expand the autonomy and accountability of the ministries. Specifically, it is necessary for the ministries to autonomously plan and employ evaluation criteria and essential requirements for High-risk High-return R&D projects, and to strengthen qualitative and in-depth evaluation by experts. For example, it is believed that the program will be able to operate under various responsibilities of the ministries in accordance with various support methods such as bottom-up, top-down, mixed-system, and discussion type, and with various supporting fields such as source technology type, industrialization type, and social problem-solving type.

Introduction of effective incentives

As a result of the survey, it was found that the lack of incentives limits the participation and improvement efforts of the ministries and stakeholders. As an alternative, expansion of budget support for excellent High-risk High-return R&D programs are proposed.

Regarding the above suggestion, as shown in Tables 10, 83.3% of those surveyed responded positively to the questionnaire item of “Sufficient budget support for ministries and R&D management agencies is necessary,” while the negative response was 8.3%. The difference from the test value was significant, indicating that it is necessary to introduce practical incentives that can lead to the planning and operation of High-risk High-return R&D programs (projects) by the ministries and R&D management agencies.

Table 10 Necessity for Sufficient budget support

Specific examples of incentives include: ① Reviewing priority support for High-risk High-return R&D programs when allocating and coordinating the R&D budget of the ministries, ② Reflecting in deliberation of planning and evaluation management expense of ministries (R&D management agencies) according to the challenging trait and creativity level, ③ Priority support for patent application or follow-up R&D in the same technology field, and ④ Priority support for technology transfer and commercialization.

Second, the increase of research autonomy and follow-up support for outstanding researchers and outstanding projects are proposed. As shown in Table 11, the positive response to the questionnaire item of “Reinforcement of compensation system” was 66.7% and the negative response was 8.3%. The difference from the test value was significant, indicating that the response to the questionnaire item was statistically positive.

Table 11 Reinforcement of compensation system

Specifically, in the case of creative R&D, for stable research and to increase research commitment, it is necessary to promote institutionalization such as granting the maximum level of autonomy of research contents and research period within the total research expenses, and granting additional points in the case of supporting follow-up projects. In addition, in the case of challenging R&D, it is necessary to acknowledge the moving targets such as the research targets and methods, considering external environment, market, and technology change, and to support the subsequent commercialization of excellent projects showing early success and an effect.

Conclusion and further research

In this empirical study, we have analyzed the current status of implementation and adoption of High-risk High-return R&D, which was first introduced in 2013. The results from the analysis of the survey showed that the recognition of usefulness and overall satisfaction were high.

These results prove that there is a slight lack of distinctive factors of High-risk High-return R&D to differentiate it from existing R&D in planning. And although it is identified as a High-risk High-return R&D Program at the time of program announcement, there is a limit to the full development of the researchers’ challenging trait and creativity due to use of the same management and evaluation systems as those for existing projects. Also, due to the mixed concept of the existing High-risk High-return R&D, the evaluating field simultaneously applies the evaluation factors of “Challenging R&D (challenging trait of research goal)” and “Creative R&D (research content’s creativity)” in the selection evaluation. And there are many institutions that conduct interim evaluation that may interfere with researchers’ engagement in the research. In addition, the High-risk High-return R&D program focuses on R&D achievements in the final evaluation. In the measures upon evaluation, Honorable Failure is exempted from penalties and an opportunity to retry is given; however, no award is implemented for outstanding research achievements. Additionally, the publication of cases of excellent and valuable failure is not systematically conducted, which needs improvement.

Furthermore, this research gives diverse improvement policies. First, we propose dividing and classifying High-risk High-return R&D into Creative R&D and Challenging R&D to secure flexibility in the operation of the system. Second, improvement of the evaluation and operation management system and securing the flexibility of program management of High-risk High-return R&D programs shall be proposed in order to expand the autonomy and accountability of the ministries. Third, expansion of budget support for excellent High-risk High-return R&D programs, the increase of research autonomy and follow-up support for outstanding researchers and outstanding projects shall be proposed to introduce effective incentives.

We are aware of the limitations and shortcomings of this research. The quantitative data was collected mainly in relation to individual recognition over the process of management of High-risk High-return R&D. For future research, it would be useful to compare the status of R&D management agencies before and after the introduction of High-risk High-return R&D.