We’re sorry, something doesn't seem to be working properly.
Please try refreshing the page. If that doesn't work, please contact us so we can address the problem.
A case study on the improvement of institution of “High-Risk High-Return R&D” in Korea
© The Author(s). 2017
Received: 17 August 2017
Accepted: 5 September 2017
Published: 29 September 2017
Despite the increase of government spending on R&D in South Korea, there have been limits in enhancing the challenging trait and creativity of research outcomes. A new approach to the current mode of R&D is considered necessary to tackle this problem. In 2015, South Korea operated fifteen programs, namely “High-risk High-return R&D,” from seven government ministries. The purpose of this study is to examine the actual conditions for adoption and to further promote early establishment and wide implementation of the new “High-risk High-return policy” in National R&D, and to suggest ways to improve it.
In this study, we have approached the case with a life-cycle perspective of plan-management-evaluation of R&D by carrying out a survey and unstandardized interviews with key staff from R&D management agencies. Based on the results of the analysis, we suggest improvements in three aspects: 1) flexible system operation, 2) government ministries’ autonomy and accountability, 3) effective incentives.
Finally, we discuss possible improvements, future directions, and the limits of this study.
It has been pointed out that the success rate of government R&D in Korea was 98.5% in 2009 and 88.0% in 2012 (Jin Seong-gi, 2013), which is higher than that of advanced countries, while the actual performance is lower.
The main reason for this phenomenon is that government support is mainly focused on completing safe projects that do not have a risk of failure, rather than selecting and supporting projects that are innovative and powerful but difficult to achieve. The presidential directive at the “Science-Technology Strategy Council” in May 2013, calling for an increase in the performance of R&D investment by activating challenging R&D programs through the establishment of a research management system that accepts honorable failure, can be seen as evidence of this issue.
Another reason is that the current research environment is not suitable for innovative research, and there is a lack of a research system that supports challenging and creative research from a long-term perspective, beyond a simple quantitative evaluation.
Since 2013, Korea has been implementing a High-risk High-return R&D system to improve these problems. However, unlike the original intention of the implementation process, it is being implemented only in a limited number of projects, and the system is being operated and managed without any distinction from existing projects.
The purpose of this study is to examine the current status for adoption, and to further promote early establishment and wide implementation of the new “High-risk High-return policy” on National R&D, and to identify the ways to improve it.
Research background & methodology
Overview of High-Risk High-Return R&D Institution
As for the progress of the High-risk High-return R&D institution, in February 2013, regulations related to the management of national R&D programs laid the foundation for High-risk High-return R&D programs.1 Based on these, in August 2013, the guideline for promoting High-risk High-return R&D was established by the 2nd regular session, NSTC (2013) and in June 2016, the guideline for promoting High-risk High-return R&D was reestablished by the 22th expert committee, NSTC (2016).
As for the main contents, each ministry promoted the development and operation of 2014–2015 High-risk High-return R&D pilot programs and presented criteria for the selection of the projects, the mid-term/final evaluation, and Honorable Failure. It was made clear at the time of the announcement that a project was a “High-risk High-return R&D project.” As an evaluation criteria during the selection evaluation, a rating ratio was applied of at least 50% of the indicators related to the challenging trait of the research goals and the creativity of the research contents, and 20% or more of the items related to the research capacity and ethics level of the research supervisor. In the case of annual and phase evaluations, it was suggested that it is not necessary to conduct an interim evaluation, and that, if conducted, the evaluation should be conducted for the purpose of adjusting and supplementing the research contents. The final evaluation focuses on the R&D performance such as the achievement of research objectives.
Guidelines on the Evaluation Standards for Honorable Failure
Criteria for Evaluation
Integrity of research process
Reason for under-achievement
∙ Challenging goals
∙ Failure to meet goal due to external factors
Appropriateness of method and process of research
∙ Evaluation of research method (whether it was retried after failure)
∙ Evaluation of research process (sincere record of research notes, demonstration of progress)
Judgment of value derived from research process
Effect of results
∙ Project performance of the results derived from research process
∙ Research in a whole new field and originality of the method
Contribution to follow-up research
∙ Research results that can help with follow-up research
After that, the results of the pilot program and the plan for the next year were reviewed twice, in April 2015 and February 2016. In the inspection, the department performance was checked and eligibility for additional points was surveyed during the department interim evaluation. In the 2016 Government R&D Innovation Plan (National Science and Technology Strategy Council, 5.12.2016), the acceleration of R&D innovation through strengthening support for challenging, creative, and strategic research was reported in order to shift the paradigm in national R&D programs.
Looking at the results of promotion of the High-risk High-return R&D program, in 2014, ten departments and agencies, including the Ministry of Science, ICT and Future Planning and the Ministry of Trade, Industry and Energy, established a plan to designate and implement 24 programs (budget KRW 581.9 billion). Of these programs, only eight programs (KRW 200.6 billion) in six departments (34.5% of the original plan) were actually implemented. In 2015, ten departments and agencies, including the Ministry of Science, ICT and Future Planning and the Rural Development Administration, planned 27 programs (budget KRW 469.1 billion). Of these, 72.8% of the programs were implemented (seven departments and agencies and KRW 342.1 billion).
2013–2015 High-risk High-return status in government R&D plans
Number of Agendas Submitted
Plenary session: 27
Steering committee: 68
Plenary session: 29
Steering committee: 73
Plenary session: 5
Steering committee: 50
8 out of 19 plans
(MSIP, MOTIE, MCST)
3 out of 24 plans
2 out of 4 plans
7 out of 42 Plans
12 out of 47 Plans
(MSIP, MOTIE, MND, SMBA)
8 out of 26 Plans
Policy trends: Domestic and overseas
In the case of Korea, the precedent studies of high-risk high-return R&D research, which covers a broader range of R&D systems including challenging and creative R&D, are as follows: First, there are studies by Lee Min-Hyung (2013, 2014) and Kim Sang-Il (2013) with a research approach from the perspective of an R&D paradigm shift. The studies made the criticism that, from the 1990s, Korea’s R&D system has not been successfully transformed from a “fast-follower type” into a “first-mover type” to lead the world. As an alternative, these studies suggested development of the national science and technology policy direction and improvement of the efficiency of national R&D programs.
Second, there is a study by Lee Ju-Ho et al. (2014) on the method of supporting national R&D programs from the perspective of bureaucratic control. This study tried to investigate the vulnerable causes of High-risk High-return research in the transition of Korea’s R&D system and problems of bureaucratic control in the process. In addition, Cho Hyun-Dae et al. (2014) pointed out that in the national R&D programs in Korea, individual ministries have strong control over the subordinate planning/management agencies, and projects are created according to the situation and needs of individual departments, hampering efficiency.
Third, through a research approach from the perspective of investment efficiency, Song Chi-Woong et al. (2013) raised concerns that researchers from universities and institutes in Korea are aiming for overly safe research and that it is not contributing to solutions for urgent practical problems or creating new opportunities. Park Ki-Bum et al. (2013) presented the direction of the R&D program to enhance the investment efficiency of the national R&D program and the competitiveness of the university. In addition, Cho Hyun-Dae et al. (2014) suggested the improvement of basic research investment allocation and the basic research fund support system.
Finally, Kim Hyun-Min et al. (2014) can be cited for a research approach from the perspective of deregulation. This study argued for the promotion of performance for realizing a creative economy, deregulation to create a researcher-friendly environment, and establishment of a rational national science and technology innovation system.
The following are examples of High-risk High-return research conducted by some advanced countries referred to a study by Lee Il-Hwan (2015). First, the United States has been expanding its support for challenging and adventurous programs to strengthen the leading research. The Defense Advanced Research Projects Agency (DARPA) is attempting a competitive R&D strategy different from the existing one in its strategy and operation method in order to conduct leading and challenging R&D (DOE, 2014). In addition, since 2007, the National Science Foundation (NSF) has emphasized the potential innovation of research in selecting projects in order to strengthen the challenging trait and adventurousness of R&D, and has expanded the scope of research innovation from 2013 on the recommendation of the National Science Board (NSB) (NSF, 2015).
Second, in Japan, strategic innovation is pursued through the flexibility of R&D program management. Since 2004, the Advanced Technology Exploration Research (ERATO)2 http://www.jst.go.jp/erato/ http://www.jst.go.jp/impact/ program of the Japan Science and Technology Agency (JST) has been reinforcing the flexible management of R&D by providing an appropriate amount of research funds and granting researchers full authority to conduct R&D, including revising research objectives.
In addition, the Impulsing Paradigm Change through Disruptive Technologies program (ImPACT) of the Council for Science, Technology and Innovation (CSTI), which has been in operation since 2013, has been supporting challenging projects that have a high probability of failure but will bring about changes that transform the social and industrial paradigm if they succeed (Yuko Harayama, 2014).
In the case of the UK, a research environment is being established to maximize creativity and encourage challenging research. For example, in 2004, the “IDEAS Factory” approach was introduced to connect research with innovative ideas. And the IDEAS Factory has been holding workshops called “sandpits” at which 20 to 30 experts from various fields collaborate to solve challenging problems.3
Europe is focusing on creating jobs through supporting technology commercialization. “Horizon 20204 http://ec.europa.eu/programmes/horizon2020/” is a program for promoting challenging R&D projects in Europe. Horizon 2020 is meant to preemptively carry out projects centering on challenging R&D projects that reflect high interest in smart growth, responding to climate change, energy efficiency, and public welfare. In addition, Horizon 2020 focuses on maximizing R&D achievements such as creativity through streamlining of administrative procedures and simplification of the structure of program operations, and emphasizes international cooperation and utilization of intellectual property rights to create performance.
As the role of science and technology becomes significant in securing the nation’s competitiveness and as the R&D budget increases accordingly, the need for a life-cycle system for the efficient management of national R&D is growing. Hwang et al. (2016) reviewed the issues on the national R&D Life-Cycle System as follows:
The arguments on the life-cycle system of the National R&D programs have addressed the need for developing a coordination system, aiming to increase the effectiveness and efficiency of R&D investment by closely linking the analysis and evaluation of the program with budget allocation (Jeong et al. 2005; Oh 2006; Lee 2006; Kim et al. 2008; Park and Lee 2009; Ahn et al. 2014).
The research analyzes the current status of adoption of the High-risk High-return R&D policy from a macro perspective concerning the life-cycle aspect of R&D, involving the process of planning (PLAN) – R&D (DO) – evaluation (SEE). Such a life-cycle perspective enables the research to contribute to enhancing the efficiency of national R&D investment by fulfilling the initial purpose of introducing the High-risk High-return R&D policy.
In this research, we carried out a survey and unstandardized interviews with key staff from R&D management agencies. The survey and interviews began on April 18, 2016 and ended on April 30, 2016. Analysis was carried out after survey results were collected from twelve organizations and interviews were conducted with twelve interviewees. In addition, we also referred to unstandardized interviews with experts in the R&D management agencies in order to bring out possible methods of improvement from the research field perspective.
The survey was analyzed on the basis of each measurement result (7-point Likert scale) of each category of the questionnaire. To prove the statistical validity of the measurement results, a one-sample t-test was carried out (Kang & Kim, 2005).
Analysis of current status of High-Risk High-Return Type R&D
Usefulness of High-Risk High-Return R&D institution
Usefulness of High-risk High-return R&D
Improvement of National R&D Program after adoption of High-risk High-return R&D
Preference for High-risk High-return R&D
Contribution to performance enhancement and investment efficiency of High-risk High-return R&D
Second, the positive response to the questionnaire item of “I favor the High-risk High-return R&D institution which was implemented twice (2014, 2015)” was 41.7% and the negative response was 25.0%. The difference between the test values was significant, indicating that the preference for adopting the High-risk High-return R&D institution was positive.
Third, the positive response to the questionnaire item of "The High-risk High-return R&D institution contributes to the improvement of investment efficiency and the improvement of the performance of national R&D programs" was 66.6% and the negative response was 25.0%. The difference from the test value was also significant, and it can be seen that the introduction of the High-risk High-return R&D institution contributes to the enhancement of investment efficiency and performance of national R&D programs.
Overall satisfaction with High-Risk High-Return R&D Institution
Overall satisfaction with High-risk High-return type R&D
Overall satisfaction with High-risk High-return R&D
Analysis of current status from the R&D life-cycle perspective
Planning of High-Risk High-Return R&D
Regarding the feasibility of the program objective, the question "Do you think that the project you are promoting corresponds with the breakthrough or innovative goal defined by the High-risk High-return R&D program?" showed results of 50.0% for “Yes, it corresponds” and 50.0% for “No, it does not correspond.” These results prove that there is a slight lack of distinctive planning factors of High-risk High-return R&D that differentiate it from existing R&D.
☆☆☆Agency, Team Leader, △△△△△: “There is no distinction between the case of giving a high score on challenging trait and creativity in the existing R&D program and the case of promoting a High-risk High-return R&D program. It is necessary to present guidelines based on clear definitions of the concepts of creativity and challenging trait so that it can be easily carried out by the ministries and R&D management agencies.”
The above non-standardized interview results can be regarded as a case in which the High-risk High-return R&D concept is divided into challenging and creative R&D, and supplementation to the existing guidelines reflecting the characteristics of the program are needed.
Management of High-Risk High-Return R&D
Management stage of High-risk High-return R&D
Guidance on the categorization of High-risk High-return R&D
Second, the question “Does your organization have its own regulations or guidelines including the contents of the High-risk High-return R&D program guidelines?” showed results of 50.0% for “No, regulations/guidelines are not provided” and 33.3% for “Yes, regulations and amendments are provided.” This can be seen as a limit to the full development of the researchers’ challenging trait and creativity by using the management and evaluation systems of existing projects.
Evaluation of High-Risk High-Return R&D
Evaluation of High-risk High-return R&D
Selection by evaluation criteria
Final evaluation focusing on R&D performance
However, due to the mixed concept of the existing High-risk High-return R&D, the evaluating field simultaneously applies the evaluation factors of “Challenging R&D (challenging trait of research goal)” and “Creative R&D (research content’s creativity),” and it is necessary to separate the standards.
Second, 25.0% of those surveyed responded positively to the questionnaire item “During the project, mid-term evaluation is not carried out for the immersion of research” while 66.7% responded negatively. This showed that there are many institutions that conduct interim evaluation that may interfere with researchers’ engagement in the research.
Third, the positive response to the questionnaire item of “When the final evaluation is made for the High-risk High-return R&D program, the focus of the evaluation is centered on the R&D achievement” was 91.7% and the negative response was 8.3% As shown in Table 6, the difference from the test value was significant, indicating that the High-risk High-return R&D program focused on R&D achievements in the final evaluation.
Measures after evaluation
Measures after Evaluation
Prize for outstanding research results
In the case of Honorable Failure, exemption from penalties and provision of opportunities to retry
Publishing cases of valuable failure
Second, 41.7% of those surveyed responded positively and 33.3% responded negatively to the questionnaire item “In the case of Honorable Failure, the researchers are exempted from penalties and given an opportunity to retry.” As shown in Table 7, the difference from the test value was significant, indicating that Honorable Failure was exempted from penalties and an opportunity to retry was given.
Third, the positive response to the questionnaire item "Cases of excellent and valuable failure are published" was very low (8.3%), while the negative response was high (83.3%). As shown in Table 7, the difference from the test value was significant, indicating that the publication of cases of excellent and valuable failure is not systematically conducted.
Discussion: Methods for improving the performance of High-Risk High-Return R&D
In the following section, we propose improvements to complement the overall weaknesses of the High-risk High-return R&D system and activate the system.
Securing flexibility of system operation
As a result of the survey on current status, it was found that the ambiguity of the High-risk High-return R&D concept and insufficient operation of the system limited the intended drive to pursue challenging and creative R&D. As an alternative, we propose securing flexibility in the operation of the system by dividing and classifying High-risk High-return R&D into Creative R&D and Challenging R&D, while maintaining the existing exception of the common management regulations.5
Specifically, creative R&D is to be limited to projects that can contribute to the creation of future value, such as academic development and creation of new knowledge, by introducing and applying creative research themes and methods. In addition, challenging R&D is a project with high industrial and economic effects and high utilization of public welfare, and is to be developed as a program with specific challenging goals and a clear purpose of utilization of performance.
Characteristics and Operation of High-risk High-return R&D
High-risk High-return R&D
Characteristics of Project
Themes and research methods are creative.
Research goals are challenging and risk-taking
Setting of Goal
Emphasis on creativity based on various research methods
Challenging goals that are difficult to achieve
(Allowing moving targets considering changes in external environment)
Centered on free participation
(researcher-centered bottom-up planning priority)
Free participation + Designated participation
(top-down planning through PD/PM)
Increased flexibility of project evaluation and research funding
(e.g. strengthening selection evaluation, shortening annual/step evaluation, ensuring continuity and accountability of evaluators, flexible use of research grants)
Undesignated project (centered on small projects), field-designated project (centered on large projects)
Various configurations according to target difficulty level
Project Performance Index
Review focusing on appropriateness of operational method
Review focusing on goal achievement
High-risk High-return R&D operation method differentiated from general R&D
High-risk High-return R&D operation method differentiated from general R&D
Improvement of autonomy and accountability of ministries
The survey above pointed out that the system is not expanded well due to the lack of willingness of the ministries and the insufficient provision of regulations and guidelines by the R&D management agencies.
As for improvements, first of all, improvement of the evaluation and operation management system of the High-risk High-return R&D program shall be proposed. Specifically, in this method, the National Science & Technology Council reviews each department’s High-risk High-return R&D program eligibility and performance annually, checks performance targets and performance indicators so that items related to the challenging trait and creativity can be reflected in the evaluation of the program and the budget review, and returns it to the budget review.
In addition, the project should not be promoted as part of a research project or sub-project; rather, it must establish a unified system of budget allocation-execution-management by promoting the program in “phased program” units in order to use it as basic data for program evaluation and budget allocation/adjustment.
Second, securing the flexibility of program management are proposed in order to expand the autonomy and accountability of the ministries. Specifically, it is necessary for the ministries to autonomously plan and employ evaluation criteria and essential requirements for High-risk High-return R&D projects, and to strengthen qualitative and in-depth evaluation by experts. For example, it is believed that the program will be able to operate under various responsibilities of the ministries in accordance with various support methods such as bottom-up, top-down, mixed-system, and discussion type, and with various supporting fields such as source technology type, industrialization type, and social problem-solving type.
Introduction of effective incentives
As a result of the survey, it was found that the lack of incentives limits the participation and improvement efforts of the ministries and stakeholders. As an alternative, expansion of budget support for excellent High-risk High-return R&D programs are proposed.
Necessity for Sufficient budget support
Need for sufficient budget support
Specific examples of incentives include: ① Reviewing priority support for High-risk High-return R&D programs when allocating and coordinating the R&D budget of the ministries, ② Reflecting in deliberation of planning and evaluation management expense of ministries (R&D management agencies) according to the challenging trait and creativity level, ③ Priority support for patent application or follow-up R&D in the same technology field, and ④ Priority support for technology transfer and commercialization.
Reinforcement of compensation system
Reinforcement of compensation system
Specifically, in the case of creative R&D, for stable research and to increase research commitment, it is necessary to promote institutionalization such as granting the maximum level of autonomy of research contents and research period within the total research expenses, and granting additional points in the case of supporting follow-up projects. In addition, in the case of challenging R&D, it is necessary to acknowledge the moving targets such as the research targets and methods, considering external environment, market, and technology change, and to support the subsequent commercialization of excellent projects showing early success and an effect.
Conclusion and further research
In this empirical study, we have analyzed the current status of implementation and adoption of High-risk High-return R&D, which was first introduced in 2013. The results from the analysis of the survey showed that the recognition of usefulness and overall satisfaction were high.
These results prove that there is a slight lack of distinctive factors of High-risk High-return R&D to differentiate it from existing R&D in planning. And although it is identified as a High-risk High-return R&D Program at the time of program announcement, there is a limit to the full development of the researchers’ challenging trait and creativity due to use of the same management and evaluation systems as those for existing projects. Also, due to the mixed concept of the existing High-risk High-return R&D, the evaluating field simultaneously applies the evaluation factors of “Challenging R&D (challenging trait of research goal)” and “Creative R&D (research content’s creativity)” in the selection evaluation. And there are many institutions that conduct interim evaluation that may interfere with researchers’ engagement in the research. In addition, the High-risk High-return R&D program focuses on R&D achievements in the final evaluation. In the measures upon evaluation, Honorable Failure is exempted from penalties and an opportunity to retry is given; however, no award is implemented for outstanding research achievements. Additionally, the publication of cases of excellent and valuable failure is not systematically conducted, which needs improvement.
Furthermore, this research gives diverse improvement policies. First, we propose dividing and classifying High-risk High-return R&D into Creative R&D and Challenging R&D to secure flexibility in the operation of the system. Second, improvement of the evaluation and operation management system and securing the flexibility of program management of High-risk High-return R&D programs shall be proposed in order to expand the autonomy and accountability of the ministries. Third, expansion of budget support for excellent High-risk High-return R&D programs, the increase of research autonomy and follow-up support for outstanding researchers and outstanding projects shall be proposed to introduce effective incentives.
We are aware of the limitations and shortcomings of this research. The quantitative data was collected mainly in relation to individual recognition over the process of management of High-risk High-return R&D. For future research, it would be useful to compare the status of R&D management agencies before and after the introduction of High-risk High-return R&D.
Regulations on the Management, etc. of National Research and Development Programs Article 33–4 (Special Provisions on High-risk High-return R&D Programs) ① An R&D program that can significantly contribute to academic advancement or enhancement of public welfare ② An R&D program that is innovative and has high industrial utility and therefore may produce high profit or form a new industrial group or market (Established 2.22.2013)
Regulations on the Management, etc. of National Research and Development Programs Article 33–4 (Special Provisions on High-risk High-return R&D programs) as ① An R&D program that can significantly contribute to academic advancement or enhancement of public welfare, and ② An R&D program that is innovative and has high industrial utility and therefore may produce high profit or form a new industrial group or market.
BYH developed concept and wrote the paper. HJJ collected data and analyzed policy trend. MHC performed statistical analyses. DCK designed the research and wrote the paper. All authors read and approved the final manuscript.
The authors declare that they have no competing interests.
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Open AccessThis article is distributed under the terms of the Creative Commons Attribution 4.0 International License (http://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license, and indicate if changes were made.
- Ahn, S.-J., Park, E.-j., & Lee, Y. (2014). Exploration of life-cycle Management for Government R&D program: The case of preliminary feasibility study on R&D program. Journal of Korea Technology Innovation Society, 17(1), 124–145.Google Scholar
- DOE. (2014). Basic energy sciences summary report: Basic energy science. DOE. Google Scholar
- Government R&D Innovation Plan, Science-Technology Strategy Council, May 12, 2016.Google Scholar
- Yuko Harayama, (2014), “Innovation Policy or Innovative Policy,” Council for Science, Technology and Innovation (CSTI) Cabinet office.Google Scholar
- Byung Yong Hwang, Jee Hyun Suh, et al. (2016), “An Empirical Study on the Improvement of R&D Competition Policy for the National R&D Program,” Proceedings, SOITmC & CSCOM 2016 Conference.Google Scholar
- Jeong, K. H., Mun, J. K., Park, M., & Park, B. (2005). A study on improvement of Evaluation & Budget Coordination System for effective NRDP. Journal of Korea Technology Innovation Society, 8(1), 183–208.Google Scholar
- Jin Seong-gi (2013), “Don’t measure the creative economy and government R&D by short-term performance,” Science & Technology 13–11, The Korean Federation of Science and Technology Societies.Google Scholar
- Cho Hyun-Dae et al. (2014). An analysis of Korea’s national supporting system of basic research and policy recommendations for leading creative R&D, STEPI policy research 2014–23. Seoul: STEPI.Google Scholar
- Kang, B.-s., & Kim, K.-s. (2005). Social science statistical analysis. Seoul: Data Solution.Google Scholar
- Hyun Min Kim et al. (2014), Management System of National R&D Program of Korea based on the research fields, KISTEP.Google Scholar
- Kim, M., Lee, H., Choi, C., Choi, K.-i., & Jeon, J. (2008). A model and its application of performance monitoring, evaluation, and management system for national R&D. Journal of Korea Technology Innovation Society, 11(4), 613–638.Google Scholar
- Kim, S. I. (2013). A research for reaction to R&D paradigm shift. KISTEP. Google Scholar
- Il Hwan Lee (2015) “Efficient Implementation of High-risk High-return R&D” Internal Project Research Paper, 2015.11, KISTEP.Google Scholar
- Lee, J. W. (2006). Management and utilization of the information from the government R&D evaluation systems, STEPI policy research 2006–07. Seoul: STEPI.Google Scholar
- Lee, Ju-Ho et al. (2014). A reform agenda for the promotion of high-risk, high-payoff research. KDI FOCUS, 49.Google Scholar
- Lee, M. H., & Lee, H. J. (2014). Developing Government R&D Project Management System Focused on problem-solving, STEPI policy research 2014–11. Seoul: STEPI.Google Scholar
- Lee, M. H. (2013). The direction and challenges of National Innovation System for realizing creative economy. STEPI Issues & Policy, 27.Google Scholar
- National Science Foundation (2015), National Science Foundation's Merit Review Process Fiscal Year 2014: NSF.Google Scholar
- NSTC. (2013a). Guideline for Promoting High-Risk High-Return R&D, submitted proposal article 5, the 2nd regular session. National Science & Technology Council, August, 2, 2013.Google Scholar
- NSTC (2016) Guideline for Promoting High-Risk High-Return R&D, the 22th NSTC expert committee proposal article 5, June 27, 2016.Google Scholar
- Oh, D. H. (2006). A study on the strategic improvement of priority-setting for Investment in the National R&D. Seoul: STEPI.Google Scholar
- Park, B., & Lee, K.-J. (2009). A study on the Government R&D Budgeting Evaluation System in South Korea. Journal of Korea Technology Innovation Society, 12(4), 819–839.Google Scholar
- Park, K. B., et al. (2013). Future science and engineering talents, STEPI policy research 2013–14. Seoul: STEPI.Google Scholar
- Chi, W. S., Sung, B. H., Yong, S. J., Kee, G. K., Jung, H. S., A Feasibility Study on the Introduction of K-ARPA System, STEPI Policy Research 2014–18. Seoul: STEPI.Google Scholar