Risk Analysis and Risk Management for the Artificial Superintelligence Research and Development Process

In James Miller, Vic Callaghan, Roman Yampolskiy, and Stuart Armstrong (editors), The Technological Singularity: Managing the Journey. Berlin: Springer, 2017, pages 127-140.

12 Pages Posted: 15 Jul 2016 Last revised: 12 Nov 2019

See all articles by Anthony Barrett

Anthony Barrett

Global Catastrophic Risk Institute

Seth Baum

Global Catastrophic Risk Institute

Date Written: August 26, 2015

Abstract

Artificial superintelligence (ASI) is increasingly recognized as a significant future risk. In the absence of adequate safety mechanisms, an ASI may even be likely to cause human extinction. Thus ASI risk scenarios merit attention even if their probabilities are low. ASI risk can be addressed in at least two ways: by building safety mechanisms into the ASI itself, as in ASI safety research, and by managing the human process of developing ASI, in order to promote safety practices in ASI research and development (R&D). While ASI researchers and developers typically do not intend to cause harm through their work, harm may nonetheless occur due to accidents and unintended consequences. Thus opportunities may exist to reduce ASI risk through engagement with the R&D process. This paper surveys established methodologies for risk analysis and risk management, emphasizing fault trees and event trees, and describes how these techniques can be applied to risk from ASI R&D. A variety of risk methodologies have been developed for other risks, including other emerging technology risks, but their application to ASI has thus far been limited. Insights from risk literatures could improve on what existing analyses of ASI risk have yet been conducted. Likewise, a more thorough and rigorous analysis of ASI R&D processes can inform decision making to reduce the risk. The decision makers include governments and non-governmental organizations active in ASI oversight, as well anyone conducting ASI R&D. All of these individuals and groups have roles to play in addressing ASI risk.

Keywords: risk analysis, risk management, artificial intelligence

Suggested Citation

Barrett, Anthony and Baum, Seth, Risk Analysis and Risk Management for the Artificial Superintelligence Research and Development Process (August 26, 2015). In James Miller, Vic Callaghan, Roman Yampolskiy, and Stuart Armstrong (editors), The Technological Singularity: Managing the Journey. Berlin: Springer, 2017, pages 127-140., Available at SSRN: https://ssrn.com/abstract=2807358

Anthony Barrett

Global Catastrophic Risk Institute ( email )

Seth Baum (Contact Author)

Global Catastrophic Risk Institute ( email )

Do you have negative results from your research you’d like to share?

Paper statistics

Downloads
97
Abstract Views
933
Rank
489,057
PlumX Metrics