Autonomous weapons: ICRC addresses meeting of experts

Convention on Certain Conventional Weapons (CCW) Meeting of Experts on Lethal Autonomous Weapons Systems (LAWS), 13-16 May 2014, Geneva

Statement of the International Committee of the Red Cross (ICRC)

Mr Chairman, Your Excellencies, ladies and gentlemen,

I would like to begin by thanking you, Mr Chairman, for giving the International Committee of the Red Cross (ICRC) the opportunity to present the highlights from the experts’ meeting organized by the ICRC in March, which addressed the technical, military, legal and humanitarian issues emerging from the debate about autonomous weapon systems.

The meeting brought together representatives from 21 States and 13 independent experts with the aim of sharing perspectives and gaining a better understanding of the issues raised by these weapons.

For the purposes of the ICRC’s meeting, “autonomous weapon systems” were defined as weapons that can independently select and attack targets, that is, they have autonomy in the ‘critical functions’ of acquiring, tracking, selecting and attacking targets.

We would like to begin by presenting a few of the highlights from the meeting; this will be followed by a brief list of additional observations by the ICRC. We should stress at the outset, that these highlights are not intended to represent a convergence of views of the participants, but instead reflect some of the key points made during the discussions.

• First, it was noted that existing civilian autonomous robotic systems have significant limitations: they are not capable of complex decision-making and reasoning; their capacity to perceive or adapt to their environment is very limited; and they therefore cannot operate outside of the most simple and predictable situations.
• It was also pointed out that some weapon systems with autonomy in the ‘critical functions’ of selecting and attacking targets are already in use today. They tend to be highly constrained in: the tasks they carry out, the types of targets they attack, and the contexts in which they are used.
• It was noted that highly sophisticated autonomous weapon systems programmed to independently determine their actions, make complex decisions and adapt to their environment – referred to by some as ‘fully autonomous weapon systems’ with ‘artificial intelligence’ – do not yet exist.
• There was recognition of the importance of maintaining human control over the ‘critical functions’ of selecting and attacking targets, although there was less clarity on what constitutes ‘meaningful human control’.
• There was no doubt among the participants that the development and use in armed conflict of autonomous weapon systems is governed by international humanitarian law (IHL), including the requirement to carry out legal reviews of new weapons.
• There was acknowledgement of the very significant challenge of programming a machine to undertake the qualitative judgements that would be required to apply the IHL rules of distinction, proportionality and precautions in attack, particularly in complex and dynamic conflict environments. This is not possible with current technology and is unlikely to be achievable in the foreseeable future.
• There were different views on the adequacy of IHL to regulate the development and use of autonomous weapon systems. Some believed that existing law is sufficient, but others argued that an explicit ban on autonomous weapon systems is necessary, or the development of a legal norm requiring, and defining, ‘meaningful human control’.
• Finally, attention was drawn to the importance of carefully considering whether the principles of humanity and the dictates of public conscience would allow life and death decisions to be taken by a machine – with little or no human control. It was argued by some that the manner in which people are killed matters, even if they are lawful targets, and that as a matter of respect for human dignity, the decision to take someone’s life must remain with humans.

This concludes the review of selected highlights from the meeting. We encourage delegations to read the full summary report, written under the ICRC’s sole responsibility, which will be made available at this meeting.

Turning to the additional observations of the ICRC; there are six key questions that, in the ICRC’s view, would merit particular attention during this meeting:

1. What should be the scope of the discussions about autonomous weapon systems?

The scope of the ICRC’s expert meeting was deliberately broad, covering any weapons that can independently select and attack targets. This excludes weapon systems whose ‘critical functions’ are remotely controlled by a human operator.

The ICRC takes the view that legal and ethical concerns are not limited to future technology but may also be raised by autonomous weapon systems developed with current technology.

The central issue is the potential absence of human control over the ‘critical functions’ of acquiring, tracking, selecting and attacking targets, and consequently over the use of force of any type.

2. What is ‘meaningful human control’?

There seems to be general recognition that human beings must retain some degree of control over the ‘critical functions’ of weapon systems and the use of force. However, there is a need for a greater clarity on what is meant by ‘meaningful human control’; and a need also for clearer answers to a number of related questions, such as: Is the required level of human control the same in all circumstances? And, how will the concept of ‘meaningful human control’ be implemented?

3. What lessons can be drawn from autonomy in existing weapon systems?

Closer examination of autonomy in the ‘critical functions’ of existing weapon systems may provide insights into the present understanding of ‘meaningful human control’ and what level of autonomy and human control may be considered acceptable or unacceptable, and in which circumstances, including in terms of the compatibility of existing systems with IHL. Therefore it may be beneficial for States to share the legal reviews of existing weapons that have autonomy in their ‘critical functions’.

4. What are the implications, for compliance with IHL, of increasing autonomy?

If weapon systems become more autonomous – with greater “freedom” to determine their operations – they may become less predictable. Unpredictability would raise serious questions with regard to compliance with IHL. For example: What assurance is there that a weapon system will always operate within the law? How can the weapon system be adequately tested and its performance for its intended use verified without predictability in its likely effects? How can the development and deployment of the weapon system be lawful if there is no guarantee that it will perform in accordance with IHL?

5. Are autonomous weapon systems necessary?

A key question for States to ask themselves is whether it is possible to accomplish the same military tasks while maintaining direct human control over the ‘critical functions’ of selecting and attacking targets (by remote operation, for instance). If that is the case, then autonomy in these functions may not be necessary, at least not in most circumstances.

6. Are autonomous weapon systems ethically and morally acceptable?

Finally, perhaps the fundamental question is whether autonomous weapon systems are compatible with the principles of humanity and the dictates of public conscience. There is a sense of deep discomfort with the idea of any weapon system that places the use of force beyond human control.

This concludes the list of additional observations by the ICRC. In closing, I would like to emphasise that the ICRC welcomes these discussions at the CCW and encourages all States to consider these key questions during their ongoing deliberations, both here and in their own domestic contexts, while developing their policies on autonomous weapon systems.

Thank you very much for your attention.

See also the report of the ICRC meeting of experts on autonomous weapon systems, held in Geneva from 26 to 28 March 2014.