We make buildings install fire extinguishers for safety. Should AI plants be forced to install something that can shut it down in an instant?
We make buildings install fire extinguishers for safety. Should AI plants be forced to install something that can shut it down in an instant?
The problem is that survival is a nearly universal instrumental goal. You may not explicitly tell it “you should protect your own existence,” but if you give it any other goal then it’s going to include an unspoken asterisk that includes “and protect your own existence so that you can accomplish that goal I gave you, since if you’re shut down you won’t be able to accomplish it.”