I explain the shutdown problem: the problem of designing artificial agents that (1) shut down when a shutdown button is pressed, (2) don’t try to prevent or cause the pressing of the shutdown button, and (3) otherwise pursue goals competently. I prove three theorems that make the difficulty precise. These theorems show that agents satisfying some innocuous-seeming conditions will often try to prevent or cause the pressing of the shutdown button, even in cases where it’s costly to do so. And (...) patience trades off against shutdownability: the more patient an agent, the greater the costs that agent is willing to incur to manipulate the shutdown button. I end by noting that these theorems can guide our search for solutions. (shrink)

I explain and motivate the shutdown problem: the problem of creating artificial agents that (1) shut down when a shutdown button is pressed, (2) don’t try to prevent or cause the pressing of the shutdown button, and (3) otherwise pursue goals competently. I then propose a solution: train agents to have incomplete preferences. Specifically, I propose that we train agents to lack a preference between every pair of different-length trajectories. I suggest a way to train such agents using reinforcement learning: (...) we give the agent lower reward for repeatedly choosing same-length trajectories. (shrink)

Some worry that advanced artificial agents may resist being shut down. The Incomplete Preferences Proposal (IPP) is an idea for ensuring that doesn't happen. A key part of the IPP is using a novel 'Discounted REward for Same-Length Trajectories (DREST)' reward function to train agents to (1) pursue goals effectively conditional on each trajectory-length (be 'USEFUL'), and (2) choose stochastically between different trajectory-lengths (be 'NEUTRAL' about trajectory-lengths). In this paper, we propose evaluation metrics for USEFULNESS and NEUTRALITY. We use a (...) DREST reward function to train simple agents to navigate gridworlds, and we find that these agents learn to be USEFUL and NEUTRAL. Our results thus suggest that DREST reward functions could also train advanced agents to be USEFUL and NEUTRAL, and thereby make these advanced agents useful and shutdownable. (shrink)

Recently, network coding emerged as a promising technology that can provide significant improvements in throughput and energy efficiency of wireless networks. Many implementations of network coding in wireless networks, such as COPE, encourage nodes to overhear to improve the coding opportunities so that they can create better opportunities for coding at the transmitter node through overhearing more packages. In this paper, we have shown that all overheard packets are not necessarily useful for coding; thus, a node can go to sleep (...) mode and shutdown the transceiver to have low power consumption. In this approach, nodes use RL mechanism for duty cycling that is aware of network coding; thereby high throughput and low power consumption can be achieved. In this paper, the duty cycling decision problem is formulated as a Markov decision process. The objective is to maximize the expected total … . (shrink)