Intelligent voice assistants and the microphone-equipped Internet of Things devices that support them are very convenient but carry significant privacy risks. Newer and future devices extend these risks by listening all the time — beyond a few specific keywords. The goal of our research is to develop privacy controls for such devices, by allowing users to specify restrictions — what should the assistant be able to hear and what is off-limits? — and for our system to be able to enforce their preferences. So far in our research, we investigated people’s expectations for these devices, developed potential privacy-preserving approaches, and prototyped transparency mechanisms. In the next phase of the project, we propose an in situ study of privacy controls for passive listening devices, evaluating their effectiveness and usability across several different dimensions and criteria.