Many political philosophers argue that interference threatens a person’s agency. And they cast political freedom in opposition to interpersonal threats to agency, as non-interference. I argue that this approach relies on an inapt model of agency, crucial aspects of which emerge from our relationships with other people. Such relationships involve complex patterns of vulnerability and subjection, essential to our constitution as particular kinds of agents: as owners of property, as members of families, and as participants in a market for labor. We should construct a conception of freedom that targets the structures of our interpersonal relations, and the kinds of agents these relations make us. Such a conception respects the interpersonal foundations of human agency. It also allows us to draw morally significant connections between diverse species of unfreedom—between, for instance, localized domination and structural oppression.