What is our epistemic access to metaphysical modality? Timothy Williamson suggests that the epistemology of counterfactuals will provide the answer. This paper challenges Williamson's account and argues that certain elements of the epistemology of counterfactuals that he discusses, namely so called background knowledge and constitutive facts, are already saturated with modal content which his account fails to explain. Williamson's account will first be outlined and the role of background knowledge and constitutive facts analysed. Their key role is to restrict our imagination to rule out irrelevant counterfactual suppositions. However, background knowledge turns out to be problematic in cases where we are dealing with metaphysically possible counterfactual suppositions that violate the actual laws of physics. As we will see, unless Williamson assumes that background knowledge corresponds with the actual, true laws of physics and that these laws are metaphysically necessary, it will be difficult to address this problem. Furthermore, Williamson's account fails to accommodate the distinction between conceivable yet metaphysically impossible scenarios, and conceivable and metaphysically possible scenarios. This is because background knowledge and constitutive facts are based strictly on our knowledge of the actual world. Williamson does attempt to address this concern with regard to metaphysical necessities – as they hold across all possible worlds – but we will see that even in this case the explanation is questionable. These problems, it will be suggested, cannot be addressed in a counterfactual account of the epistemology of modality. The paper finishes with an analysis of Williamson's possible rejoinders and some discussion about the prospects of an alternative account of modal epistemology.