When experimental participants are given the chance to reflect and revise their initial judgments in a dynamic conversational context, do their responses to philosophical scenarios differ from responses to those same scenarios presented in a traditional static survey? In three experiments comparing responses given in conversational contexts with responses to traditional static surveys, we find no consistent evidence that responses differ in these different formats. This aligns with recent findings that various manipulations of reflectiveness have no effect on participants’ judgments (...) about philosophical scenarios. Although we did not find a consistent quantitative effect of format (conversation vs. static survey), conversational experiments still provide qualitative insights into debates about how participants are understanding (or misunderstanding) the scenarios they read in experimental studies, whether they are replacing difficult questions with questions that are more easily answered, and how participants are imagining the scenarios they read in ways that differ from what is explicitly stated by experimenters. We argue that conversational experiments—"Socratic questionnaires"—help show what is going on “under the hood” of traditional survey designs in the experimental investigation of philosophical questions. (shrink)