Cognitive modeling of human behavior has advanced the understanding of underlying processes in several domains of psychology and cognitive science. In this article, we outline how we expect cognitive modeling to improve comprehension of individual cognitive processes in human-agent interaction and, particularly, human-robot interaction (HRI). We argue that cognitive models offer advantages compared to data-analytical models, specifically for research questions with expressed interest in theories of cognitive functions. However, the implementation of cognitive models is arguably more complex than common statistical (...) procedures. Additionally, cognitive modeling paradigms typically have an explicit commitment to an underlying computational theory. We propose a conceptual framework for designing cognitive models that aims to identify whether the use of cognitive modeling is applicable to a given research question. The framework consists of five external and internal aspects related to the modeling process: research question, level of analysis, modeling paradigms, computational properties, and iterative model development. In addition to deriving our framework from a concise literature analysis, we discuss challenges and potentials of cognitive modeling. We expect cognitive models to leverage personalized human behavior prediction, agent behavior generation, and interaction pretraining as well as adaptation, which we outline with application examples from personalized HRI. (shrink)