Orange heart shapes are commonly perceived as slightly reddish, which is an example of the memory color effect (MCE). Given that the MCE is a modulation of visual memories of typical colors of familiar objects, it can be considered to be a top-down effect. Whether cognitive penetration can explain MCEs has been actively debated since Macpherson argued that the belief that hearts are red alters orange perception. This paper aims to provide a credible explanation of the MCE that is consistent (...) with scientific studies. I first prove that Macpherson’s theory is not properly supported by the psychological evidence that she provides. I also show that her model contradicts a report on the neural correlate of the MCE. Next, I examine color constancy, which enables the perception of colors as constant under various illuminations, as a possible noncognitive principle for the MCE. The relationship between them, however, is not well established. Using statistical analyses of scenes around us, Purves suggests that perception is determined by the most frequent percepts. I argue that Purves’ view better explains the noncognitive aspects of visual memory and is consistent with the finding of the neural correlate of the MCE. (shrink)