When giving directions or describing an environment, people can assume perspectives other than their own. The present study was designed to determine the degree to which this non-egocentric perspective-taking reflects a generalized representation or an effortful inferential process. We employed a spatial perspective-taking task to explore this distinction and to evaluate how describing a non-egocentric perspective affects subsequent memory. One participant (the director) viewed a display of objects from a single perspective and described the display to a second participant (the matcher) from a perspective that varied by 0°, 45°, 90° , 135°, or 180° from the viewing perspective. Following the description, the director's memory for the spatial layout was tested using judgments of relative direction, scene recognition, and map drawing. Results revealed that the ability to describe the space from a non-egocentric perspective did not reflect a generalized representation of the space. Participants imagined and recognized familiar views faster and/or more accurately than novel views. Moreover, we found that different tasks showed different degrees of facilitation for the visually perceived and described views, suggesting that different experiences (seeing and describing) affect spatial representations in qualitatively different ways. Together, these findings support an egocentric spatial memory system and emphasize the importance of understanding differences in our spatial experiences as well as differences in the tasks we use to probe spatial memory.