Averaging across observers is common in psychological research. Often, averaging reduces the measurement error and, thus, does not affect the inference drawn about the behavior of individuals. However, in other situations, averaging alters the structure of the data qualitatively, leading to an incorrect inference about the behavior of individuals. In this research, the influence of averaging across observers on the fits of decision bound models (Ashby, 1992a) and generalized context models (GCM; Nosofsky, 1986) was investigated through Monte Carlo simulation of a variety of categorization conditions, perceptual representations, and individual difference assumptions and in an experiment. The results suggest that (1) averaging has little effect when the GCM is the correct model, (2) averaging often improves the fit of the GCM and worsens the fit of the decision bound model when the decision bound model is the correct model, (3) the GCM is quite flexible and, under many conditions, can mimic the predictions of the decision bound model, whereas the decision bound model is generally unable to mimic the predictions of the GCM, (4) the validity of the decision bound model's perceptual representation assumption can have a large effect on the inference drawn about the form of the decision bound, and (5) the experiment supported the claim that averaging improves the fit of the GCM. These results underscore the importance of performing single-observer analysis if one is interested in understanding the categorization performance of individuals.