We introduce a novel biologically inspired feature descriptor, Co-occurrence of Local Anisotropic Gradient Orientations (CoLlAGe), that captures higher order co-occurrence patterns of local gradient tensors at a pixel level to distinguish disease phenotypes that have similar morphologic appearances. A number of pathologies (e.g. subtypes of breast cancer) have different histologic phenotypes but similar radiographic appearances. While texture features have been previously employed for distinguishing subtly different pathologies, they attempt to capture differences in global intensity patterns. In this paper we attempt to model CoLlAGe to identify higher order co-occurrence patterns of gradient tensors at a pixel level. The assumption behind this new feature is that different pathologies, even though they may have very similar overall texture and appearance on imaging, at a local scale, will have different co-occurring patterns with respect to gradient orientations. We demonstrate the utility of CoLIAGe in distinguishing two subtly different types of pathologies on MRI in the context of brain tumors and breast cancer. In the first problem, we look at CoLlAGe for distinguishing radiation effects from recurrent brain tumors over a cohort of 40 studies, and in the second, discriminating different molecular subtypes of breast cancer over a cohort of 73 studies. For both these challenging cohorts, CoLlAGe was found to have significantly improved classification performance, as compared to the traditional texture features such as Haralick, Gabor, local binary patterns, and histogram of gradients.