Although it is generally assumed that brain damage predominantly affects only the function of the damaged region, here we show that focal damage to critical locations causes disruption of network organization throughout the brain. Using resting state fMRI, we assessed whole-brain network structure in patients with focal brain lesions. Only damage to those brain regions important for communication between subnetworks (e.g., "connectors")--but not to those brain regions important for communication within sub-networks (e.g., "hubs")--led to decreases in modularity, a measure of the integrity of network organization. Critically, this network dysfunction extended into the structurally intact hemisphere. Thus, focal brain damage can have a widespread, nonlocal impact on brain network organization when there is damage to regions important for the communication between networks. These findings fundamentally revise our understanding of the remote effects of focal brain damage and may explain numerous puzzling cases of functional deficits that are observed following brain injury.