In the bone marrow, the special microenvironment niches nurture a pool of hematopoietic stem cells (HSCs). Many HSCs reside near the vasculature, but the molecular regulatory mechanism of niches for HSC maintenance remains unclear. Here we showed that the induced deletion of CXCR4, a receptor for CXC chemokine ligand (CXCL) 12 in adult mice, resulted in severe reduction of HSC numbers and increased sensitivity to myelotoxic injury, although it did not impair expansion of the more mature progenitors. Most HSCs were found in contact with the cells expressing high amounts of CXCL12, which we have called CXCL12-abundant reticular (CAR) cells. CAR cells surrounded sinusoidal endothelial cells or were located near the endosteum. CXCL12-CXCR4 signaling plays an essential role in maintaining the quiescent HSC pool, and CAR cells appear to be a key component of HSC niches, including both vascular and endosteal niches in adult bone marrow.