Aggregation of cancers among families with highly penetrant genetic mutations such as hereditary nonpolyposis colorectal cancer is well-described. However, there is a paucity of data regarding familial aggregation of hormonal cancers (cancers of the breast, endometrial, ovarian and prostate) and colorectal cancer (CRC) in the general population. We investigated the association between having a first-degree family history of breast, endometrial, ovarian, or prostate cancer and CRC risk. Population-based CRC cases and controls were recruited by the Ontario Familial Colorectal Cancer Registry (OFCCR). Logistic regression was conducted to obtain odds ratio (OR) estimates and 95% confidence intervals (95% CIs). First-degree family history of breast cancer was associated with a modest, borderline statistically significant increased CRC risk (age-, sex-adjusted OR = 1.2, 95% CI = 1.0, 1.5). The magnitude of CRC risk was greatest if more than one first-degree kin had breast cancer (age-, sex-adjusted OR = 1.7, 95% CI = 1.0, 2.0), as well as if the kin was diagnosed at >50 years of age (age-, sex-adjusted OR = 1.4, 95% CI = 1.1, 1.8). Family history of ovarian cancer was associated with reduced CRC risk (multivariate-adjusted OR = 0.6, 95% CI = 0.3, 1.0). Although statistically significant increases in CRC risk were observed in the age-, sex-adjusted OR estimates for family history of endometrial and prostate cancers, the associations were no longer significant after multivariate-adjustment. In conclusion, individuals with a first-degree kin with breast cancer may have a modest increased risk for CRC compared to individuals without.