Orally applied nondigestible carbohydrates (NDC) have been associated with immune-modulating effects and other health benefits. The effects of prebiotic carbohydrates have recently received much attention, but other NDC have been reported to induce immune modulation as well. Many different effects have been shown on parameters of innate and specific immunity, mostly in animal experiments or in vitro. Data from clinical trials are limited, but promising studies have reported beneficial effects on mucosal and systemic immunity in humans. NDC are fermented to various degrees by the intestinal microbiota. Therefore, immune-modulatory properties have often been attributed to microbiota-dependent effects, especially in the case of prebiotic NDC. However, some NDC have been reported to bind to specific receptors on cells of the immune system, suggesting microbiota-independent, immune-modulatory effects play a role as well. This review aims to provide an overview of the published immune-modulatory effects in vitro and in vivo induced by NDC such as fructans, galactooligosaccharides, beta-glucans, pectins, and resistant starch. In addition, issues related to the underlying mechanisms are discussed: interaction between bacteria, their metabolites and the immune system, as well as direct effects of NDC via lectin receptors.