People with genetic or acquired defects in the biosynthesis of bile acids may suffer from cholestasis. Patients with a deficiency of 3 beta-hydroxy-delta 5-C27-steroid dehydrogenase/isomerase from 3 beta, 7 alpha-dihydroxy- and 3 beta, 7 alpha, 12 alpha-trihydroxy-5-cholenoic acids, the sulfated and partially glycine-conjugated forms of which are found in their urine and bile. 3-Oxo-delta 4 bile acids are detected in the urine of patients with a deficiency of 5 beta-reductase. It has been postulated that these unusual bile acids might act as cholestatic agents in these patients. The aim of the present study was to test this hypothesis in an in vitro system, since the abnormal bile acids would be metabolized in in vivo experiments. Basolateral (sinusoidal) and canalicular plasma membrane vesicles were isolated from rat liver. A rapid filtration method was used to determine transport of cholyltaurine in the presence of model bile acids into the isolated vesicles. It was found that 3 beta, 7 alpha-dihydroxy-5-cholenoic acid and 7 alpha-hydroxy-3-oxo-4-cholenoic acid both inhibited the apical, ATP-dependent transport system for cholyltaurine in a competitive manner with K(m) values of 15 microM and 16 microM, respectively. Radioactively labeled 3 beta, 7 alpha-dihydroxy-5-cholenoyltaurine and 7 alpha-hydroxy-3-oxo-4-cholenoyltaurine were not transported by the same transport system. The same types of experiments were performed with basolateral plasma membrane vesicles. It was found that, in contrast to the canalicular ATP-dependent bile acid transport system, only 7 alpha-hydroxy-3-oxo-4-cholenoyltaurine was a competitive inhibitor of the sodium-dependent transport system for cholyltaurine with a K(m) of 16 microM. Studies with radioactively labeled 7 alpha-hydroxy-3-oxo-4-cholenoyltaurine and 3 beta, 7 alpha-dihydroxy-5-cholenoyltaurine revealed that 7 alpha-hydroxy-3-oxo-4-cholenoyltaurine was transported in a sodium-dependent manner into basolateral rat liver plasma membrane vesicles, whereas 3 beta, 7 alpha-dihydroxy-5-cholenoyltaurine was not transported in a sodium-dependent way. These results support the hypothesis that the unusual bile acids found in patients with defects in bile acid biosynthesis might act as cholestatic agents by inhibiting the canalicular ATP-dependent transport system for bile acids which constitutes the rate-limiting step in the overall process of bile acid transport across hepatocytes. Furthermore, the experiments demonstrated that, despite similar substrate specificities, the basolateral sodium-dependent and the apical ATP-dependent transport system for cholyltaurine might have different recognition sites for bile acids.