Article Text

Download PDFPDF

Microbiology of bacterial translocation in humans
  1. C J O’Boyle,
  2. J MacFie,
  3. C J Mitchell,
  4. D Johnstone,
  5. P M Sagar,
  6. P C Sedman
  1. Combined Gastroenterology Service, Scarborough Hospital, Scalby Road, Scarborough, North Yorkshire, UK
  1. Mr MacFie.

Abstract

Background—Gut translocation of bacteria has been shown in both animal and human studies. Evidence from animal studies that links bacterial translocation to the development of postoperative sepsis and multiple organ failure has yet to be confirmed in humans.

Aims—To examine the spectrum of bacteria involved in translocation in surgical patients undergoing laparotomy and to determine the relation between nodal migration of bacteria and the development of postoperative septic complications.

Methods—Mesenteric lymph nodes (MLN), serosal scrapings, and peripheral blood from 448 surgical patients undergoing laparotomy were analysed using standard microbiological techniques.

Results—Bacterial translocation was identified in 69 patients (15.4%). The most common organism identified wasEscherichia coli (54%). Both enteric bacteria, typical of indigenous intestinal flora, and non-enteric bacteria were isolated. Postoperative septic complications developed in 104 patients (23%). Enteric organisms were responsible in 74% of patients. Forty one per cent of patients who had evidence of bacterial translocation developed sepsis compared with 14% in whom no organisms were cultured (p<0.001). Septic morbidity was more frequent when a greater diversity of bacteria resided within the MLN, but this was not statistically significant.

Conclusion—Bacterial translocation is associated with a significant increase in the development of postoperative sepsis in surgical patients. The organisms responsible for septic morbidity are similar in spectrum to those observed in the mesenteric lymph nodes. These data strongly support the gut origin hypothesis of sepsis in humans.

  • bacterial translocation
  • mesenteric lymph nodes
  • serosal scrapings
  • enteric bacteria
  • postoperative sepsis

Statistics from Altmetric.com

Request Permissions

If you wish to reuse any or all of this article please use the link below which will take you to the Copyright Clearance Center’s RightsLink service. You will be able to get a quick price and instant permission to reuse the content in many different ways.

Gut translocation of bacteria is defined as the passage of gastrointestinal microflora across the lamina propria to local mesenteric lymph nodes (MLN) and from there to extranodal sites. It has been proposed that bacterial translocation may initiate a cytokine response from macrophages encountered in mesenteric lymph nodes.1 In certain conditions this may predispose the host to the development of septic morbidity.

Many studies in animals using a wide variety of experimental situations support this concept of the gut as the originator of septic complications and that bacterial translocation may be an important intermediary mechanism in the development of such sepsis. The evidence in humans to date has been less convincing but the well recognised phenomenon whereby nosocomial infection is frequently a consequence of gut derived organisms such as Escherichia coli does lend credence to the hypothesis. Few previous studies have investigated a causal relationship between microbiologically confirmed bacterial translocation in humans and subsequent septic morbidity.

We have previously reported an incidence in bacterial translocation of 10.3% in a consecutive series of 242 patients undergoing laparotomy for a variety of surgical complaints.2 When translocation occurred the incidence of septic complications was 28% compared with 11.5% when it did not (p<0.05).

The aim of this study was to examine the spectrum of microbiological flora identified within the MLN, serosal scrapings, and peripheral blood cultures at the time of laparotomy and to determine whether there was a similarity between these bacteria and the causative organisms identified from the patients in whom septic postoperative complications occurred.

Patients and methods

All patients presenting to the Combined Gastroenterology Service at Scarborough Hospital who required laparotomy were entered into this study. Patients were excluded if there was any evidence of intraperitoneal sepsis or contamination, if sampling of lymph nodes was not possible or clinically inappropriate, or if antibiotics had been administered preoperatively. Where there was any doubt regarding intraperitoneal sepsis an intra-abdominal swab was taken for culture and the patients were excluded if this was positive for bacteria. All patients gave informed consent to participate in the study which was approved by the Scarborough Locally Organised Research Ethics Committee.

At the start of laparotomy a serosal scraping was obtained from the antimesenteric border of the terminal ileum. A lymph node from the mesentery of the terminal ileum was excised by the use of a separate clean surgical blade. The samples were transported promptly in separate containers in sterile saline solution for microbiological analysis. Simultaneously, a peripheral blood sample was obtained by means of venepuncture. This was placed into aerobic and anaerobic blood culture bottles. Prophylactic antibiotics were only given to the patients after these samples were obtained.

TISSUE CULTURES

Serosal scrapings and lymph nodes were separately homogenised in saline by means of a stomacher (Seward Medical, London, UK). Each homogenate was inoculated onto Columbia blood agar and cysteine lactose electrolyte deficient (CLED) media for aerobic incubation and incubated in air with 5% carbon dioxide and onto Wilkins-Chalgren blood agar with neomycin and Columbia blood agar for anaerobic culture. Anaerobic cultures were incubated in a Don Whitley anaerobic workstation in an atmosphere of 80% nitrogen, 10% hydrogen, and 10% carbon dioxide. Aerobic cultures were incubated at 37°C for two days and anaerobic cultures were incubated at 37°C for five days.

BLOOD CULTURES

Aerobic and anaerobic blood culture bottles were incubated constantly at 37°C for seven days. The aerobic bottles were additionally mechanically agitated for the first 48 hours. Bacterial growth in the blood culture bottles was detected on a Bactec NR730 machine (Becton Dickinson Europe, Meylan Cedex, France) which samples CO2 in the head space of the culture vials twice daily for the first three days and then once daily for the last four days of the incubation period. The machine indicates a positive culture whenever the culture bottle head space CO2 concentration exceeds baseline threshold levels. All blood culture vials with positive readings had 0.5 ml of media extracted which was examined by Gram staining and further inoculated onto Columbia blood agar, chocolate Columbia blood agar, and CLED media, and incubated in air with 5% carbon dioxide, and onto Columbia blood agar; incubation was for 48 hours in an anaerobic atmosphere as described previously.3

Isolates grown from mesenteric lymph nodes, serosal scrapings, and peripheral blood samples were identified by characteristic colonial and microscopic appearances and by using standard microbiological tests. Whenever required, full speciation of organisms belonging to the family Enterobacteriaceae was performed using API 20E strips (bioMerieux, France). Enterococcus faecalis and other streptococci were identified as follows: microscopic and colonial appearances; Lancefield grouping by the Streptex kit (Murex, UK); and where appropriate by using API 20 Strep strips (bioMerieux, France). Obligate anaerobic isolates were identified by characteristic colonial growth characteristics, senstivity to metronidazole, microscopic appearances, and rapid ID32A strips (bioMerieux, France). Obligate aerobic isolates and non-fermenting Gram negative bacilli were identified using API 20NE strips (bioMerieux, France). Yeasts were identified by characteristic colonial and microscopic appearances, the germ tube reaction, and API 20AUX strips (bioMerieux, France). Bacillus species and coryneforms were identified by standard microbiological tests.4

Organisms belonging to the genus Staphylococcus were identified by microscopic and macroscopic appearances, catalase reaction, coagulase reaction, and DNAse reaction.

Organisms were broadly grouped into indigenous gastrointestinal flora and bacteria which would not normally be found resident in the gut. Anaerobic organisms were identified by their ability to grow in a strict anaerobic atmoshere. Their Gram reaction susceptibility to metronidazole and biochemical reactions (API) were identified by their microscopic and macroscopic morphology germ tube reaction and by the Candida API biochemical reactions.

Postoperative sepsis was defined as the presence of clinical, haematological, or radiographic evidence of infection where positive bacterial culture was obtained from a septic focus.

STATISTICAL ANALYSIS

All data were expressed as median and interquartile range. Comparisons between non-parametric groups of data were made using the χ2 test or the Mann-Whitney U test.

Results

Tissue samples were obtained from a total of 448 consecutive patients (216 men and 232 women, median age 67 years, range 14–94 years) who met the inclusion criteria. Table 1 presents clinical details of the patients. The clinical details and point prevalence of bacterial translocation for the first 242 patients in this series have been previously published.2

Table 1

Patient details at presentation—individual patients may be included in more than one category

Bacterial translocation occurred in 69 patients, giving a prevalence of 15.4%. Table 2 lists the clinical presentations of patients in whom translocation occurred. Patients were stratified according to the presence or absence of clinical conditions which have been associated with translocation in previous studies (table 3). There was no significant difference in the prevalence of translocation for malignancy, inflammatory bowel disease, jaundice, significant preoperative weight loss, or the administration of total parenteral nutrition preoperatively. In contrast, there was a significant increase in the prevalence of bacterial translocation in patients who were aged greater than 70 years (21% versus 11%), in patients who were operated on non-electively (21% versus 12%), and in patients presenting with distal bowel obstruction at laparotomy (41% versus 13%).

Table 2

 Prevalence of positive cultures and of postoperative sepsis by clinical subgroup. Translocation is defined as positive nodal culture with or without positive serosal culture

Table 3

The prevalence of bacterial translocation in different preoperative clinical settings

A total of 98 separate positive cultures was obtained on the 69 positive nodal samples. Multiple organisms were cultured in 22 (32%) patients. E coli was the most commonly identified organism and was found in 38 (55%) patients (table 4). Organisms belonging to the family Enterobacteriaceae (including E coli) comprised 61% of positive cultures. Coagulase negative staphylococci were the most frequent non-enteric bacterial species to translocate and were present in 18 (26%) nodal samples. Aerobic and facultative anaerobic bacteria as well as obligate anaerobes were identified. Interestingly there was no evidence of fungal translocation. Aerobic bacteria (which includes the facultative anaerobes) were cultured in 91% (89/98). Seventy seven per cent (75/98) of cultured organisms were typical of indigenous intestinal flora. The remainder consisted of organisms which would not frequently be associated with colonisation of the gastrointestinal tract: coagulase negative staphylococci,Staphylococcus aureus, Bacillus spp, coryneforms, andPseudomonas aeruginosa. Enteric bacterial isolates were part of a multiorganism growth in 56% of cultures compared with 36% of non-enteric bacterial cultures (p<0.02). The obligate anaerobesClostridium perfringens and Bacteroides fragilis were isolated invariably as part of a multiorganism colonisation in association with facultative anaerobes.

Table 4

 Organisms translocating to mesenteric lymph nodes (98 isolates in 69 patients) and to serosa (48 isolates in 38 patients)

Forty eight positive cultures were obtained on the serosal scrapings from 38 patients (8.5%). The prevalence of positive serosal scrapings in node negative patients was 2.9% (11/379) compared with 39% (27/69) in those patients with culture positive nodes (χ2 = 68.4, p<0.0001). The Enterobacteriaceae were present in 52% of cultures and E coli was again the most commonly identified organism, occurring in 37% of patients. Of the 27 patients who had positive nodal cultures and positive serosal scrapings, 23 (85%) had similar species at both sites.

Blood cultures were positive in 9/69 patients (13%) in whom translocation to lymph nodes was identified. Three of these patients grew similar species from their nodal samples. E coli, Citrobacter freundii, and coagulase negative staphylococci were the organisms cultured. Two of these patients developed postoperative septic complications; one was diagnosed with an E coliurinary tract infection and the other grew coagulase negative staphylococci from a discharging wound.

Postoperative septic complications developed in 104 patients (23%) (fig 1). Eighteen patients (17%) developed more than one complication. Enteric organisms were cultured from septic foci in 77 patients (74%). Members of the family Enterobacteriaceae (E coli, Klebsiella spp, Proteus spp,Enterobacter spp, Citrobacter spp, Morganellaspp) were isolated in 70 of these cases (table 5). Forty five per cent of patients who had evidence of bacterial translocation developed sepsis compared with 19% in whom no organisms were cultured (p<0.001). Septic morbidity was more frequent when a greater diversity of bacteria resided within the MLN (50% versus 45%) (table 6), but this was not statistically significant. In those patients with evidence of translocation at laparotomy and who developed postoperative sepsis, 32% (10/31) grew the same genus of bacteria from both the preoperative nodal sample and the postoperative septic focus (table 7): E coli was isolated in four cases of urinary tract infection and two wound infections, coagulase negative staphylococci was isolated in two cases of septicaemia, P mirabilis caused a wound infection, and K oxytoca a urinary tract infection. Table 2 summarises the prevalence of positive cultures and of postoperative sepsis by clinical subgroup.

Figure 1

Postoperative infective complications (n=104 patients, 23%). Eighteen patients experienced more than one septic event.

Table 5

 Microbiology of postoperative infections. More than one organism was cultured in a number of patients

Table 6

 Septic complications in relation to number of species cultured from nodal samples

Table 7

 Patients in whom the same organism was cultured from nodal sample and site of sepsis (n=10)

Discussion

This is the largest series examining gut translocation of bacteria in humans. It confirms that translocation does occur and is more common in the presence of distal bowel obstruction, when patients are elderly, and when urgent surgery is required. The prevalence of bacterial translocation in our 448 patients was 15.4%. The presence of distal intestinal obstruction at laparotomy was the strongest predictor of translocation and occurred in 41% of patients (table 3). This is in keeping with a number of previous studies in humans.2 ,5 ,6 Age greater than 70 years and the necessity for urgent surgery were also associated with significantly increased translocation rates (both 21%). This is the first study which shows an association between bacterial translocation and these two parameters. Distal bowel obstruction might be expected to be more prevalent in the elderly as the incidence of colorectal carcinoma is higher. This does not explain the increased translocation found with age in our series as there was no significant difference in the incidence of distal bowel obstruction in older patients compared with those aged less than 70 years (9% versus 6%, NS). Furthermore, the incidence of translocation in patients aged less than 70 years who did not have distal obstruction was just 8.4%. If we exclude urgent laparotomy cases from this group the translocation rate was just 7% (11/155).

We recognise that a study such as this may underestimate the true incidence of bacterial translocation and that more extensive tissue sampling or culture of portal venous blood might have shown a greater number of positive cultures. We chose to undertake sampling only from ileal serosa and ileal lymph nodes because previous animal work suggests that these would provide the most productive tissues for analysis,7 and we did not feel that more extensive sampling was clinically appropriate or ethically acceptable. Furthermore, it could be argued that the identification of enteric organisms in the tissues sampled in this study was a consequence of bacterial contamination through mechanisms other than translocation. We consider this unlikely for a number of reasons. Firstly, the spectrum of bacteria isolated is typical of enteric bacteria. Secondly, the blood cultures were positive in only a minority of patients subsequently confirmed as being positive for translocation. Thirdly, while it is recognised that bacteria may appear in extraintestinal tissues after bowel manipulation8 we have avoided this by immediate sampling of tissue after opening the peritoneum. Fourthly, patients with peritoneal contamination from any cause were excluded from this analysis. Finally, the significant association between positive cultures obtained from MLNs and serosal samples suggests movement of bacteria occurs from within the intestinal lumen to the outside. If the converse were true one would anticipate a higher prevalence of positive serosal samples than was observed in mesenteric lymph nodes.

Both human and animal research has suggested that the presence of obstructive jaundice,9 inflammatory bowel disease,10 the administration of total parenteral nutrition (TPN),11-13 and malignant disease14 are all associated with an increased incidence of bacterial translocation. We have found no evidence to support these findings. The differences in bacterial translocation between those patients with and without inflammatory bowel disease and those who did and did not receive TPN were negligible, while jaundice and malignant disease were associated with reduced translocation rates (not significant).

Three primary mechanisms that promote bacterial translocation have been identified: intestinal bacterial overgrowth; increased permeability of the intestinal mucosal barrier; and deficiencies in host immune defences. These mechanisms may not be mutually exclusive and more than one may be relevant in individual patients.

BACTERIAL OVERGROWTH

In this study Gram negative facultative anaerobes, and particularly E coli, were the most common organisms identified in mesenteric lymph nodes and serosal scrapings. Obligate anaerobes were cultured less frequently. In healthy subjects the distal ileum harbours aerobic and anaerobic bacteria at a concentration of approximately 108 organisms. Distal to the ileocaecal valve the bacterial concentration is of the order of 1012 colony forming units per ml, with anaerobic bacteria outnumbering aerobes by 1000-fold.15 ,16 The predominant isolates areBacteroides spp, Bifidobacterium spp,Eubacterium spp, Lactobacillus spp, andClostridium spp. Intraluminal population density of bacteria would therefore not appear to be the most significant influence on bacterial translocation as one would expect significantly greater numbers of obligate anaerobes to be identified in extraintestinal tissues.17 ,18 Previous authors have found a very low incidence of bacterial translocation by obligate anaerobes.19-21 This has prompted suggestions that obligately anaerobic bacteria are the principal inhibitors of bacterial overgrowth and translocation of E coli and other potentially pathogenic bacteria.17-19 These observations are based again on animal studies where it was noted that a relative increase in numbers of oxygen tolerant bacteria in the distal bowel was associated with an increase in translocation of these organisms without a concomitant increase in anaerobic bacterial translocation. We cultured obligate anaerobes from the mesenteric lymph nodes in 13% of patients (9/69). This is a high proportion of these organisms when compared with previous studies and would appear to refute the theory of an entirely protective role for these organisms in humans. This would be in keeping with the possibility that obligate anaerobes actually translocate at the same rate as facultative anaerobes across the intestinal epithelium but that the obligate anaerobes are more likely to be destroyed in microenvironments which are not oxygen deficient and which may well prevail within the mesenteric lymph nodes or en route to them. Our finding that whenever obligate anaerobes were isolated, they were part of a multiorganism culture involving facultative anaerobes, might reflect in some way poor oxygenation or perfusion of these tissues.

INTESTINAL PERMEABILITY

Migration of organisms across the bowel wall may occur by pinocytosis in epithelial cells and has been proposed as the principle mechanism of translocation in the presence of an intact mucosal barrier.22 ,23 However, many studies have identified alterations in intestinal permeability in critically ill patients. Using dual sugar probe techniques the most frequent finding is that lactulose absorption is increased while the absorption of the smaller molecules is unchanged.21 This suggests that changes in intestinal permeability are caused by loss of tight junction integrity or by areas of cell loss at villous tips. Theoretically this might predispose to passive movement of bacteria across the intestinal epithelium on the presumption that bacteria would follow the pathway of large molecules. In support of this Berg has shown in non-published data that non-viable latex particles of similar diameter to E coli translocate with similar efficiencies to viable indigenous bacteria.23 Such a mechanism could account for the diversity of microorganisms which we found in mesenteric lymph nodes.

IMMUNE STATUS

Under normal circumstances translocating bacteria should be phagocytosed within the mesenteric lymph nodes. This would explain the failure to culture viable bacteria from the MLN of healthy rodents.20 However, if the host is immunocompromised the normal defence mechanisms may fail, permitting egress and survival of these bacteria at distant extraintestinal sites, the inference being that with increasing severity of illness bacterial translocation occurs because of the inability of the host to deal adequately with the numbers of bacteria present. Much experimental evidence exists to support this concept. Bacterial translocation is more common in athymic mice than in cont- rol animals, suggesting a specific role for T cell mediated immunity in inhibiting translocation.24 Immunosuppressant agents and leukaemia have been associated with increased translocation to mesenteric lymph nodes in rodents25-27 and leukaemia has been associated with increased translocation to blood in humans.28Deficiencies in humoral immunity (serum immunoglobulins) have been implicated in the penetration of pathogenic Vibrio cholerae and Salmonella enteritidis into the lamina propria.29 ,30 The possibility that bacterial translocation may not be an all or none phenomenon has been previously suggested.31 Experimental evidence exists in human studies which attest to the invariable presence of E coli β galactosidase in the cytoplasm of mesenteric lymph nodes but positive cultures in only 5% of the same patients.32 This would suggest translocation and subsequent control in the mesenteric nodes. The association between multiorganism colonies and an increasing incidence of postoperative sepsis as shown in this series is compatible with this speculative mechanism of translocation and is probably a manifestation of immunosuppression.33 Furthermore it is of some interest to note that our results do show that the incidence of bacterial translocation was greatest in patients aged over 70, in those undergoing urgent surgery, and in patients with intestinal obstruction, all of whom might be expected to have a degree of impaired immunity. Bacterial translocation may then serve as a promoter of septic morbidity but may not actually be the initiator.

An important finding of this study is the association between bacterial translocation and postoperative sepsis. Forty five per cent of patients in whom bacterial translocation occurred developed a septic focus compared with 19% of patients who showed no evidence of bacterial translocation. Furthermore, the commonest translocating organisms, the Enterobacteriaceae, were also the organisms responsible for postoperative septic complications in 67% (70/104) of cases. Of those patients in whom bacteria were cultured from the MLN and who developed postoperative sepsis, 32% (10/31) grew at least one phenotypically similar species type at both sites (table 7). This is the first compelling evidence from human studies that confirms a potential pathological significance for bacterial translocation. Despite this, we recognise that our data do not prove causality between translocation and septic morbidity and that there are some obfuscating findings in our data. For instance, the occasional culture of non-enteric organisms such as coagulase negative staphylococci from MLNs in patients subsequently identified as having organisms similar to the predominant organism in septic foci is difficult to explain on the basis of intestinal translocation. We accept that these organisms may be skin contaminants and thereby could reflect poor sampling technique. Our experimental protocol endeavoured to avoid such problems and the low incidence in this series of such growth suggests that this was not a major problem. We feel that on no account do these findings invalidate the overall observations made from this study, but rather reflect the inherent methodological problems associated with investigating a pathophysiological phenomenon in a clinical setting with an inevitably heterogeneous patient population. Confirmation of proof of causality between translocation and sepsis would necessitate specific species subtyping of bacteria from the lumen of the gut, MLNs, or other extaintestinal tissue and subsequent septic foci with all their respective temporal relationships in terms of sampling times recorded. This will be logistically very difficult to achieve but does emphasise the need for additional prospective studies in the clinical setting as well as in the controlled environment of the laboratory.

These data also provide a rationale for the use of selective gut decontamination. The principle behind this treatment is that by selective elimination of enteric Gram negative Enterobacteriaceae using specific antibiotic therapy, translocation rates and subsequent postoperative sepsis may be reduced. While some studies have shown significant reductions in septic morbidity the results remain controversial and there is no firm evidence to date of enhancement of survival.34 Our results suggest that selective gut decontamination may have a role to play in the prevention of gut derived sepsis. Our findings of significant translocation by anaerobic bacteria suggest a role for therapy directed against these organisms. We found no evidence of candidal translocation in any of our tissue samples. However, Candida species were cultured from 10% of postoperative septic foci in this study. We have previously reported the presence of Candida spp in gastric microflora35 and although others have shown thatCandida spp can translocate36 ,37 our data would suggest that opportunistic infection caused by Candidaspp probably occurs by a mechanism other than translocation. Consequently, treatment directed against intestinal yeast may be unnecessary. Finally, selective gut decontamination does not influence translocation of bacterial endotoxin which may itself be responsible for continued cytokine release and may influence host susceptibility to postoperative infection. The implication therefore is that manipulations of microflora alone are insufficient to overcome the effects of altered barrier function and diminished immune defences. It is interesting to speculate, therefore, that selective gut decontamination directed only at enteric aerobic and anaerobic bacteria, possibly in conjunction with immune enhancing agents, may provide an optimal treatment strategy for critically ill patients which may significantly decrease septic morbidity.

Conclusion

This is the largest study of bacterial translocation in humans. Unlike many previous studies in humans bacterial translocation was estimated by culture of viable extragastrointestinal organisms rather than by indirect assessment of microbial particles or bacterial endotoxin. Organisms cultured in the MLN and serosa reflect the spectrum of bacteria resident within the gut lumen. We have shown a significant association between bacterial translocation and postoperative morbidity. The majority of septic events were caused by indigenous gut organisms. This strongly supports the “gut origin” hypothesis of nosocomial infection. The wide variety of organisms cultured suggests a non-specific passive mechanism of absorption, the rate of which probably depends on the nature and severity of illness. Bacterial translocation probably occurs in healthy individuals but is not clinically significant in the presence of a fully functional immune system. Isolation of viable bacteria in MLN once translocation has occurred is dependent on the immunological competence of the host and individual bacterial virulence factors which prevent their destruction. In more severely immunocompromised individuals, such as the aged and those with distal bowel obstruction, there is a failure to prevent the propagation of these viable bacteria from the MLN to extraintestinal sites which become sites of postoperative sepsis. Thus bacterial translocation appears to be an important early step in the promotion of sepsis in debilitated postoperative patients, rather than the sole initiator. Elucidation of the defence mechanisms by which intestinal microflora are confined within the gut lumen and translocation rates are kept to a minimum will be a significant advance in the understanding of gut derived infection.

Acknowledgments

The authors would like to thank Iris Davidson for her invaluable administrative skills, Dr Paul Buckley and Dr Ian Tring and the rest of our anaesthetic colleagues, Sister Carol Wadsworth and the theatre staff, and Mr Paul Sudworth, Chief Technician, Department of Microbiology. This study was supported in part by a grant from the Yorkshire Region Health Authority.

References