Article Text
Abstract
AIMS To examine the volume and potential impact of gastroenterology research outputs from 1985 to 1998 from 14 developed countries; the overlap with research in cancer, infectious diseases, and genetics; and the funding sources for this research. To determine if countries' research outputs correlated with their burden of corresponding diseases and inputs to their research.
METHODS Selective retrieval of papers from the Science Citation Index and manual look up of a sample to determine funding sources. Classification of journals by four categories of research level (clinical/basic) and potential impact (low/high).
RESULTS Gastroenterology represents about 8% of world biomedical research but over 11% in Italy, Japan, and Spain. Its potential impact is highest (but declining) for the USA. It has increased noticeably in most European countries, particularly in Finland. Gastroenterology research has become more clinical in Japan, Spain, Australia, and the Netherlands but more basic in Canada, Germany, Finland, Israel, and South Africa. Funding comes primarily from national governments, followed by national private non-profit sources and industry but little industrial funding occurs in some countries. There is a strong and positive correlation between reported deaths from gastrointestinal neoplasms and countries' outputs of research in gastrointestinal oncology.
CONCLUSIONS Bibliometric analysis can reveal differences between countries in their research in a subject when a common methodology is applied to an international database. Variations in research methods in different countries can plausibly explain some of the variation in the potential impact of the work.
- bibliometrics
- funding
- impact
- mortality
- research
Abbreviations used in this paper
- A
- number of authors per paper
- C0–4
- five year citation count
- CF
- calibration factor
- D
- number of addresses per paper
- F
- number of funders per paper
- GASTR
- gastrointerology (filter)
- GENET
- genetics (filter)
- ICD
- International Classification of Deaths
- INFEC
- infection (filter)
- NFP
- private non-profit
- ONCOL
- oncology (filter)
- PIC
- potential impact category
- RL
- research level (clinical to basic)
- SCI
- Science Citation Index
Statistics from Altmetric.com
In this study we sought to explore the position of gastroenterology as a science in various European countries over the past 15 years and compare the situation with that in other leading countries. Gastroenterology is defined as the biomedical subfield that focuses on the development, anatomy, function, and malfunction of the gastrointestinal tract, liver, biliary system, and pancreas, and includes relevant animal model work. For this purpose we considered the number of papers published in peer reviewed journals and measures of their potential influence on other scientists through the citation scores (impact factors) of the journals in which they appeared. We also examined the inputs to research:
the size of the team—that is, the number of authors (A);
the degree of collaboration—that is, the number of addresses (D);
the extent to which research had been approved by funding agencies—that is, the number of acknowledgements of financial support (F); and
whether the work was clinical or basic—its research level (RL).
We also wanted to answer the following three questions:
How much overlap is there between gastroenterology and research in oncology, infectious diseases, and genetics, and is this related to the relative incidence of gastrointestinal disease in different countries?
Are the numbers and potential impact of European gastroenterology papers increasing or decreasing relative to those of the rest of the world?
What are the funding sources for research in gastroenterology in different countries and is there a relation between explicit research funding and quality of scientific output?
In an earlier study,1 the state of gastroenterological research in the UK was analysed in some detail through its output of published papers. The methodology consisted of selective extraction of gastroenterology papers from a general bibliographic database, theScience Citation Index (Institute for Scientific Information), and analysis of their characteristics in terms of their inputs. These included the numbers of authors and addresses listed on each paper, the RL (expressed as a number from 1=clinical to 4=basic) and, importantly, the number and type of funding sources acknowledged by the authors or implicit from their addresses.
The present study used a similar methodology to the earlier one but was extended to cover the outputs of some 14 countries active in gastrointestinal research; to examine a longer time period (1985–98; 14 years); and to look at three subject areas within gastroenterology, as mentioned above. A similar analysis was made of the input factors obtained in the different countries and the output variable of journal impact category to see if the results of the UK analysis would apply internationally. If the variation between journal impact category for different countries could be plausibly explained in part by differences in the mean values of input parameters, then the residual differences between nations could more appropriately be attributed to qualitative differences in the conduct of their research. This would allow research policy to be better informed by lessons drawn from international comparisons.
Methods
The first task was to devise a bibliographic search strategy or “filter” that would selectively retrieve papers relevant to gastrointestinal research from the Science Citation Index ( SCI ). It is important that such a filter is not confined to lists of specialist journals or the majority of relevant papers will be omitted.2 The earlier paper described the development of this particular filter (GASTR) which also consisted of title keywords and was developed by three senior representatives of the British Society of Gastroenterology. It collected many papers relevant to gastroenterology but inevitably it retrieved some papers irrelevant to the subfield (that is, it lacked precision or specificity) and it also failed to identify some papers that would have been deemed relevant by the experts who defined them (that is, it lacked recall or sensitivity). These factors were determined by the experts marking lists of papers and were respectively found to be 0.83 and 0.87. The ratio between them gives the calibration factor (CF) or the estimated true number of papers in the subfield divided by the number actually retrieved, which was 0.95—that is, the filter retrieved 5% more papers than the “true” total.
Three other subfield filters were also used in this study, covering, respectively, papers in genetics (GENET), infection (INFEC), and oncology (ONCOL). The first two were developed by scientific staff at the Wellcome Trust (Dr Barbara Skene and Dr John Stephenson); oncology was contributed by Dr Lesley Walker of the Cancer Research Campaign. The filters consisted of both specialist journal titles and title keywords, the latter often in combinations.3 GENET had a CF of 1.04 and covered nearly 14% of the biomedical literature; INFEC had a CF of 1.29 and covered 16% of biomedicine; and ONCOL had a CF of 1.24 and covered 15% of biomedicine.
Papers were retrieved from the SCI if they were articles, notes, or reviews. These are regarded as reports of research findings; they include “letters” toNature and a few other journals but not ordinary correspondence or editorial material. Integer counting was used to attribute papers to different countries—that is, a paper with addresses in France and England would be counted as one for each country. Attention was focussed on the outputs of 14 countries, of which nine were in Europe (Finland (FI), France (FR), Germany (DE), Italy (IT), the Netherlands (NL), Spain (ES), Sweden (SE), Switzerland (CH), and the UK) and five were elsewhere (Australia (AU), Canada (CA), Israel (IL), Japan (JP), and the USA). Care was taken to exclude papers from, for example, Prince of Wales Hospital (Australia or Hong Kong) from the total for Wales (in the SCI, UK addresses are given as England, Wales, Scotland, and North Ireland) and from Beth Israel Hospitals in the USA from the total for Israel. The outputs were determined for each country and for each year in a 14 year period, 1985–98, for gastroenterology and for the three other subject areas as the number of papers that satisfied both the GASTR and one of the other filters—that is, the intersection of the two filters. Because of annual fluctuations, the results are normally presented as three year moving averages.
Details of the papers (authors, addresses, source—that is, journal name) were downloaded from the SCI to an MS Excel 97 spreadsheet for three two year periods to analyse the inputs (number of authors (A), addresses (D), and RL) and outputs in more detail. The periods were 1985–86, 1991–92, and 1997–98. The world outputs of papers grew by about 30% over this period from approximately 33 000 papers in 1985–86 to 43 000 in 1997–98 (after deletion of papers without authors and addresses).
The journals were classified in two ways: by RL (from 1=clinical to 4=basic) and by potential impact category (PIC, from 1=low to 4=high). The methodology was similar to that used for analysis of UK papers described in the earlier paper. RL is assigned to journals by CHI Research Inc. on the basis of expert opinion, comparison with journals already classified, and journal to journal citation patterns: the journals most frequently cited by papers in a journal are normally considered to be at the same RL or one level more basic.4Table 1 shows examples of journals at each of the four levels, both in gastroenterology and in general medicine. Not all journals had an assigned RL: papers in these journals were excluded from this part of the analysis. Although it would, in principle, be preferable to allocate papers individually to RLs rather than use an average value for the journal in which they were published (for example,Gut clearly contains papers ranging all the way from clinical observation to basic research) no effective system for doing this has been developed (other than inspection of each title, which would not be practicable).
The PIC was based on mean five year citation counts (C0–4) to papers in a journal, taken from the Journal Expected Citation Rates file published by the Institute for Scientific Information. Table 2shows the criteria for categorisation and examples of journals at each PIC value.
Papers from each of the 14 countries were then characterised by mean values of RL and PIC in each of the two year periods for the purposes of a graphical display of the results, although strictly these values are not valid measures because the underlying parameters are categorical and not continuous variables. However, differences in the distribution of PIC values between countries and between different two year periods were evaluated using a χ2 test with three degrees of freedom to see if they were statistically significant.
The PIC of the journals in which papers are published appears to depend, at least for gastroenterology papers from the UK,5on inputs to the research—that is, number of authors (A), number of addresses (D), number of funding bodies acknowledged (F), and RL. From the earlier study,1 the coefficient relating journal impact category with A was determined as 0.08—that is, other factors being equal, an increase of one person in the list of authors would tend to increase the journal impact category by 1/12th of a unit. This may not seem very much but as the mean differences between countries in PIC values are often only about 0.1 or 0.2, and the distributions are mostly statistically significantly different from each other, it can be important. The mean number of authors, and the standard error of the mean, were then determined for papers from each of the 14 countries, together with New Zealand (NZ) and South Africa (ZA) for which data were available, and for the world as a whole. The mean number of addresses was also determined for gastroenterology papers from each of these 16 countries.
Other information sought for the papers was details of funding sources. For the UK, all biomedical papers are looked up routinely and their funding acknowledgements recorded.6 ,7 Details of funding have also been recorded for all Australian, New Zealand, and South African biomedical papers for 1993–94,8 and for samples of biomedical papers from the six other G7 countries (Canada, France, Germany, Italy, Japan, and the USA) for 1995 (Grant J, personal communication). For the G7 countries, samples of gastroenterology papers were retrieved from the databases on which the biomedical papers were held by use of the gastroenterology filter. For the remaining six countries (that is, Finland, Israel, the Netherlands, Spain, Sweden, and Switzerland) it was necessary to select samples of gastroenterology research papers (for the years 1996 and 1997) and look them up in libraries de novo. Funding bodies were recorded as three letter codes (trigraphs) and categorised by sector into three main types:
national government,
national private non-profit,
industrial (including pharmaceutical companies).
Because the pharmaceutical industry tends to be multinational, with research laboratories and extramural research programmes in many countries, no account was taken of the country of registration of the companies involved.
There were also papers without funding acknowledgements. In the UK these papers would have come mainly from hospitals (often supported by public funds) and universities (again, often with core support from the national or state government). Data are presented on the percentage of gastroenterology papers with funding acknowledgements and also on the mean number of funding bodies acknowledged per paper (F). This parameter is known from the UK study to have a positive influence on the impact category of the journals in which the papers are published. Although the effect falls off as F increases, initially the coefficient relating impact category to F is 0.18 so that an increase of unity in the number of funding acknowledgements has more than twice the effect on impact category as an extra author.
Finally, in order to make a comparison between the amount of gastrointestinal research in each country and the corresponding burden of gastrointestinal disease, statistics were sought on the death rates per 100 000 population for the 14 countries in a recent year that were attributable to “digestive” causes. In most countries the majority of these deaths are due to neoplasms. The main source of such data is the World Health Organization (WHO) World Health Statistics Annual, which lists causes of death on the International Classification of Deaths (ICD9) system. (For Switzerland, data were taken from a later volume which uses the ICD10 classification system.) For the present exercise, gastrointestinal neoplasms (stomach=091, colon=093, rectum and anus=094, liver=095) were grouped together; all other digestive causes (codes 341–9), including typhoid (code 011), and other intestinal infectious diseases (codes 010, 012–016, 019), were taken as another group.
Results
Gastroenterology research accounts for about 1 in 12 biomedical papers worldwide. The annual number of papers retrieved by the gastroenterology filter has risen from about 16 500 to about 22 000 between 1985 and 1998. As a subfield, it is growing slightly more slowly (+2.4% per year) than biomedicine as a whole (+3% per year). The relative commitment to gastrointestinal research has consequently fallen in most but not all countries: in Japan and in Australia it has risen.
Table 3 shows the relative contribution of the 14 countries' gastroenterology research to world output and to national outputs of biomedical papers over the 14 year period; they are ranked in order of this percentage. The “change” columns indicate whether these percentages have been increasing or decreasing by means of symbols (+, ++, ++++++, −, −−, −−−−). It is striking that Japan has increased its relative contribution very substantially (++++++) and that of the USA has declined sharply (−−−−). The UK contribution has also declined (−−), mainly because of decreased commitment to gastroenterology as its biomedical output is almost constant as a percentage of the world total. Spain (++), Italy (++), and Germany (+) have increased their contribution in gastroenterology despite a declining commitment to the subfield because their biomedical outputs have been rising rapidly.
The extent of multidisciplinarity within gastrointestinal research, as shown by the percentage of gastroenterology papers that also fall within one of the other subfields, varies by country, and table 4 shows the overall percentage of gastroenterology papers from each of the 14 countries that lies within the three subject areas. The countries are ranked in descending order of overall multidisciplinarity relative to the world mean. This has been rising steadily, from 3% to 8% in genetics, from 12% to 19% in infection, and from 19% to 26% in oncology. However, there are some notable exceptions. Thus in genetics (GENET), Spain has increased its joint output between 1985–87 and 1995–97 the most (+60%) relative to the world average, followed by Finland (+44%), and the UK (+43%). By contrast, Israel has actually reduced its output of joint papers in this subject compared with a world growth of +143%. In infectious diseases, Finland has increased its joint output over the same period most (+48% on top of the world growth of +44%), followed by Spain (+40%). Australia's joint output has grown relatively the least (−28% relative to the world, or just +3% absolutely), but for the first seven years it was much the highest joint output in the world, reflecting the discovery of the effects ofHelicobacter pylori in that country.9 ,10 In oncology, the biggest relative growth has been in Spain (+24% on top of the world growth of +29%). However, the percentage of gastroenterology papers that are concerned with oncology is still quite low (17%), particularly compared with the situation in Japan where from 1996–98 the proportion of oncology papers in the gastroenterology output was as high as 42% when allowance is made for the oncology correction factor.
Categorisation of gastroenterology papers by RL, from clinical to basic, and by PIC, from low to high, is shown in figs 1 and 2 for the 16 countries (that is, including New Zealand and South Africa) and for the world in 1997–98. The countries are listed in descending order of mean RL (that is, the most basic at the top) and mean PIC (that is, the highest potential impact at the top) (See comment in methods section on the practice of calculating mean values for RL and PIC.) Overall, nearly 20% of the world papers are classified as basic research, but Canada published 30% of its papers in these journals whereas Finland, Italy, and Israel perform less than 15% of basic research in the subfield. In terms of journal PIC, the world distribution approximates closely to the idealised one of 10% in PIC4, 20% in PIC3, 30% in PIC2, and 40% in PIC1 journals that the classification scheme was designed to produce: the actual percentages are 11.1%, 17.7%, 29.6%, and 41.6%. The USA publishes 18% of its papers in PIC4 journals whereas the UK only achieves just under 10% in this category of journals.
There have been some notable changes both in RL and PIC. For four countries, outputs have steadily become relatively more clinical: Japan, Spain, the Netherlands, and New Zealand; for another four they have become relatively more basic: Canada, Germany, Finland, and South Africa. In terms of potential impact, the countries showing the most marked and steady improvement in the quality of their outputs are the Netherlands, France, Sweden, Finland, Italy, Spain, and Germany; by contrast, the potential impact of the outputs of Australia, Japan, and New Zealand have steadily declined relative to the world average.
As expected, the mean number of authors per paper increased steadily with time for all countries, as did the mean number of addresses. The world mean values in the three time periods rose from 3.7 to 4.4 and 5.1 authors and from 1.8 to 2.0 and 2.3 addresses. Italian papers, followed by those from Japan, Spain, and France, have the most authors, and New Zealand the fewest. Finnish papers showed the most collaboration (most addresses) followed by the Israelis; New Zealand and Japanese papers had the fewest addresses.
Data on the funding of gastroenterology research papers in the different countries are shown in fig 3. For each country, the percentage of papers funded by the national government, by national private non-profit (PNP) sources, by industrial sources, and without funding acknowledgements is shown. (Data were also collected on funding by foreign governmental, including international, and foreign PNP sources but are not shown.) Total apparent funding may exceed 100% (because of multiple funding of some papers) or be below 100% (if there are many papers funded by foreign governments and non-profit sources). Israel is the most notable example of a country with funding from abroad but several European Union countries also benefit to some extent from such funding. The very low government support for Finnish research is anomalous as much of the funding is directed through the Academy of Sciences which is formally classified as a PNP organisation. Overall, national governments funded 49% of papers (when the percentages were weighted by the numbers of papers from each country in 1997–98), national PNP sources funded 31% of papers, industry funded 18%, and almost 40% of papers had no funding acknowledgement. (The percentage for Finland has been increased by half the PNP total to allow for the anomalous treatment of the Finnish Academy; correspondingly, the PNP total has been reduced by the same amount.) The mean number of funders per paper for each country is shown in table5. The very high value for Sweden is mainly attributable to the numerous endowed foundations that exist in that country, many of which give small grants, notably for equipment.
Table 6 shows the incidence of mortality (not standardised for age distribution) per 100 000 population for males and females together from digestive causes. The columns refer to neoplasms and other diseases; the countries are ranked by total mortality rates in a recent year. Finally, fig 4 shows the output of each of the 14 countries in oncology+gastroenterology for 1995–98, multiplied by the 1.24 × 0.95 correction factor (taken from table 1), and expressed as a percentage of biomedical papers, plotted against mortality from gastric cancer taken from table 6. The correlation is significant and positive atr 2=0.90. It remains large (0.66) even if the data point for Japan is excluded.
Discussion
The three questions posed in the introduction can now be considered in the light of the results that have been obtained. With regard to overlap, there is clearly a substantial oncology component in gastroenterology research and it now accounts for just over 25% of the gastroenterology total (but over 40% in Japan). Infection accounts for just under 20% of gastroenterology research but genetics has yet to make a big contribution to the subfield and represents less than 10%. For all three subfields, the overlap with gastroenterology has steadily increased worldwide, showing that the subject has become more multidisciplinary. Most European countries appear to be performing satisfactorily in terms of both quantity and quality of output, and the latter is increasing in all of those studied, except the UK.
Is there a correlation between relative commitment to cancer research in gastroenterology and the incidence of gastroenterological neoplasms? Figure 4 appears to show that countries are taking account of the burden of gastric cancer in making decisions on the importance of research in the area. This may be because individual researchers in countries with a high incidence are influenced by their personal knowledge of individual cases, which gives them an incentive to seek solutions. It may also reflect a conscious political choice to tackle a major health problem. There are some outlying countries, as would be expected: Italy has a higher output than the burden of disease would suggest and the UK and Canada a lower output.
Another comparison of output and disease burden is shown in fig 5, in which the ordinate (y axis) shows the output of gastroenterology papers outwith oncology in the same years and the abscissa (x axis) shows the death rate from other gastrointestinal diseases. There is still a positive correlation but it is not as strong as for gastroenterological neoplasms. Notable outliers are Australia and Spain (with higher outputs than expected), and the USA, the Netherlands, Finland, the UK, and Germany (with lower outputs than expected).
The impact of gastroenterology papers from the different countries is shown in terms of the PICs of the journals in which they are published in fig 2. In 1997–98, the USA was still the leader in terms of mean PIC values but it looks as if several European countries may overtake it early in the next century—probably the Netherlands, Finland, and Switzerland.
It is of interest to see if the national differences in mean PIC values for 1997–98 can be explained, at least in part, by differences in number of authors (A), number of addresses (D), mean numbers of funders (F), and mean RL. (Data on funding are taken from 1995 papers, or from papers from 1996–97.) The best fit from a similar equation to that used for the UK study is given by:
PIC=0.075×A+0.1×D+0.106× F−0.005×F2+0.1×RL+1.02
when the squared differences between observed and estimated PIC values are given a relative weighting equal to the square root of the number of papers from each country. The observed mean PIC values and those calculated on the basis of this equation are shown for the 16 countries in fig 6. There is a positive correlation between the two, showing that the differences in mean input parameters for each country are indeed able to account for some of the differences in the observed PIC values.
Countries with data points above the line are performing better than would be expected on the basis of the mean input parameters: they notably include the USA, the Netherlands, the UK, and Switzerland. It is probably no coincidence that these countries operate on the basis of competitive funding with many different sources able to support investigator initiated research projects. Conversely, countries whose data points are below the line are performing less well. Although New Zealand and South Africa, together with Israel, Spain, and Italy fall into this category, it should be emphasised that there were very few papers from the first two of these countries (144 and 158, respectively) and so the error bounds on the observed PIC values will be unusually high. Japan and Italy are disadvantaged in this analysis because they both have unusually large numbers of authors per paper, which may reflect more the national culture in terms of listing contributors to the research than the reality of who actually did the work.
The equation is similar to the one relating journal weighting to input parameters for UK gastroenterology papers in terms of the coefficients for A and F. This suggests that numbers of authors and numbers of funders have a real and positive association with research impact, as was found by a very different study on papers submitted to a single journal.11
What are the conclusions to be drawn for the conduct of gastroenterology research? Firstly, there appears to be an association between the incidence of gastrointestinal disease in different countries and the amount of research that takes place, especially in gastric cancer. However, several countries, notably the UK, are performing less than an international comparison would suggest is appropriate. In view of the slow relative decline in gastroenterology research worldwide over the past 15 years, it would appear that an increased level of support could be justified, at least in some countries.
Secondly, the research agenda in gastroenterology still seems to be largely dominated by the traditional sources of disease—cancer and infection—and takes little account of the enormous changes in genetics that are currently taking place as a result of the deciphering of the human genome. This suggests that gastroenterologists need to make more of an effort to work with colleagues in the field of genetics, which could be to the long term benefit of both.
A third conclusion is that research carried out by large teams, with inter-laboratory collaboration and several funding sources, may have more influence on other researchers because it is likely to be published in journals of higher impact. This suggests that funding bodies should aim to support larger teams of researchers working in cooperation rather than lone investigators or small teams. It also suggests that a plurality of funding sources—government agencies, charities, foundations, and industry—can provide a lively and vigorous environment in which researchers can compete. This appears to be conducive to the production of high quality work.
Acknowledgments
This study was commissioned by the British Society of Gastroenterology, and was financially supported by the Dutch branches of Astra-Hässle, Byk-Gulden, Janssen-Cilag, Schering-Plough, and by the Netherlands Society of Gastroenterology. The authors are grateful to Rob Cottrell, who organised examination of papers in libraries to determine their funding, and to Vishal Gulati, who wrote macros to determine the numbers of authors and addresses on each paper, both of the Wellcome Trust's Policy Unit.
Abbreviations used in this paper
- A
- number of authors per paper
- C0–4
- five year citation count
- CF
- calibration factor
- D
- number of addresses per paper
- F
- number of funders per paper
- GASTR
- gastrointerology (filter)
- GENET
- genetics (filter)
- ICD
- International Classification of Deaths
- INFEC
- infection (filter)
- NFP
- private non-profit
- ONCOL
- oncology (filter)
- PIC
- potential impact category
- RL
- research level (clinical to basic)
- SCI
- Science Citation Index