Very quick note on things that are used but not cited

In most of the discussions of using usage as a metric of scholarly impact, the example of the clinician is given.  The example goes that medical articles might be heavily used and indeed have a huge impact on practice (saving lives), but be uncited. There are other fields that have practitioners who pull from the literature, but do not contribute to it.

So it was with interest that I read this new article by the MacRoberts:

MacRoberts, M., & MacRoberts, B. (2009). Problems of citation analysis: A study of uncited and seldom-cited influences Journal of the American Society for Information Science and Technology, 61, 1-12 DOI: 10.1002/asi.21228

The article provides great examples from the field of biogeography (the distribution of plants and animals over an area - they tell me). It is typical for researchers in this field, when writing articles in peer-reviewed journals, to not cite their data sources.  Some of the data sources are flora  - "a list of plant species known to occur within a region of interest." The flora might be books, government reports, notes in journals or some other sort of gray literature.

The authors give a couple examples - one is their article - and show how these articles are uncited according to Web of Science, but heavily used and well incorporated into databases, books, and pamphlets. As they say, the purpose of the article has been achieved.

Not only are these things used directly, but once their contents are incorporated into databases, the database then goes on to serve maybe thousands of people. The sources are often listed in notes or in an appendix but no citation. This content that is sucked up into books or databases provides no traceable usage link - as far as I can tell. If we can't even determine the impact of the article - a container for an idea - how can we understand or evaluate the impact of the author and his or her knowledge contributions?

It's been noted elsewhere (see the article for citations/discussion) that the largest influence on a scientist often comes from informal communication partners - colleagues and co-workers. This is not cited, either.  So the question becomes, if we are truly interested in evaluating a scientist on his or her influence, we have to come up with new methods that look at how their ideas have been used - it is not enough to look at article citations or downloads.

(as an aside: the authors quote a website that bemoans the difficulty of locating floras. Certainly if they were cited, that would help!)

4 responses so far

  • Hi Christina
    I am working on a semantic approach to identifying what I call ``lineages of works,'' which is not exactly like influence, but clearly has some strong affinities with it. There are serious but not insurmountable conceptual problems that need to be solved, or at least rendered harmless. Implementation of the idea will require expert human readers, but I think can eventually be taken up by machine readers. My purpose is to aid researchers in resource discovery. I thought you and your readers might be interested in this. Check back now and again at the web site listed above (the front page is a blog) for updates on my progress. If only the blogosphere weren't so interesting and I would get back to writing, I would have finished the essay about this that I am now writing.... sigh.

  • I guess the key question is whether uncited influence, within a field, is correlated with cited influence. It seems likely that the most influential papers in a given field will be used in both cited and uncited ways. So even if there's a fair bit of uncited use, the metric is still somewhat useful, so long as enough papers get cited at all.
    Of course, if you try to do comparisons of papers or researchers from different fields, that breaks down. Perhaps that could be corrected for somehow?

  • Christina Pikas says:

    Their point - which I didn't convey that clearly - is that in this particular field, there is no correlation. That is, uncited documents are heavily used. There are lots of ways to correct citation metrics from field to field, with varying success.

  • Jim Croft says:

    Finding Floras is a no-brainer. There may not be faceted website yet, but there is an oldskool turning page type book, a guide to the standard Floras of the world, and a search through any half decent online library catalogue would quickly reveal an answer.
    But the problem is deeper and more systemic than that. When it comes to documentation of sources and concepts, no discipline is more fastidious and self-citing than biological taxonomy. Every prior reference since the 1750s is sleuthed and documented in excruciatingly boring detail, yet these citations are never captured by the indexers because they are embedded in the formality of the species descriptions and not in the surrounding narrative, in the interests of brevity and economy. Most times this represents rigourous scientific confirmation or elaboration of prior art but it is never picked up as citation, even though full bibliographic details are provided. Taxonomists have long been dissatisfied with the big citation indices as a reliable metric of their productivity and impact.
    There is a strong move now towards putting data online and treating it as a publication. Interestingly, our pedestrian data and other resources on the website get 3-4 orders of magnitude more attention in a day than our most erudite journal publication would get in a lifetime.