you could probably savely claim that they “show” rather than “seem to suggest” – otherwise good
looks interesting. is there any critique or limitations to be associated with this diagram? you might be able to provide a quick summary of the main message of this diagram in paragraph 10 in addition to what the reader might interprete its message to be.
Hey Joe, interesting paper which summarises the failings (and uses) of traditional mentrix in classifying impact. I am assuming that the paper (by mentioning alternative forms of impact) attempts to draw out these altmethods by mentioning CoPs in academia. I miss a more comprehensive overview of emergent impact metrics although places like Mendeley are mentioned. while PhD students understand text well, there is a lack of images/illustrations/summary tables. rock on
—> story by Reinhart, Rogoff, & Herndon…. if you haven’t mentioned this before don’t mention new material in the conclusion…
Altmetrix as well as traditional metrix may both be contingent to the reality we live in, since it may be that due to current values and believes an article may be less desirable to cite or influence, even though it might be a well argued piece. Also, what if the article can hardly be found and its this which is the barrier to impact. I guess then it shows that ideas kept in silos are in hibernation.
this all seems to be more part of the literature analaysis and not yet of the discussion. putting the alt metrix definition, limitations and advantages much further would be useful to link it up more closely with the discussion on impact and to then focus on new metrix and CoP. also, I think your discussion on the definition of impact might be made shorter just by focusing on the main point that we need some way of “measuring impact” — one old one are the trad methods… bring them closer to the impact section. I start to repeat myself
I don’t remember if you mentioned the altmanifesto at the beginning, if not you should…
possibly mentioned too late? also I would argue it to be worth if you were to summarize some of the discussion around traditional methods by an illustration, such as a table or sketch…
I agree to your above comment as this paragraph appears to be a key statement for your paper and I would recommend to use it as an introduction to the discussion of new metrix in combination with the stuff on CoPs that you got earlier….
Joe, wouldn’t harm to put a reference for CoP theory in here at this early point, even if it comes later (something which I can’t tell at this point)
hey Joe, have you defined “value” anywhere? If so, it wasn’t too clear for me.
it might be worth to use the same thing here and use it at the beginning when talking about the h-index etc. you might have done that – beg you pardon if you did, since my memory is exhibiting holes on occasion
mate, I wonder if it might be useful (actually I think it would be highly useful) to note how the CoP connects back to “impact” which hasn’t been mentioned here much, but would introduce some punchlines
mate, I think on page 1 or so you refer to “social learning” in the context of CoP, while situated is the correct form, as also mentioned here – I might be mistaken
Ironically the blog post linked to talks about a new feature for Google Reader, that was made during 20% time. Google Reader is soon to be retired……
The disciplines that are mentioned here are specifically management, design and computing, as these are the schools that HIghWire is associated with within Lancaster University.
Although I think most people will be totally fine without reading up on how to work with CommentPress (I think its quite intuitive), if you are uncertain I really recommend reading the How to read a CommentPress document page (linked at the bottom of this paragraph).
Yes, quite right. Thanks Sophie!
Note that throughout the whole document CoP may refer to a (singular) community of practice, or the plural version communities of practice.
Nesta is an independent charity that works to increase the innovation capacity of the UK. (from Wikipedia)
The Blackwell report (available here) is a really intriguing document, in a more involved study (longer word count) it would make sense to refer to much more heavily.
Refer to infometrics and scientometrics for more information on areas related to bibliometrics that are specialised for particular purposes.
The H-index is based on the work of J.E. Hirsch (detailed in An index to quantify an individual’s scientific research output).. it is based on quite complex maths/stats, and does give a more meaningful measure than purely counting citations, although for the reasons explored in more detail on the next page it is still somewhat flawed. Also as Arnold & Fowler explore, it’s quite easy to manipulate… something that is surely tempting for researchers and their institutions.
The reason I say altmetrics do ‘by implication’ have social elements, is that thy’re often calculated based on data from ‘social’ websites.
The value of ‘scaffolding’ in papers such as this is disputable but I generally elect to leave it in, I like to make sure that my intentions clearly built slowly through the title, abstract, and introduction. At the end of the introduction seems to be a sensible point to explain what is coming next (and why) just in case the slow evolution has lost some people along the way.
There are many more studies (referred to later in the section on tradmetrics) that support the premise that data from citation indexing is difficult to interpret.
Good point: I’ll do that. I might – to be provocative – actually reference the Wikipedia article for the introduction. You’ll see them later… but there are references to peer-reviewed literature on CoP later on.
Maybe the explanation of what is meant by the Wolff quote should be expanded?
The explanation about the Wolff story has been altered now so that it explains the pertinent facts (Wolff’s report changed policy, but was unpublished and there is no record that it changed policy… but seems extremely likely – how does this reflect in its impact)
This image is taken from an ebook/extensive blog published by the London School of Economics here is a fantastic resource for understanding impact as it relates to social sciences in particular. Some pages of the book have a healthy discussion about the intricacies of the field.
The diagram itself, I believe, is a good representation of what is generally meant by impact: semiotically to me this diagram shouts complexity, interconnectedness, multidimensionality, dynamic.
The reference to the REF does not refer to the REF’s own interpretation of impact, but is only included to demonstrate that ‘whatever they interpret impact as’… is important.
I’m wary of having multiple references to a newpaper, in particular one that is so politically aligned as The Guardian. The Guardian does, however, have a wealth of interesting articles on research, higher, and further education. Some interesting examples are these:
Could be improved by referring to a broader scope of the CoP literature, this is very Wenger-centric. Potential to refer to the various studies of only/digital/anonymous CoP experiments (which usually show encouraging results).
The theoretical background, and pathways to imagining how something could be implemented, have been strengthened with the inclusion of the references to virtual communities (case study in the paper is from Caterpillar) and online communities (case study is for faculty development, in a University) and finally the paper on ‘Growing up digital’ that analyses how the web influences learning systems.
I have not read the Wenger & Lave book (although have skimmed some chapters) hence I referred to the Francois & Quek book review, which is from the Proceedings of the CHI conference (2011) and as such one would assume a well-reviewed piece.
The references to Wenger (2006) are from his personal website, which is not peer-reviewed, although the content is mainly derived from peer-reviewed writings.
I have around 50 notes that liken elements of Communities of Practice and Social Learning Systems, to artefacts of scholarly publishing, unfortunately with a restricted word count they could not all be included.
The 2002 Brown paper explains in detail how the web has affected learning: many of his points seem to be applicable to the learning that happens in the academic community. The concept of the web is quite interchangeable with scholarly publishing networks.
I refer to ‘tacit value’ as I believe there is value, but that it is implied, silent, and muted.
Analyses is the plural for analysis… So, in reply to your second comment, yes, I think there are far too many analyses in this paragraph, I’ll fix it up….
There’s many different kinds of study!! :o) That’s part of what this is arguing. The communities of practice stuff is all grounded in ‘situated learning’ – which I think 9 out of 10 times would be very different to the kind of learning that’s been required to produce this….
Hope it made sense. Thanks for the corrections too, they’re appreciated!
I’ve fixed that paragraph, removed some references to analysis and broken it up into two paragraphs. Thanks for pointing it out.
I believe that ‘tradmetrics’ is a term not used previously, so has been coined in this paper.
The list of papers exploring the pros and cons of various studies were mostly part of a to-and-fro discussion between academics that seemed to start in the late 70s and continue until the early 90s. This is a perfect example of how slowly knowledge progresses, which in part is due to the slowness of publication/review/citation indexing cycles. Digital/web technology can certainly improve upon this (and has done already).
I think there is space for a qualitative study of what academics (maybe at different career stages, and across different fields) believe the significance of citation counts, H-indexes and journal impact factors are: something I couldn’t find in my literature searching. Anecdotally I know that almost all the academics I have spoken to consider these things very important factors in the ‘game’ of academia.
I have anecdotally heard of ‘citation circles’ where scholars will agree to cite each others work, as it would be mutually beneficial. This is only one way that the system can be gamed. It would be interesting to try and quantify the scale of these circles. You could consider this kind of citation an inherently social one, so in some way could be described by the CoP theory. Maybe this is a new kind of disingenuously constructed social connection.
Priem & Hemminger do not ‘pull any punches’ as regards traditional metrics. Maybe their rhetoric is a little too strong and may ‘put off’ traditionalists.
This is speculation, but seems to be reasonable based on the prior remarks and references.
This paper is trying to cram a lot of thinking into a relatively small space. I think this argument in particular should be explored in isolation, and could then inform further study. The first of the three research questions (in the conclusion) tries to address this.
The classification of citation function papers are extremely interesting. They’re coming from a computer science/natural language processing perspective, and one discusses the use of annotations (in part inspiring me to publish this document with annotations such as this one).
There are three approaches:
My belief is that none of these will really work until semantic publishing is ubiquitous… whatever system is employed needs to be interoperable.
The hypothes.is (http://hypothes.is/) project touches on the annotation angle and may prove to have influence on his problem.
The Cameron paper is quite visionary, although in practice, 15/16 years after publication, we don’t seem to have taken any steps towards his vision. I wonder why?
I have referenced by own work here. In part this is a tongue-in-cheek acknowledgement of the fact that the majority of scholarly writers do this, not always for entirely innocent reasons. In this case I do think the piece of writing is directly relevant (I’m exploring the evolution of traditional metrics, and altmetrics, and how opaque these practices are).
Self-citation is often given as a reason why anonymous peer-review is flawed: you can usually figure out who somebody is because they will repeatedly cite their own work.
Disingenuous self-citations present another good argument for citation-annotation or citation-classification systems to become more prevalent.
Google’s Ngram Viewer shows how often keywords occur in historic literature. If you plug in various keywords you can see how different topics ‘trend’ at different points in history. The examples below demonstrate how topics covered in this paper have been discussed… the Ngram graphs correlate nicely with the narrative presented in this paper.
You can also view them all on the same graph…. here.
Viewing peer-view etc as ‘filters’ is a useful way to analyse their real-world meaning.
Speed is a huge criticism of tradmetrics. If you consider somebody may be given tenure based on their citation-counts, what if their citations are all 30 years old? Some modified metrics (i10 rating is one example) do take time into account, but the challenge is in interpretation and understanding: statistics aren’t commonly appreciated.
The subtle differences between social interaction, connection and signature are somewhat difficult to imagine, but they’re crucial when critiquing altmetrics in terms of communities of practice.
A signature could be me retweeting somebody mentioning their work.
An interaction is the very fact I have retweeted.
The connection is a combination of the two.
Nanopublishing is an area I haven’t explored at all, although it is certainly related to this area. One example of its utility is that a ‘citable unit’ could be a paragraph, rather than a whole paper. Applying this technolog could be a game-changer for scholarly publishing.
Nanopublishing could easily be build using linked data, and the ideals of the semantic web. The TED talk and the Bizer et. al. paper are explaining how the semantic web ‘vision’ is developing (at 2009).
These are all big ideas, and could have a profound effect on ‘impact’.. they are also long-term projects, but deserve due consideration now.
I refer to my a paper my brother contributed to for the ‘Database Wiki Project’. Am I citing the paper because I’d like to help my brother enhance his citation counts, or am I citing it because it is relevant? I believe the answer is both. Some of the ideals of the database wiki project could be applied to the problem of curating scholarly literature.
The story of Thomas Herndon is current. It is also (I think) may be being exaggerated by the media somewhat. Nonetheless it does show that a paper that has had some impact, was peer-reviewed, and yet contained very basic errors. Presumably a more junior scholar would have had their workings scrutinised more than Reinhart and Rogoff did.
It also highlights the fact that although this was a paper based on numbers and data, the actual data wasn’t included in the paper (or in the review process). How thoroughly can a review process be, where the actual data isn’t even viewable?
It is a testament to Renhart and Rogoff that they shared their data with Herndon, there was obviously no harm intended by either of them.
Could an alt-review system be constructed so that it both mitigated this kind of risk, and also facilitated communities of practice?
Paul Ralph – quoted here – has taught me, and is a young academic that I have huge respect for and in some ways has inspired my interest in this field. I have not asked for permission to quote him, so I am hopeful that he doesn’t take any sort of offence (the quote is from a public facebook page).
I wonder how can the sentiment embodied in Paul’s words be translated into an empirical study, that might be able to move peer-review towards being ‘about methodology’ (as Paul puts it) rather than politics. This might be the best strategy for attacking the entrenchment…
Here I’m trying to point out that if CoP are relevant to how much impact a paper carries, and also if openness (of review, access, etc) are relevant to these CoP… then potentially a fantastic strategy for cultivating (and measuring) impact would be to move towards much more open publishing (in terms of review and access).
Marcus Du Sautoy says of peer-review, in the Guardian:
“We live in an age where everything has to be sealed and delivered and complete when it’s delivered and complete when it meets a journal and, in fact, that’s not how science is done”
I really like the idea of more ‘living’ documents. For this specific text however, I think a paragraph about the possibilities and perspectives for this new kind of text – both in academia, popular media and elsewhere.
I presume you should have the word ‘area’ in the second line of paragraph 5
Line 13 analyses – I am unable to find that word anywhere.
Sjoe Joe that is a lot of’ analysis’ in one paragraph
I have learnt a lot by reading your paper and have decided I am far better suited to working than study!
May 23, 2013 at 9:08 pm
See in context
April 28, 2013 at 9:24 pm
April 28, 2013 at 2:39 pm
April 28, 2013 at 2:32 pm
April 28, 2013 at 2:28 pm
April 28, 2013 at 2:25 pm
April 28, 2013 at 2:22 pm
April 28, 2013 at 2:21 pm
April 28, 2013 at 2:18 pm
April 28, 2013 at 2:11 pm
Website content © Communities of Impact 2018. All rights reserved.