All Comments

Comments on the Pages

  • Tradmetrics; citation analysis (22 comments)

    • Comment by Sophie on April 22nd, 2013

      Line 13 analyses – I am unable to find that word anywhere.

      Comment by Sophie on April 22nd, 2013

      Sjoe Joe that is a lot of’ analysis’ in one paragraph

      Comment by joe on April 22nd, 2013

      Analyses is the plural for analysis… :) So, in reply to your second comment, yes, I think there are far too many analyses in this paragraph, I’ll fix it up….

      Comment by Sophie on April 22nd, 2013

      I have learnt a lot by reading your paper and have decided I am far better suited to working than study!

      Comment by joe on April 22nd, 2013

      There’s many different kinds of study!! :o) That’s part of what this is arguing. The communities of practice stuff is all grounded in ‘situated learning’ – which I think 9 out of 10 times would be very different to the kind of learning that’s been required to produce this….

      Hope it made sense. Thanks for the corrections too, they’re appreciated!

      Comment by joe on April 25th, 2013

      I’ve fixed that paragraph, removed some references to analysis and broken it up into two paragraphs. Thanks for pointing it out.

      Comment by joe on April 26th, 2013

      I believe that ‘tradmetrics’ is a term not used previously, so has been coined in this paper.

      Comment by joe on April 26th, 2013

      The list of papers exploring the pros and cons of various studies were mostly part of a to-and-fro discussion between academics that seemed to start in the late 70s and continue until the early 90s. This is a perfect example of how slowly knowledge progresses, which in part is due to the slowness of publication/review/citation indexing cycles. Digital/web technology can certainly improve upon this (and has done already).

      Comment by joe on April 26th, 2013

      I think there is space for a qualitative study of what academics (maybe at different career stages, and across different fields) believe the significance of citation counts, H-indexes and journal impact factors are: something I couldn’t find in my literature searching. Anecdotally I know that almost all the academics I have spoken to consider these things very important factors in the ‘game’ of academia.

      Comment by joe on April 26th, 2013

      I have anecdotally heard of ‘citation circles’ where scholars will agree to cite each others work, as it would be mutually beneficial. This is only one way that the system can be gamed. It would be interesting to try and quantify the scale of these circles. You could consider this kind of citation an inherently social one, so in some way could be described by the CoP theory. Maybe this is a new kind of disingenuously constructed social connection.

      Comment by joe on April 26th, 2013

      Priem & Hemminger do not ‘pull any punches’ as regards traditional metrics. Maybe their rhetoric is a little too strong and may ‘put off’ traditionalists.

      Comment by joe on April 26th, 2013

      This is speculation, but seems to be reasonable based on the prior remarks and references.

      Comment by joe on April 26th, 2013

      This paper is trying to cram a lot of thinking into a relatively small space. I think this argument in particular should be explored in isolation, and could then inform further study. The first of the three research questions (in the conclusion) tries to address this.

      Comment by joe on April 26th, 2013

      The classification of citation function papers are extremely interesting. They’re coming from a computer science/natural language processing perspective, and one discusses the use of annotations (in part inspiring me to publish this document with annotations such as this one).

      There are three approaches:

      • (subjective) human citation classification, using a schematic
      • automatic citation classification (using language processing techniques)
      • citation-annotation (such as this annotation) that doesn’t ‘tick a box’ but does give a  qualitative account of particular citations

      My belief is that none of these will really work until semantic publishing is ubiquitous… whatever system is employed needs to be interoperable.

      The hypothes.is (http://hypothes.is/) project touches on the annotation angle and may prove to have influence on his problem.

      Comment by joe on April 26th, 2013

      The Cameron paper is quite visionary, although in practice, 15/16 years after publication, we don’t seem to have taken any steps towards his vision. I wonder why?

      Comment by joe on April 26th, 2013

      I have referenced by own work here. In part this is a tongue-in-cheek acknowledgement of the fact that the majority of scholarly writers do this, not always for entirely innocent reasons. In this case I do think the piece of writing is directly relevant (I’m exploring the evolution of traditional metrics, and altmetrics, and how opaque these practices are).

      Self-citation is often given as a reason why anonymous peer-review is flawed: you can usually figure out who somebody is because they will repeatedly cite their own work.

      Disingenuous self-citations present another good argument for citation-annotation or citation-classification systems to become more prevalent.

      Comment by joe on April 28th, 2013

      Google’s Ngram Viewer shows how often keywords occur in historic literature. If you plug in various keywords you can see how different topics ‘trend’ at different points in history. The examples below demonstrate how topics covered in this paper have been discussed… the Ngram graphs correlate nicely with the narrative presented in this paper.

      Citation index

      Citation analysis

      Research impact


      Comment by joe on April 28th, 2013
      Comment by PlausibleApe on April 28th, 2013

      it might be worth to use the same thing here and use it at the beginning when talking about the h-index etc. you might have done that – beg you pardon if you did, since my memory is exhibiting holes on occasion

      Comment by LongerLastingApe on April 28th, 2013

      I agree to your above comment as this paragraph appears to be a key statement for your paper and I would recommend to use it as an introduction to the discussion of new metrix in combination with the stuff on CoPs that you got earlier….

      Comment by IsAGreatApe on April 28th, 2013

      this all seems to be more part of the literature analaysis and not yet of the discussion. putting the alt metrix definition, limitations and advantages much further would be useful to link it up more closely with the discussion on impact and to then focus on new metrix and CoP. also, I think your discussion on the definition of impact might be made shorter just by focusing on the main point that we need some way of “measuring impact” — one old one are the trad methods… bring them closer to the impact section. I start to repeat myself

      Comment by LeavesTheMoonApe on April 28th, 2013

      possibly mentioned too late? also I would argue it to be worth if you were to summarize some of the discussion around traditional methods by an illustration, such as a table or sketch…

  • Introduction (10 comments)

    • Comment by joe on April 24th, 2013

      Nesta is an independent charity that works to increase the innovation capacity of the UK. (from Wikipedia)

      Comment by joe on April 24th, 2013

      The Blackwell report (available here) is a really intriguing document, in a more involved study (longer word count) it would make sense to refer to much more heavily.

      Comment by joe on April 24th, 2013

      Refer to infometrics and scientometrics for more information on areas related to bibliometrics that are specialised for particular purposes.

      Comment by joe on April 24th, 2013

      The H-index is based on the work of J.E. Hirsch (detailed in An index to quantify an individual’s scientific research output).. it is based on quite complex maths/stats, and does give a more meaningful measure than purely counting citations, although for the reasons explored in more detail on the next page it is still somewhat flawed. Also as Arnold & Fowler explore, it’s quite easy to manipulate… something that is surely tempting for researchers and their institutions.

      Comment by joe on April 24th, 2013

      The reason I say altmetrics do ‘by implication’ have social elements, is that thy’re often calculated based on data from ‘social’ websites.

      Comment by joe on April 24th, 2013

      The value of ‘scaffolding’ in papers such as this is disputable but I generally elect to leave it in, I like to make sure that my intentions clearly built slowly through the title, abstract, and introduction. At the end of the introduction seems to be a sensible point to explain what is coming next (and why) just in case the slow evolution has lost some people along the way.

      Comment by joe on April 26th, 2013

      There are many more studies (referred to later in the section on tradmetrics) that support the premise that data from citation indexing is difficult to interpret.

      Comment by FunnyApe on April 28th, 2013

      Altmetrix as well as traditional metrix may both be contingent to the reality we live in, since it may be that due to current values and believes an article may be less desirable to cite or influence, even though it might be a well argued piece. Also, what if the article can hardly be found and its this which is the barrier to impact. I guess then it shows that ideas kept in silos are in hibernation.

      Comment by MadApe on April 28th, 2013

      Joe, wouldn’t harm to put a reference for CoP theory in here at this early point, even if it comes later (something which I can’t tell at this point)

      Comment by joe on April 28th, 2013

      Good point: I’ll do that. I might – to be provocative – actually reference the Wikipedia article for the introduction. You’ll see them later… but there are references to peer-reviewed literature on CoP later on.

  • Conclusions and Research Questions (10 comments)

    • Comment by joe on April 26th, 2013

      Nanopublishing is an area I haven’t explored at all, although it is certainly related to this area. One example of its utility is that a ‘citable unit’ could be a paragraph, rather than a whole paper. Applying this technolog could be a game-changer for scholarly publishing.

      Comment by joe on April 26th, 2013

      Nanopublishing could easily be build using linked data, and the ideals of the semantic web. The TED talk and the Bizer et. al. paper are explaining how the semantic web ‘vision’ is developing (at 2009).

      These are all big ideas, and could have a profound effect on ‘impact’.. they are also long-term projects, but deserve due consideration now.

      Comment by joe on April 26th, 2013

      I refer to my a paper my brother contributed to for the ‘Database Wiki Project’. Am I citing the paper because I’d like to help my brother enhance his citation counts, or am I citing it because it is relevant? I believe the answer is both. Some of the ideals of the database wiki project could be applied to the problem of curating scholarly literature.

      Comment by joe on April 26th, 2013

      The story of Thomas Herndon is current. It is also (I think) may be being exaggerated by the media somewhat. Nonetheless it does show that a paper that has had some impact, was peer-reviewed, and yet contained very basic errors. Presumably a more junior scholar would have had their workings scrutinised more than Reinhart and Rogoff did.

      It also highlights the fact that although this was a paper based on numbers and data, the actual data wasn’t included in the paper (or in the review process). How thoroughly can a review process be, where the actual data isn’t even viewable?

      It is a testament to Renhart and Rogoff that they shared their data with Herndon, there was obviously no harm intended by either of them.

      Could an alt-review system be constructed so that it both mitigated this kind of risk, and also facilitated communities of practice?

      Comment by joe on April 26th, 2013

      Paul Ralph – quoted here – has taught me, and is a young academic that I have huge respect for and in some ways has inspired my interest in this field. I have not asked for permission to quote him, so I am hopeful that he doesn’t take any sort of offence (the quote is from a public facebook page).

      I wonder how can the sentiment embodied in Paul’s words be translated into an empirical study, that might be able to move peer-review towards being ‘about methodology’ (as Paul puts it) rather than politics. This might be the best strategy for attacking the entrenchment…

      Comment by joe on April 26th, 2013

      Here I’m trying to point out that if CoP are relevant to how much impact a paper carries, and also if openness (of review, access, etc) are relevant to these CoP… then potentially a fantastic strategy for cultivating (and measuring) impact would be to move towards much more open publishing (in terms of review and access).

      Comment by FinalTwoOrThreeCommentsApe on April 28th, 2013

      —> story by Reinhart, Rogoff, & Herndon…. if you haven’t mentioned this before don’t mention new material in the conclusion…

      Comment by MeetTheApeApe on April 28th, 2013

      hey Joe, have you defined “value” anywhere? If so, it wasn’t too clear for me.

      Comment by FinallyCommentApe on April 28th, 2013

      Hey Joe, interesting paper which summarises the failings (and uses) of traditional mentrix in classifying impact. I am assuming that the paper (by mentioning alternative forms of impact) attempts to draw out these altmethods by mentioning CoPs in academia. I miss a more comprehensive overview of emergent impact metrics although places like Mendeley are mentioned. while PhD students understand text well, there is a lack of images/illustrations/summary tables. rock on :)

      Comment by Joe on May 23rd, 2013

      Marcus Du Sautoy says of peer-review, in the Guardian:

      “We live in an age where everything has to be sealed and delivered and complete when it’s delivered and complete when it meets a journal and, in fact, that’s not how science is done”

  • Communities of Practice (8 comments)

    • Comment by joe on April 22nd, 2013

      Could be improved by referring to a broader scope of the CoP literature, this is very Wenger-centric. Potential to refer to the various studies of only/digital/anonymous CoP experiments (which usually show encouraging results).

      Comment by joe on April 25th, 2013

      The theoretical background, and pathways to imagining how something could be implemented, have been strengthened with the inclusion of the references to virtual communities (case study in the paper is from Caterpillar) and online communities (case study is for faculty development, in a University) and finally the paper on ‘Growing up digital’ that analyses how the web influences learning systems.

      Comment by joe on April 26th, 2013

      I have not read the Wenger & Lave book (although have skimmed some chapters) hence I referred to the Francois & Quek book review, which is from the Proceedings of the CHI conference (2011) and as such one would assume a well-reviewed piece.

      The references to Wenger (2006) are from his personal website, which is not peer-reviewed, although the content is mainly derived from peer-reviewed writings.

      Comment by joe on April 26th, 2013

      I have around 50 notes that liken elements of Communities of Practice and Social Learning Systems, to artefacts of scholarly publishing, unfortunately with a restricted word count they could not all be included.

      Comment by joe on April 26th, 2013

      The 2002 Brown paper explains in detail how the web has affected learning: many of his points seem to be applicable to the learning that happens in the academic community. The concept of the web is quite interchangeable with scholarly publishing networks.

      Comment by joe on April 26th, 2013

      I refer to ‘tacit value’ as I believe there is value, but that it is implied, silent, and muted.

      Comment by TwoFootedApe on April 28th, 2013

      mate, I think on page 1 or so you refer to “social learning” in the context of CoP, while situated is the correct form, as also mentioned here – I might be mistaken

      Comment by ThinkingApe on April 28th, 2013

      mate, I wonder if it might be useful (actually I think it would be highly useful) to note how the CoP connects back to “impact” which hasn’t been mentioned here much, but would introduce some punchlines

  • What is Impact? (7 comments)

    • Comment by joe on April 22nd, 2013

      Maybe the explanation of what is meant by the Wolff quote should be expanded?

      Comment by joe on April 25th, 2013

      The explanation about the Wolff story has been altered now so that it explains the pertinent facts (Wolff’s report changed policy, but was unpublished and there is no record that it changed policy… but seems extremely likely – how does this reflect in its impact)

      Comment by joe on April 26th, 2013

      This image is taken from an ebook/extensive blog published by the London School of Economics here is a fantastic resource for understanding impact as it relates to social sciences in particular. Some pages of the book have a healthy discussion about the intricacies of the field.

      The diagram itself, I believe, is a good representation of what is generally meant by impact: semiotically to me this diagram shouts complexity, interconnectedness, multidimensionality, dynamic.

      Comment by joe on April 26th, 2013

      The reference to the REF does not refer to the REF’s own interpretation of impact, but is only included to demonstrate that ‘whatever they interpret impact as’… is important.

      Comment by joe on April 26th, 2013

      I’m wary of having multiple references to a newpaper, in particular one that is so politically aligned as The Guardian. The Guardian does, however, have a wealth of interesting articles on research, higher, and further education. Some interesting examples are these:




      Comment by ContraditoryApe on April 28th, 2013

      you could probably savely claim that they “show” rather than “seem to suggest” – otherwise good

      Comment by CriticalApe on April 28th, 2013

      looks interesting. is there any critique or limitations to be associated with this diagram? you might be able to provide a quick summary of the main message of this diagram in paragraph 10 in addition to what the reader might interprete its message to be.

  • Preface and how-to guide (6 comments)

    • Comment by joe on April 11th, 2013

      Ironically the blog post linked to talks about a new feature for Google Reader, that was made during 20% time. Google Reader is soon to be retired……

      Comment by joe on April 11th, 2013

      The disciplines that are mentioned here are specifically management, design and computing, as these are the schools that HIghWire is associated with within Lancaster University.

      Comment by joe on April 11th, 2013

      Although I think most people will be totally fine without reading up on how to work with CommentPress (I think its quite intuitive), if you are uncertain I really recommend reading the How to read a CommentPress document page (linked at the bottom of this paragraph).

      Comment by Sophie on April 22nd, 2013

      I presume you should have the word ‘area’ in the second line of paragraph 5

      Comment by joe on April 22nd, 2013

      Yes, quite right. Thanks Sophie!

      Comment by Lea on April 28th, 2013

      I really like the idea of more ‘living’ documents. For this specific text however, I think a paragraph about the possibilities and perspectives for this new kind of text – both in academia, popular media and elsewhere.

  • Altmetrics; appreciating the scholarly ecosystem (4 comments)

    • Comment by joe on April 26th, 2013

      Viewing peer-view etc as ‘filters’ is a useful way to analyse their real-world meaning.

      Comment by joe on April 26th, 2013

      Speed is a huge criticism of tradmetrics. If you consider somebody may be given tenure based on their citation-counts, what if their citations are all 30 years old? Some modified metrics (i10 rating is one example) do take time into account, but the challenge is in interpretation and understanding: statistics aren’t commonly appreciated.

      Comment by joe on April 26th, 2013

      The subtle differences between social interaction, connection and signature are somewhat difficult to imagine, but they’re crucial when critiquing altmetrics in terms of communities of practice.

      A signature could be me retweeting somebody mentioning their work.

      An interaction is the very fact I have retweeted.

      The connection is a combination of the two.

      Comment by JumpingIntoTheWaterApe on April 28th, 2013

      I don’t remember if you mentioned the altmanifesto at the beginning, if not you should…

  • Abstract (1 comment)

    • Comment by joe on April 24th, 2013

      Note that throughout the whole document CoP may refer to a (singular) community of practice, or the plural version communities of practice.

Source: http://communitiesofimpact.joesart.org/all-comments/