The Methods Section

I remotely attended the OpenSciNY conference last Friday and was extremely impressed with the creativity and care going into current open science projects and resources.  The third speaker, Jean-Claude Bradley, was especially impressive, and I came away with some new ideas for scientific literacy instruction.

First off, Bradley discussed open notebook science, and how his ultimate goal is to make scientific research as transparent and accessible as possible. His research notebooks, and those of the students he teaches, are all available online and updated in real time.  Raw data from the notebook can be seamlessly connected to experimental reports.   All results are kept in easily shared formats (Google Docs, Excel) so that others can quickly view, comment, and use them.

But why go through all of this?  How do open practices benefit scholarship as a whole?  Many will argue that having experimental results easily accessible online is a detriment to scholarship; who is vetting this material? Where is the security of peer review?

Bradley flatly stated that peer review cannot and does not equal infallible results.  Review is not proof, and scholarship is not peer review.  The term ‘peer review’ has no standard definition between publications.  In most cases, reviewers are anonymous; readers are not privy to their qualifications or experiences. Journals promising peer review rely on the reader’s interpretation and preconceived notions as to how that practice reflects on article quality.  Without any standardization, peer review becomes less of a practice to ensure quality and more of a marketing tactic for publishers.  In 2009, bloggers at the Scholarly Kitchen were able to get an extremely ridiculous article published in an open, “peer reviewed” electronic journal simply by paying for an Open Access fee.  Obviously, peer review does not equal quality.

Even the most trusted journals publish articles with extreme errors, and I’m not talking about grammatical mistakes.  Bradley used the Journal of the American Chemical Society as an example of a highly respected periodical that has let some very questionable articles slip through peer review unquestioned.  For a prime example, check out the NaH debacle here. (Link to the now withdrawn paper.)

Unfortunately, many librarians and instructors are supporting the illusion of peer review.  Professors require students to use only peer reviewed sources in their papers; librarians explain how peer reviewed sources are the only true ‘scholarly’ sources.  Students need to be able to critically evaluate research articles themselves.  Providing a list of criteria for scholarly articles (peer reviewed, many references, etc.) may be doing students a disservice.  Sure, seeing that an article cites no references is a good reason to be skeptical of its contents, but students need to understand why references are important.

Not only that, but science librarians need to address more research specific issues in classes.  The value of raw data cannot be overlooked, as it often is in many published articles.  When the raw data for any experiment is easily available to readers, the science behind the experiment becomes transparent, allowing readers to draw their own conclusions.  Bradley stressed this point during his presentation and discussed how open notebook science allows readers to quickly find raw data for all of his research.  I think many science librarians overlook this important issue; we tell students that the methods and materials section of a scholarly article needs to be clear and thorough.  One should be able to complete the experiments by using the methods section as a recipe.  However, if the author’s raw data is not available, it is impossible to know if there were any mistakes made in the results.

I think the biggest lesson I learned from attending OpenSciNY was that even though I was a working chemist 3 years ago, I was already out of the loop when it came to the current scientific research community.  If librarians want to be subject specialists, they must attend events and conferences outside the library realm.  I know how to search the literature and evaluate research, but before this conference, I had only heard of open science in passing.  I’ll be making it a point to skip some library conferences in favor of engaging with the scientific community this year.  Finding out the needs of working professionals can only help me better serve the next generation of scientists.

For more information on Dr. Bradley’s research check out his blog, Useful Chemistry, and the wiki/open lab notebook for his classes and research.