Skip to main content
Ask A Librarian

WVUL eResource Collections Evaluation Toolkit

Summary

The task of gathering and compiling the data for the  vendor packages was assigned to KARM (Technical Services) at WVUL.   The datasets include COUNTER JR1 (number of successful full-text article requests by month and journal (usage)) and JR5 (number of successful full-text article requests by year-of-publication (YOP) and journal) usage reports, ILL statistics, and overlap analysis. The process for gathering and compiling is described below.

Gathering & Compiling Usage Statistics

  1. Create spreadsheet for the usage statistics.  The spreadsheet should include the following columns:
  1. Title
  2. ISSN, if available
  3. Renewal Cost
  4. Usage 20xx, add a column for each  year to be analyzed
  5. Usage Average
  6. Cost per Average Use
  7. Create a separate column for each YOP to be reviewed
  8. YOP Average
  9. CPAU Rank
  10. Usage Rank
  11. YOP Rank
  12. Rank Average
  13. Rolling Sum
  14. Overlap
  15. Coverage
  16. Overlap Packages
  17. ILL Lending 20xx, for each year to be analyzed

    Follow link for example spreadsheet, https://drive.google.com/open?id=0B5sPtZ_YKSWGSlNtS25YeWZYV2c

  1. Title data, ISSN, and renewal cost will provide the basis for the spreadsheet and will typically be vendor provided as part of the renewal process.
  2. Harvest usage statistics by logging into the administrative site for the vendor you wish to analyze.  Follow instructions on the vendor’s to download the COUNTER JR1 and JR5 reports for each year that will be included in your analysis.
  3. Load usage statistics into appropriate columns using an application such as Microsoft Excel’s VLOOKUP. A tutorial can be found here. VLOOKUP will attempt to match the data by title where exact matches are found and import the resulting data to populate each column. This is a great time savings as you aren’t required to copy and paste data into each cell. However, data returning N/A  must be matched by hand.
  4. Calculate the usage average using Excel’s AVERAGE tool in the column, Usage Average.
  5. Populate the Cost per Average Use (CPAU) column by dividing the Renewal Price with the Average Usage.
  6. Import the YOP data to the spreadsheet using the methods related in step 4, populating columns corresponding to each YOP report.Note: For  the example, 2016 usage was imported from the YOP 2016 report and 2015 usage from YOP 2015.   
  7. Calculate the average using the instructions in step 5 and populate column, YOP Average.
  8. Click the Select All button in the top left hand corner of the spreadsheet or click in a populated cell and click CTRL + A. While selected, add a filter to every column by clicking on Sort & Filter>Filter. This step will allow you to sort your data without skewing the data in the rows.
  9. Unselect the entire spreadsheet, by clicking in an empty cell.
  10. Calculate CPAU Rank by clicking the filter drop down arrow at the top of the CPAU column and sort Smallest to Largest. In the CPAU Rank column enter 1 in the top cell and 2 in cell below to create a pattern. Select those two cells and drag the fill handle down the column. Note: If there are cells with zero values, begin ranking at the first title that has a positive CPAU.
  11. Calculate Usage Rank by clicking the filter drop down arrow at the top of the Usage column and sort Largest to Smallest. In the Usage Rank column enter 1 in the top cell and 2 in cell below to create a pattern. Select those two cells and drag the fill handle down the column. If there is zero usage, manually enter the largest rank for all of those titles (tied).
  12. Calculate YOP Rank by clicking the filter drop down arrow at the top of the YOP column and sort Largest to Smallest. In the Usage Rank column enter 1 in the top cell and 2 in cell below to create a pattern. Select those two cells and drag the fill handle down the column. If there is zero usage, manually enter the largest rank for all of those titles (tied).  
  13. Populate the Rank Average column by calculating the average of CPAU, Usage, and YOP ranks using the instructions in step 5.
  14. Populate the Rolling Sum column by entering the following formula in the top cell, =SUM(B$2:B2). Adjust the formula if needed to where B reflects the Renewal Cost column and drag down. This formula will calculate the rolling sums of the titles as you sort and is useful if you are working toward a specific budget number.

 

Measuring Overlap Between the eResources Package and Full Text Journals

  1. Conduct overlap analysis using either a vendor provided tool or by hand. To conduct by hand, search for the title in your holdings, either print or electronic. Identify any overlaps that occur with your holdings using this to populate the columns, Overlap, Coverage Dates, and Overlap Packages.  
  2. In the Overlap column, record Full, Partial, or None overlap, where full overlap indicates exact coverage matches, partial indicates some overlap, and none indicates no overlap.
  3. In the Coverage Dates column, record coverage dates for the titles being analyzed.
  4. In the Overlap Packages column, record each overlap package and the respective coverage.

Gathering & Compiling ILL Statistics

  1. Request or gather ILL lending statistics for at least three years, if available, for the vendor or package being analyzed to identify the percentage of usage which can be attributed to the filling of off campus ILL lending requests.
  2. Import the ILL lending data using the methods related in step 4, populating columns, eg. ILL Lending 2016, ILL Lending 2015, and ILL Lending 2014. 

Trends & Anomalies

Evaluating usage trends can help to identify anomalies in the data collected  that skew the usage results. During our analysis of  JR1 data, a bar chart of the usage was created, see below. A tutorial for creating a bar chart can be found here. Adding a trendline to this chart was illuminating as it displayed a marked increased in Vendor #3 usage beginning in 2015. Institutionally, there was no functional change that would account for that marked increase. It is surmised that compromised user accounts were responsible for skewing the data and is a reminder to continually evaluate all data, including vendor provided data.



Data Analysis Methods

The data gathered can be reviewed in a variety of ways. Each institution will need to consider which method or other combinations of criteria are best suited for its needs. Remember that effective analysis should be replicable, affordable (both cost and time), and ideally simple. Here are 3 methods and criteria that could be used:

A. Usage and Cost Driven

This is the method WVU used. After compiling the data as described above, apply the usage-driven selection criteria below.

  1. Eliminate any titles with complete overlap.
  2. Sort by Usage Average remove those < 100 and identify > 500 uses to keep (if any).
  3. Remove those titles where the listed cost of the journal is higher than the average from table 9 of the  New World, Same Model | Periodicals Price Survey 2017 ¹.  
    1. This was shifted up to accommodate science, health science, and engineering titles.
  4. If necessary, consider average YOP.
    1. YOP data is especially useful if there is significant partial overlap as it indicates which year articles are being requested from. For example, if there is partial overlap with a title in an aggregator package, and the YOP data shows that most requests fall within the aggregator coverage, you might choose to cancel the title.
  5. Remaining titles were selected using a rolling sum, as calculated in step 15 above.

B. Weighted Ranking (weighted arithmetic mean):

Analysis method where some factors contribute more than others to the average rank.

  1. Weighted ranking would take criteria of your choice such as discipline, number of students/faculty in discipline, or cost per use and inflate the value of that factor(s) to alter the outcome of the analysis.

C.  Journal Ranking Metrics:  

Criteria widely used to evaluate an academic journal's impact and quality.

  1. Impact Factor
  2. Eigenfactor
  3. SCImago Journal Rank
  4. H-index
  5. Expert survey
  6. Publication Power Approach (PPA)
  7. Altmetrics
  8. diamScore
  9. Source normalized impact per paper (SNIP)
  10. PageRank
  11. JRank

D.   Institutional Publication and Citation Practices

Criteria used to correlate faculty publications and citations with institutional usage.

  1. # of publications by university faculty/researchers in each journal
  2. # of citations by university faculty/researchers in each journal