En este apartado se resume la información relevante acerca de Snowball Metrics. Tras una revisión del proyecto Snowball, hemos considerado que es una aproximación metodológica y práctica muy completa, que se ha puesto a prueba y que ha contado con respaldo de empresas como Elsevier y organizaciones como Eurocris. Snowball no obliga a disponer de métricas con un cálculo y un origen de datos concreto, sino que propone una definición general de las métricas, agrupadas en categorías, para que se decida en cada caso en qué consiste la métrica en un ámbito determinado y con unos acuerdos concretos.
En conclusión, Snowball Metrics proporciona el marco para conseguir los objetivos del módulo de indicadores-
Los documentos base de los que se ha extraído esta información se encuentran disponibles a continuación:
20140616_Snowball_Metrics_for_CERIF_XML_version_1.1.zip
Snowball Metrics.
Snowball Metrics should be seen as a balanced scorecard of metrics from which a selection can be made to help understand institutional strengths and weaknesses in a particular area
The Snowball Metrics Exchange service
Snowball Metrics can be used within a single institution to give useful information about trends over time, but their real value and motivation is for benchmarking and that requires institutions to be able to see each other’s Snowball Metrics.
… maximum benefit depends on users being able to understand their position relative to their peers on a wider set of metrics, including those that rely on institutional data such as Applications Volume.
This need has given rise to the Snowball Metrics Exchange service, a free “broker service”, which has been built by Elsevier such that:
- Any institution who is using Snowball Metrics can become a member of the Snowball Metrics Exchange service.
- The institutional members will be responsible for generating their Snowball Metrics according to the recipes. The metrics could be calculated using a bespoke system, in a spreadsheet, or in a commercial tool.
- Each institution can be a member of one or more benchmarking clubs: groups of institutions which have agreed to exchange metrics with each other.
- Institutions may choose to accept or decline requests to share all or some Snowball Metrics with benchmarking clubs or individual institutions; this is entirely under their control.
- Institutions will use the “I’ll show you mine if you show me yours” facility in order to exchange equivalent Snowball Metrics with each other.
- Only Snowball Metrics values will be exchanged. The data underlying the metrics will never be exchanged, and will remain behind the institutions firewalls.
The Snowball Metrics Exchange service is made of three components
- The Snowball Metrics Uploader (“Uploader”) that includes an API and allows an institution to encrypt and upload Snowball Metrics from their own institution for future exchange.
- The Snowball Metrics Exchange (“SMX”), which acts as the ‘broker’ to relay the encrypted metrics from an institution’s Uploader for download by entitled institutions. Within the SMX, an institution can manage the metric entitlements for each institution that has agreed to share metrics with them.
- The Snowball Metrics Downloader (“Downloader”) that allows an institution to download encrypted metrics shared by entitled, participating institutions.
Snowball Metrics Landscape
The outcomes are what the business wants or needs to achieve. The outputs are the actions or items that contribute to achieving an outcome
Snowball Metrics recipes
The hierarchy of the original set of 10 Snowball Metrics
The recommended use of Snowball Metrics
Triangulate the decision.
Snowball Metrics offers a balanced scorecard based on a broad set of metrics:
- It is strongly advisable to “triangulate” within the quantitative input into a decision. Every metric has weaknesses, but these can be compensated for by the strengths of (an)other metric(s). It is the responsibility of the user of Snowball Metrics to ensure that a metric’s weaknesses are compensated for by another metric or input.
- There is a broad diversity of questions that metrics could be used to help address. Existing scorecards often tend to be based upon output and citation metrics, largely since comprehensive commercial databases are readily available, and / or financial metrics, since they are relatively easy for universities to measure. Snowball Metrics draw a much more comprehensive and rounded view of institutional performance across the full range of research activities.
- They are unlikely to distort the research process in unanticipated ways through encouraging too much focus on a particular activity. For example, it is well known that rewarding researchers solely for publishing a high volume of output encourages researchers to slice their work in more, smaller pieces to be able to publish a higher volume24, which is probably not a true reflection of the desired outcome. Snowball Metrics offers the option to select several metrics to encourage a balanced outcome for universities.
- Some metrics may be more or less relevant to different disciplines. Many metrics, such as Applications Volume and Awards Volume, are equally useful across all fields, when the disciplinary denominator is used and an appropriate selection of peers for benchmarking is made. Other metrics, such as Citation Count and Collaboration may be more valuable in STEM25 areas and less so in the social sciences and arts and humanities.
- Some practitioners of metrics prefer simple, straightforward metrics, often based on total counts, such as Citation Count and Altmetrics. Others prefer more complex metrics, that often inherently correct for variables such as those between disciplines, for instance Field-Weighted Citation Impact.
Overview and denominators
Denominator definitions:
- Institution
- Discipline
- Researcher
- FTE (full-time equivalent) count
- Research student
- Funder-type
- Funding-type
- Time period
- Full-time or part-time research students
- Home or overseas research students
- Gender
Display of Snowball Metrics
A Snowball Metric is one which has been defined and agreed by research focused universities as being useful in supporting strategic planning by enabling benchmarking between institutions. These metrics have tested methodologies to ensure that they can be generated with a reasonable amount of effort that is not manually intensive.
Primary data sources
SGI, Scopus, WoS. Consistency between institutions
Granularity of denominators
Some metrics, when calculated for the more granular denominators, and especially for smaller institutions, will be based on few data points. These small data sets cause fluctuation in the metric between time periods, and may make the metrics less reliable for benchmarking.
Counting
Whole counting is used to generate Snowball Metrics. The method of counting isimportant when a data element has more than one denominator associated with it.
Citation counts
Some Snowball Metrics depend on counts of citations. These citation counts are typically the total number of citations received since publication up to the date of the current data extract.
Outputs included in the calculation of a Snowball Metric
Every output in a data set would ideally be associated with the information needed for it to be included in the calculation of every Snowball Metric. In practice this is probably not the case; outputs in institutional repositories do not always have associated counts of citations or affiliation information, and outputs are not always part of serials that have journal metrics values, for example.
Currency conversion
The international benchmarking of financial recipes such as Applications Volume, Awards Volume and Income Volume depend on an exchange rate mechanism.
Denominators
Institution denominator
An institution is defined as the sum of all units of an organization included in or with an organization’s consolidated annual financial statements, and headed by one president, chancellor, or equivalent.
The physical location of a campus does not define whether something is or is not part of a university; overseas campuses are included, because they are recorded within the data elements of the institutional systems
Discipline denominator
The discipline denominator enables benchmarking between institutions at a more granular level than that of the entire institution. A meaningful discipline-level denominator has the following characteristics:
- It is a structure that has the same meaning at all institutions.
- It draws on data that are readily available to an institution.
- It uses information that is reasonably current.
Researcher denominator
A researcher is any faculty or staff member who could act as the principal investigator of a funding application and who spends >0% time on research.
This definition includes all those working in research-focused universities who have time allocated to research of any kind.
This definition excludes trainees including undergraduate and graduate students, post-doctoral researchers, and staff or faculty with limited-term or temporary appointments such as visiting scholars. Will it be like this in Hercules?
FTE (full-time equivalent) count denominator
FTE count indicates the extent of a researcher’s workload that is focused on research.
FTE count is used to provide the option to normalize for different sizes of institutions, and disciplines within those institutions
Research student denominator
A research student is any student studying for either a doctoral award or a masters degree by research, having achieved a first degree as a condition of entry. A research-based higher degree is a postgraduate programme comprising a research component (including a requirement to produce original work) that is larger than any accompanying taught component when measured by student effort.
This definition excludes trainees including graduate students undertaking taught courses, post-doctoral researchers, and staff or faculty with limited-term or temporary appointments such as visiting scholars.
Funder-type denominator
This denominator is applied to:
- Applications Volume: to the count, price, or amount applied for.
- Awards Volume: to the count and value of awards.
- Income Volume: to the income spent.
- Market Share: to research income.
Funding-type denominator
Funding-type is a denominator for the Research Student Funding recipe
Time period denominator
Calendar year, Financial year, Quarter
Full-time or part-time research students denominator
This is the method by which a student is being taught their course
Home or overseas research students denominator
This provides the option to look at the student population at a more granular level, separating home and overseas students.
Gender denominator
The gender denominator is used to monitor equal opportunities issues.