Proposal for a Campbell Collaboration Training Group within the Campbell Methods Group 

Betsy Jane Becker                                 Therese Pigott      

Michigan State University                 Loyola University of Chicago 

This proposal for a Campbell Collaboration (C2) Training Group is organized as follows. We begin with a rationale for the group, based on the five general functions of Campbell Methods groups.  Next we outline six areas where a C2 Training Group (C2TG) would be anticipated to make contributions, and for each area we list potential activities. We then sketch out a possible framework for ongoing evaluation of the functions of the group.  We provide contact information and biographical sketches for the two conveners and we conclude with a list of supporters of the proposal. 

Rationale 

The Campbell Collaboration is organized to support the conduct and maintenance of systematic reviews of diverse problems in the social sciences.  Materials from the C2 Methods Group state 

“Methods groups within the Campbell Collaboration have five general functions:

·                      providing advice on methodology and methods policy

·                      providing training and support

·                      conducting empirical research

·                      helping to monitor the quality of systematic reviews prepared within the Collaboration

·                      serving as a forum for discussion.” 

The C2 Training Group will serve as the formal body in the C2 Methods Group for addressing the goals of providing training and support for review groups within C2.  As C2 review groups in different substantive areas (i.e., education, criminal justice, etc.) begin to conduct and possibly solicit C2 reviews, the substantive experts conducting those research reviews will need instruction on methodology for systematic reviews, from problem formulation through the reporting of results.  The C2 Training Group will serve as a clearinghouse for review groups who need training in any aspect of systematic reviews.  Review groups would request training in a specific area or set of areas, and the C2 Training Group would enable or assist those reviewers in locating a trainer (or group of trainers) to respond to the need.   

The membership of the Training Group will draw from the international pool of scholars associated with C2 who have extensive experience in conducting systematic reviews, and those who actively conduct research on methods for meta-analysis.  The C2 Training Group is anticipated to provide both formal training courses and shorter term instruction on other issues involved in systematic reviews.  The C2 Training Group will also instruct its own members on new techniques for systematic reviews so that a core group of trainers is available across the countries represented in C2 who could provide instruction on a variety of review topics.  As part of the training process, the C2 Training Group will also compile a set of training materials that could be used for instructional purposes.

In sum, the C2 Training Group would establish and provide 

1.  a clearinghouse for training opportunities,

2.  specialized methods training,

3.  training for trainers and a forum for interaction among trainers,

4. training workshops resented at C2 meetings and other venues,

5.  a collection of training materials, and

6.  follow-up and evaluation of training opportunities. 

Proposed Activities of the C2 Training Group

1. Clearinghouse for training opportunities.  

C2 reviewers will likely encounter many unanticipated issues as they conduct reviews.  For some reviewers, work on a C2 review may be their first experience with a systematic review.  Other reviewers, though experienced in systematic reviews, may face methodological or statistical issues that are beyond the scope of their experience.  The C2 Training Group will serve C2 reviewers who need further training in some aspect of systematic reviews.   

We anticipate that reviewers will request assistance on a specific problem encountered during a review, and that reviewers may not have contacts with other researchers who have experience with the same problem.  The C2 Training Group could serve as a liaison with review groups, organizing and coordinating trainers or groups of trainers to respond to a specific need.  The C2 Training Group will maintain a list of trainers who have expertise in particular areas of systematic reviews.  At the end of this proposal an appendix lists individuals who have indicated their support of this proposal or a potential interest in providing training, along with their areas of expertise to date. This list will be expanded as additional individuals express interest. 

We have several specific ideas about how a training clearinghouse would operate, particularly given that at this time there is no funding specifically aimed at the support of this effort.  One idea is to create a website where trainers could post contact information, professional references and evidence of their training experience (see item 6 below), information on scheduled courses and workshops, and, possibly, “self-help” training materials.  Persons available for extended consultation (presumably for some form of compensation) could also indicate that in the web site.  The site would be incorporated into the existing C2 Methods web site. 

Second, the C2 Training Group will sponsor a listserv, to be advertised on relevant C2 web sites, for persons interested in C2 Training activities. We anticipate that both trainers and potential trainees would enroll in this listserv. Trainers would post to the listerv the schedules of workshops offered, courses available at C2 meetings and elsewhere, and the like.  In addition it is possible that some help may become available via postings to the listerv itself, but there is currently no compensation available to guarantee that a competent trainer would respond to every question posted.  This listserv could be created at Michigan State University, which already sponsors a listserv for persons interested in technical (statistical) meta-analysis issues. 

 2. Specialized methods training.

 As stated previously, we anticipate that C2 reviewers will encounter challenging issues at every stage of the review process.  The C2 Training Group could provide specialized methods training given the broad expertise represented by its members.  Some examples of topics for specialized training appear below.

              A.     Problem formulation

a.       Identification of important variables and coding issues

b.      Judging conceptual relevance of studies

c.       Planning and conducting power analysis for the synthesis

              B.     Literature search/data collection

a.       Identifying and using reference databases

b.      Organizing information about searching and search products

c.       Using informal and secondary channels for data collection

              C.     Data evaluation

a.       Minimizing and dealing with publication bias

b.      Judging quality of studies

c.       Evaluating coding decisions

d.      Computing effect indices

              D.     Data analysis

a.       Analyzing results of qualitative studies

b.      Handling missing data

c.       Combining estimates of effects

d.      Fixed versus random effects models

e.       Analyzing dependent effects

f.        Analyzing explanatory models of effects

g.       Conducting sensitivity analyses

h.       Using Bayesian methods

E.      Interpretation and presentation

a.       Creating graphical displays of results

b.      Theory testing in systematic syntheses

We will request that members of the Training Group identify their competencies in these and other areas of expertise required for the conduct of systematic reviews and meta-analyses.  By providing a list of available trainers with their areas of expertise the Training Group could facilitate the process of C2 reviewers (or review groups) making contact with trainers or consultants with relevant expertise.  Clearly one important area of concern to the Methods Group is that of data analysis, and we anticipate that many Training Group members will have expertise in this area.

3. Training for trainers and a forum for training discussions.

The C2 Training Group would serve as a research community where expertise and training experience could be shared.  Given the international character of C2, efficient use of resources would require trainers to have a broad base of expertise to meet the needs of the wide ranging review groups. 

In order to ensure quality and consistency of training across trainers, the C2 Training Group will develop a mechanism for the approval of trainers and training materials. This will be done with an eye toward eventual issuance of C2 certifications. The Training Group will collaborate wit the C2 Methods Group Coordinating Committee in the development, approval, and execution of these mechanisms.

Further, the C2 Training Group anticipates the need for meetings of the trainers, to share expertise and provide training to each other on aspects of research reviews. Online or electronic contact would also be desirable (e.g., a chat forum or other archive for trainers).  The sharing of knowledge across the C2 Training Group would ensure that the methods used in C2 reviews represent the most current techniques for systematic reviews.  Eventually C2TG may be able to offer training workshops for potential trainers, but at this time we anticipate this activity will first occur via participation in the training listserv, other electronic media, or via more informal routes.

4. Workshops.

It is anticipated that C2 Training would be of interest to attendees of the various C2 conferences. Thus the C2 Training Group would likely be called upon to organize and present training sessions at meetings of the C2 review groups.  Having an international collection of capable trainers as participants will enable the C2 Training Group to contact and recruit trainers who could provide such workshops at C2 conferences.  Many of the supporters of this proposal have offered workshops (of varying lengths) and term-length courses at universities around the world.

It seems likely that C2 reviewers will want a range of training experiences. Those experiences may vary in both scope and timing (i.e., length of the training and the stage of the review process at which the training is desired).  Some reviewers may want to interact with a trainer at the very earliest stages of their work – before even submitting a protocol for C2 approval. Others may feel confident about their capacities to formulate a problem and collect studies but may desire assistance in data evaluation (coding) or data analysis.  Also it seems probable that some reviewers will want shorter more intensive experiences (e.g., 1 or 2 day workshops) while others may want a more involved or longer-term opportunity (a week-long or semester-long class).  We anticipate that with a collection of members in C2TG the group will be able to provide this broad range of training opportunities. As part of these opportunities, we will also explore the possibility of producing videotaped sessions (with revenues going the Campbell Collaboration and royalties to participants as a potential model)  

5. Collection of materials.

Many of the individuals who have agreed to support and participate in the C2 Training Group have taught courses on research synthesis in university settings as well as shorter courses and  workshops in a variety of contexts.  We propose to gather syllabi and course outlines for existing courses from members of the group who are willing to contribute such materials. Because many potential members of the Training Group already have their own materials, we do not believe that it is necessary to require that they use other (e.g., common or “approved”) materials. 

Once a sizeable collection of training materials is gathered, some documents may eventually be made available online in the form of self-help materials.  Until such time as a collection of materials is assembled we will provide links to materials that are available on the websites of individuals who are in the training list (depending of course on the willingness of the Training Group members to allow such links).

The gathering of training resources would also support the C2 Training Group’s goal of developing a community of trainers who have access to the most up-to-date research on review methods.  Again at this point the effort would depend on the collaborative collegial efforts of members, as at this time little funding is targeted for this particular effort.

6. Follow-up with recipients of training and evaluation of all training opportunities.

Clearly if the C2 Training Group is to guarantee the quality of the training provided, evaluation of the trainers, their materials and associated C2-sponsored training opportunities will be critical.  To assure that those who volunteer to serve as trainers have some level of qualification for the job we will ask that potential members submit a list of a few training experiences (e.g, workshops or classes) they have offered, and that they provide names of people who could serve as references (e.g., principal investigators of projects for which workshops have been held, former students, etc.)  This information will be made available in the resources attached to the list of available trainers. 

As C2 sponsored training sessions begin to occur, we anticipate that evaluations of those sessions would be carried out. At first this might consist of simple evaluation forms or email correspondence with those who join the C2TG listserv.  The evaluation could be more-or-less voluntary, and probably less structured, until policies and procedures for evaluation are established.  As activities proceed, we anticipate more formal evaluation will be possible, and that materials for that effort would be developed.  Later we foresee that trainers who do not provide evaluative evidence (or who are evaluated as poor or ineffectual trainers) could be removed from the list of endorsed trainers, though we hope this would be a rare occurrence.

Limitations of this Proposal

A variety of issues will need to be resolved once the C2 Training Group is established.  For example, one thorny issue concerns compensation for training and consultation provided by C2 trainers. While we do not have a specific proposal for a policy on compensation at this time, we do not believe that C2 should be involved in individual negotiations concerning compensation to trainers or consultants, though we do believe compensation is appropriate, particularly for extended workshops and courses or long-term consultation.  Our current plan is to post prominently on the C2TG web site a statement that compensation for workshops, classes and extensive consultation is appropriate and should be negotiated between the trainer and potential client before any extensive work is undertaken.

Also while we hope to maintain some kind of oversight and quality control over the training opportunities made available through C2, it is not clear how forward the Training Group should be about this matter, and whether or how C2TG should become involved in disputes about the quality of training, should they arise.  Again policies concerning this issue would need to be developed once the group is established.

There is one final caveat we wish to add to our proposal, to be clear from the outset about this limitation.  Because there is currently no funding available to either of the conveners of the C2TG, we have designed the initial activities of the group to be largely low maintenance and self-sustaining.   It is possible that in the future funding may become available.  If funding were available the proposed activities of the C2TG could easily be elaborated, but rather than propose many activities that cannot currently be guaranteed, we have started small with the idea that later expansion would be possible.

Conclusion 

It is our hope that the formation of a group for training within the C2 Methods Group will enable more individuals to become proficient in the methods for systematic review and the analysis of data gathered in the review process.  Through the six activities outlined here and others that may develop in the future, this group will enable more researchers to make the best use of existing evidence, as well as lead to more informed decisions about what new or additional evidence is needed to inform critical policy decisions across the social sciences.  Also this group will facilitate communication among scholars interested in providing the best educational opportunities for persons interested in learning about systematic reviews and meta-analysis.  We hope you will approve the formation of the proposed Campbell Collaboration Training Group within the C2 Methods Group. 

Contact Information for Conveners

Betsy Jane Becker, Professor
Synthesis Research Group (SynRG)
College of Education
462 Erickson Hall
Michigan State University
East Lansing, MI 48824-1034
Email: bjbecker@msu.edu
(517) 355-9567  (Phone)
(517) 353-6393  (Fax) 

Therese D. Pigott, Assistant Professor
Research Methodology
School of Education
Loyola University Chicago
1041 Ridge Road
Wilmette, IL  60091
Email: tpigott@luc.edu
(847) 853-3301 (Phone)
(847) 853-3375 (Fax)

Biographical Sketches of Conveners 

Betsy Jane Becker is a professor of Measurement and Quantitative Methods in the College of Education at Michigan State University (MSU), where she has taught since 1983.  Becker is an applied statistician who has developed and studied methods for meta-analysis and conducted her own research syntheses.  Her early work on combinations of significance values won the Outstanding Dissertation Award of the American Educational Research Association in 1985.   

Becker’s work in research synthesis has focused on probability value summaries, indices of change, and the synthesis of correlations. Her current work on research synthesis concerns methods for combining and analyzing correlation matrices from series of studies, and using the summary results to build synergistic models across studies.  Becker has conducted her own research syntheses (published in such outlets as Review of Educational Research and Journal of Research in Science Teaching) and she has collaborated with researchers from a wide range of disciplines on quantitative reviews for such journals as Journal of Family Psychology, Journal of Sport and Exercise Psychology, and Contraception. 

Becker teaches one of MSU’s seminar-long classes on meta-analysis, and has also taught (or co-taught) workshops on meta-analysis nationally and internationally since 1983.  Also she teaches a regular summer course on meta-analysis for the University of Michigan’s Inter-university Consortium for Political and Social Research.  Five of Becker’s 11 doctoral graduates have completed dissertations on methods for research synthesis, and she has served on over 50 doctoral and master’s committees, several of which concerned meta-analysis. Finally Becker is the organizer and main faculty sponsor of the Synthesis Research Group (SynRG) a cross-disciplinary research group of faculty and graduate students at Michigan State University. 

Therese D. Pigott is assistant professor of Research Methodology in the School of Education at Loyola University Chicago, where she has taught since 1998.  She earned her BA in psychology from the University of Notre Dame, and her PhD in Education from the University of Chicago.  Prior to joining the faculty at Loyola University Chicago, she served as associate program officer at the Spencer Foundation and as assistant professor in the Department of Foundations and Research at National-Louis University in Evanston, Illinois. 

Her research interests include statistical methods for meta-analysis and statistical analyses with missing data.  With Larry V. Hedges, she has recently published an article on power analysis in meta-analysis in Psychological Methods.  She is currently on the editorial board of the Journal of Educational Psychology, Psychological Methods, and Psychological Bulletin.  In addition to research in applied statistical methods, she is currently using the Early Childhood Longitudinal Survey to investigate issues of kindergarten transition, funded by a grant from the American Educational Research Association.  She is also involved as an evaluator in the federally-funded Illinois Professional Learners Partnership, a multi-year, multi-institutional collaboration to educate quality teachers for high-need schools.

Pigott teaches a yearly seminar-long course on meta-analysis and one on hierarchical linear models at Loyola University Chicago.  She collaborates regularly with colleagues and students on applications of research synthesis, missing data analysis and hierarchical linear models.