Policymakers, voters, and researchers need empirical evidence to effectively understand and assess policies and programs. To support evidence-based policymaking, many federal agencies are establishing or strengthening independent evaluation offices. But because evidence-building activities extend well beyond the sponsored evaluation research of any particular agency, clearinghouses that collect and assess existing studies are also critical partners in evidence-based policymaking.
In 2018, the Office of Planning, Research and Evaluation, within the Administration for Children and Families of the US Department of Health and Human Services (HHS) introduced the Pathways to Work Evidence Clearinghouse, which collects and curates studies on helping job seekers with low incomes succeed in the labor market. (Full disclosure: studies I have coauthored are included in Pathways to Work.)
Pathways to Work builds on similar efforts to assess education research at the US Department of Education’s What Works Clearinghouse (WWC) and employment and training research at the US Department of Labor’s Clearinghouse for Labor Evaluation and Research (CLEAR). Pathways to Work covers many interventions that are also included in WWC or CLEAR, particularly research on education and training programs, employment retention services, and work-based learning. However, it also includes studies on the effectiveness of case management interventions, health services, and supportive services and the effects of these interventions on employment and mobility.
Surfacing evidence on combinations of services
Users of the Pathways to Work clearinghouse can search for evidence either by intervention or by study. An intervention is defined as a relatively specific “bundle of services or policies implemented in a given context,” such as Job Corps or Project QUEST (Quality Employment Through Skills Training), that has been evaluated in at least one rigorous research study. In some cases, a policymaker might be interested in replicating or scaling up a specific intervention, so reviewing studies at the intervention level is of real value. But often, policymakers are interested more generally in investing in a type of service or combinations of services to support people with low incomes. Because interventions often combine a number of different services, it can be difficult for policymakers and analysts to summarize the research literature on a particular service or set of services that is most relevant to their needs.
The Pathways to Work clearinghouse solves this problem by allowing users to filter their searches on the combination of services that interests them. For example, if I were interested in the evidence on apprenticeship or on-the-job training programs that also provided case management for participants, I can easily select those filters and identify the 15 interventions in the clearinghouse that offered both. This capacity to filter the clearinghouse by component services is important for navigating the often complicated world of programs to support people with low incomes.
Rating system needs more explanation
One limitation of Pathways to Work not shared by WWE or CLEAR is that it provides no results or exposition of studies that it rates as having a “low” evidence rating. Pathways to Work also provides no explanation for why a given study received the rating it did, regardless of whether the study is rated “high” or “low.” In some cases, the justification for a study rating in the clearinghouse can be inferred. A study using a randomized control trial that earns a high evidence rating seems to be rated that way because it uses a strong method and implements that method well. However, all quasi-experimental studies and some randomized control trials are rated low by the clearinghouse.
Reviews of low-rated studies include no explanation for why the methods of those studies were deemed unreliable and no details on the results of the study. Some explanation of the rating methodology is given in a protocols landing page and a more detailed protocols document (PDF), but neither explain why a specific study was rated in the way it was.
Providing policymakers and their staff with a clearer understanding of why studies are given a specific rating would do more to help them craft policies based in evidence.
The relative merits of different quasi-experimental methods are hotly debated by social scientists and policy evaluators. Some evaluators believe that reliable evidence can only be derived from experimental studies, and others believe there is much to learn from quasi-experimental research and even point out that experimental studies introduce risks of their own. The DOL’s CLEAR balances these perspectives and gives users tools to understand the evidence by including a detailed description of different considerations for interpreting a study’s findings, even when they give it a low rating. Like Pathways to Work, WWC excludes some studies that do not meet its review standards. However, unlike Pathways to Work, WWC does accept strong quasi-experimental studies as meeting its review standards and provides a detailed review of those studies.
The Pathways to Work clearinghouse is an important tool in evidence-based policymaking in the area of career pathways and mobility for people with low-incomes. But more explanation around their ratings methodology and why studies receive the ratings they do would help this important tool reach its full potential.