The WorkAdvance Demonstration Testing a Sector-Based Training Program James A. Riccio 3rd MSF – CSC Social Sector Conference Singapore 13 September 2017
[email protected]
Promoting upward economic mobility for disadvantaged adults The issue • Stagnant wages for non-college educated adults since the 1970s, despite economic growth • 4-year college degree not the answer for all, but improving skills (human capital) is essential • Yet “training” often viewed as ineffective
One approach: WorkAdvance 2
WorkAdvance design and evaluation build on a history of rigorous studies • Mandatory “welfare-to-work” programs
• Case management and job placement (“work first”) • “Workfare” / community service jobs (temporary)
• Wage supplements • Post-placement coaching for retention/advancement
• Human capital components (typically via referral to other agencies): − Basic education (reading and math) − Occupational skills training 3
General lessons from careful evaluations The “good news” • Moderately successful in speeding up job entry and increasing earnings / reducing benefit receipt • Can be cost-beneficial for taxpayers and participants
But generally… • Little poverty reduction (unless wages supplemented) • Positive effects sometimes limited to certain subgroups • Little improvement in upward mobility 4
Sparse evidence proving effectiveness of “training” for disadvantaged adults Challenges • Many participants don’t complete training or receive a credential • Credential may not be valued by employers • Skills may not be in (or remain in) high demand
• Often no bridge between training completion and job in relevant field
Can sector-based strategies work better? 5
Distinguishing features of “sectoral” strategies • Training and placement into jobs in high-demand sectors with strong career pathways
• Dual-customer emphasis: Participants and employers • Substantial employer engagement at all stages of programming: − − −
Consult on curriculum design Conduct mock interviews with participants Provide hands-on training
• Provider brings together multiple employers in a field to collaborate on developing a qualified workforce 6
An earlier influential study of sector-based approaches Sectoral Employment Impact Study by Public/Private Ventures • 3-site randomized trial with experienced providers of sector programs • Strong positive effects on employment, wages, income, and other outcomes • Small samples, short-term results (2 years of follow-up), survey data only 7
The WorkAdvance Demonstration Scope
• Test of 4 sector-based programs operated by different organizations in 3 states • Longer-term and more comprehensive evaluation
Funded by a consortium
• Private foundations • Federal government (Social Innovation Fund)
Sponsored by • NYC Center for Economic Opportunity (Under former Mayor Michael Bloomberg) 8
WorkAdvance: Basic Model
Sector-Focused Training
+ Post-Placement Assistance
9
Essential elements of WorkAdvance model Total program duration: About 2 years • Careful screening to determine suitability and readiness for training • Sector-tailored career readiness preparation • Sector-focused occupational skills training • Sector-focused job development and placement • Post-placement support for retention /advancement • Some soft skills coaching throughout 10
Locations and sectors Per Scholas
St. Nicks Alliance
Madison Strategies Group
Towards Employment
New York City
New York City
Tulsa
Cleveland
Information technology
Environmental remediation
Transportation
Health care
Manufacturing
Manufacturing
11
Providers: All nonprofits, but with important differences
Per Scholas
St. Nicks Alliance
Madison Strategies Group
Towards Employment
New York City
New York City
Tulsa
Cleveland
New to Tulsa
New to manufacturing and sectoral training
Most experienced (15 years sectoral training)
Train, then place into job
Traditional workforce development Train, then place into job
Initially:
Initially:
Place first
Place first
Later:
Later:
Train first
Train first 12
Examples of occupational skills training Per Scholas
St. Nicks
Madison Strategies
Towards Employment
NYC
NYC
Tulsa
Cleveland
15 Weeks
5 – 12 weeks
4 – 32 weeks
2 – 17 weeks
At private tech. schools or community colleges
At private tech. schools or community colleges
On site • A Plus • Network Plus
On site or at private tech. schools
• Asbestos removal • Aviation • CDL with hazmat manufacturing endorsement • Comm Driver’s Lic. • Pest control • CNC machining • Diesel mechanic • Welding
• CNC machining • Welding • Phlebotomy • Patient care assist. • Nurse assistant • Medical billing 13
The Evaluation
14
Intake and random assignment June 2011-June 2013
Recruit applicants Extensive Screening: Suitability for Training Applicants agree to participate in study
Random assignment
WorkAdvance group
Control Group 15
Screening and intake process: 80% screened out (before random assignment) 100 90 80
Percentage
70
Example from Per Scholas Beginning of process
(Other sites similar)
60 50 40
• End of screening • Random assignment
30 20 10 0
16
Sample characteristics – All sites combined Sample size Total Program group Male
2,564 1,293 73%
Single, never married Age
67% 34 years
Parent, with custody
32%
Education • Less than HS diploma/GED • Only HS diploma/GED • Some college or more
6% 38% 56%
Currently employed
20%
Prior criminal conviction
24%
1717
Sample characteristics – by site Per Scholas
Madison Towards St. Nicks Strategies Employment
Sample size Total Program group
690 349
479 242
697 353
698 349
Male (%)
81
85
84
41
Some college or more (%)
63
44
58
57
Employed (%)
13
11
27
27
Prior criminal conviction (%)
10
20
40
25 18
Data sources for “impact” study UI WAGE (ADMIN.) RECORDS • Full sample (N=2,564) • Through 3 years after random assignment
YEAR 2 SURVEY • 80% response (N=2,058)
• Average follow-up at month 23
19
Impacts on completion of training in targeted sectors 100 90
Percentage of respondents
80
“Impact”
70 60 50
57.9 51.8
40
+45.9 ***
30
47.9
+31.6 ***
20
20.2 10
12.0
+31.2 *** 8.1
0
Per Scholas WA group
39.3
St. Nicks Alliance
Control group
Madison Strategies Group
+37.0 *** 10.8
Towards Employment 20
Impacts on other participation outcomes Career readiness (%) Per Scholas
89 60
Towards Employment
36
+ 29 *** 90
St. Nicks Madison Strategies
Job search assistance (%)
64
45
+26 *** 90
44
52
+50 *** 87 +35 ***
25
33
82 +46 ***
75 +30 *** 77 +52 *** 72 +39 ***
Post-employment services (%) 25
64 +39 *** 69
24
+45 *** 86
14
+72 *** 69
23
+46 *** 2121
Cost of WorkAdvance
Expenditures on program operations • $5,200 - $6,700 per participant across the four sites • Full benefit-cost analysis still to come
22
Impacts on Economic Outcomes
23
The bottom line Overall positive effect on earnings (all sites combined), but important variation across sites One site (Per Scholas): Large impacts
One site (St. Nicks): Few impacts Two sites (Madison Strategies and Towards Employment): Promising impacts (later cohort) 24
Impacts on Year 3 earnings – full sample % who earned $20,000+ per year
Average earnings + $1,864 ***
18,000
(+ 12%)
16,000 14,000 12,000
$17,481
10,000
$15,616
8,000 6,000 4,000 2,000 0
WA group
100 90 80 70 60 50 40 30 20 10 0
6.4pp*** 40.0%
33.6%
Control group 25
25
Impacts on UI employment rate over 3 years Percentage
St. Nicks
Per Scholas
100 80
60 40
Year 3: -4.6pp
Year 3: +6.8pp**
20 0 **
Percentage
100
***
** * ** *
Madison Strategies
*** *
Towards Employment
80 60
40 20
Year 3: +3.5pp
Year 3: +3.3pp
0 ** RA 2 3 4 5 6 7 8 9 10 11 12 13 WA group
** ** ** RA 2 3 4 5 6 7 8 9 10 11 12 13 Control group 26
Impacts on % employed in targeted sector in all sites (at time of survey) 100
Percent of respondents
90 80
Biggest impact
70 60
64.6 +16.5
61.1
***
50 40 30 20 10
+40.7 *** 20.4
31.8
+11.9 ***
48.2
50.1 +16.4 ***
33.6
19.9
0
Per Scholas WA group
St. Nicks Control group
Madison Strategies
Towards Employment 27
Increasing employment rate in targeted sector is not enough to increase average earnings Whether jobs in the sector pay more also matters Increase in sector employment (vs. control group)
x
Higher wages = in sector (vs. control group)
Impact on earnings
28
IMPACT ON % WITH JOB IN SECTOR
Per Scholas
HOW WELL x SECTOR PAYS
x
OVERALL = IMPACT ON EARNINGS
$$
=
$$$
=
+40.7 pp
x
St. Nicks +11.9 pp
29
Impacts on UI earnings over 3 years Per Scholas
6,000 5,000 4,000 $ 3,000 2,000 1,000 0
+27%
Year 3: - $304
Year 3: +$4,929*** ***
6,000 5,000 4,000 $ 3,000 2,000 1,000 0
St. Nicks
** ** *******************
Madison Strategies
**
Towards Employment
Year 3: +$1,371 * * RA 2 3 4 5 6 7 8 9 10 11 12 13 WA group
Year 3: +$802 * ** * ** RA 2 3 4 5 6 7 8 9 10 11 12 13 Control group 30
Impacts on % with job that offered advancement opportunities 100
“Employer offered many opportunities for advancement” (Current or most recent job – 2yr survey)
Percentage of respondents
90 80 70 60
+13.0 66.1 ***
50
+1.8 61.1
53.1
59.3
+10.7 66.1 *** 55.4
+8.4 55.6 ** 47.2
40 30 20
10 0
Per Scholas WA group
St. Nicks
Control group
Madison Strategies
Towards Employment 31
Impacts on % with job with other desirable characteristics (Current/most recent job – 2yr survey) Per Scholas Satisfied with job
***
Employer-provided health insurance
St. Madison Nicks Strategies
Towards Employment
** **
Hourly wage > $15
***
*
Regular work schedule
**
*
Regular permanent job
*
** 32
Do these factors influence effectiveness: •
The maturity/experience of the providers?
•
The shift to a training-first focus?
•
Characteristics of participants?
33
33
Impacts on Year 3 earnings, by cohort and site 25,000
20,000 15,000
Late Cohort
Early Cohort +7,123 *** -2,772
+1,074 +1,871
+3,603 *** +2,313
-801 -635
10,000
5,000 0
3434
Impacts on % who earned $20,000+ in Year 3, by cohort and site 100 90
Late Cohort
Early Cohort
80 70
60
+17.5 ***
+0.6
50 40 30
-7.1
+0.4
+9.6
+10.4 **
+10.8 **
-2.2
20 10 0
35
Impacts on Yr 3 earnings, by initial work status -327
+3,112 ***
20,000 15,000 10,000
(+20%) 18,594 18,267
+1,932 * (+14%)
18,750 15,638
15,607
13,674
5,000 0
Fully attached Semi-attached Long-term unemployed At baseline: Fully attached: Employed, or unemployed < 1month Semi-attached: Unemployed 1-6 months Long-term unemployed: Unemployed more than 6 months
36
Impacts on characteristics of current or most recent job, by work status at baseline Fully Attached
SemiLong-term Attached unemployed
Satisfied with job Saw opportunity for advancement at job
* ***
***
Had regular work schedule
*
***
Earned $20,000+ in Year 3
***
*
37
Limitations of a sector approach Not appropriate for all • Training in a just 1 sector won’t suit a diverse target group
• May need “bridge” programs to help more qualify • If can’t afford not to work, may preclude FT training
Must be nimble and have good labor mkt info •
Adapt quickly to changes in industry and employer demand
Steep learning curve for more traditional workforce providers 38
What’s next for evidence-building on sector strategies? For WorkAdvance • • •
Longer-term follow-up (5 years) Benefit-cost analysis Final report: 2020
Need better evidence on: •
Soft-skills coaching and post-employment features of sector programs (the “add-on” value)
•
Replication across more providers, sectors, labor markets
Other randomized trials are underway •
Results due over next few years 39
For more information Most recent MDRC reports on WorkAvance Encouraging Evidence on a Sector-Focused Advancement Strategy; Two-Year Impacts from the WorkAdvance Demonstration. (2016) By Richard Hendra, David H. Greenberg, Gayle Hamilton, Ari Oppenheim, Alexandra Pennington, Kelsey Schaberg, and Betsy L. Tessler Can Sector Strategies Promote Longer-Term Effects? ThreeYear Impacts from the WorkAdvance Demonstration. (Research Brief) (Forthcoming) By Kelsey Schaberg 40
