Rehabilitation Research and Training Center on Employment Policy for Persons with Disabilities
Employment Policy for Persons with Disabilities
Edited Transcript of November 8, 2007 Public Policy Forum
Employment and Disability Policy Initiatives in the UK: Results and Lessons for the U.S.
Stephen Bell, Abt Associates
Michael Daly, Department for Work and Pensions, U.K.
Richard Dorsett, Policy Studies Institute, U.K.
Susan Purdon, National Center for Social Research, U.K.
Bruce Stafford, University of Nottingham, U.K.
David Stapleton, Mathematica Policy Research, Inc.
For further information about this policy forum contact:
tel (607) 254-8311
The collaborators would
like to thank the National Institute on Disability and Rehabilitation Research
(NIDRR) for funding our work on this paper. The opinions expressed are the
speaker's own and do not represent official positions of NIDRR or
The contents of this policy forum were developed under a grant from the Department of Education. However, these contents do not necessarily represent the policy of the Department of Education, and you should not assume endorsement by the Federal Government. (Edgar, 75.620 (b).
This policy forum is being sponsored in part by the Rehabilitation Research and
The Co-Principal Investigators are:
Susanne M. Bruyere - Director, Employment and Disability Institute, School of Industrial and Labor Relations, Extension Division, Cornell University
Richard V. Burkhauser - Sarah Gibson Blanding Professor and Chair, Department of Policy Analysis and Management, College of Human Ecology, Cornell University
David C. Stapleton - Director, Mathematica Policy Research, Inc.
Communication Access RealTime
Translation (CART) Services for this event
were provided by Natalie C. Ennis,
Disclaimer: CART is provided in order to facilitate communication accessibility and may not be a totally verbatim record of the proceedings.
Welcome and Introduction
If you have not yet grabbed the materials, if you would do so, we are going to start any minute.
Okay. We're going to get started. We will probably have a few more people coming in. And we will just proceed.
I am Susanne Bruyere from
Center on Employment Policy for People with Disabilities. Our topic today is "Employment and Disability Policy
Initiatives in the
We are very pleased to have visitors from abroad, and I'm going to give Stephen Bell an opportunity to introduce our guests, who we're very happy to have here and who are doing multiple things while they're here so that is why we were able to capture all of them at once coming from such a distance. But Steve will share some more information with you. You also have that in your flyers. And I will do a bit to introduce Steve.
Before I do
that, I want to explain a little bit about the National Institute on Disability
and Rehabilitation research funded project to
We have some
upcoming policy forums. I'll come back at the end to talk more about those.
What I would like to do now is to turn this over to Stephen Bell, who comes to
us from Abt Associates in
With that, I will turn this over to you.
I don't know
about perfect facilitator. I think I was chosen because I may be the only one
in the room who knows all four of our guests from the
I'm really excited about the conversation we're going to have here, something I've been wanting to be a part of for some time.
As happy as I
am to be here and as long as this has been due, the benefit of coming now is
that we're going to hear about some very recent policy findings from a handful
of very worthwhile initiatives in
is in some ways the godfather of all of these studies. He has been with the
For the last
ten years, Mike's focus has been on evaluating welfare to work programs,
addressing population disabilities. He's overseeing in some capacity one or
more, I don't know, of the studies, and recently moved into a new role to
design a unique set of benefits to address the extra costs of disability, out
of pocket costs. As close to understanding
Thank you for that, Steve.
I'm also very
pleased to be here. In the introduction, Susanne said she hoped to learn from
things done in the
confess, actually, the true number of years I've been with the
I have actually been involved with all of these studies at one time or another. Some of them from the start and some of them at the finish, but all of them at one point or another.
I don't want to spend too much time on my introductory remarks, because I think the main interest for you is hearing my three copresenters with detailed studies. There are just a few things I was asked to cover by way of background. Why did we fund these studies?
I was just
saying, I don't want to take up too much time on
introductory remarks so we can get to the meat of the presentations. I was just
asked to say a few things in an introductory way. Starting with the question
of, why did we fund these studies? What was it about the
I think, though, this has been substantially reinforced by the habit of current government of signing up to high level targets. Easy things like eliminate child poverty, for example, increase employment rates to 80%. Narrow the employment rate gaps between various disadvantaged groups and the whole population. And indeed in one sense to bring about equality for disabled people. That's probably not quite such a hard target because we haven't quite worked out what that means.
But the importance of those targets is it really places the focus on outcomes, not inputs. I think in the past, there has been a distinct tendency for not just politicians but also for senior officials to support their pet programs, things they wanted to bring in and which they saw as effective, regardless of maybe what the long term evidence is on their outcomes. But once you have as the focus achieving 80% employment rate, there's a limit how much you can fudge that. What that means is that politicians want to know when a program is introduced, what is this actually going to contribute to meeting our targets? How many children will this lift out of poverty? And if it's not working, eventually that will come out and they will want to know that as early as possible and as clearly as possible so we can focus on something that does work.
Second question, why disability in particular? These points are not necessarily in order of preference. I think firstly, there are social reasons. There is a strong design for social reasons to bring about fuller participation of disabled people in all aspects of society. That's been expressed in one sense by the introduction of disability discrimination act of 1995 which has many similarities but also many differences from the Americans with disabilities act.
More recently, the prime minister's strategy unit looked across the whole range of policies affecting disabled people and came out with a report with the fairly obviously interpreted title of improving life chances of disabled people. And that was where the goal was set out effectively of equality for disabled people by 2025.
Those were the social reasons. There are also financial reasons. A number of the studies that we're talking about today have as one of their measures of success moving people off of incapacity benefits.
Rough equivalent to your disability insurance benefits. We have something like 2.7 million people signed up on those incapacity benefits at home. If you relate that to population of around 60 million, that's a pretty substantial number. The annual cost of that is in excess of 12 billion pounds, which is about $24 billion a year. So there's a strong financial imperative for government to try to bring those numbers down.
There are broader economic reasons. Like many countries, we're facing the problems of an ageing society, increasing numbers of people dependent on pensions and other benefits. And the principal way we have identified to manage that increasing burden is to increase the employment rate. Some probably fairly specious sums show if you can raise it from the current 75% to something closer to 80%, then dependency ratios are kept in control. These are all working together, strengthening the economy, saving money, et cetera, all by doing pretty much the same things. How long that happy coincidence will last is another question. So I think it's probably important to strike them together.
Why these particular studies? Just looking at the job retention and rehabilitation pilot, we have had a long standing belief that early intervention to help people with disabilities to remain in work rather than to move back into work after a period of absence is important and ought to be effective. But actually the evidence we have on what works was almost entirely absent. So we set up a pilot to try to test some interventions which Susan will talk about.
The New Deal for Disabled People represented not our first program to help disabled people move into work, but it was the first really large scale intervention designed specifically to help disabled people. So not an employment program to help people who are out of work, some of who might be disabled, but set up specifically to help disabled people.
The third one, pathways to work, represents our first serious attempt to introduce an element compulsion for people. Our aim here is to help enough people so we can start making a noticeable inroad into those capacity benefits, into the low employment rates of disabled people. So three rather different programs, all addressing slightly different issues around employment of disabled people.
I think I've said enough if not too much. We'll move over to Richard. Mike.
Mike. Richard Dorsett is from the policy studies institute in
Steve, and good morning, everybody. In this section, I will be talking about
the reengineering incapacity benefits and the effects of the pathways to work
reforms. Pathways to work is the label among people
with sickness and disability within
Mike has given a bit of a motivation for why this is important. I just reiterate the point that this represents the first real instance of imposing benefit conditionality in a work focus benefit conditionality for people claiming these benefits.
outline of what I'm going to talk about is I'll begin with a quick overview of
the capacity benefits in the
incapacity benefits in the
broad terms, there are some parallels between the IB/IS
distinction in the
interesting are the differences. I put three big ones here. In the
states. Not at all contingent on benefits.
And finally, a subtler point is that the PCA does not require an individual to be incapable of work. They have to just satisfy the criteria of having a significant condition or disability. There's no mention of the substantial gainful activity like you have in the states.
All right. Mike already said
that we have about 2.7 million claimants of incapacity benefits in
So that's the motivation why policy response was needed. And pathways to work is the main component addressing that. There's a number of elements to this. I'll just draw out the main components here. The first is that it offers early focused intervention. Now, if I haven't said this already, pathways applies for the new claimants of incapacity benefits. It's being piloted for some existing claimants but we're mostly talking about people starting a claim here. Under pathways to work, people have to attend series of work focus entities. From this point of view, it's significant, this benefit conditionality.
There's also an attempt to speed up the PCA process and to try to have the PCA result available within 12 weeks. I don't know if that's been entirely successful, but another aspect of this is to try to focus more on capability under the medical assessment rather than simply qualifying the benefits in terms of their condition.
this stage of work focused interviews, further participation in pathways is
purely voluntary. Pathways is assembled what is referred to as a choices
package of specialist support. This is preexisting and new programs with
tailored supports as needed. The big two within this, the NDDP, this is the New
Deal for Disabled People that I won't say very much about because Bruce will be
talking about this. But it basically provides help into work and support once
in work for a period of six months. The more innovative program under pathways
to work is the condition management program, which is, as is its aim, to help
people manage their disability or illness. This is something that's delivered
by the medical service in
The third element of this is to provide clearer incentives. The key component here is what's called the return to work credit. This is a payment of 40 pounds per week for people entering work. The payments are paid for a year and people have to be working jobs of 16 or more hours per week and the jobs have to be paying less than 15,000 pounds a year.
Okay. I'm not keeping an eye on time.
Okay. The evaluation approach. I won't dwell on the technical piece, but just to say that pathways was introduced on a pilot basis initially. Here we focus on those areas where it was introduced in April 2004. The evaluation is based on both survey data and administrative data. This is essentially structure sod we can observe what happens to people from the very start of their claims for a period of 18 months subsequent to that. And we have two cohorts of new claimants or people starting their benefits claim. One cohort, four pathways introduced, and one after. This was within specially chosen comparison areas. With this structure, we are able to evaluate the effect uses differences approach.
present the results of the evaluation, this is a graph I just wanted to show
because it was very instrumental in shaping people's expectations that pathways
was going to have a significant effect. So the chart shows three lines. The
black line â€“ all lines show the six month off flow rate. The black line shows
the off flow rate in those areas where pathway was introduced in October 2003. The blue line, the same for those areas where pathways was
introduced in April 2004. And the bottom line is everywhere else in
This slide here shows the evaluation results. Before getting into the detail of this, I'll just say something about the structure of the table. The first column gives the impact estimate. And the significant effects are marked with stars. So one star is significant at a particular level and two stars is significant at a more exacting level.
The level of significance, the P values given in the second column, the thing I wanted to draw out mainly here was the third column labeled "base." Base could have been termed counter factual. This is the estimate of what these particular outcomes would have been had pathways not been introduced. So take the first row, for example. The effect of pathways on whether somebody was in paid work a year and a half after first inquiring about their claim was increased by 7.4 percentage points from a level of 29.7%. So that was a significant effect. So there's a significant employment effect.
In terms of earnings, the effect is positive but not significant. In terms of incapacity benefits, we see a small negative effect but again it's not significant.
We also asked in the survey, we asked people about their current health status and whether their health problem affected their day to day activity. And the fourth row of results here shows that wasn't a significant effect. We asked the subsequent question of whether it affects their day to day activity a great deal. It seems here that the pathways program is reducing the effect that people experience a severe constraint on their day to day activities.
In terms of the employment effects, we can look in more depth as to how the eventual effect evolved. This chart here plots the estimated effect over time. For reasons aside, we can't really look at the four months before. But the thick blue line plots the evolving impact on employment over time. So at the end of the period, what it demonstrates is that this effect has been growing steadily rather than the initial surge that subsequently tailored back. The green shows the counter factual or what was labeled "base" on the previous table.
We can do the same for benefit impacts. This is actually taken from administrative data rather than survey data because the survey questionnaire was designed with the fact that we could use high quality benefit information from the admin data. Again, the counter factuals are shown in green, plotted on the right hand axis. So half the example will no longer be on the sample.
This shows again, at the end of the 18 month period, we have this effect of about 1.7 percentage points. But what's noticeable is that there's a strong effect at roughly the five, six, seven month point where the probability of being off benefit at this stage is increased about 6 percentage points. This clearly maps back into this earlier table that I showed.
Okay. So to
offer some concluding comments, pathways to work is a significant development
in labor market policy in
In terms of how this feeds into policy, there's a sustained rise in employment but not a reduction in benefit suggested in terms of gravitating towards this 80% employment target that Mike mentioned earlier on. The results are quite positive. In terms of achieving the one million reduction in benefit roles in IB, the results are less positive. Fewer people are reporting their health condition limits their ability to go about their day to day activities. So this is consistent with what you might hope would happen with condition management program.
A couple reminders when attempting to extrapolate any of these points. First, the benefit population here includes in the early days people that haven't yet passed the medical assessment of PCA. So it's a different population from the states.
Secondly, as I already mentioned, the availability of Medicaid and Medicare and the dependency on receiving SSI and SSDI alters the structure. So it's another important distinction that needs to be borne in mind. And that's it.
Are we taking questions? Yes, go ahead.
Just a clarification. I'm struck by, half of the people are not on the benefit rolls a year and a half later. Is that because they didn't get on or they didn't pass the test? Or do some actually go back to work? Is there a way to tell the difference?
It would be a
combination. That's the last point I was making is that people who get the
benefit before actually passing the PCA. So when I enter work
or if that condition has actually ceased to be a barrier or because I failed
the PCA, or for a number of other reasons. That's why you would expect
to see a higher off flow rate amongst the population that we're considering in
Does the incapacity test have a durational requirement? Or is it simply as of the time?
As of the time.
Are we supposed to be using a mic for the webinar or not?
They have it on the table. Oh, I see.
Okay. So my question is, can you explain how you could increase employment without decreasing benefit receipt?
Can you repeat the question? The audience can't hear the questions. Thank you.
So the question is, how do you reconcile the fact that there's an employment effect but no benefit effect in the long run. We scratched our heads about this as well. I mean, we're limited in the extent to which we can come up with a conclusive answer to this. One thing we did do was to create joint outcomes where you combined them. What was observed was that there was an increase in the proportion of people who 18 months after starting their benefit, the start of their claim, were employed but not on benefits and a reduction in the proportion who were not employed and not on benefit. And those two effects pretty much cancelled each other out. So what it seems to be suggesting is that the employment effect is manifesting itself through those people who wouldn't have been on the benefit anyway.
Two more here and then we'll move on.
You have compared the additional cost of providing the service and the administrator cost?
So the question is, have we compared the costs of administering the service with the benefits. And the answer to that is, that's a part of the evaluation. I have only reported on one part of the evaluation, which is the impact analysis. But the program evaluation involves qualitative analysis. The impact analysis I presented attempts to distinguish the relative effects of component within pathways to the overall impact and importantly, the cost benefit analysis. But the stage of cost benefit analysis is that it's imminent.
Just to add
something to that, while we are still waiting, the formal cost benefit
analysis, the decision has been made to roll out the pathways program across
the whole of
I think you
made an excellent distinction between the
I think there
may be some interesting comparisons between your results and
I was wondering whether you can tell a little bit about the difference between IB and IS and the relative number of people who are on IS and what kind of interactions.
Okay. Just to
repeat the question, there was a comment making comparisons across the
And a request for some expansion on the distinction between IB and IS, which I'm probably not entirely qualified to give but I'll make an attempt at it anyway. So the IB and IS differ, as I said, in terms of the eligibility criteria. So IB requires a sufficient contribution, national insurance contribution, over I think it's the previous three years. And the IS is available to people who don't have that. People applying to go on incapacity benefits all channel through the same route. If you haven't satisfied the earnings contribution, you still apply for IB. If you eventually pass the PCA, you'll be granted what's called credits only IB, which is that you become a claimant but not a beneficiary. So you don't actually receive any payment. An income support is a benefit available to people on low income. There's an idea of a sudden amount of money that you require for living costs which is dictated by your personal characteristic and various premium available for adult dependents and child dependents and type of disability and these kinds of things, and that income support will provide the necessary income to allow you to achieve that. So it's common for people, nearly everybody claiming income support will also be getting incapacity benefit itself on a credits only basis.
Richard, could you fill in the figures on the relative size of folks getting the benefits?
I can't remember the exact figures, but roughly speaking, about 40% of the total 2.7 million are on income support rather than incapacity benefit. That is a proportion which has been increasing over time. I think the other thing that's probably worth saying is that overall, people on the means test income support tend to have worse characteristics. By definition, they're more likely to be people who have little or little work history or an interrupted work history. Also more likely people of mental health problems. Worryingly, there are differences with age and gender as well.
Final point. Is this correct, that movement between the two types of benefits is not as common for an individual over time as it is here? There's no over lap.
The only over lap, if you're receiving incapacity benefits, it is possible you can claim some income support in addition to that. So if you have enough children or housing costs or whatever that push your entitlement over what you're getting in benefits. But there's no movement between the two in the sense that if you have the national insurance contributions to entitle you to claim incapacity benefit, that record is maintained while you're on benefit so you'll never lose it. And if you haven't got it when you start, then you can't accumulate while on the benefit. So one or the other.
Okay. We took a little longer there, but this information is so beneficial.
Purdon will speak about the job retention and rehabilitation pilot on which I
had the privilege to work with Susan and her colleagues at the National Centre
for Social Research in
She has worked on, in addition to what I've just mentioned, the New Deal for lone parents evaluation and an evaluation of education for disadvantaged families. On top of which, she specializes in the design of social surveys on topics in the university.
Yes. Jack of all trades. The trial I'm going to talk about is called the job retention and rehabilitation pilot. We never thought of a snappy title for it. As Steve said, it was managed from the national center of social research but it was actually a collaboration which Steve and Dave did on what they brought as technical advice and led on the cost benefit analysis of that. It was a huge team effort.
this trial was actually designed as a full randomized controlled trial. They
are common in the
It was a trial aimed at people who were off work sick for somewhere between 6 and 26 weeks. To be on the trial, you still had to have a contract for employment but you've been off work for a considerable amount of time. And this is very yellow. You can't see it very well.
The main objective of the trial was to offer help to people so that they would get back to work. And a successful return to work was defined as a return for 13 or more weeks. It was entirely voluntary, this trial. And people have to self nominate. So you have to know about the trial and put yourself forward.
And there were four randomization groups: A health intervention. The idea was, if you could help people's problem out, they could go back to work. The second was workplace intervention. If you were in that one, you didn't get help with your health; what you got was help with your workplace. So the idea with that one was to fix the workplace, not the person. The third randomization group was combined, workplace and work health help. And fourth was a control group. The trial ran for a two year period from April 2003 to April 2005.
The three interventions, what I've done here is tried to characterize what they were. Within reason, the providers of the services were allowed to offer whatever they thought was best. They had budgets but they could offer what they wanted to. What happened to people typically if they were in the health group, they would be offered things like physiotherapy, if they had musculoskeletal problems. They might get referred to specialists, and that would be paid for them. Or they might get some complementary therapy.
If you were assigned to the workplace group, typically people got ergonomic assessments or the service provider would liaise with your employer to help you get back to work, making changes in the workplace for you.
If you were in the combined group, you could get any of the things in the two groups above or something else if you like. An awful lot of people in this group got cognitive behavior therapy. The best way to think about this was the service providers had three models to deliver: Health, workplace, or combined. The combined was meant to be what they thought was best practice. So what we were trying to best practice against those two. The reason we had the three interventions is there was no understanding when we started this as to which model would be best both in terms of getting people back to work and which would be most cost effective. So if you were to roll this out, would you roll out something that was primarily a health led intervention or would you roll out something that was a workplace intervention? So that was the debate we thought we were having.
What we thought would win is that we were convinced the combined one ought to come away as the outright winner because that was best practice.
The trial didn't recruit as many people as we hoped but we did have in total almost 2,900 people. We assigned 711 per group.
Now, inevitably, not everybody randomized took up an intervention. Some people didn't like what was being offered to them so they dropped out. Some people said they wanted help with their health and were actually assigned to the workplace. Not everybody got what they wanted. In the percentages, you can see 78% of people with health took it up but only 55% of people offered workplace did not take it up. Most people wanted their health problems sorted out, not help with their workplace. Some people didn't take it up because they actually went back to work very quickly. So those numbers encompass them.
And not everybody randomized was actually interviewed. We sent an interview around to people's homes to find out whether they had gone back to work. But not everybody would cooperate with that. So it was a randomized trial but we didn't have equal size groups at the end to compare. You might say that slightly undermines the design or not.
Okay. This just gives a rough indication of how much each of the interventions cost. So spend per client over a six month period. You can see the post expensive was the combined intervention at just over 2,000 pounds per person. That was followed by health at about half that. And the cheapest of all was the workplace, or arguably the absolutely cheapest was the control group where you offer nothing. That's zero. So again, you would expect combined one is the most expensive so that one should win.
This is what we find. This was our big horrible shock around this. This is our key outcome. Percentage return to work, 13 weeks. If you can spot a difference in those figures, you tell me where it is. In every single group, about 45% went back to work. On that basis, the control group did just as well as any of the other groups. All our theories about how this was going to work completely went out the window. We thought the combined would be the outright winner. We thought the others would be behind that and the control way behind. But they are completely the same. When we first ran these off, we ran it about 17 times to check we got it right. Eventually we concluded there is no impact. None of these interventions worked.
So faced with that, which we thought this might be a short report to write at the end of it, we tried to dig deeper to find if there were subgroups for whom it did work. The problem there is that we ran into rather small sample sizes. If you looked at the group off work because of injury, it was only a small percentage of the total group but there is an impact. Here you can see 36% of the control group went back to work. If you were in one of the other groups, it would be at least 14 percentage points higher than that. So for that subgroup, it appeared to work and it was particularly powerful if you were assigned to the workplace group. The theory we've had is that employers were willing to take people back.
Now, if you got a positive result on one thing and no impact overall, you have to have a negative result somewhere. This is our horrible negative result. This was the most high profile finding we had. Just border line significant. If you were off work with a mental health problem, the best thing we could have done to you was put you in the control group. If you were in the control group 59% went back to work. If you were in one of the other groups, you had less than 50% chance of going back to work. This was a huge shock. We were worried that we were just kind of fishing around looking for results. But if it's right, then it's rather worrying.
We did collect an awful lot of other data on other outcomes of people. One of the things we expected to happen although it wasn't the aim of the trial, we did expect the trials to improve people's health, and so we looked at various health outcomes. They're all self reporting. I just picked this one out at random which was our finding on depression scores. The scores of 8 to 10 means you have mild depression. 11 plus, then it's moderate to severe depression. What we would have liked to have seen is the interventions reduce the scores of 11 plus because that's when you start worrying about people. On that grounds, the final column, you can see there was actually no impact or more or less no impact on the severe depression. There did appear to be some impact on mild depression. So if you were in the control group, then you had a 25% probability of having mild depression which would go down to 20% with the interventions. So there was some small impact on health. Again, because of sample numbers, we were on borderline significance on this but that is a significant finding just about.
So what we concluded was that the interventions had a moderate impact on people's health but pretty much no impact overall on their employment chances.
Now, where that left us, because keep in mind, when we designed the trial, we had no expectation that this would happen. So we didn't plan for this to happen very well. What we were left with was scrabbling around with the evidence that we had to work out what had gone wrong and why the interventions weren't working. The best evidence we had was we did in depth qualitative interviews with small samples of people in the intervention groups so we knew what they were saying about the interventions. And the things that we kind of pieced together from this, one thing that came back as a recurrent theme was that the groups didn't like the interventions they were offered in many cases. They thought they weren't intense enough, that they weren't well enough matched to what they perceived as their issues.
The other thing that came out was that people assigned an intervention quite often said that their relationship with the service provider wasn't particularly well managed. So that they were often left to contact the provider to say, what's happening to me now. And what was coming back was a sense that people assigned to an intervention often would â€“ the intervention wouldn't entirely happen because they weren't following it through. It is kind of anecdotal. Not everybody said that. But there is some sense that people were left to be very proactive about their intervention, and they weren't being so, if that makes some sense.
The third piece of evidence, because you can imagine, where we were, the finding that we were left really having to explain was the failure with the mental health group. The best piece of evidence we have on this and the thing we think might have happened is that if you were assigned an intervention, the main aim was to try to keep you in your same job as you were in when you started. So bearing in mind everybody had a contract in employment. What the providers were trying to do was get people back to the same job. The people with mental health problems, in the control group, what they did was got themselves a new job. So that is one possible explanation as to why the control group did better. Essentially, they just left their current employers. What we haven't done is followed them up over time to find out whether that was a short term solution for them and what then happened is that they lost that employment as well. We don't know.
So that's where we are. I'll finish there. It was an extremely interesting experience but we're finding it to be a little disappointing, to say the least.
Let me hand the mic around. We now have a roving microphone.
Art Sherwood. Thank you. What about the black hole of the drop outs? Did you try to characterize them? You said a lot of people dropped out because they didn't like what happened.
We followed them up through the interviews. I know we haven't done any analysis looking at whether they were very different in terms of their outcomes to other people. Maybe we should. Did you have a particular hypothesis, by the way?
Very nice presentation. Thank you. I have some personal interest in seeing the results. Mr. Brown must be pleased that it is his job to do the randomization.
Having said that, I think it would be too early for him. One of the things these kinds of interventions may do is to slow down people's return to work. I mean, after all, you need to make a point and so forth. So it is entirely possible that long term effects would be negative as well. But I think it's also possible that they may turn out different. I was wondering whether you could match in some medical records.
One of the things we haven't done, and it feels slightly ridiculous, but we could track everybody through the trial through their benefit records. We could actually find out whether people turned up on the benefits system over time. We haven't done that. I'm not entirely sure why. We got permission to link their data from people, so we can do it.
In terms of whether or not it was long enough, in some instances, the intervention could have delayed the return to work. The service providers didn't think that was a plausible explanation themselves. They thought they had had long enough with people to get them back to work. So they were very surprised by the findings. So though I think it's plausible, I would be surprised if that turned out to be the answer.
Just to answer that briefly, I think the intention always was that if we could show the increase of people who returned to work for at least 13 people, we would follow that up to see if that was sustained for longer. But as Susan said, this one particular failure was not something we planned for.
What we did do, we looked to various other possible measures of outcomes by the proportion who would return to work within six months and that also showed no impact. This particular institutional reason for wanting to make sure that you got people back to work within the 26 weeks, because after that time, you move from broadly speaking from being entitled to sick pay to being on incapacity benefit. So the real thing was, can you actually get people to work, back to work, while they still got the contract before they make that slip into long term benefits? So that's possible that if we work with people for longer, we could have more of an effect. But there's an institutional reason for that 26 week cut off.
There's a question in the back. Is it on?
I was wondering, you mentioned intent to treat. I don't know if it would have addressed the detail of the population with mental illness in that you could have had the people who stayed with it have a different set of conditions. But they could be holding on because of other reasons. Not necessarily because they necessarily got a benefit from the employment but because they benefit from the social contact. So you might be getting a population, and attrition could be very systematic.
Because the mental health finding is so controversial, I guess, or disappointing, we have recently commissioned a piece of work to look at that group. That's one of the things being looked at. I roughly know that the findings are that that's probably not the explanation, as far as we can tell. I think the explanation that's kind of sticking in there is the changing job distinction. But I do take your point.
The problem we have is that we're getting into small sample sizes. So you quickly run out of statistical power to check things out.
One last question here. We will have time at the end as well.
I'm wondering if the intervention might have been more successful if it were planned with quantitative research with this focus.
If the intentions were granting qualitative research?
That included people with disabilities.
organizations who did the service provision were people who had worked in the
field for a long time. They are all professionals in the field. What we took
for granted, maybe rightly or wrongly, was that they understood current theory
about what was best practice in interventions. So within the constraints of
what they were allowed to do, they were providing what they considered to be
best current practice in the
Or involve people with disabilities in that research.
We probably didn't do that. The interventions weren't prescriptive at all. In the headings, there were things you couldn't do or could do but the providers could do whatever they wanted. There wasn't a definitive intervention about that, if that makes sense. Probably not.
I'm going to cut off the discussion at the moment.
Last out of
the gate, though I think the longest acquaintance in my case, happy
acquaintance, is Bruce Stafford, professor of public policy at the
Steve. Good morning, everyone. This presentation is about the New Deal for
Disabled People. That's a national program that was designed to help people
with disabilities and health conditions move into sustained employment. And as
Steve has already mentioned, it was an evaluation that was conducted by a U.K./U.S. consortium that includes everyone from the
I want to say at the outset that this work focuses on extensively the work of my colleagues. And I want to thank the department for sponsoring the evaluation as well.
In terms of sort of key headline findings, I suppose there are two things I want to highlight. First, it is possible to claim that NDDP worked in the sense that it was successful in reducing benefit receipt and increasing the employment of participants on the program. And that from a sort of societal point of view, the benefits exceeded the costs.
second sort of headline finding I suppose is that this was achieved
notwithstanding that there were some difficulties in implementing the program.
And in terms of sort of policy lessons, I suppose there are two key issues we
want to highlight. The first of which would be the funding and contractual
regime used in NDDP, which the Department for Work and Pensions didn't
necessarily get right but I think there are potential lessons for both the
And the sort
of second key thing I suppose is that there are lessons to be learned about the
broad institutional framework within which NDDP was delivered in the
And what I want to, I suppose, highlight is the role that was played by an organization called Jobcentre Plus. They provide employment services and benefit services to people with health conditions and disabilities.
Before going on to the findings, some sort of background information about New Deal for Disabled People. It was piloted in 1998, two versions of it. In effect, a third variant of it was extended nationally in July 2001. And as I've already said, it's a program that was designed to help people claiming incapacity related benefits. That's principally incapacity benefit and income support that Richard has already mentioned. Principally designed to help people move into employment.
It was a voluntary program. It was delivered by around 60 providers, known as job brokers. And job brokers are a mixture of private, public, and voluntary sector organizations who competitively bid to deliver the program.
A key feature of the program was outcome related funding. Basically, each job broker received an expanded registration fee each time someone registered to take part in the program. Initially, that was 100 pounds. But in October 2003, that was increased to 300 pounds. However, most of their funding came from payments they received when someone ended employment and sustained that employment for a period of time.
Now, the amount that each job broker received for these job entry and resustainment employment varied, and it depended on what the provider of the organization negotiated with the department during the procurement process.
There's a large scale mixed method evaluation. We did surveys of the eligible population. Surveys of the people who took part, which in some of the literature, if you look at it, we sometimes call them registrants, not participants. There were also surveys of job brokers and employers who were involved in the program. There was qualitative research of people who took part of the staff involved both in Jobcentre Plus and job broker organizations. And also there was some qualitative research of the employees as well.
The impact analysis and cost benefit analysis both focused on administrative data. For a number of the components in the evaluation had a longitudinal component to it. And the evaluation period spanned July 2001 to around about November 2006.
There were a
number of different sort of implementation issues that I could discuss. The
first that I want to take up is take up rates of the
program. The reading that I've done with the evaluation of the Ticket to Work,
plus what we've said about the New Deal for Disabled People, would suggest that
take up rates could be higher. The low take up rate in the Ticket to Work seems
to be more of an issue. The take up rate in the
It was probably higher in these pathway work areas because of the more intensive mandatory work focused regime that Richard already mentioned.
So while NDDP was a voluntary program, providing structured setting within which recipients considered sort of future employment options and aspirations, has increased program take up.
I should say, though, that other research that's been done does show that these work interviews are not uncontroversial. Claimants can complain about the timing of these interviews being inappropriate given their health condition. And in some cases, the research suggests that they've been used to reassure longer term claimants of returning to work rather than focus on Richard's work.
The second sort of implementation issue I want to take up is about the funding contract regime. This regime had a major influence on the job brokers. Now, without going into the sort of details of the funding regime and how job broker performance was monitored, many providers over time increasingly focused on those who were closer to the labor market. And this was because job brokers had to be confident that she could convert registrations on the program to job entries within a reasonably short period of time. Otherwise, their reported performance against contracted targets and the specified minimum performance requirement would be on demand.
Now, as I've in a sense touched on, there were funding changes in October 2003, and these were in part a response to job brokers, some job brokers, claiming that they had difficulties recovering their costs. And that this funding gap led a few of them to withdraw from the program. For others, it could lead to them searching for different ways to subsidize the program. It could lead to them wanting to increase the number of participants they referred to other organizations for service. It could reduce the resources of the program. And it could lead to an increase in caseloads.
However, as I've already mentioned, this did lead to an increased focus on people who were close enough to it.
And thirdly, like many implementation studies, the evaluation in a sense highlights the importance of information given and of effective communication. There were initially some problems in working relationships between some job brokers and some local Jobcentre Plus offices. But these seem to have been largely addressed over time with increase of contact between the staff involved. Less successfully dealt with was the choice element of the program design. On the NDDP potential participants were meant to have a choice of different provider, but in practice, they rarely had enough information to make an important choice about which provider to use.
In terms of
outcomes, the administrative data shows that of the 2,600 registrations between
July 2001 and November 2006, 43%, or just under
11,000, had resulted in jobs by November 2006. Indeed, the proportion increased
over time. So for example, of those registering between July 2001 and 2002, 32%
found work within 12 months. But this increased to 44% for those registering
between December â€˜04 and November â€˜05. Using a sort of match comparison
The graph on the left here shows that for incapacity benefit receipt, there was a reduction over a 12 month period of 16 percentage points for longer term participants, and 13 percentage points for more recent participants.
The graph on the right shows an increase in employment over the same period of 11 percentage points for longer term participants and by 7 percentage points for more recent participants.
These figures are sort of reproduced in this table. It's possible to look at impacts on benefits and employment rates. We also can look at the likely impacts following the changes that we introduced in the funding and contractual regime in October 2003.
So what this table shows is at the top, incapacity benefit receipt. It shows the percentage point reduction for longer term benefit recipients and more recent recipients, 24 months and 36 months. It also shows incapacity benefit amount, the reduction in pounds on benefit expenditure, again, for 24 and 36 months. And it shows the increase of employment at 24 and 36 months.
Also on the table, the other figures relate to what happened before and after October 2003. So the estimates for the 36 months show that in general, the reduction in benefit receipt continued both longer term and more recently for beneficiaries, although diminishing in the more recent claimants.
For employment, it appears that the longer term recipients leveled off to around about 10 or 11 percentage points in the third year. But remained about 8 percentage points reduction for more recent claimants. Reflecting the falls in benefit receipt that are also savings in amount in benefit. So for example, the average monthly benefit saving initially grew. By month 24, it was around 81 pounds for longer term claimants and 51 pounds for more recent recipients.
The analysis that compares impacts before and after the October â€˜03 funding changes suggests that impacts on benefit receipt and hence benefit amount were larger post October but had a more modest effect on the employment rate. However, these impacts can't be attributed to the program definitively. Am I running out of time?
Okay. Because it could be due to for instance changes in the state of the labor market for disabled people or it could be because it's reflecting that the program has matured over time.
The evaluation also involved a cost benefit analysis. And estimates here are presented on a participant basis. Again, for longer term recipients and more recent participants. The cost benefit analysis is done from three different perspectives: Government's, participant's, and societal perspective, which is a sum of the two. The analysis is slightly complicated because there are three different methods used or approaches used to estimate the costs of the program. The first one is actually based on the actual amount that was paid out by the government to it job brokers. The second, which on this table is labeled upper bound estimates is based on a cost survey of the job brokers cost. And the third one labeled lower bound estimates is the preferred methodology used by the evaluation team is based on an adjustment made of job broker costs. Two of the job brokers who participated, it is suspected, overestimated their costs. However, regardless of whatever cost approach you use, from the government perspective, it's clear that there were benefits to the government entity paid. That's less true from the participant's perspective where the benefits were modest. But if you looked at it from the societal point of view, there was a net benefit to society from the induction of the program.
Steve, I know you're hassling me. If I could just quickly â€“
suppose the two key conclusions that I wanted to put across is that, in part,
this is based on my reading, a partial reading of what you've done in the
states, Ticket to Work program, but it seems to me, the job brokers had an
advantage compared to the employment networks because of the existence of
Jobcentre Plus. Because what it was able to do was to assign people to service.
And because of the mandatory work focus regime that's been introduced, it was
able to increase the take home of the program. So looking at NDDP, I don't
think there's necessarily anything that the job brokers did, but there are
lessons there for the
sort of conclusion is around outcome related funding. One of the reasons
justifying this in the
Further from the labor market, what provision is made for them.
Finally, I think one can argue that there is a role for an NDDP type service for the client group but it would need to be part of the wider regime.
Thank you, Bruce. If there are any burning questions on Bruce's information, good. Because I want to pass it along to David Stapleton, as the discussant who needs no introduction.
Can people hear me okay? Okay. Thanks. Steve should have introduced me by the fact that my first work on disability was project network which was a project that Steve was in charge of the evaluation of.
So it's my fault.
I'm not going
to say a lot about your results or try to analyze or worry about that. But I
wanted to contrast what has gone on in the
It's particularly interesting the contrast, the New Deal for Disabled People with Ticket to Work. I've been involved in the evaluation of Ticket to Work. But as Bruce has told you, it has features that sound a lot like Ticket to Work. There are job brokers. Ticket to Work has employment networks, wide open to nonprofits, for profits, government agencies, et cetera. They have performance based payments, based on whether the person gets a job and gets off the rolls. And they're stretched out over a long period of time. It's a voluntary program essentially although there are some, in some areas at least, where there's an expectation of people trying to work and go through this process.
So lots of ways, it sounds like Ticket, at least in terms of some of the basic features. But here, the difference ends. What we've seen is evidence in increased exits in the rolls, it increased employment by a comparable amount, and the costs are lower than the benefits when you measure it from the government's perspective, from the participant's perspective, and from the broader social perspective. We're doing evaluation of Ticket to Work, as I said, and we have an executive summary of the third report over there. What you will see is that we can find no evidence that it has increased employment to beneficiaries or increased benefits. And of course the rates were very low, probably about a third of what they are in your program.
That doesn't mean it can't be improved. In fact, the hopeful thing about the NDDP is that maybe we can improve the program's design. But you also have to ask whether it's the other aspects of the policy environment that are resulting in the difference in the outcomes for Ticket versus NDDP. Bruce alluded to these. Especially the health insurance. Ticket is set up so that people who participate will maintain their Medicaid and Medicare for a long time. So I don't know that that's such a big deal. But maybe it is.
I think these
job focused interviews that you talked about that people had to go through, at
least in the pathways areas, that may be an important
difference. There may be differences in incentives. People may feel more secure
about being able to come back to the rolls in the
I think the pathways to work demonstration is very interesting for me, as is the job retention and rehabilitation program. People in this country have been saying for a long time what we really need to do is early intervention and get people while they're still attached to the employer, get them back to work and keep them off the rolls or at least shorten their stay on the long term roll.
I think there
are some promising results as far as employment is concerned, but we don't see
the results we would like to see in terms of the benefits. Certainly Susan's
results are puzzling. But I think the important point here is that you've made
a lot of progress and we know a lot more because of what is going on in the
Would members of the panel like to respond to the discussant?
If you like, I could say something about what we've got planned. Just to respond briefly to David, it would be interesting to hear further about what we've got planned, one thing that we definitely have underway is the national implementation of the pathways to work program, which for reasons which it would probably be indiscrete of me to speculate on too much. In the 60% of the country which is not already covered by the extended pilots, it has been decided that would be contracted out to private and voluntary sector, with contractors rather than government agency. There's various political considerations going into that which I won't get into. But we're definitely committed to doing that.
The other thing we've committed to doing is replacing with single benefit, employment support allows it will be called. Where to start were this. We always thought that it was important to have a change in benefit in order to change people's expectations that although, as Richard pointed out, there is an important distinction that receipt of our benefit is not contingent on being incapable of significant work as in the U.S. Nevertheless, the fact that you pay somebody something called incapacity benefit gives people a strong signal that you assume they're incapacitated. And it's very difficult to engage with somebody in a conversation about going back to work at the same time they're trying to prove to that you they can't go back to work. So we want to change the benefit but also change the expectations. We're also taking the opportunity to remove some of the many anomalies that there are and difference in people being on the two different types of benefit. For example, both current benefits are operated annually in line with inflation, but using different indices so they drift apart over time. So for reasons which no doubt made good sense to somebody at some point but at the moment they just confuse people.
There is also simplification going on. But the main difference that was introduced in the welfare reform act was to take the legal powers not only to oblige people, as under the pathways to work program to attend work focused interviews, but to make a compulsory for most of them to undertake some kind of work related activity between those interviews in line with an action plan which they will agree with a personal advisor.
of the current financial situation for
That's the broad direction of travel we've got. The other thing I would say in terms of innovations is that what I would ideally like to do is to spend more money. We have things like NDDP which work and would work for more people if we spent more money on it.
Anything else from the panel in relation to Dave's comments? Let me mention, I have to leave right at 11:00, but Susanne will take over at that point.
Anything more in response to Dave or should we open for questions?
This question is about protocols to getting people to work. There was a piece that ran in the Washington Post a couple of weeks ago describing how thousands of people with disabilities were employed and set aside in federal government programs. I guess they were successfully successful in their work because they were doing work in the new executive office building. My question is, are there any protocols you know that are in the pipeline where the government has the job to match the abilities of the disabled person with a job and train them to do it?
Everybody is looking at me. There are a few things along those lines. I think it's fair to say in the sense of a major initiative, no. There are a number of approaches which have been used by various organizations, including job brokers, which have a systematic approach of trying to identify the most appropriate jobs for an individual person based on their abilities. Which I guess is partly what you're thinking about. And where necessary, to provide them with whatever support they need to be able to do that, which might be some preemployment training, it might be paying for adjustments, it might be some initial job coaching. But I think the direction we've been moving in, as I think is probably apparent from the presentations, is rather than the government getting involved in trying to design that approach is saying to organizations like our job brokers, if you think you've got something which works, then by all means, go ahead and do it. We will pay you for the outcome.
thing that I would mention, which is relevant, is something which we're working
on with various organizations, including employers
forum on disability in the
I know you have a question and you did as well. I'll go between you. I think it should be on.
I think it might be. I think one of the compromises we have to have on the trial is because we had no way of identifying people, we had no lists, we had to ask people to come forward. We had no way of approaching them. Which means we did have a trial of people who were motivated. And certainly had we found an impact, it would have affected what we inferred from that about more general groups.
Whether it is the solution to the puzzle, I'm not sure. Certainly the service providers said to us they knew they had motivated people and they thought when they started this that would be the group they ought to be really successful with because they were working with people who wanted actively to go back to work and if they couldn't change things for that group, they would have no chance for people who were not motivated. So I don't know what we would have found had we done a trial of unmotivated people. But what people thought would happen is that it would actually be less successful with that group but we can't prove it. I don't know what the answer is.
Another question, and then we will go to our remote participants. Thank you.
I'm with the office of disability. I have two points. One is in regards to comparison that David Stapleton provided before he left about comparison with the work incentive program. I also think we need to look at TANF and those individuals as well. The second though, a major challenge that we keep finding, especially with Ticket to Work, expressed by many people with disabilities, has to do with a fear of the loss of their healthcare benefit. And you may have addressed it and I didn't catch that. But I'm just wondering how you did include that in your analysis or in your review, and I appreciate the comment.
Just in terms of on the first point, yes, there is an issue. This is something that my colleagues in the past on the board, we have a very distinct tendency to classify people by what benefit they're in receipt of rather than looking at people as individuals. And there are a lot of disabled people claiming benefits other than incapacity benefits who would benefit from participation in some of these programs. It's something that I won't say that we have any active plans there, but it's certainly something that people have considered.
In terms of
the second point, the health benefits, compared to the
Can I add
something? Just a couple of points. I mean, one,
specifically from the pathways side, one of the components of pathways is the
possibility to waive a repeat medical assessment if someone exits benefit and
returns to it. It reduces some of the risk of starting again from square one
and reentering the program and jumping through all the hoops. But on the first
point about comparisons with U.S. programs, there's a fundamental difference
that there's not an obvious program to compare something like pathways to
because it's a mandatory program, at least in part, for the population of SSI
and SSDI recipients in the states, it's not a comparable program. One of the
elements from the
I would like to take one or two more questions. Can you stay another five minutes? That's not a problem? We have one question here, and our administrator at a distance has been accruing questions. But one more question here.
Hi. My name
is Marlene. I have a couple of points. I'm unemployed and I'm motivated. I work
in the trenches. I work with people with disabilities. Being an educator, and I
did a presentation in the
Any comments from the panel or anyone in the audience in response to the comment?
I think we've lost some of our participants who perhaps could have responded to that. Thank you for your comment.
Jeff, are you there and can you tell me if you have questions from our participants at a distance?
Yes, I'm here, and actually we don't have any questions.
Okay. Terrific. Then with that, I am going to thank our panelists. Anyone attending the association of policy analysis and management conference, I believe tomorrow morning you have another panel. Am I right about that?
If you are
here in D.C. and are attending that conference, you have another opportunity to
see our guests. But our sincere thanks for joining us here while you were in
Just on behalf of all of us, thank you for giving us the opportunity to present this to a wider group of people. And if anyone wants to pick up on these issues afterwards, get in touch.
With your permission, I will put your contact information and your Power Point slides on our website so that people can follow up with you directly and also get the benefit of seeing the materials.
Yep, Thank you very much.
Thank you very much.
more information about the Rehabilitation Research and
Susanne M. BruyÃ©re
Employment and Disability Institute
201 Dolgen Hall