New research on selection rates of black and minority ethnic UK parliamentary candidates shows the “ethnic penalty” is disappearing for the next generation of MPs, argues Sunder Katwala.

“Nobody can say when we might see a British Barack Obama. In many ways, Obama could be a once in a lifetime strike of political lightning. But Obama’s election will throw the spotlight on progress on race in British politics. The question we can try to answer is this: how far do candidates from ethnic minority backgrounds still face higher hurdles because of their race?

The conventional wisdom on this subject is pretty gloomy. There are currently 15 non-white MPs: there would be 60 if the House of Commons was to reflect proportionately the ethnic mix of the country. So the fact of under-representation is clear.

Beyond that, most of the conventional wisdom turns out to be wrong.

The evidence from current Parliamentary selections offer good grounds for hope that we are reaching a ‘fairness tipping point’ where black and Asian candidates do not face an ‘ethnic penalty’ which means the barriers are higher than for white candidates.

Where Labour is selecting new candidates for the next election, BME candidates have this time around been selected at a rate which exceeds their proportion in the population: this is especially true in the party’s most winnable seats. The Conservative Party is making good progress from a low base, particularly selecting candidates for its most winnable seats.

There have been complaints that special measures to select more women have not been replicated for BME candidates. Yet the evidence shows, perhaps counter-inutitively, that relatively rapid progress is being made on ethnic representation while a gender penalty for female candidates remains more stubborn. So perhaps the prospects for a British Obama – or at least for fair chances in British politics – are much better than anybody has realised.

Ethnic minority representation: the conventional wisdom

The core arguments made by MPs and campaigners on minority represenation are summarised in the Operation Black Vote report commissioned by Harriet Harman and published. This formed the basis of Keith Vaz’s case for all minority shortlists in introducing a private members’ bill.

The key points can be summarised:

• There has been very little progress since 1987, so that it will take 75 years to have a ‘Parliament that looks like Britain’ if special measures are not introduced.

• Black and Asian candidates face significantly higher hurdles in being selected because of their ethnicity.

• There will be no chance of a level playing field until similar measures to the all women shortlists (adopted by the Labour Party) are introduced, because international comparative evidence suggests proportionate representation of women has almost always required positive discrimination or equality guarantees.

• The failure to act demonstrates that race is not taken as seriously as gender – so that BME candidates fall further behind when progress is made elsewhere.

Much of this can be taken as a broadly accurate description of the situation of fifteen and even ten years ago. However, the evidence from recent Parliamentary intakes and current selections suggests that it is decreasingly relevant or accurate.

How can we determine progress towards fair chances?

Almost all public debate on this issue focuses on just on the headline number: how many black and Asian MPs there are in the House of Commons.

That is important: the overall under-representation is clear evidence that it has been more difficult for black and Asian citizens to become MPs. But this does nothing to illuminate a central issue: whether the ethnic penalty is being reduced over time, or alternatively, remains stubbornly high so that different strategies are urgently needed.

Looking at Parliament as a whole only tells us so much. The pattern of political careers means that the number of BME MPs today reflects the results of candidate selection contests over several Parliaments since the election of the Father of the House back in 1964: more than two-thirds of MPs have served for over a decade, and one in twenty since before 1979.

So to get a snapshot of whether candidates today have fair chances, we need to look at the cohorts of new entrants – the class of 2005, of 2001 and so on – assessing whether the extent to which BME candidates or women are elected proportionately to their share of the population in new intakes. Because party selection in a winnable seat is overwhelmingly the almost exclusive route to the House of Commons, it makes sense to assess progress in each of the major parties. If candidates had fair chances of selection, we would expect that to be reflected over time in BME candidates to be selected in 1 in 13 seats, and women in half of the seats.

A fairness tipping point?

In the Labour landslide of 1997, four new black and Asian MPs joined the five already in Parliament, all representing Labour (as the sole Asian Conservative lost his seat in the landslide). The number of black and Asian MPs rose from six to nine.

But this increase was solely a consequence of Labour strength and Tory weakness in the new Parliament. This obscured the lack of progress towards diversity in the Labour Party in 1997. Labour had elected 183 new MPs to Parliament in the 1997 landslide. So only 2.2% were from ethnic minority backgrounds. Strikingly, Labour fielded only 13 non-white candidates (including the 5 existing MPs) out of 639. Less than 2% of candidates who were not already MPs were non-white. Neither the Parliamentary intake, nor the pool of Labour candidates, were any more diverse proportionately than the pre-1997 Parliamentary party.

In the two general elections after 1997, the number of Labour BME MPs has risen from 9 to 13, and Parliament as a whole from 9 to 15. But this apparently slow rate of change obscures how much progress towards fair chances for BME candidates has sped up.

In 2005, Labour elected only 40 new MPs. This time, 3 Asian and black MPs made up 7.5% of the group. Labour fielded 32 BME candidates in all (5.1%). And that rose to 8% in selecting candidates who not already in Parliament. Both new candidates and new MPs were more than three times more likely to be non-white than had been the case in 1997, even though with Oona King losing her seat, there was only one more non-white Labour MP than in 2001 and only four more than in 1997.

In this Parliament, Labour has held 26 selection contests in seats which the party already holds. In four cases (15.4%) it has selected non-white candidates. It can be confident of winning each of those seats: the least safe would require a 9% swing for the Conservatives in Streatham. And Labour is also likely to regain Bethnal Green and Bow following the implosion of George Galloway’s Respect party.

Overall, Labour is selecting BME candidates at a rate of 10.5%. (This rises to 16.7% in marginal seats where Labour is within 10%, and falls to 8.2% in unwinnable seats).

Conservative progress - while LibDems stall

Just as importantly, this is the first time good progress is being made in more than one party. This is essential if the presence of black and Asian MPs in Parliament is to be entrenched, rather than vulnerable to changes in the fortunes of particular parties. The Conservatives had an all-white parliamentary party after 1997 and 2001, having had one Asian MP (out of 336) after 1992.

However, they did elect 2 non-white MPs in the 51 strong class of 2005. They made up 3.9% of the new intake and 1% of the Parliamentary Party as a whole.

Now, in 32 selections replacing retiring Conservative MPs, the party has selected 3 BME candidates (9.4%), 2 of them women, Helen Grant in Maidstone. Kent, and Priti Patel in Witham, Essex. The party has selected 7 further BME candidates in its 175 target seats (4.3%), 3 of them in the top half of the list. Overall, the party is selecting BME candidates in 4.9% of new selections so far.

These figures – strong progress at the top, and less further down the candidates list - reflect the ‘top down’ way in which the Conservative leadership has tried to catalyse change from a low base. This may not yet fully reached the grassroots.

The LibDems will not elect any minority MPs next time around. They have only briefly had one Asian MP, when Parmjit Singh Gill won the 2004 Leicester South by-election before losing it after 10 months at the General Election. He is the only candidate in anything like a competitive seat.

In nine selections in LibDem held seats, the party selected five women but no BME candidates. The party does select BME candidates at a slightly higher rate (5.6%) than the Conservatives but very heavily in areas with high ethnic minority populations where the LibDems have little chance. The party needs to select a non-white candidate in seats like Winchester, York or St Ives, not just areas where there are high minority populations like Birmingham or East London.

From ethnic penalty to fair chances?

Those figures show that Labour can claim to be defeating the ethnic penalty in Parliamentary candidate selections at all levels (Labour seats, marginal seats and unwinnable seats) this time around.

Warnings against complacency should be listened to to ensure progress can be sustained. And progress in Parliamentary selections does not seem to be being reflected in local politics.

But complacency is not the only risk. We have had the strange situation of the most vocal advocates for increased BME representaion telling aspiring candidates that they do not and can not have fair chances and a level playing field despite the evidence suggesting that they already do.

The evidence does challenge the claim that there could never be fair chances without all minority shortlists. It was a reasonable hypothesis to suggest that what has been shown to be true historically of gender would also prove true of race – but the dynamics of race and gender under-representation could turn out to be quite different.

Indeed, though this should not be overstated, strong black and Asian candidates might just have had a slight edge in Labour selections this time around. That could just be a statistical quirk: the sample sizes are small. But it could also in part suggest that a good part of the Labour selectorates, aware of the historic under-representation, are motivated to challenge this and to vote for strong black and Asian contenders. Nor would that mean that unreconstructed attitudes have been eradicated – but that they can be outweighed by a commitment to equality from party leaderships and members.

If it did prove possible to sustain a pattern where BME candidates were selected over time in 10 - 15% of cases, that would hardly be something to worry about: it would simply mildly accelerate progress towards a Parliament that looks like Britain while still taking some time to overcome the historic legacy of under-representation. (If fair chances in selections and Parliamentary intakes could be achieved and sustained, how quickly this happened would depend on the rate of retirements and turnover of existing MPs, though many may feel that this is a secondary issue if fair chances are in place).

The all minority shortlists debate

I have been a critic of all minority shortlists on the grounds of both their necessity and their potentially regressive effects on race politics.

Current black and Asian MPs are almost equally divided on the issue. To some extent, this reflects a generational division. Those pioneering candidates who had to overcome much steeper barriers to break through twenty years ago are sceptical of claims that so much has now changed. They might well feel that younger candidates are naïve to feel so confident of competing and winning on a level playing field. But the evidence suggests that new candidates – largely the children of the 1970s rather than the 1950s – are right to asset this confidence, though most would acknowledge that they can do so only by standing on the shoulders of those who made the historic breakthrough possible.

All minority shortlists may well have made a difference 20 years ago – when they were not seriously on the agenda – they have become possible at a time when they are no longer necessary, certainly in the Labour Party. And the evidence from the use of all women shortlists is that a ‘ceiling’ effect is a danger, with BME candidates faring worse in other seats.

And they are also largely a red herring in the Liberal Democrats. Nick Clegg challenges his party in stating that he will reluctantly consider all minority shortlists if progress can not be made by other means. But it is difficult to see how all minority shortlists would help the LibDems to advance unless they are going to deploy them in marginal constituencies with very low ethnic minority populations. A different approach – probably more similar to the Conservative A list strategy – is more likely to tackle the problem the party faces of selecting BME candidates in winnable seats.

The Conservative Party’s challenge is not to see increasing ethnic and gender diversity as a one off event as part of the party’s brand decontamination, if progress is to be sustained. A fairly top down approach carries the risk of not being owned by the party, but successive leaders – the process began under IDS and Michael Howard after the election of 37 white men and 1 woman in the Tory class of 2001 – deserve credit for catalysing progress.

Increasingly Labour MPs and candidates including Chuka Ummuna and Sadiq Khan and Jon Trickett have been arguing for tackling race, gender and class cohesively in trying to achieve fair chances across all of these dimensions while avoiding a politics of competitive grievance between the advocates of fair chances for women, black and Asian candidates or those from poorer backgrounds.”

Sunder Katwala is the general secretary of the Fabian Society and will speak at the America Votes, Europe Responds conference on November 8.