In politics, opportunities for supreme power are rare and fleeting. Yet rather than making challengers to Sir Keir Starmer more ruthless, this truth seems to have made them more cautious. The health secretary, Wes Streeting, resigned from the cabinet but did not launch a leadership bid. Rather than provoke a contest, Mr Streeting’s message to Sir Keir was that since his authority was gone, his duty was to depart and enable an orderly transition rather than cling to office.

If the Labour leadership were truly up for grabs, winning it would require opportunism, a feel for elite collapse and a willingness to defy both the party establishment and orthodoxy. Those who successfully seize the crown – Lloyd George, Harold Macmillan, Margaret Thatcher and Boris Johnson – recognise their moment and act decisively. These leaders were also not subject to the Labour party rulebook.

Sir Keir’s grip is loosening, but replacing a sitting Labour prime minister is institutionally and politically difficult. Not least because any successor would still need to unite large sections of the parliamentary party and trade union movement, as well as the activist base and wider membership. In the meantime, Britain faces a damaged prime minister, a fractured ruling party and no clear route out of a political crisis – just as another brutal cost-of-living squeeze takes hold.

If Sir Keir stays in post and Mr Streeting or anyone else wants to challenge him, then under Labour’s rules they need a fifth of Labour’s MPs to back them. Sir Keir automatically gets on to the ballot as the incumbent leader. Getting 81 MPs to publicly back a coup is extremely difficult unless the leader’s sway has already evaporated. Sir Keir could follow Mr Streeting’s advice and resign for a caretaker leader. That may now be the least damaging option. But Labour’s rules make swift succession difficult, requiring candidates to canvass support from constituency parties and trade unions.

Polling by Persuasion UK suggests Labour’s crisis is existential, not a problem of presentation or leadership style. A sharper version of Sir Keir’s politics won’t resolve the problem. While MPs panicked over Reform UK, the polling showed Labour’s voters mostly stayed home or turned left to the Greens and Lib Dems. Many defectors felt abandoned by Labour’s visionless triangulation. Simply replacing the health secretary and ploughing on regardless, as implied in Sir Keir’s letter to Mr Streeting, would be a mistake.

If it is to renew itself, Labour needs a leadership contest. Ideological fights can deepen division. But Sir Keir took office before Labour had resolved questions over fiscal restraint, social fragmentation, whether green transition can raise living standards and whether competence alone can hold together an electoral coalition. That is why figures like the Greater Manchester mayor, Andy Burnham, and the former deputy prime minister Angela Rayner matter beyond personal ambition: each represents a different emotional tone and understanding of state, nation, economy and society. Blocking Mr Burnham’s Westminster return would make Labour look scared of renewal.

In 1968, when Sir Keir’s favourite Labour prime minister, Harold Wilson, looked weak after devaluing the pound, many MPs wanted his chancellor, Roy Jenkins, to take over. Jenkins dithered and the moment passed. Wilson remained prime minister. Labour’s history suggests that politicians frozen by norms or fear rarely wear the crown. Unlike Jenkins, Mr Streeting forced Labour’s succession crisis into the open. Mr Burnham is now testing whether it can become something larger than elite discontent.

Labour is in the deepest trouble. A juicy leadership drama ignites all Westminster-watchers, another spellbinding live-action theatre of rising and falling stars, duels, betrayals of trust, new alliances and old ones broken.

Some would pull back from this vortex. Is regicide absolutely necessary when “stability” is what people and markets say they want and vox pops groan, “Not another one!” After less than two years, with worse turmoil ahead from the Trump war, now, really?

Whatever comes of Wes Streeting’s attempt to trigger a contest, this sixth game of thrones for No 10 in a decade is inevitable and unavoidable. Labour has to confront what voters said deafeningly in the local elections: not Labour and, crushingly, not Keir Starmer. He is in that bourn from which no traveller returns: political death. No one ever came back from such public rejection. Ignoring it is not an option, just wishful thinking. As Mark Carney famously warned, “Hope is not a plan, nostalgia is not a strategy.” Labour needs a plan and a strategy, a chance to start again with candidates laying out their maps and their melodies.

Politics is a miserable business much of the time. Starmer doesn’t deserve this, but with a dignified timetable he must be gone by autumn. The threatening spectre of Nigel Farage means there is no room for sympathy, nor time to wait for change either in Starmer or in public opinion of him: it will never come. Though he has been an unsuccessful – and unlucky – leader, I like and respect the man, but the public doesn’t. Better by far if he doesn’t fight a contest, but when the time comes moves to the Foreign Office, where he’s best suited – a uniting gesture just as when Ed Miliband was invited back to the frontbench.

No rule says Labour will bounce back under a new leader, even though Andy Burnham (alone) has a net positive rating for popularity, and now, courtesy of Makerfield MP Josh Simons, a possible route back to parliament. Like Starmer, no contender beats Farage as “best prime minister” in polling this month. That means little. Until a leader arrives with a new agenda and new style, what does anyone know? What we do know is that more of the same is certain annihilation. Labour members realised it as they saw their councils fall.

The king’s speech was dismissed as almost irrelevant, it being unclear who “my government” will be in a few months’ time. That’s a mistake. Starmer rightly called his measures radical. Powers to fast-track EU agreements, restrictions on council house sales, the ending of new leaseholds, bans for “conversion therapies” for gay and transgender people and an overhaul of Send provision – these are small emblems of what is happening already.

To recap: in under two years Labour has ended the two-child benefit cap, projected to take 450,000 children out of poverty, provided breakfast clubs for primary schoolchildren in England, made 500,000 extra children in England eligible for free school meals. Free nurseries liberate families to work, with a crucial early years educational boost and up to 1,000 new Best Start family hubs in England. The young have come first, with arts back on the curriculum, a lost youth service restarted with 250 centres to be built or refurbished in England – and new further education colleges with extra construction courses and apprenticeships. This is Labour turf; it’s not nearly enough, but it’s a start at tackling neglected training for the half of young people in England not bound for higher education. British Steel is set to be nationalised, alongside train operators, with rail fares frozen in England and pay-per-mile road pricing for all electric cars by 2028.

Business and its press protest at a real living wage raised by 6.7%, plus a 4.1% increase for the minimum wage, shifting power towards working rights that ends zero-hours contracts and beefs up union recruiting. Rightly the national insurance increase to raise £25bn fell on employers not workers. The Renters’ Rights Act protects 11 million people in England from no-fault evictions. Net migration plummeted by 78%. Green energy surges ahead with priority investment. Today’s NHS waiting-list figures show the fastest fall in 16 years, aided by more than 2,000 extra GPs and 170 new community diagnostic centres.

No space here for everything, but Starmer’s government has done the good that only Labour governments do. Enough? No, but it’s dangerously frivolous to dismiss its advances. How unjust that only 26% feel that Starmer has brought any change compared to its Conservative predecessors; 60% feel there’s been little to no change. That’s plain wrong.

Expectations that “change” made by politicians can lower the cost of living may be impossible. After 14 years of an austerity that stripped the country bare and economic blows that left standards of living stagnant for 20 years, there will be no quick return to the days of yearly improvements and children destined for lives better than their parents. The International Monetary Fund warns the Iran war will hit Britain harder than any other industrialised nation. The UK inflation rate tops the G7 chart. Brexit has lost us a colossal 6 to 8% of GDP.

A wise new leader would be honest and summon a sense of emergency for radical remedies. Ask the Institute for Fiscal Studies director, Helen Miller. She calls for a big-bang tax reform: everything all at once rather than picking off one thing at a time. A great property tax renewal, abolishing stamp duty, beginning again on council tax, ignoring southern losers when more would gain elsewhere. Others suggest bonds to pay for defence and housing would attract savers, with exemption from inheritance tax, yielding more than the Treasury would lose. A one-off wealth tax could raise £160bn, a shock-and-awe levy to set a new tone. Time to abandon the pension triple lock that will cost £15.5bn by 2029: spend that on housing young families instead.

Some essentials cost nothing: electoral reform, abolishing the House of Lords, while accelerating rejoining the EU. It’s baffling that this government has been so timid. In a stagnant time when voters reach for anything new, Starmer’s caution was the wrong message for this era. That’s the lesson for the next leader. But make sure voters know all that Labour has already done.