ymeon Brown and David Lammy went to the same primary school in Tottenham, north London. One went on to a selective secondary school, and became the second black person to hold one of Britain's great offices of state. The other became a journalist who would go on to survey the country's black professional class more thoroughly than anyone ever had – and who remembers being told by Lammy that he could "never become prime minister, because you sound like where you come from".

It was 2010, and Brown, interested in politics, was interning for the then Labour backbencher. "And he was right, certainly at the time," Brown says now.

It's a small exchange, but it sits at the centre of everything Brown's new book, The Good, the Black and the Boujee, is trying to work out. What separated the two boys from the same Tottenham streets was class. And class, Brown argues, is the elephant in the room in almost every British conversation about race.

"In the book, I was trying to reconcile two of what seem to be contrary events," he says, as we chat over Zoom. "On the one hand, Britain's black population has never had more cultural stature and economic capital – in politics, music, film, publishing – but those same years produced the Black Lives Matter protests and the toppling of Edward Colston's statue into Bristol harbour, and black people saying that it is harder than ever to succeed."

Kemi Badenoch, Stormzy, Daniel Kaluuya, Olivia Dean: are these a reflection of a more equal Britain, or are we simply more comfortable looking at elite black faces while nothing structural changes underneath them? Their success is real, but a handful of people crossing class boundaries is not the same as collective progress.

"Black people in Britain are hyper-visible, but also, in many cases, disenfranchised," Brown says. "And that creates a tension – class has been missing from conversations about race."

To get underneath that tension, Brown interviewed about 1,500 members of Britain's black middle class about their education, employment, families, aspirations, spending, social values and, of course, class. He touches on some questions that feel taboo within this community, such as why middle-class black Britons tend to date white people. What his findings reveal is something Brown thinks Britain has never quite reckoned with: the fact that, as he says, "to be black in Britain is so often to be assumed to be working class", and that assumption has masked how the black Britons who come closest to power are actually, disproportionately, middle class.

Not everyone was pleased that he went looking. "Bona fide black elites don't like the idea of this book," Brown tells me matter-of-factly. Some people, once they heard what he was working on, stopped returning his calls. "People felt it was going to be an exposé."

The objection, as he understood it, came from the idea that to be black and middle class doesn't mean you are anywhere near as privileged as if you were white and middle class. "That may be true," Brown says, "but it doesn't mean that we shouldn't interrogate it."

Britain is a nation obsessed with talking about class and allergic to admitting it has any issue with it. Everyone wants their success to look like the product of hard work and nothing else; nobody wants to concede they had a head start. Black Britons, it turns out, are no exception. "It is very rare to hear someone say, 'I'm black and middle class,' in Britain," Brown says. "The most common response I got from interview requests was: 'Thanks, but I'm not middle class.'"

Part of the reason is that class itself refuses to sit still long enough to be defined. It is nebulous and mercurial, with economic and cultural dimensions that don't always point the same way. "The middle class itself has always been quite fragmented," Brown says. "There's always been variety in terms of social, economic and political capital."

Perhaps the most visible example of the changing face of Britain is in politics. Britain has had its first black leader of a major political party; it has had a black chancellor and foreign secretary. And yet, for all that representation has grown, Brown thinks it has brought remarkably little benefit. "Ultimately, these people are representing their class interests," he says. He runs through the roll call: "David Lammy, educated at a school founded by Henry VIII; Kwasi Kwarteng, Eton and Oxbridge; Miatta Fahnbulleh, the current cabinet's only black minister, privately educated, Oxbridge … Black politicians actually have to have more elite credentials than white candidates," he says.

Brown's point is less that every politician on his list was born privileged than that advancement often depends on acquiring the educational and cultural credentials prized by the establishment. Lammy, for example, was raised by a single mother in Tottenham after his father left the family; he attended the sought-after state-funded King's School in Peterborough, which paved his way to university in London, then Harvard Law School.

School turns out to be one of the clearest mirrors of class that black Britain has. Among black professionals Brown surveyed, almost one in four had attended a fee-paying school at some point – and, contrary to the popular image of the scholarship kid, 72% of them had their fees paid outright by their parents.

"Representation has been the end goal, and I think people have found it unsatisfying," Brown continues. For him, the muted response among some black and leftwing voters to Lammy's departure from government illustrated the limits of symbolic progress.

Brown sees this as part of a wider pattern of a black British elite that increasingly lacks political imagination, or a model of success without assimilation and establishment acceptance. "I was a child of the 90s; it was a very radical period," he says. "People were critical of corporate behaviour; my parents and their friends had been galvanised by the anti-apartheid movement. There was a kind of radicalism to black sensibility. But black Britain has now been co-opted by corporate and individualist ideals."

Outside politics, Brown argues that often conversations that are putatively about race are actually about class. He remembers, for example, the debates that used to dominate the heyday of "Black Twitter", about issues such as "proximity to whiteness" – the notion that lighter skin brought you closer to acceptance. "Completely missing from this is the notion of class," Brown says. "Because then who has greater proximity to whiteness, Eton-educated Kwasi Kwarteng, or mixed-race Mark Duggan from Tottenham?"

The arguments over black dating patterns – who is dating outside their race and why – tend to present themselves in the same way. "People primarily date class before race," he says. "It's the most visceral discourse that is happening online, and I thought it was useful to add a data intervention." Brown found that black women, for example, are more likely to be middle-class professionals than their black male counterparts. "What many women were saying was: 'I've got this level of professional success, I've hit these particular class markers, but I can't find a man who also equals this on the terms that I want,'" he says.

Brown grew up in the 90s, in a terraced house on the edge of Tottenham's once notorious Broadwater Farm estate, an area he describes as "a hub of grime, street life and black radicalism". His parents were both second-generation British-Jamaicans – his father a "career criminal" and his mother a childcare worker. He describes early years in social housing, when his mother delivered the Yellow Pages for extra cash. He started his professional life as an assistant archivist at the Black Cultural Archives before joining the Guardian as a researcher on a Scott Trust bursary, and later moved to Channel 4 News, where he is now home affairs correspondent. All of which is to say that he now operates in a decidedly middle-class milieu.

When I ask him how he would describe his class status, he laughs. "Everyone always wants to ask me that!" He describes himself, in the end, not as middle class but as "multi-class", adding: "I'll never really be read as middle class, because of my accent and other things."

Whatever the terms, Brown is part of an emerging, often young, black professional class that hasn't assimilated into the frameworks of white cultural sensibilities. He can be found at book launches populated by the "black literati", and festivals such as Detty December in Nigeria, or Paris's Fête de la Musique – as Brown puts it, a "middle-class cultural life driven by black cultural products".

The Cambridge sociologist Ali Meghji, whose research on Britain's black middle class predates Brown's, argues that black middle-class identity in Britain tends to take one of three forms: those who don't see racism as a significant issue and feel a stronger affinity with people of the same class; those who strategically code-switch to match white middle-class taste; and those who deliberately steer their cultural life towards black diasporic culture.

Brown himself, and many of his subjects, fit firmly into the third category. Like the traditional white middle class, it's a sort of "if you know, you know" crowd. And like any middle class, it is also full of gossip, drama and internecine fighting. ("There's so much beef within the black literati, you wouldn't believe," Brown laughs.) "When I go to a book launch, I don't want to drink warm wine and eat cheese," he continues. "I want to turn up, hear some bashment, dancehall and soca."

Yet the flourishing of this confident black professional culture is only one half of the story Brown tells. "We've never had a greater cohort of black people with this level of cultural or economic capital, but we still have a huge working class that is also, in some cases, disenfranchised from citizenship," he says. He points to unresolved Windrush cases as an example. Black Britain, Brown argues, has become more stratified – even as the country continues to talk about it as though it were a single, undifferentiated group.

That flattening also shapes the way Britain talks about the working class. Political debate routinely treats the "white working class" as a uniquely left-behind constituency, even though black and other minority-ethnic working-class people face the same pressures, often compounded by racial discrimination. The label can turn shared material problems into a racial contest, inviting struggling groups to see one another as rivals rather than people with common interests.

And for all the pleasures, rivalries and contradictions of black professional life, the economic foundations of middle-classness are shifting beneath everyone's feet. Three decades of wage stagnation and rising costs have made the category less secure – and membership of it worth considerably less than it once was. "If we think about what it means to be middle class, whatever your race, at this moment in time," Brown says, "it means to be poorer, less influential, and doing work of lesser stature than 30 years ago."

The irony, then, is that just as black Britain has developed a sizable middle class – and a culture distinctly its own – the middle-class bargain itself is beginning to fall apart.

The Good, the Black and the Boujee: How Black Britain Became Middle Class by Symeon Brown is published by Dialogue (£25). To support the Guardian, order your copy at guardianbookshop.com. Delivery charges may apply.