The former Palmer and Harvey head office in Hove is to be turned into 92 one-bedroom flats.
Stonegate Homes, a developer from Dorking, in Surrey, wants to convert P&H House, in Davigdor Road, Hove.
Planning law allows owners to turn offices into homes using “permitted development rights”.
The change of use means that planning permission has been granted for almost 300 flats in the area despite the concerns raised by neighbours.
Brighton and Hove City Council told Stonegate: “Asbestos-containing materials within the building are a contaminant of concern.”
The council has set a number of conditions, including plans for assessing and dealing with any contaminants at the site and agreeing a travel plan.
Stonegate said that the site had 110 parking spaces – more than enough for the 92 homes proposed.
It already had planning permission to turn the building into 57 flats.
The higher number will add to the worries of neighbours who formed a campaign group – Hove Gold – to fight plans for high-rise blocks in the area. They want low-rise family homes instead.
But planning permission was granted for 150 flats in four blocks of flats in nearby Lyon Close in March.
And 52 flats are due to be built next to P&H House on the empty plot that was previously the Hyde Housing office.
Just to the west, in Cromwell Road, on the corner of Palmeira Avenue, plans have been submitted for 80 flats and an 80-bed hotel.
Goldsmid ward councillor Jackie O’Quinn said: “I always knew they (Stonegate) had a planning permission but they chose not to do anything about it.
“Then in May or June they had withdrawn an application for 82 flats.
“Then they put in the application for 92 flats and we can do nothing at all because it is ‘permitted development’.
“They say they tried to rent them as offices but cannot, so they convert them to residential.”
The Hove Gold campaigners are concerned about the impact of more people using the number 7 bus, the nearby Charter Medical Centre and adding to the pressure on local schools.