A cross-party group of parliamentarians has accused the government of inflicting "profound harm" on people being housed at Napier Barracks.
The former Army accommodation in Folkestone is one of a number of sites being used to house people seeking asylum in Britain who would otherwise be destitute.
During investigations detailed in a report published this week, the All-Party Parliamentary Group on Immigration Detention (APPG) found evidence of what it describes as "appalling treatment and conditions" at Napier Barracks, Penally Camp in Wales and the Tinsley House Immigration Removal Centre near Gatwick Airport.
The barracks, part of the wider Shorncliffe Army Camp and which previously housed service personnel for short stays, has the capacity to hold 523 people, but the Home Office refuses to say how many people are currently housed there.
Although the Home Office insists individuals are not detained at Napier, the APPG says conditions there – including visible security measures, surveillance, lack of privacy and poor access to healthcare, legal advice and means of communication – make it akin to 'quasi-detention'.
For survivors of torture, trafficking or other serious forms of violence, such conditions can cause them to relive past abuses and be highly traumatic.
The report, produced by a cross-party panel of 10 MPs and peers, therefore recommends that Napier Barracks be closed as asylum accommodation "with immediate and permanent effect".
Bridget Chapman, a spokesman for charity Kent Refugee Action Network, welcomed the latest report and says it has vindicated opposition to the use of the barracks.
She said: "Many of the people placed at Napier will have been subjected to incarceration and torture in places like Libya, Iran and Eritrea.
"It is re-traumatising to place them in what is obviously an ex-military facility, and in a prison-like setting. The report shows that the impact on the mental health of vulnerable people has been huge and it is unforgivable that they were put somewhere that the Home Office knew would impact on their well-being.
"I note the Home Office is still using the line that the barracks were good enough for British soldiers. Let's be absolutely clear, they weren't good enough which is why they had been taken out of service years ago."
In March, pictures released by the Independent Chief Inspector of Borders and Immigration (ICIBI) showed what were described as "inadequate" and "impoverished" living conditions at the barracks.
A photograph of one part of the facility – described as an isolation room – showed damaged brickwork and peeling paint and was described as "unfit for habitation" by the ICIBI.
The local authority expressed its disappointment at the decision, as did MP Damian Collins.
A Home Office spokesman said: "Military barracks sites were previously used to house military personnel. To suggest they are not good enough for asylum seekers is an insult.
"Residents are not detained at Napier, they are provided with three meals a day and have their basic needs catered for.
"Our new Nationality and Borders Bill will create an immigration system that is fair but firm and fix our broken asylum system.
"This includes using accommodation centres, which will build on existing capacity whilst ensuring individuals have simple, safe and secure accommodation while their claims and returns are being processed."
We have closed the ability to comment on this story due to the number of abusive and racist posts received on this topic.
We appreciate it is a very divisive issue but must ensure our comments adhere to house rules.