Reece Stone, 13, from Gravesend, has been keeping the birds for three years.
However, he has had to get rid of the shed where they lived after a neighbour complained and it was discovered the family had not sought planning permission.
Reece Stone, 13, has been told he cannot keep his racing pigeons by Gravesham Borough Council
The Stone family had just moved into their council home when the structure was built.
They claim they told council officers about Reece's birds when they were shown their new property and built the shed to house them, believing it would not be a problem.
The family have now taken down the pigeon loft after being given a deadline by the council.
Reece's dad Keith said the Ifield School pupil was "really upset".
Since the trouble with the pigeons began, he said his son - who has special needs - has been feeling down.
Reece Stone with one of his beloved racing pigeons
Reece is a member of a pigeon racing club that meets at Gravesend Rugby Football Club, in Donald Biggs Drive.
Mr Stone said: "It's made him feel down and affected his work at school.
"It's been put in his book that he doesn't concentrate because of his birds. It's affecting his learning.
"The birds give him responsibility and it helps him with his numbers, he gets so much out of it. He lives and breathes for his birds."
Reece has had to keep the pigeons cooped up in crates for the past two-and-a-half weeks – conditions in which they should not be stored in for more than two or three days.
Some of Reece's racing pigeons at home in Gravesend
The youngster often races his pigeons in tournaments, but has had to cancel his registration with the Royal Pigeon Racing Association because he changed address.
He cannot renew his registration and start racing again until he has proper conditions in which to house the birds.
Reece said: "I feel upset. The birds make me happy and they're important to me.
"I like to race them over to France and I enjoy being part of my racing club. It keeps me out of trouble.
"I'm worried about my birds because of the poor conditions and they're not getting out."
The Stone family have written to the RSPCA about their concerns over the health and safety of their winged friends, hoping they will help them fight their cause.
A Gravesham council spokesman said: "The tenant had no permission to build a structure to house pigeons at either council property. This is against the tenancy agreements which the tenant had signed.
"After receiving several complaints after the construction of the latest large outbuilding, the council advised the tenant that they were in breach of their tenancy.
"The tenant has formally complained to the council after we requested that the building be removed and was told that they had to seek permission. Keeping up to 12 pigeons is allowed under the tenancy provided they are kept in a purpose-built loft and provided permission is granted.
"The council understands that the tenant intends to keep about 90 pigeons and has never asked for any permissions. The tenant was told they must remove the outbuilding by January 14 or face legal proceedings. They have the right to appeal.
"The council has sympathy for the tenant but has to bear in mind the number of animals proposed to be kept and the effect on others."