Gills' chairman Paul Scally says the practice of footballers taking the knee before matches should come to an end.
He was speaking this week after a portion of the 10,000-strong crowd booed England players on Sunday.
The Three Lions' stars led by stand-in skipper Marcus Rashford knelt ahead of the clash with Romania.
For the second match in a row, boos could be heard coming from sections of the crowd, as well as clapping.
Priestfield chief Mr Scally fears the gesture is "irritating" normal, non-racist fans and called for it to be brought to an end.
He said: "I'm totally against it. The point has been made. There is no need to do this anymore.
"It is having an adverse effect on football.
"The vast majority of people in this country are honest, decent and non-racist people. They are against racism. They are finding this somewhat irritating and unnecessary."
Gillingham players have taken part in the peaceful protest during their League One campaign this past year.
England manager Gareth Southgate urged supporters to back the players' stance before the game which ended in a 1-0 victory to his team.
Jeers could also be heard from a minority of the crowd at England's friendly win against Austria on Wednesday last week.
Prime Minister Boris Johnson has backed the knee gesture, despite concerns from members of his party it may reflect far-left political tones following the murder of George Floyd in America.
This Sunday, England will play Croatia in their first Euro 2020 game at Wembley before 22,500 fans.
The gesture of players taking the knee ahead of kick-off has become a familiar sight in English football since last summer.
It aims to highlight racial equality and highlight social justice issues faced by the black community.
Cory Boothman is a safeguarding officer for Medway Culture Club, which promotes diversity.
He says the booing is "disappointing and disheartening", likening taking a knee to a two-minute silence.
He said: "If people disrespect the silence, it gives off the same feeling.
"Booing is discrimination, taking a knee is a stance to be respectful about the racism people are facing in football.
"Some people don't see the correlation between taking the knee and the two-minute silence but it is really important.
"I feel booers need to be educated. Campaigners should engage with fans and ask them to attend sessions about why taking the knee is important or perhaps be banned from the games for life."
He believes racism is prevalent in football and has been for many years.
He added: "Taking the knee is deeply embedded in that, it is the little stand-off against racism that can make a massive difference.
"We shouldn't brush taking the knee under the carpet, we need to kick the discrimination out of football.
"I like to hope it is only a few people that are booing.
"When the stadiums reopen fully there may be more due to power in numbers.
"It must be really hard for black footballers as all they want to do is play for their country and racism can really affect them negatively."
The 39-year-old believes the booing is due to a mixture of different factors – people "might be racist and they might not understand the politics involved either".
"I feel that booers lack education, there is a lot of confusion with politics with the BLM movement.
"There is a long way to go but at least we are going in the right direction by taking a knee, people are now more aware of racism in the sport."
And this week Mr Scally also told fans he can't afford to refund season ticket money.
He says the issue has given him sleepless nights but, in the current climate, Gills simply don’t have the money.