Norwich signed Gilmour on loan from Chelsea in the summer as the Scottish international searched for regular first-team football.
Gilmour made a significant breakthrough last season under former Chelsea manager, Frank Lampard, and was regularly featured under current boss, Thomas Tuchel, but was jeered by his own fans during Tuesday’s 3-0 defeat at Crystal Palace.
And Agbonlahor feels the January transfer window will be the right time for Gilmour to leave Norwich and return to his parent club.
“I can’t believe the audacity of the Norwich fans to chant that. I hope it is a small section from the away fans, I hope that home fans wouldn’t sing that,” Agbonlahor told talkSPORT.
“He’s a 20-year-old footballer, we have seen what he’s done in a Scotland shirt, seen what he’s done for Chelsea.
“If I am Chelsea, I am bringing him back now. Pack your stuff, get yourself back and we are going to find you a club in January that are going to play to your strengths.
“He is playing with Championship players, they are not good enough. He needs to play with players who want to play football and his sort of football.
“He is going to waste six months of his career staying there until the end of the season. I would drag him back.”