Is this a sign of something?

Sounds like overfeeding which will lead to the buildup in the gravel as well as mess with your nitrate/nitrite levels as the mess decomposes.