Characterization of Smooth Symbol Classes by Gabor Matrix Decay