I think you are signaling that you feel inadequate about your shooting. My attire (plain shirts) definitely signals that I think jerseys look lame, but I don't get any vibe of any sort from shooters in jerseys. they seem exactly like the people without jerseys except dressed funny.
For what its worth I am quasi sponsored and I have a T-Shirt from my sponsors. I wear them as I see fit at 2 day matches and switch it around every match I go to.
I like being the unique guy without a jersey, and I have had more people come up to me and ask about the companies that support me because they see a difference in a t-shirt vs a jersey, than some of the other people I shoot with.
Maybe its an intimidation thing?