I'm kinda new but from common sense and what I've seen thus far, some gear is better than no gear. Are you going to ditch the jacket and ride with just a t-shirt anyways? Vest is better than a t-shirt. Who cares what you look like.

I was considering, since it gets 120F in the summer time where I live, getting a vest, and elbow and knee pads, but the cost of all of it ended up being a bit more than just getting a jacket.