i took mine off. the bike looks a hell of a lot better now. i think i used a 5 or 6mm allen key. fit perfect

 

had to drill it out with an ez out...turned out to be a 6mm which didnt come in the allen key set I bought, stripped it with a 7/32

 

I'd guess you had an incomplete set of metric's then... happens with a lot of sets. Try and find a 23mm, often not around. Sort of like trying to find a 1" 3/16

A 7/32 might have fit better than the metric set you had - but that didn't mean it was the right size.

 

+1