Yes take them off,but you have a few options.
If doing it yourself at home-
You will need
1 can of oven cleaner
1 can of aircraft stripper
220 -300 sand paper
wire brush
Your paint(spray can)
You can go to bare metal and use the anodize paint that looks great.
I would use a wheel paint as it should hold up much better.
Option 2 (best option)
Take them and have them sand blasted and powder coated,should run $75-$100
Call your local PC co.
You can buy an old oven of CL for free-$50
and google diy power coating.
If your going to PC a good amount of parts then worth the money.