Spacers for Swing arms?

I have a 96 2x4 300 fourtrax that I bought recently. The swing arm was not on the ATV when I purchased it, as the previous owner had removed it to replace the bearings. Here's where my dilemma lies...When I put the new bearings in and reinstalled the swing arm, there is about 3/8" gap between the seal on the swing arm bearing and the frame on the driver side, and about 3/16" gap on the passenger side. You can see the swing arm bolts easily.
Is there a spacer that goes in this area for this model?
It does not show on any exploded diagram I have looked at. I have a '98 that I built and it does not have this issue.
Can I just stack some large washers in there?
What's the best fix?
The design of the frame would not let it bend that far, plus the ATV was in pretty good condition, other than the swing arm bearings, which one would think would indicate that it was not abused.
Any thoughts?