Either change collision sets, or if yyou want circle gears, use the special red-blue-circle method(Litter the surface in same size, same collision, no selfcollision circles, every second has liquid collision off)
It looks like the wheel is going too fast and the axle is bending. I'm not sure how to fix it, though. Try using the CSG menu to cut a circle in the wheel and put another circle to use as an "axle".
In the script menu for the axle of the small circle, change "legacyMode" value from 1 to 0 (zero). That, by itself, will reduce the bumping. Another thing to do is to select the small circle and axle and move them away from the large circle. The small circle is bumping around because you have it pressed up against the large circle. Do those two things and the small circle will rotate smoothly.
The only thing you have to do is change the legacyMode of the axle that stretcehs from 1 to 0. If you are doing a circle gear transmission you want the circles overlapped just like you have them