I'll use gloves when putting ASEs into my Dansco, otherwise I really don't like handling coins that way. They feel much less secure than using my fingers. If you do use gloves frequently, make sure they're clean. Likewise for bare hands, of course.
It seems that unless one's hands were completely free of any oils or other substances, even contact with the edge could lead to some issues 30 years down the road. But by that time, who cares?
I would assume that the graders wash their hands fairly frequently. I know if I am doing something with high quality coins (of which I have few), I wash my hands before starting. Gloves present problems from loss of dexterity to damage if the gloves themselves are dirty. I teach a chemistry lab and usually discourage the use of gloves for most procedures, the exceptions being where quite toxic substances are being handled.