I was just waiting for someone to make a comment about their "testils", since nobody will want to use a diminuitive form after reading this...
I prefer "dentils" because it is shorter to type and to say and because "denticles" sounds pretentious to my ear for no reason other than my personal prejudices. But I accept both as correct.
You apparently realize it but not sure some others do. When you trace the word dentil to its Latin root, denticle is the diminutive form. In other words, denticle has been nothing more than a small dentil for a couple of thousand years now.
And for some reason I just have to be different. I always called them denticals. Which apparently isn't even a word.
Occasionally I see the word "dentilated" used in the following fashion: "The coin's obverse rim is dentilated whereas the reverse's rim is plain. You will find "dentilated" and "dentilled" in dictionaries as accepted usages but I don't find "denticled" or "denticlated" listed at all. Not that any of this matters - use whatever you please. English is a very malleable language that changes all the time.
Happy to see the minutia of formulaic grammar are critically important discussions on this forum. I am beginning to feel right at home, and thanks for the welcome @GenX Enthusiast . He is the preeminent forum Forensic Grammatician from what I've been told . . . . . . Z