Yes, and also depending on the buyer. Some forms of PMD can increase a coin's value (a rare contemporary counterstamp, for example, or a well-done hobo-nickel carving), but that's getting a bit picky about the definition of "damage".
I agree with you both. It's tough to put a value on any coin with damage. Some folks will buy a coin with damage because they need it but can't afford a problem free coin. Some won't even look at a problem coin. You get a better idea of value when you try to sell it.
I have a large cent with two crudely carved holes. My guess is it served as a button. My mind creates a scenario of a Civil war soldier needing a button to pass inspection.
It does range with the degree of sensitivity of the buyer. From, "That's not bad but you should knock a dollar off for me........" To, "I notice when using my 30X loop that the King might have lost an eyelash and it was struck onto the coin. Can you give it to me for half-off?"