It most cases it is a matter of opinion. Personally, I find that if the countermark does not deface the obverse or reverse, it lends more credibility and some idea as to where the coin traveled and when. They can make the coin's history more interesting. Also, there are collectors of coins with countermarks, so they would tell you it makes them more valuable to them.