Guido, the general rule is, if it comes off in mild soap and water, then it's "dirt and grime". If that doesn't affect it, then it's not dirt, it's toning, oxidation, patina or tarnish, which shouldn't be removed. There are exceptions to the rule:
Soap and water shouldn't be used on pure-copper and aluminium coins. If those get dirty, you're pretty much restricted to removing "dirt" with a dry brush.
Coins with plastic "goo" stuck to them, from sticky-tape or cheap and nasty coin albums, can be rinsed with acetone or some such similar non-acidic, non-caustic solvent.
Badly corroded coins are likely not going to be worth very much, no matter what you do to them. Some forms of corrosion, such as rust on iron, verdigris on copper and "cancer" on aluminium, has a tendency to spread over time and coins with corrosion like that should probably be treated to stop the coin getting worse.
Don't say "infinitely" when you mean "very"; otherwise, you'll have no word left when you want to talk about something really infinite. - C. S. Lewis