Your Favorite Common Low-Denomination Coins?

New Zealand 1,2, and 5 cent pieces. You should be able to pick up nice examples very cheaply.

1 cent last issued for circulation in 1987 (1988 in sets only) and demonetized 1 may 1990

2 cent last issued for circulation in 1987 (1988 in sets only) and demonetized 1 may 1990

5 cent last circulation issue 2004 seen in sets 2005, and 2006.

Funny thing is in 1967 when NZ went to decimal currency the following coins were issued:

1 cent
2 cent
5 cent
10 cent
20 cent
50 cent



Now NZ has

10 cent
20 cent
50 cent
100 cent($1.00
200 Cent($2.00)

All NZ has to do now is issue a circulation five dollar coin and then move the decimal point one place to the left and NZ would be back to the 1967 point again

By "discontinued", do you mean ended production or demonetized? Judging by your mention of the Canadian "penny", it's probably the former.
And if it's indeed the former, you might try to acquire a Russian 1 copeck (last produced in 2009). They were worth 0.03 of a US cent when discontinued (maybe 0.035-0.04 of a modern cent), making them among the lowest face value coins ever produced (AFAICT, discounting extreme hyperinflation cases like Yugoslavia and Zimbabwe, only the Philippine 1 centavo went lower, and that one was a coin only produced for accounting reasons; the Russian copeck actually circulated, and indeed still occasionally appears in circulation).
Another possibility would be the Czech hellers (i.e. all the coins below 1 koruna - 10h, 20h and 50h). Also, Israeli 1 agora and 5 agorot (the 1 might be tricky to find though).