When Does A New Century Begin For A Numismatist?

Solstices and Equinoxes "work" because of Leap Years.

This is not really accurate. First, as mentioned, solstices and equinoxes work because of leap years. Second, the Gregorian calendar only sets rules for leap days, it is by no means accurate down to a nano-second (irregularities in the earth's rotation will see to that). We started using leap seconds long after Pope Gregory was dead and buried (they actually started in my lifetime).

But just carl, what do I do with my 1900, 2000 and eventually,
2100 year coins?

I think I will be going with the mathematical (?) method and put
1900 in the 20th century and 2000 in the 21st century. It just seems
to make more sense.

I'm in the "01" starts the century and "00" ends it camp.

A new century begins on the fist day of the first month of the first year of the century. thus it begins on 01/01/01. 001-100 fist century. 101-200 second century and so on

On his/her 100th birthday.

The first century is obviously 1-100 (and yes, there are coins from it, even dated ones, though naturally those dates are in non-AD calendars).
That means the second century is 101-200, the third is 201-300, and so on, up to the 21st century being 2001-2100.

As for the 2000s, to me this refers to the decade that ended in 2009 (similarly the 1900s were 1900-1909, and so on).

I like DBM's idea, incidentally.

whats a year among a 100
my 20th century type set goes from 1900-to 2000