I would speculate that there are no less than a hundred thousand collectors who live, breathe, and die for ancients, and probably millions or more who have at least one or two ancient coins in their collection.
I only have two or three ancient coins. It is a field that I would love to expand my collections in -- but the barriers to entry are many and sometimes high.
Some thoughts from my inexperienced point of view:
It seems that there are many, many fakes, and they can be very difficult to detect.
The books and literature to attribute your finds are expensive and often out of print.
Many sellers are not located in the US, resulting in large shipping fees and customs issues.
A large majority of ancient and medieval coins are not certified by a reputable
TPG, which discourages modern numismatic investors who are hesistant to collect and invest in ancient and medieval coins without an expert having looked at the coin and declared it genuine/correctly graded/attributed.
The history behind ancient coins is not emphasized. (from the Phoenicians, Greeks, Byzantine Empire, Celts and Romans, the Saxons...the great Islamic conquests and the Crusades to take them back, the Holy Roman Empire, the German States, the Spanish & French & English kingdoms...most of this is not taught to our children in school anymore, indeed ever, except perhaps very briefly mentioned.) Without this to encourage young collectors' interests and demand for coins there is not much impetus for young numismatists to start ancient/medieval coin collecting. There are great stories to be told, but no one really tells them anymore.
There are not really very many standardized price guides. Modern coin collectors in the US are used to having many price guides that tell them wholesale and/or retail values for dates and mint marks. This is a barrier to bidding/buying ancient and medieval coins because you often have no idea if the price the seller is asking is a great deal, a reasonable price, a bit high, or way overpriced. No one wants to spend $200 on a coin and then find out that you can buy them all day long for $30. The counterpart to this is that dealers are reluctant to stock ancient inventory; they don't know much about it either, and they have limited resources for pricing and margins. This negatively affects both acquisition and resale. As a result, ancient coin dealers tend to focus only on ancients and specialize in that area, isolating them from other collectors who might become interested in ancients if they saw the coins in showcases and books at their local coin shops.
There is a lack of newsstand publications which focus on Ancient/Medieval collecting. The
ANA's Numismatist occasionally runs articles, but there is nothing on the order of a COINage or
Coin World or
Coins Magazine for ancient coinage.
A number of unscrupulous sellers feel the need to flat-out mislead non-informed collectors about some things over and over: apparently every medieval Spanish coin or bit or cob was "pirate treasure", every prutah or lepton is a "Bible coin", cleaning that lot of dug LRB's is an easy and fast process that will let you find fabulous rare coins for cheap, etc.
The denomination and composition of popular ancients such as Roman/Greek coins can be difficult to grasp without a bit of effort; diobols, tetradrachms, dupondii, denarii, tremisses, solidi...etc. This leads to mislabelling and misidentification by sellers and buyers. Silvered AE's get sold as AR's, there's billon, copper, brass, electrum, and the various gold alloys.
Finally, pricing for popular ancient coins in general tends to be quite expensive, perhaps excepting antoniniani/radiates/LRB's; with middle to higher grade coins, even uncertified, going for $50 to $500 or much more. This discourages young collectors without high disposable incomes from joining the hobby of ancient coins; even a theoretically-simpler set such as "All Roman Emperors after the year 200 AD" is going to hit you in the pocketbook a bit.
Just my opinions, and hopefully not too much of a sidetrack.