In my experience (as someone who is trying to put together a complete set of gem-quality moderns), the worth of collecting modern circulation strike quarters depends on grade and eye appeal. The higher grade and better looking, the more worthwhile.
2005 or so to date are very easy to obtain, and there appears to be a lot that get third party graded at MS66 or 67. Whether this is grade inflation, or from careful harvesting of mint rolls, I can't say.
2001 to 2004 are fairly easy to obtain in MS65, but not in MS66.
1999 and 2000 Millenium series circulation strikes are fairly difficult to obtain in MS65, and full Trends prices will likely get you outbid on eBay. MS64's are fairly easy to get.
1980 to 1996 are for the most part quite difficult to obtain in MS65, and quite expensive. I think these didn't get much love from collectors when they came out, so not very many real nice ones got set aside (or the RCM
was having quality issues). Much easier to obtain in MS64.
1968-1979 are reasonably easy to obtain in MS65, and significantly less expensive than the 1980's. MS64's are even more common.
If you don't care about whether or not its a circulation strike and just want to fill holes, you might want to consider the mint sets for 1992, 1999, 2000 and the Olympics. These look very sharp and can usually be found for usually around $12 or so per set.