Advice For Newbie

I suggest getting a recent Red Book.
The assumption is you're mostly interested in US coins.
Page through it and see what catches your eye.
Although the prices listed are not particularly accurate they are close enough so that you can determine what will fit your wallet and tastes.

And as another possibility you might want to consider a type set.
One coin from each design that will represent the whole issue.
It's a relatively popular way of collecting.
For instance for Lincoln cents you would have:
-- a VDB cent (1909 only)
-- a wheat ear reverse (1909 through 1958)
-- a zinc-coated steel (1943 only)
-- a memorial reverse (1959 to date)
-- and there are several more recent designs.
Those are of course inexpensive coins but as you go back in time even the most common dates in a design will satisfy your "not too cheap" criterion.

Kanga! You beat me to it.

I would start by getting a higher grade 1921 Morgan in about MS 64 to 65. Those will be in the $100 range. Look at the APMEX page if you can`t wait for a Morgan. If you are looking for a quality Morgan with good quality strike and the like, check out some ebay auctions. I would recommend looking for a certified PCGS or NGC, at least MS 62 for some semi-available issues. If you like toning, I suggest waiting a longer time for one you want. I personally like toning on the periphery or edges of the coin. I wish you luck in your coin career, it will be a hobby you will enjoy for a long time!

I am a cent collector so I will naturally try to steer you to copper collecting. Most Large cents 1793 - 1857 are reasonably priced in circulated condition and are even available in MS. You can get some really cool coins from the earliest days of the U.S. mint for under $100.

You can't go wrong with 19th century gold type coins in AU/BU.