Collecting Discussion - Key, Semi Key, Or Commons First?

My idea of collecting is, start with the commons first, if you can find them, but don't shy from keys and semi-keys just because they are so, especially if they appear for common prices.
(I'm more of a type collector than a date collector, so this is not as much of a problem to me, but key types - think 1859 cents - are nearly as much a thing as key dates, especially in earlier series, just harder to formalize directly.)

It is also sometimes the case ( Flying Eagle cents and, to a lesser extent, 20 cent pieces are examples among US series) that all dates are either relatively common, or extremely rare. You're not going to start your FE cent date set with a key date unless you get really really lucky, and/or have an awful lot of money.
(This is almost the case with Standing Liberty quarters too - but in that case there are a few other big keys and semi-keys as well; you don't need to hunt for the 1916 when you can start with the 1927-S or whatever, and get the 1916 later when you have that kind of money.)
There are also similar cases in many modern series when the key dates are NIFC (there's not much of a practice in the US of doing set-only coins, aside from S-mint proofs, so perhaps the closest analogue is the 1996-W dime; there are, however, a few such recent Canadian coins). In this case, you're probably going to buy a set to get that NIFC date anyway (if you even include it in your list), so there's no real reason to start with it, even if it's technically the key.

It really depends... I think really it's more of a "feel" thing... when you see coins you like, buy them for your set. It might be a $2 coin, or it might be a $500 coin. Be flexible with your budget, but not with what you're looking for.

If someone wanted to collect Barber dimes, would you recommend them to wait (and save up) for an 1894-S before trying to get any others? If it was Liberty nickels instead, would you advise them to start with an 1913?

There really are cases where the key date is, in fact, nearly impossible. In the more modern series it's at least usually theoretically possible to find any key date if you have enough money - in earlier series it can easily be that a particular year or variety literally isn't available for sale for a decade or more unless you hound the current owner (and even then you might have a better chance of getting a restraining order than the coin you need). I think there's been a variety that got recently offered on public auction for the first time since the 1890s - can't recall what it was unfortunately.

As @january1may indicates, some in a series may be effectively impossible to get. When you are starting collecting a series, clearly you have to "count the cost" of getting the complete set before you start, and/or accept that your set will always be missing a couple.

If I had the means, I would definitely go with the harder coins first and on my journey to find them I am sure I would come across some of the common coins and add them as well as long as they were a cheap good deal. I have a number of holes in my type set missing and once I get back to it I am going to try and just save up for the Seated dollars or the older large cents before getting other stuff (like upgraded walker/barber halves, 3 cent silver, etc)

Regardless of one's choice, "Look before you leap."

Yes, the general advice is to buy the key dates first, and I think that advice is accurate for experienced numismatists who understand grading, coin markets, etc... However, I'm a fairly new coin collector and I immediately dismissed that advice because of my naivete in all matters related to coins. No way was I willing to shell out hundreds of dollars for a 16D Mercury before I really understood grading and the market. Making a newbie mistake on a $5 coin is called learning, making a newbie mistake on a $800 coin is called foolish.