When& Why Did The Specimen Year Set Develop Different Finish

In 1996 the Mint changed to a parallel line background
on the Specimen Sets.

Charlton says it was changed in 1996. It was also changed between 2009 & 2010 which is where the lynx set mule 10c came from. it also says "<the 1996-2009> finish was a variety of that used by the Bullion Department on their maple leaf coinage. However, this finish was surfacing on giftware coinage, so it is thought the time had arrived to again make specimen set coinage a distinct finish, brilliant relief on the obverse portrait and reverse design, with a frosted relief on the legends, and date, all on a lined matte background."

As you say, the regular specimen sets frequently go for less than issue however the bird loonie series and the young wildlife toonie series frequently go for more.

Don't confuse the BU (PL, NBU etc) sets which were made with a 'special' process with the UNC sets that, as you say, basically contain the same coins as rolls. Pre-2006 sets are BU. In 2006 the specialty sets (birthday, O Canada etc) were changed to circulation. In 2011 the basic pliofilm set was also changed to UNC and the production of BU coins was discontinued.

I think they're more popular with US & International buyers. The first bird loonie set was the flying loonie in 1997; both the 'regular' style specimen set and the O Canada set. The next was the family of loons in 2002 for the 15th anniversary. There were 2 in 2004; the elusive loon coin & stamp set and the 'regular' Jack Miner's Goose Sanctuary set. 2005 to date there has been one issued each year.
1997 w/ Flying Loon
2002P Family of Loons
2004P Elusive Loon coin & stamp set
2004P Jack Miner Canada Goose set
2005P Tufted Puffin
2006P Snowy Owl
2007 Trumpeter Swans
2008 The Common Eider
2009 The Great Blue Heron
2010 Northern Harrier
2011 Great Grey Owl
2012 25th Anniversary of the Loonie - Loon with Chicks
2013 Blue-Winged Teal
2014 Ferruginous Hawk
2015 Blue Jay