I think the coin was labelled correctly when you bought it. Whenever I see a "VF-XF" type label or see a coin with that kind of difference in the wear between obverse and reverse I treat the coin as the lower grade when valuing them in my collection. So I'd be calling it a VF if I owned it even if the obverse was XF...but that's just me :)
Nice coin, I just saw one of these yesterday and am thinking of purchasing.
Thanks for sharing!