I also don't really understand why there should be any premium attached to a serial number...since it is simply a result of sequencing...
That's a fair thought. I'm guessing you don't own a serial #1 note or a solid serial number or a full up (or down) ladder serial number. You're absolutely correct though, it's a result of sequencing. I wish more collectors thought like you so I could finally afford one of these notes.
To the OP, welcome to CCF. I recommend you collect what YOU like. If you like it, keep it. What's the downside ?