I don't think it's just about dexterity, it's also about frustration level. Some people get easily frustrated, others don't, and I think that's the big difference between hardcore and casual gamers -- the casual ones are the ones who get easily frustrated, which prevents them from enjoying hardcore games, which kind of rely on frustration (dying, learning, and trying again) as a game mechanic.