Question about sugar consumption?

so i know that sugar is terrible for you and is basically empty calories, so i basically don't eat anything sugary (like chocolate or cookies) except for one day a week (usually friday or sat). however, i notice that lots of the healthy things i eat contain lots of sugars... like, i eat tons of fruit a day, so is this bad just because of the sugar? or even in cereal there are sugars... is it that some sugars are ok for you and others aren't? like, refined sugars are bad but other sugars can be alright? because it's nearly impossible to limit your intake to 40 gs a day, even if you don't eat anything sugary.