If you think that people don't want to be in the meeting, don't invite them, or make it clearly optional.
It's a lot easier to tell someone "I don't think it's necessary for you to join, but I invited you because I didn't want you to feel excluded in case you want to be part of it" than for that someone to say "you know, you required my presence in your meeting, but I think I won't bring you anything and it's a loss of time for me".