People think I got a soft voice on the phone, what should I do?

Over the phone, people I am talking to for the first few times often ask me to repeat myself. It can be frustrating hearing this so I should do something about it. But when I am talking confidently I'll still get it sometimes. I've been attending a speech club too. I'm wondering if it's confidence I need the most because even when I'm being treated rudely on the phone, I feel like I'm the mature person by being able to take it and talk with a strong voice. If you can help I'd appreciate it. Please no mean answers.