Be cautious about dropping your agent, but don't be foolish and hold on to an unhappy relationship. Over the years, I've talked with writers who have switched agents. In most cases, they probably did the right thing. In a few instances, the unhappy writer didn't like the second one any better. Or the third or even the ninth.
Here's my advice—which I wish I had followed myself: Ask yourself, "What do I want from an agent?" Do you want someone who emails or phones every day? That's not reasonable, and most writers won't find such a person, unless the writer is getting mega-buck advances.
The agent who fired me didn't have a personality that worked well with mine: I'd call him bluntly aggressive. I wanted an agent whose personality reflected mine—someone I liked personally and assumed editors would as well.
Take time to think through what kind of agent you want. During the period when I knew my relationship would end with my first agent, I seriously looked at the kind of personality I wanted to work with the next time. I needed someone that I could email or phone and bounce off ideas and get a reply such as, "Sounds good" or "I don't think so," and then I could take it to the next step. I didn't need an agent who said, "Send me a proposal, and I'll let you know." I didn't want to invest a large amount of time into a product that wouldn't sell.
There are excellent reasons to fire your agent.
Be sure you choose the right one.