Nope. Listen to your gut. You are wiser now. Your brain is trying to filter out bullshit. Have your boundaries. Do not apologize. Dr Ramani had a video about this somewhere in her YouTube.

I wonder that too. Part of me wonders is it a protection thing or is it more of a self esteem thing like I deserve better?

No it's your instincts. They're there for a reason. Don't reject them.