Verifying phone number makes it possible to log in without using authenticator !?! WHY

This is so stupid.. I have always had authenticator on so I would not fall victim to a sim swap attack. Recently I verified my phone number on Binance, thinking it would somehow be better for my security, or at the very least now make it worse. Now it is possible to log into my binance account WITHOUT using authenticator, but merely using a code sent to a phone number making me vulnerable to sim swap attacks.. Is this the most stupid ting ever or am i missing something?