Security with cell phone / ledger live app

Hi , I’m new to bitcoin . I recently purchased a ledger and was in there process of transferring my bitcoin from the exchange to the wallet . From what I understand if your transfer your bitcoin to a hard wallet the only way someone could steal it is if they know your seed phrase. However, with the ledger live app if someone stole my phone couldn’t they just use the app to transfer the bitcoin to there address and everything is stollen. Therefore, they would not even need access to the actual wallet just the app. I feel like I’m missing something .