If I leave XPO and have a balance in my HSA, do I have to keep my account with Fidelity?

No. You can transfer your balance to another HSA administrator or leave it with Fidelity. It’s your choice.