That can be the case if the router is set to automatically assign to any service type, because then it can connect to 2G/3G instead of LTE. Depending on the model of the router, you need to go to its interface page and enter Internet settings – there you can change Preferred service type from Automatic to LTE only/4G only, and that will force the router to connect only to LTE, hence improving speed and stability.