Just to add, despite restricting to a single IP, any other external IP can still reach the logon page of the router. If they enter the correct username/password, it simply rejects and wont let them in. Whilst it achieves answering the question, its not the most secure way of doing it, especially as time goes on, router is no longer supported and weakness(es) are discovered.
The other suggestion of using VPN access to dial into the internal network to hit the management port is more secure but agree this limits who can access it within your office. However, this actually can work in your favour from a security point of view where you can have a dedicated laptop/PC that its sole job is to dial into routers to manage. This allows a process to follow and control who has access to them at a given time.
