Software development is like gardening. The codebase is like a garden, where some things thrive, while others die and end up as compost. And this is normal. Things that do not work out, should be removed.
Refactoring is 1. a disciplined activity, not a free-for-all and 2. external behavior does not change, tests still pass and do not need refactoring.
Refactoring is needed any time the code design principles are not respected, such as duplication, nonorthogonal design, outdated knowledge, usage, performance, and the tests pass.
It is best to do refactoring at the moment of doing the task. This is because unnecessary dependencies are removed, there is any way not enough time, and it is easier to do small refactorings during the task. If the refactoring takes up more time, it is OK to schedule time for it and make sure it is done during the sprint. Fail to refactor early and risk having to invest more time down the road to fix the problem.