When we examine the problem closely, we find that "time" is not the unitary phenomenon we may have supposed it to be.
Backlogs make everyone feel good. The trouble is that 90% of the things in the feature backlog will never get implemented, ever. So every minute you spent writing down, designing, thinking about, or discussing features that are never going to get implemented is just time wasted.
There is a saying to the effect that there are three variables in engineering: time, functionality, and resources -- pick two. In fact, there is a fourth variable: debt.
The cost of paying back technical debt comes in the form of the engineering time it takes to rewrite or refactor the code or otherwise fix the problem. If the interest you ultimately accrue is less than the cost of paying back the debt, there is no point in paying it back in the first place. The problem is that it can be difficult to know in advance which debts will ultimately have the highest cost.
Human beings cannot comprehend very large or very small numbers. It would be useful for us to acknowledge that fact.