You don’t have the time if that “empty” time is allocated for recovery. Downtime is not free time.

Time allotted for rest is still time that is already accounted for which is unable to be spent elsewhere.

There shouldn’t be any confusion.