I suggest that:
If Last judge want go away he must close rated duel. When last rated end he can leave.
If a new judge come in rated duel will be open again and the judge that want go away can leave.
If the only judge want go away in few mins he close rated duel and can start a countdown for already started rated match.
If the only judge want go away immediatly or their line is bad, temporanly an expert player or, if necessary, with high rated that not play but is connected and active can take the decision.