If you're mostly trying to memorize that true implications are true, sure. But the answer to the converse question "Are all invertible matrices unitary?" is "no."
But memorizing the stronger statement that a unitary matrix U has inverse U* is more useful anyway.