It's an idiom. Of course, you can legitimately compare apples to oranges. I, for one, love apples but hate oranges, and I might be able to articulate why if I set myself to it.