If the CFO is saying to use Excel, you use Excel. This isn't the hill to die on, and fighting it makes a horrible first impression if you're new.
The OP didn't really frame how this question came up. But using what all your co-workers use and staying in the same ecosystem has a lot of value, even without sensitive data involved. If the whole company uses Excel, no one wants to deal with the one person using Sheets, and vice versa.