Lets see if the new kid can perform here
Of course you can get any of the above arbitrarily wrong, but supposing you don't, you have performance baseline. You can get cute and prefetch a few images or maybe downsample them and store the thumbnails somewhere, but that's something users might be unhappy about.
I don't know, can you handroll a jpeg implementation that can beat libjpeg?
Personally the image viewer I'm happiest with has been Quick Look in macOS, largely down to performance.
Memory leaks could happen anywhere in the process. Allocate a surface for the bog standard decoder to draw on, forget to release the surface, boom you have a leak.
It possibly doesn't help that raw files are likely to be especially heavy.
burnoutdv should report a bug if not done already.