1. The original method uses two cameras on the back, taking a picture from both simultaneously and using parallax to construct a depth map, similar to human vision. This was introduced on the iPhone 7 Plus, the first iPhone with two rear cameras (a 1x main camera and 2x telephoto camera.) Since the depth map depends on comparing the two images, it will naturally be limited to the field of view of the narrower lens.
2. A second method was later used on iPhone XR, which has only a single rear camera, using focus pixels on the sensor to roughly gauge depth. The raw result is low-res and imprecise, so it's refined using machine learning. See: https://www.lux.camera/iphone-xr-a-deep-dive-into-depth/
3. An extension of this method was used on an iPhone SE that didn't even have focus pixels, producing depth maps purely based on machine learning. As you would expect, such depth maps have the least correlation to reality, and the system could be fooled by taking a picture of a picture. See: https://www.lux.camera/iphone-se-the-one-eyed-king/
4. The fourth method is used for selfies on iPhones with FaceID; it uses the TrueDepth camera's 3D scanning to produce a depth map. You can see this with the selfie in the article; it has a noticeably fuzzier and low-res look.
You can also see some other auxiliary images in the article, which use white to indicate the human subject, glasses, hair, and skin. Apple calls these portrait effects mattes and they are produced using machine learning.
I made an app that used the depth maps and portrait effects mattes from Portraits for some creative filters. It was pretty fun, but it's no longer available. There are a lot of novel artistic possibilities for depth maps.
Yes, they will capture them from the main photo mode if there’s a subject (human or pet) in the scene.
> I made an app that used the depth maps and portrait effects mattes from Portraits for some creative filters. It was pretty fun, but it's no longer available
What was your app called? Is there any video of it available anywhere? Would be curious to see it!
I also made a little tool, Matte Viewer, as part of my photo tool series - but it’s just for viewing/exporting them, no effects bundled:
One of the example pictures on TFA is a plant. Given that, are you sure iOS is still only taking depth maps for photos that get the "portrait" icon in the gallery? (Or have they maybe expanded the types of possible portrait subjects?)
My previous comment was about the default Photo mode.
If you have a recent iPhone (iPhone 15 or above iirc) try it yourself - taking a photo of a regular object in the standard Photo mode won’t yield a depth map, but one of a person or pet will. Any photo taken from the Portrait mode will yield a depth map.
You can find out more about this feature by googling “iPhone auto portrait mode”.
Apple’s documentation is less helpful with the terminology; they call it “Apply the portrait effect to photos taken in Photo mode”
https://support.apple.com/guide/iphone/edit-portrait-mode-ph...
I’d thought that would be heavier than just record the depths.
Ah, that makes sense, thank you!
I think there might be a few typos of the file format?
- 14 instances of "HEIC"
- 3 instances of "HIEC"
I could probably make a better camera app with the correct aperture math, I wonder if people would pay for it or if mobile phone users just wouldn't be able to tell the difference and don't care.
Those methods come from the world of non-realtime CG rendering though - running truly accurate simulations with the aberrations changing across the field on phone hardware at any decent speed is pretty challenging...
That's a bit cynical. Blurring the background can make the foreground object stand out more, objectively (?) improving the photo in some cases.
However, you could just make the app connect to localhost and hoover up the user's data to monetize and then offer the app for free. That would be much less annoying than showing an ad at launch or after every 5 images taken. Or some other scammy app dev method of making freemium apps successful. Ooh, offer loot boxes!!!
What real optics does:
- The blur kernel is a function of the shape of the aperture, which is typically circular at wide aperture and hexagonal at smaller aperture. Not gaussian, not triangular, and the kernel being a function of the depth map itself, it does not parallelize efficiently
- The blurring is a function of the distance to the focal point, is typically closer to a hyperbola; most phone camera apps just use a constant blur and don't even account for this
- Lens aberrations, which are often thought of as defects, but if you generate something too perfect it looks fake
- Diffraction effects happen at sharp points of the mechanical aperture which create starbursts around highlights
- When out-of-focus highlights get blown out, they blow out more than just the center area, they also blow out some of the blurred area. If you clip and then blur, your blurred areas will be less-than-blown-out which also looks fake
Probably a bunch more things I'm not thinking of but you get the idea
The blurring is also a function of the distance, it's not constant.
And blowouts are pretty convincing too. The HDR sources probably help a lot with that. They are not just clipped then blurred.
Have you ever looked at an iPhone portrait mode photo? For some subjects they are pretty good! The bokeh is beautiful.
The most significant issue with iPhone portrait mode pictures are the boundaries that look bad. Frizzy hair always ends up as a blurry mess.
i.e. train a very small ML model on various camera parameters vs resulting reciprocal space transfer function.
Just in case you were doing 3d modeling work or photogrammetry and wanted to know, like I was.
I did end up getting the Creality Ferret SE (via TikTok for like $100) for scanning small objects, and it's amazing.
I’ve also heard good things about Canvas (requires LiDAR) and Scaniverse (LiDAR optional.)
I’d be fine with paying for it, but it’s clear that they want to employ basic dark patterns and false advertising.
As another commenter pointed out, they used to be captured only in Portrait mode, but on recent iPhones they get captured automatically pretty much whenever a subject (human or pet) is detected in the scene.
I make photography apps & tools (https://heliographe.net), and one of the tools I built, Matte Viewer, is specifically for viewing & exporting them: https://apps.apple.com/us/app/matte-viewer/id6476831058
FWIW I personally hate the display of HDR on iPhones (they make the screen brightness higher than the maximum user-specified brightness) and in my own pictures I try to strip HDR gain maps. I still remember the time when HDR meant taking three photos and then stitching them together while removing all underexposed and overexposed parts; the resulting image doesn't carry any information about its HDR-ness.
It also reads EXIF to "scale" the image's physical dimensions to match the field of view of the original capture, so wide-angle photos are physically much larger in VR-Space than telephoto.
In my opinion, this button and feature alone justifies the $4000 I spent on the device. Seeing photos I took with my Nikon D7 in 2007, in full 3D and correct scale, triggers nostalgia and memories I've forgotten I had for many years. It was quite emotional.
Apple is dropping the ball on not making this the primary selling-point of Vision Pro. It's incredible.
Anyways, never heard of oiiotool before! Super cool
> Copyright © 2014 - 2025 Mark Litwintschik. This site's template is based off a template by Giulio Fidente [1]
The theme is for the Pelican [2] static site generator.
Lidar is a let down. First I would expect that Lidar would trickle down to non-pro devices. Come on apple FaceID got introduced in iphone X and next year it was in all iphone models. Lidar was introduced in iphone 12 pro and still only pro devices have it. As 3rd party dev it makes me reluctant to make any app using it if it limits my user base by 50%.
I'm also disappointed they didn't improve FaceID or Lidar in the last ~5 years (Truedepth still only 30fps, no camera format to mix 30fps depth + 120fps rgb, still big latency, Lidar still low resolution, no improvement to field of view)
Chimera
The old gpu is an aberration and odd place to skimp. If he upgraded to a newer nvidia gpu it would have linux driver support and he could ditch windows entirely
And if he wasn’t married to arcgis he could just get a mac studio
ArcGIS is the best general purpose solution.
If doing mission planning including flight guidance for aerial operations, FalconView is the least worst.
Telling an experienced GIS user to drop ArcGIS is like telling a somehow-still-living B.B. King to stop using Gibson guitars.
And by "better supported" he probably meant "isn't a huge pain in the ass that breaks every other kernel upgrade due to dkms or the package manager shitting the bed" when it comes to Nvidia drivers. Doesn't matter if it's a 1080ti or a 5090.
Is it in any way relevant?
I chose not to write it earlier but my candid thoughts were that he seemed too old to still be doing this. Its just younger enthusiasts and professional gamers that do this, and the younger enthusiasts eventually get enough money for a mac and choose that.
The PSU is completely overkill, as if he was going to get GPU's. But then he has this completely outdated and old, power inefficient GPU in it instead, which is nothing to brag about and doubly warrants an explanation if the rig is to be explained at all. All while newer GPU's from the same company solve all of his driver and OS problems.
Flabbergasted is the right emotion.
First: the image recognition model is unlikely to have seen very many depth maps. Seeing one alongside a photo probably won't help it recognize the image any better.
Second: even if the model knew what to do with a depth map, there's no reason to suspect that it'd help in this application. The lack of accuracy in a image-to-calorie-count app doesn't come from problems which a depth map can answer like "is this plate sitting on the table or raised above it"; they come from problems which can't be answered visually like "is this a glass of whole milk or non-fat" or "are these vegetables glossy because they're damp or because they're covered in butter".
The labels on your food are +/-20%. If you are analyzing all of your meals via camera, it's probably not too far off that over a week.
And 20% per week is the difference between maintaining your wheight and gaining 1.6kg of pure fat a month. (Or losing that much)
All you need to know is whether your weight is trending how you want it to trend and, if not, adjust calories appropriately. I feel like a clever enough model could take pictures of food, your heart rate and your daily weigh in and suggest ways to achieve your weight goal, like "have less fries" etc.
> I'm running Ubuntu 24 LTS via Microsoft's Ubuntu for Windows on Windows 11 Pro
this is like hearing someone buying yet another automatic super car.
The details he documented don’t seem relevant for what he did and nothing performed would seem to stress even a low end ancient system.
Definitely felt like a brag to me.
Only thing that makes me think otherwise is he also documented the line counts of the scripts. Seems more like a bizarre obsession with minutiae… (would have been more meaningful to document the git commit/branch for a GitHub project instead of the line count!!)