Search for a command to run...
Widely known as fotiecodes, an open source enthusiast, software developer, mentor and SaaS founder. I'm passionate about creating software solutions that are scalable and accessible to all, and i am dedicated to building innovative SaaS products that empower businesses to work smarter, not harder.
No comments yet. Be the first to comment.
Lisbon, If you have ever been here, you know the streets are a wild mix of steep hills, narrow cobblestone alleys, and confusing one-way roads. Navigating this city takes serious skill. The yellow tra
If you have been paying attention to the AI space for the last few months, you know the fatigue is real. We have spent three years typing into a box, getting a paragraph of text back, copying it, past
Understand the distinctions between Retrieval and Cache-Augmented Generation; find out how CAG improves speed while RAG handles real-time information
I’m sitting here with a hot cup of coffee, about halfway through Google’s new paper on "Nested Learning," and I have to be honest: I need to get this out of my head and onto the screen. Usually, when a big lab drops a paper, I skim the abstract, nod ...
I have always wondered how Apple does their spatial photos on iOS and especially on the new Apple Vision Pro. If you have tried the headset, or even just tilted your phone while looking at a "spatial" capture, you know that weird, distinct feeling of depth. It feels like the memory has been lifted out of the screen.
But how do you get that kind of depth from a flat image? And more importantly, how do you do it fast enough that a user doesn't get bored waiting for a loading bar?
I recently came across a new research paper from Apple titled SHARP (Single-image High-Accuracy Real-time Parallax). While Apple rarely comments on exactly what code is running inside the Vision Pro, this paper feels like a blueprint for the magic we are seeing. It describes a system that takes a single photo and turns it into a high-quality 3D scene in under a second.
Let’s break down how this works, why it beats the current trends, and what it means for those of us obsessed with digital memories.
Here is the main issue with 3D reconstruction: you usually have to pick two out of three options:
Fast
High Quality
Single Input Source (just one photo)
Most recent breakthroughs in this space have leaned heavily on diffusion models (the same tech behind DALL-E or Midjourney). Methods like Gen3C or ViewCrafter are incredible at hallucinating missing details. If you show a diffusion model a picture of a house from the front, it can guess what the backyard looks like.
The downside? They are slow. We are talking minutes to generate a scene. Plus, the quality can get a bit "dreamy" or blurry when you look closely.
SHARP takes a different route. The researchers at Apple weren't trying to let you walk around the entire house. They wanted to support "nearby views." Think about the experience of looking at a spatial photo. You aren't walking into the photo; you are tilting your head, shifting your posture, and seeing around the edges of the subject.
For this specific goal, SHARP is a monster. It generates a 3d representation in less than a second on a standard GPU and renders it at over 100 frames per second.
Instead of building a traditional mesh (triangles) or using a heavy Neural Radiance Field (NeRF), SHARP uses 3d gaussian splatting.
If you aren't familiar with the term, imagine throwing millions of tiny, semi-transparent colored blobs (gaussians) into 3d space to represent an object. It’s a technique that has exploded in popularity because it renders incredibly fast.
The SHARP network works in a single "feedforward" pass. You feed it one image, and it spits out about 1.2 million of these 3d gaussians.
The system is clever about how it gets there. It uses a mix of pre-trained tools and custom modules:
Feature encoder: It breaks down the image using a backbone from "depth pro" (another impressive project).
Depth decoder: It predicts two layers of depth. Why two? To handle occlusions, like when an arm is in front of a torso.
Gaussian decoder: This is the heavy lifter. It refines the position, scale, rotation, color, and opacity of all those tiny 3d blobs.
One detail I loved in the paper is how they handle the fact that guessing depth from one photo is basically impossible to do perfectly. Is that car small and close, or huge and far away? a computer often struggles to tell.
SHARP includes a learned depth adjustment module. Instead of just guessing and hoping for the best, the network learns to find a scale map that resolves these conflicts during training. It basically acts as a reality check for the depth estimation, ensuring the final 3D scene doesn't look warped or stretched.
The paper compares SHARP against several state-of-the-art baselines, including those slow diffusion models I mentioned earlier.
The results are stark. When measuring for perceptual quality (using metrics like LPIPS and DISTS), SHARP reduces error rates by 25–34% compared to the best prior models.
But the real killer feature is the efficiency. It lowers synthesis time by three orders of magnitude. In the time it takes a diffusion model to set up the scene, SHARP has already finished the job and you are actively looking at the result.
The paper notes that while diffusion models are great for "hallucinating" views from far away, they struggle with the sharp, photorealistic details needed for the kind of subtle head movements you make in a headset. SHARP keeps the fine structures crisp.
Going back to my original curiosity about the Vision Pro, this paper connects a lot of dots.
Apple mentions explicitly that the goal is to support "interactive browsing of personal photo collections." They want you to be able to swipe through your library and see 3D instantly. If the tech took 30 seconds per photo, no one would use it.
By using a regression-based approach (one pass through the neural net) rather than an optimization approach (churning on the same image for minutes), they make the feature usable in real-time.
The limitations are clear, of course. You can't turn around and see what's behind the camera. but i don’t think that is the point, the goal is to provide a "headbox" that allows for natural posture shifts. It anchors the virtual camera to your physical movements, making the memory feel solid and real rather than like a flat sticker floating in space imo.
It is fascinating to see the research that likely powers the features we take for granted. We often think of "spatial computing" as just better screens, but the software stack required to fake depth from a 2d jpeg is incredibly complex.
SHARP shows that you don't always need the heaviest, trendiest ai model (like diffusion) to solve a problem. Sometimes, a highly optimized, single-pass network using the right representation (3d gaussians) is the better tool for the job.
Now, every time I tilt my head while looking at a photo on my iphone, I’ll be thinking about those 1.2 million tiny gaussian blobs adjusting in real-time.
below is the link to the official paper and github repo;