pearsonkyle / Sharp-coreml

Sharp Monocular View Synthesis in Less Than a Second (Core ML Edition)

This software project is a communnity contribution and not affiliated with the original the research paper:

Sharp Monocular View Synthesis in Less Than a Second by Lars Mescheder, Wei Dong, Shiwei Li, Xuyang Bai, Marcel Santos, Peiyun Hu, Bruno Lecouat, Mingmin Zhen, Amaël Delaunoy, Tian Fang, Yanghai Tsin, Stephan Richter and Vladlen Koltun .

We present SHARP, an approach to photorealistic view synthesis from a single image. Given a single photograph, SHARP regresses the parameters of a 3D Gaussian representation of the depicted scene. This is done in less than a second on a standard GPU via a single feedforward pass through a neural network. The 3D Gaussian representation produced by SHARP can then be rendered in real time, yielding high-resolution photorealistic images for nearby views. The representation is metric, with absolute scale, supporting metric camera movements.

This release includes a fully validated Core ML (.mlpackage) version of SHARP, optimized for CPU, GPU, and Neural Engine inference on macOS and iOS.

Rendered using Splat Viewer

Getting started

📦 Download the Core ML Model Only

pip install huggingface-hub
huggingface-cli download --include sharp.mlpackage/ --local-dir . pearsonkyle/Sharp-coreml

🧰 Clone the Full Repository

This will include the inference and model conversion/validation scripts.

brew install git-xet
git xet install

Clone the model repository:

git clone [email protected]:pearsonkyle/Sharp-coreml

📱 Run Inference on Apple Devices

Use the provided sharp.swift inference script to load the model and generate 3D Gaussian splats (PLY) from any image:

# Compile the Swift runner (requires Xcode command-line tools)
swiftc -O -o run_sharp sharp.swift -framework CoreML -framework CoreImage -framework AppKit

# Run inference on an image and decimate the output by 50%
./run_sharp sharp.mlpackage city.png city.ply -d 0.5

Inference on an Apple M4 Max takes ~1.9 seconds.

CLI Features:

• Automatic model compilation and caching
• Decimation to reduce point cloud size while preserving visual fidelity
• Input is expected as a standard RGB image; conversion to [0,1] and CHW format happens inside the model
• PLY output compatible with Splat Viewer , MetalSplatter , and Three.js

Usage: \(execName) [OPTIONS] <model> <input_image> <output.ply>

SHARP Model Inference - Generate 3D Gaussian Splats from a single image

Arguments:
    model              Path to the SHARP Core ML model (.mlpackage, .mlmodel, or .mlmodelc)
    input_image        Path to input image (PNG, JPEG, etc.)
    output.ply         Path for output PLY file

Options: 
    -m, --model PATH           Path to Core ML model
    -i, --input PATH           Path to input image
    -o, --output PATH          Path for output PLY file
    -f, --focal-length FLOAT   Focal length in pixels (default: 1536)
    -d, --decimation FLOAT     Decimation ratio 0.0-1.0 or percentage 1-100 (default:  1.0 = keep all)
                                Example: 0.5 or 50 keeps 50% of Gaussians
    -h, --help                 Show this help message

Model Input and Output

📥 Input

The Core ML model accepts two inputs:

• image : A 3-channel RGB image in uint8 format with shape (1, 3, H, W) .

  • Values are expected in range [0, 255] (no manual normalization required).
  • Recommended resolution: 1536×1536 (matches training size).
  • Aspect ratio is preserved; input will be resized internally if needed.

• disparity_factor : A scalar tensor of shape (1,) representing the ratio focal_length / image_width .

  • Use 1.0 for standard cameras (e.g., typical smartphone or DSLR).
  • Adjust slightly to control depth scale: higher values = closer objects, lower values = farther scenes.
  • If using the sharp.swift runner, this input is automatically computed from your image dimensions.

📤 Output

The model outputs five tensors representing a 3D Gaussian splat representation:

The total number of Gaussians N is approximately 1,179,648 for the default model.

🌍 These outputs are fully compatible with Splat Viewer and MetalSplatter .

🔍 Model Validation Results

The Core ML model has been rigorously validated against the original PyTorch implementation. Below are the numerical accuracy metrics across all 5 output tensors:

Validation Notes:

• All outputs match PyTorch within 0.01% mean error.
• Quaternion angular errors are below 1° for 99% of Gaussians.

Reproducing the Conversion

To reproduce the conversion from PyTorch to Core ML, follow these steps:

git clone https://github.com/apple/ml-sharp.git
cd ml-sharp
conda create -n sharp python=3.13
conda activate sharp
pip install -r requirements.txt
pip install coremltools
cd ../
python convert.py

Citation

If you find this work useful, please cite the original paper:

@inproceedings{Sharp2025:arxiv,
  title      = {Sharp Monocular View Synthesis in Less Than a Second},
  author     = {Lars Mescheder and Wei Dong and Shiwei Li and Xuyang Bai and Marcel Santos and Peiyun Hu and Bruno Lecouat and Mingmin Zhen and Ama\"{e}l Delaunoy and Tian Fang and Yanghai Tsin and Stephan R. Richter and Vladlen Koltun},
  journal    = {arXiv preprint arXiv:2512.10685},
  year       = {2025},
  url        = {https://arxiv.org/abs/2512.10685},
}