phanerozoic / argus-3d

Argus-3D

Class-agnostic 3D bounding box detection on a frozen EUPE-ViT-B backbone. Given a posed RGB image and camera intrinsics, returns 7-DoF boxes (cx, cy, cz, w, h, d, theta) for the objects in the scene.

The head is built by feature-dim discovery and unsupervised clustering, not gradient training. No 2D bounding boxes, no class labels, no segmentation map as final output. Camera-frame 3D boxes only.

Architecture

Image (768x768)
  -> EUPE-ViT-B (frozen, reused from phanerozoic/argus)
  -> patch tokens (2304, 768) on a 48x48 grid
  -> instance head: ridge over 768 dims  -> per-patch foreground score
     depth head:    ridge over 768 dims  -> per-patch metric depth (m)
     k-means modes: 8 cluster centers    -> per-patch object-type assignment
  -> threshold instance score
  -> upsample mask to 768x768, connected components
  -> for each component:
       unproject pixels to 3D using depth + K
       DBSCAN-split for instance separation
       PCA-on-xz for yaw, percentile extents for (w, h, d)
       blend extents toward the matched cluster's size prior
  -> camera-frame 7-DoF box list

Components

File layout

instance_head.safetensors   # ridge dims + coef + intercept + threshold
depth_head.safetensors      # ridge dims + coef + intercept
size_priors.safetensors     # 8 cluster centers + 8 (w, h, d) priors
config.json                 # input_res, patch_grid, prior_weight, etc.
argus_3d.py                 # Argus3D class
infer.py                    # CLI dispatcher

Usage

from argus_3d import Argus3D
import numpy as np

model = Argus3D.from_pretrained("phanerozoic/argus-3d", device="cuda")

K = np.array([[850, 0, 395], [0, 850, 510], [0, 0, 1]])
boxes = model.detect("room.jpg", K)               # list of Box3D
boxes = model.detect("room.jpg", K, depth=d)      # supply RGBD sensor depth
out   = model.perceive("room.jpg", K)             # fg score map + depth map + boxes

for b in boxes:
    print(b.cx, b.cy, b.cz, b.w, b.h, b.d, b.theta)

Eval

CA-1M val sequence ca1m-val-45662921 . Class-agnostic per-scene 3D IoU after multi-view fusion across 284 frames (stride-4 sampling of 1135 total). The head produces its own instance hypotheses; no ground-truth 2D bounding boxes are used. Sensor depth is supplied; the discovered depth head can be used in its place.

Per-stage discovery metrics:

Backbone

EUPE-ViT-B from Meta FAIR (arXiv:2603.22387) via phanerozoic/argus . The backbone is frozen and not modified by this repo.

License

FAIR Research License (non-commercial), inherited via the EUPE-ViT-B backbone.