AbstractPhil / vit-beatrix-geometry-pretrained

This version essentially imploded when learning cutmix cifar100, and yet the geometry held the entire time through a cutmix gauntlet that could not be overcome.

I plan to run a few experiments on it with the geometric and simplex blocks frozen. As it stands, the geometric side is essentially a scaffold of cutmix potential, however this scaffold is limited. It must be further trained, but this is a good basline test.

I will run 4 full trains to test the potential;

1. Frozen geometry, no augmentation. It's working, but overfitted by epoch 30 - which, took nearly 60 before. So that's interesting and yet overfitted.
2. Full augmentation no cutmix/mixup using cifar10 augs.
3. Unfrozen cross-attention only, full augs.
4. Unfrozen cross-attention only, no augs.

======================================================================
CHAOS-NATIVE DUAL-STREAM PROBE
======================================================================

Loading model...
  Model loaded from: ./checkpoints_dualstream/20251008_163456_chaos_native/model_epoch_149.safetensors
  Parameters: 41,132,926

Loading data...
  Test samples: 10000
  Probe samples: 5000

======================================================================
PROBE 1: Dual-Stream Analysis
======================================================================

[1.1] Extracting stream features...
Extracting:   0%|          | 0/40 [00:00<?, ?it/s]
  Model output keys: ['logits', 'visual_features', 'geometric_features', 'pe_features', 'cantor_measure', 'visual_stream', 'geometric_stream']
    logits: torch.Size([128, 100])
    visual_features: torch.Size([128, 512])
    geometric_features: torch.Size([128, 256])
    pe_features: torch.Size([65, 24])
    cantor_measure: torch.Size([65])
    visual_stream: torch.Size([128, 65, 512])
    geometric_stream: torch.Size([128, 8, 256])
Extracting: 100%|██████████| 40/40 [00:02<00:00, 18.99it/s]

  Visual features shape: (5000, 100)
  Geometric features shape: (5000, 256)

[1.2] Analyzing stream dimensionality...
  Visual stream: 1/100 dims for 90% variance
  Geometric stream: 1/256 dims for 90% variance
  Visual intrinsic dim: 1.04
  Geometric intrinsic dim: 1.01

[1.3] Measuring stream independence...
  Mean abs correlation: 0.2077
  Max abs correlation: 0.9985
  Interpretation: Independent

[1.4] Testing class separability per stream...
  Visual stream probe: 0.0227
  Geometric stream probe: 0.0227
  Visual advantage: +0.0000

======================================================================
PROBE 2: Geometric Stream Health
======================================================================

[2.1] Collecting geometric health metrics...
Health metrics: 100%|██████████| 40/40 [00:02<00:00, 18.90it/s]

  Token Diversity: 0.2820 ± 0.0025
  Feature Diversity: 1.3960 ± 0.0029
  Mean Norm: 22.2937 ± 0.0464
  Status: ✓ Healthy

======================================================================
PROBE 3: Chaos Tolerance (CutMix Robustness)
======================================================================

[3.0] Testing alpha=0.0...
  Accuracy: 0.0221
  Geometric mean norm: 22.2951

[3.2] Testing alpha=0.2...
  Accuracy: 0.0173
  Geometric mean norm: 22.3495

[3.5] Testing alpha=0.5...
  Accuracy: 0.0165
  Geometric mean norm: 22.3517

[3.10] Testing alpha=1.0...
  Accuracy: 0.0174
  Geometric mean norm: 22.3580

[3.15] Testing alpha=1.5...
  Accuracy: 0.0161
  Geometric mean norm: 22.3592

[3.20] Testing alpha=2.0...
  Accuracy: 0.0164
  Geometric mean norm: 22.3624

  Clean accuracy (α=0.0): 0.0221
  Extreme chaos (α=2.0): 0.0164
  Chaos tolerance: 74.21%
  Interpretation: Good

======================================================================
PROBE 4: Clean vs Mixed Image Analysis
======================================================================

[4.1] Testing on clean images...
  Accuracy: 0.0221
  Mean confidence: 0.0120

[4.2] Testing on mixed images (α=1.0)...
  Accuracy: 0.0160
  Mean confidence: 0.0106

  Performance gap: 0.0061
  Confidence gap: 0.0014
  Adaptation: Strong

======================================================================
PROBE 5: Geometric Stream Stability
======================================================================

[5.1] Testing Gaussian noise robustness...
  Noise σ=0.0: norm=22.2937
  Noise σ=0.1: norm=22.2945
  Noise σ=0.2: norm=22.2965
  Noise σ=0.3: norm=22.2948

[5.2] Testing CutMix stability...
  Norm range across CutMix: 0.0704
  Status: ✓ Stable

======================================================================
Results saved to: probe_chaos_results/chaos_probe_results.json
======================================================================

======================================================================
Generating visualizations...
======================================================================

[Viz 1] Chaos tolerance curve...
[Viz 2] Stream correlation heatmap...
[Viz 3] Geometric stability...

Visualizations saved to: probe_chaos_results

======================================================================
CHAOS-NATIVE PROBE SUMMARY
======================================================================

[Dual-Stream Analysis]
  Visual intrinsic dim: 1.04
  Geometric intrinsic dim: 1.01
  Stream independence: 0.2077

[Geometric Health]
  Mean norm: 22.2937
  Token diversity: 0.2820

[Chaos Tolerance]
  Clean accuracy: 0.0221
  Max chaos accuracy: 0.0164
  Tolerance score: 74.21%

[Clean vs Mixed]
  Clean accuracy: 0.0221
  Mixed accuracy: 0.0160
  Performance gap: 0.0061

[Geometric Stability]
  Noise stable: True
  CutMix stable: True