FluidInference / phonon-2-coreml

phonon-2-coreml

Core ML build of FermionResearch/Phonon-2 , Fermion Research's quantization-aware re-training of nvidia/parakeet-tdt-0.6b-v3 for English, in which every encoder weight takes one of five learned values per output row. Same tokenizer, 15 s window and Decoder / JointDecisionv3 contract as parakeet-tdt-0.6b-v3-coreml ; loaded in FluidAudio via AsrModelVersion.phonon2 .

Every encoder here holds the checkpoint's exact five-value weights — as fp16 palettes (iOS 18 / macOS 15 constexpr_lut_to_dense ), or as a sparsity mask plus a palette over the non-zero weights ( constexpr_lut_to_sparse + constexpr_sparse_to_dense ). No re-quantization noise is added, and all five produce identical transcripts. The decoder and joint are re-exported from the checkpoint's int6 tables; preprocessor and vocabulary are the v3 ones.

Usage (FluidAudio, merged in #980 )

let models = try await AsrModels.downloadAndLoad(version: .phonon2)  // iOS 18+ / macOS 15+, downloads Encoder.mlmodelc
let asr = AsrManager()
try await asr.initialize(models: models)
let result = try await asr.transcribe(audioFileURL)

swift run fluidaudiocli transcribe audio.wav --model-version phonon2
swift run fluidaudiocli asr-benchmark --subset test-clean --model-version phonon2

To use another encoder file, download the model directory, rename the chosen Encoder_*.mlmodelc to Encoder.mlmodelc and load it with AsrModels.loadLocal(from:version: .phonon2, encoderComputeUnits:) .

Encoder files

One 15 s window on an M5 Pro (macOS 27); RTFx is FluidAudio asr-benchmark on full LibriSpeech test-clean with the encoder on the Neural Engine. All five are exact and give the same transcripts.

Two facts decide the choice. The Neural Engine's palette cost grows with the number of palettes, not their bit width, so 8 rows per palette is faster than v3's own encoder while per-row palettes are 3× slower. And the GPU materializes sparse weights at every load ( 150 s of CPU, not cached), while the ANE compiles them once ( 1 min) and caches. So: Neural Engine (the FluidAudio default, and the only option for iOS background work) → Encoder.mlmodelc ; GPU → Encoder_lut3.mlmodelc (small) or Encoder_lut6.mlmodelc (fast everywhere); smallest possible → Encoder_sparse-g1 .

Other files: Decoder.mlmodelc (RNNT prediction net, fp16, iOS 17+), JointDecisionv3.mlmodelc (single-step joint + top-64, fp16, iOS 17+), Preprocessor.mlmodelc and parakeet_vocab.json (identical to v3), NOTICE / LICENSE (upstream attribution, CC-BY-4.0).

Accuracy vs parakeet-tdt-0.6b-v3

Full LibriSpeech, FluidAudio asr-benchmark , M5 Pro, both models run back to back with the default (ANE) encoder. WER is corpus-level (total edit distance over total reference words); RTFx = total audio / total processing time.

v3 is the more accurate model on LibriSpeech by 0.20 / 0.50 points, reproducing the upstream card's own gap to its teacher (+0.20 / +0.79 under the Open ASR Leaderboard protocol, where Phonon-2 in turn beats v3 on AMI meetings and VoxPopuli). Phonon-2 is English only; v3 covers 25 languages. The absolute values are above the card's because FluidAudio decodes in 15 s windows with a simpler normalizer, which both models pay equally.

Conversion fidelity: on the first 100 test-clean files the Core ML transcripts differ from a NeMo fp32 full-context decode of the same checkpoint by 0.34 % WER (1.83 % vs 1.79 %).

60-minute long-form file (Earnings-22, four concatenated calls)

transcribe on the 3600 s earnings22_top4_1h.wav (M5 Pro, default ANE encoder, best of 2–3 runs, processing time excludes model load). Reference = the concatenated Earnings-22 chunk transcripts, same normalizer.

On conversational long-form audio Phonon-2 is the fastest model we ship (1.45× v3's throughput, on par with Ultra) but the least accurate of the four: Ultra and Redux both beat v3 here while Phonon-2 trails it by 0.8 points, consistent with the upstream card's Earnings-22 row (6.96 % vs its teacher's 5.85 %). All five Phonon-2 encoders produce the same transcript.

Licence and attribution

The weights are a derivative of NVIDIA's parakeet-tdt-0.6b-v3 re-trained by Fermion Research and are distributed under CC-BY-4.0, as upstream. NOTICE reproduces Fermion Research's change list and training-data attribution. The encoders are exact re-encodings of the upstream weights (grouped-channel palettes, or a sparsity mask plus palettes over the non-zeros); the decoder and joint are the upstream int6 tables in fp16.