This codec would have been great 27 years ago! I suspect it wouldn't have gotten past the patent lawyers in 1999 though. It seems cute today but the entire codec space was a patent minefield, the N64 mp3 decoder was implemented twice (once based on the reference codec, once clean) with Nintendo/Rare then paying RAD game tools to license their decoder... of which none of their code was used but they had a license from the mp3 patent holder (Fraunhofer) which then shielded Nintendo from lawsuits. Fraunhofer wanted something like $250k to license directly which wasn't going to fly.

I'll also note the frame rate issues in Conker were mainly down to raw RDP fillrate issues, which was for sure the weakest part of the N64 hardware, it could draw beautiful triangles (for the time) but oh so so slowly. There was a decent amount of speech during gameplay using the RSP mp3 codec. The cartridge size was still an issue, 64Mb got a good amount of mono speech but was still very much at a premium.

>It seems cute today

Maybe in audio, but video still seems like an absolute nightmare. Look at the absolute silliness with like having to pay $.99 to play h265 video in Windows.

AV1 is royalty free and better!

Real example, I recorded the world series using an ATSC antenna, and I re-encoded a 90 second sample using a bunch of different settings on a 32-thread znver5:

  # Codec        Flags                     Size         Speed
  - -----        -----                     ----         -----
    Original     (Raw ATSC mpeg2video)     77MB (100%)  N/A
  1 libx265      -preset slower -crf 24    37MB  (48%)  0.245x
  2 libsvtav1    -preset 3 -crf 44         36MB  (25%)  1.87x
  3 libsvtav1    -preset 1 -crf 35         19MB  (25%)  0.556x

(NB: The same CRF results in a higher quality video at slower presets, this was for archival so we're using the some of the slowest presets).

(1) and (2) are completely indistinguishable from the original unless I go frame by frame looking for transient motion artifacts. (3) is only distinguishable in one shot where a dark background was washed out. Not only does AV1 produce results just as good as H265, it encodes quite a bit faster for the same size/quality.

That said, I've watched so much h264-encoded video over the past decade I think I've been trained to perceive the subtle visual "snow noise" as a sign of a high quality video: I find how AV1 blends backgrounds more visually objectionable than the dithering in H265, even though I think it's objectively higher fidelity. Similar to how I find high frame rate video "fake" after having watched NTSC TV for the first half of my life.

These were the CRFs necessary to get the same visual quality with libx265 at faster presets and the resulting sizes:

  hevc-slower-24          37MB (used above)
  hevc-slow-21            56MB
  hevc-medium-20          57MB
  hevc-faster-19          52MB
  hevc-veryfast-18        60MB

You can get it for free though: https://www.free-codecs.com/guides/how_to_download_hevc_video_extension_for_free.htm

you used to be able to get it for free on windows[0] but it seems that might not be available anymore.

Of course there is always VLC

0: https://apps.microsoft.com/detail/9n4wgh0z6vhq

> having to pay $.99 to play h265 video in Window

what?

Almost nobody actually pays for this, since every OEM includes a volume license that's baked into the price tag. VLC includes FFMpeg so it just works ootb.

Video codecs have historically been like this, to some extent or another. It's really just a manner of if the manufacturer has absorbed it into the price tag or pushed it to the consumer.

No one likes paying for a codec license anyways, but since patents are at play, they kind of have to.

That leads to absolute idiocy like this https://www.reddit.com/r/technology/comments/1p3p1hy/dell_and_hp_disable_hardware_h265_decoding_on/

https://www.free-codecs.com/guides/navigating-hevc-video-extensions-for-windows.htm

>raw RDP fillrate issues, which was for sure the weakest part of the N64 hardware

I thought weakest part was the size of the texture buffer.

Nobody expects the N64! Our two weakest parts are our raw RDP fillrate, and our tiny Texture buffer.

Ok, from a performance perspective the RDP fillrate was the biggest issue. For artists the texture buffer was a massive challenge.

If the RDP had been clocked 4x faster and had 4x the texture size then it would have been incredible… it makes beautiful pixels!

i've heard the rambus dram latency was also a big deal but does that have any effect on the fillrate as well? or is that mostly a bottlneck affecting the cpu?

In case anyone else got to the demo thinking the quality is awful -- the vocals are incomprehensible and keep cutting out -- it turns out it's like that in the original track:

https://www.youtube.com/watch?v=i1gVxKhdGPs&t=57s

i got confused at first, and then noted that this was an awful piece of music to pick for a compression demonstration.

not the music taste itself, but perhaps he should have gone for something more known that would be easy for A/B tests.

Tom’s Diner by Suzanne Vega would've been a neat choice, as it was the track used to assess and refine the MP3 compression algorithm during it's development.

Good point! Replaced the video.

It was all great until they mentioned LLM usage.

The demoscene and homebrew scene is dying as we speak.

I wonder if this would also be a great fit for the MAS3507D! Supposedly you can replace the program with your own in a 1-kiloword RAM (the default MP3 decoder is in ROM, and probably larger). If you can fit this codec in 1024 MIPS instructions, perhaps you can do the same in 1024 of these DSP instructions…

Too bad there is seemingly zero public documentation on the format of these programs, though (and no emulator or similar).

Great read, thanks!

Pardon my ignorance but one thing I'm curious about, if the decoder fills almost the entire 4k RSP instruction memory (IMEM) - wouldn't that be something games need for graphics? If so, then is this still usable for say a game?

Like the DMEM, you can replace the IMEM at any time. In this case, when you're done with processing a block of music, you can DMA the code for 3d transforms or anything else to the RSP.

Libdragon handles this transparently for you with a queue[1].

[1] https://libdragon.dev/ref/rspq_8h.html

Beautiful! I long wanted to write my own audio codec, this might be a nice challenge.

Sorry but what? Half-Life on an N64?? it looks incredible!

There's an explosion of N64 ports lately, with a huge mix of new libraries (libdragon) pure fan passion, and LLM assistance.

Some I'm tracking:

- Duke Nukem 3D (ie, the fully-functional PC version) https://github.com/BigChat88/N64DukeNukem3DPort

- Portal 64 Still alive https://github.com/mwpenny/portal64-still-alive

- Resident Evil 64 (ie, not RE2, but the first one) https://www.youtube.com/watch?v=_xRiwy8Z6S0

- Silent Hill (still an early WIP) https://www.youtube.com/watch?v=v7fM06jmHso

I haven't personally tested, but these look interesting:

- Simpsons Arcade https://www.youtube.com/watch?v=zH9JYQVzKDE

- X-men Arcade https://www.youtube.com/watch?v=ah3nB8MP4CM

- TMNT Arcade Collection https://www.youtube.com/watch?v=CiR5-ZpLtyI

At this point, they're coming out faster than I can play them. I'm in the middle of the 2nd Episode of Duke 3D right now and it's quite nice. The original 64 version was always fun, but mid-level saves are really a game changer.

> There's an explosion of N64 ports lately

There's an explosion of ALL ports lately. It's crazy.

Ocarina of Time only got released today officially but it's already playable on ARM Macs. It's not even a port per se but executing the Switch ARM code natively. And of course it's also playable on emulators regardless of the platform