I'm also releasing a hack of P-47 Aces, which fixes the sound engine to play sound effects on time. You can get that here: http://daifukkat.su/hacks/p47aces_sfx/. The original behavior is pretty deficient in my opinion, and the fix ended up rather interesting and helping emulation, so I'm glad I went and did it. Though it was amusing to see the end stage screens with constant explosions, I'm hoping having properly timed sound effects will make the game even more enjoyable.

I'm hoping to start posting on this blog again more, some fun stuff coming ahead. For now, happy end of year festivities, whatever they may be. To a pleasant 2020~

Update 2019/12/28: I fixed the P-47 Aces fix, so now it works everywhere.

Completely out of left field, this one. I was at a friend's place the other day, and he had Mitchell Corporation's The Karate Tournament running. I'd never played the game but thought it looked interesting so I gave it a few rounds and thought it was pretty damn interesting.

Then I was told that the dumped version doesn't have English voice acting and has much slower "cutscenes". Of course, this means it's time to break out the ROM reader.

Another friend dumped the ROMs and I added it to MAME, and now it's coming your way. The driver that emulates this isn't in the best shape, so I intend to go fix it up at some point, but for now it seems to work fine, I noticed no significant problems.

Credits to The Dumping Union and Anonymous for this one. Get it while it's hot: http://daifukkat.su/files/karatour.zip, and a compiled MAME that can play it: http://daifukkat.su/files/karatemame.7z

EDIT: All done now!

A few nights ago I decided to start testing that, but as usual my scope exploded and woops now I'm checking instruction cycle timings and DMA intricacies. I wrote a test harness and log parsing tool that I can now use to test the timings of anything and easily get data and specifics off of a real WonderSwan. Naturally, I used my WonderWitch for this task, as it provides a very easy way for me to write code and get it running on a real console, and has a very small iteration overhead, allowing me to test many versions very quickly.

I'd like to explain my method of getting precise cycle timings, but I'm afraid I need to give a bit of back story first. Over the past few weeks at work I've been implementing a libc for our platform to get rid of the pile of garbage known as newlib. In the process of this, I had to implement rand(), which I decided to temporarily implement as a simple LFSR. With LFSRs on the brain, and being very mindful that any seed produces the same series of outputs that never have duplicates until the entire period has been exhausted, I devised a plan for getting precise timings from the WonderSwan.

Now, the WonderSwan has a noise mode for one sound channel on its sound chip. It's implemented as a 15-bit LFSR with 2 taps, one configurable and one fixed. The default tap configuration has a period of 32767 cycles, which is maximal length for a 15-bit LFSR. There are two properties of this LFSR that make it particularly interesting: the update speed is controlled by the channel's frequency register, and the current value of the LFSR can be read by the CPU. The first property is even more interesting when one considers the range of frequencies allowed by the register. The output frequency is f = MCLK / (2048 - reg), where the register can range from 0 to 2047. This means we can have the LFSR update at a rate of one update per master clock.

The more adept of you may already see where I'm going with this. Essentially what I can do is start the LFSR at maximum frequency, save the LFSR value, perform some operation, then save the LFSR value after that. From these saved LFSR values, I can convert them back to a cycle index using the property of LFSRs that causes all outputs to be unique. Now that I have a cycle index for before and after the operation, I can subtract the latter index from the former, and get a 100% accurate time delta with complete precision between the two sample points. With a small test, I can also check the cycle overhead of sampling the LFSR, and discard that overhead from real test results. This gives me perfect timings of any operation I wish to perform.

From here, I can simply schedule a series of DMA transfers of varying lengths and retrieve the timings of each, then correlate the data to get a final algorithm. From my tests, a DMA transfer takes 5 + 2n cycles, with n being the number of words to transfer. I discovered a few other things in the process, but those will be put up into a new version of WSMan soon enough.

I've open-sourced my hardware testing infrastructure and tools on my bitbucket at https://bitbucket.org/trap15/wswan_hwtest.

I made a hack for system11 that makes Metal Slug 2 not a laggy piece of garbage. In the end, the old rumor that it was caused by "updating the game logic twice per video update" was incorrect. The real issue was much more dumb, and much more puzzling. The way they wrote their code to lock to 30 FPS is something like this:

void vblank_callback(void)
{
  if(vblank_counter != 0xFF)
    vblank_counter++;
  if(vblank_counter != 2)
    return;
  vblank_occurred = 0xFF;
  // do things...
}
void game_update(void)
{
  // do things...
  for(;;) {
    if(vblank_occurred & 0x80) {
      vblank_occurred &= ~0x80;
      break;
    }
    vblank_occurred &= ~0x80;
  }
  vblank_counter = 0;
  // do things...
}

At first, this code seems reasonable, and not particularly odd or troublesome. But now think of the problem case: what happens if the game update misses the 'every odd frame' update window? Now the game is waiting two entire frames to get back on sync. Because of this, instead of a drop from 30 FPS to 20 FPS, it is now effectively a drop to 15 FPS. If it drops to 15 FPS, it's not affected, but if it would drop to 10 FPS, it drops to 7.5 FPS. This is very very significant, and is what makes the game run at a snail's pace. Most of the time the game is not even dropping to 15, it's only displaying as such because of the poor frame limiter code. Yes, this is actually why it runs so poorly.

Now that I've discussed the problem, let's discuss how I fixed it. Essentially, I replaced their code with the code listed below. This new code is faster to execute, simpler to understand, and works better than the original. This is what I implemented in my hacked version:

void vblank_callback(void)
{
  vblank_counter = 2; // this needs to be done for other functions to update properly
  vblank_occurred++;
  // do things...
}
void game_update(void)
{
  // do things...
  while(vblank_occurred < 2); // wait for 2 frames to have elapsed
  vblank_occurred = 0;
  // do things...
}

You may look at system11's post on the release at http://blog.system11.org/?p=1442, including a very enlightening comparison video and instructions on how to replace the ROM on a physical cartridge.

I've made a more subtle hack this time around, opting to merely fix problems in the original game, rather than turn it into something entirely different. With NEW VER!, I have added built-in 30Hz autofire that doesn't break the flamethrower, rebalanced the powerup order, changed how long items last on screen, and rebalanced difficulty, in a way that I think makes the game both more fun, and more recoverable. That said, it's still an extremely difficult game, and that was one thing I actively tried to keep with these changes. In the end, I think this makes the game much more fair, and much more playable and approachable.

You can get it here: http://dodonpachi.daifukkat.su/hacks/samenew/.

I've released one last version of Ketsui Arrange, which removes the suicide bullets added in 1.5, and tweaks various parameters and colorations. This is the definitive version of Ketsui Arrange, and it will be the last.

I don't have much else to say about this, you can get it here: http://dodonpachi.daifukkat.su/hacks/ketarr/.

• WonderWitch [FreyaOS 1.2.0].ws
• WonderWitch [FreyaOS 1.1.5].ws

Once again, massive thanks to Ryphecha for putting in a lot of work to clean these dumps up for release!

The time has finally come for the release. This will be my final post on the NMK004 dump. As of this post, the patches have been added to MAME, and will definitely be included in the next stable release.

Let's start out with what everybody has been waiting for, the NMK004 internal ROM dump. You may obtain the dump here: http://daifukkat.su/files/nmk004.zip.

Next, I'm also releasing my dumping tools. This is split into two repositories, both hosted on bitbucket. OPNCAP is located at https://bitbucket.org/trap15/opncap, and nmk004-trojan is located at https://bitbucket.org/trap15/nmk004-trojan.

Lastly, I have a special secret extra bonus, but you'll have to put up with story time for a bit. A few months ago, I bought a Hacha Mecha Fighter PCB off of eBay. I knew it looked somewhat strange, being mostly EPROMs instead of mask ROMs, and lacking the NMK-110 protection MCU that's usually on the PCB. When I got it, I realized almost instantly that it had to have been a bootleg conversion from Thunder Dragon. After playing and comparing to PCB recordings of a genuine board, I knew that what I had was gold: an unprotected version of Hacha Mecha Fighter. I never bothered to dump it until I started on this project, but I felt now was the right time. So I dumped the program ROMs and sent them off to Haze to have them added to MAME. As expected, it works perfectly in MAME, and now it's possible for anyone to play it properly. In the future, I'd like to actually figure out the protection on Thunder Dragon and Hacha Mecha Fighter, but having unprotected versions works just as well.

You can get this ROM set from http://daifukkat.su/files/hachamfb.zip.

Title screen of Hacha Mecha Fighter

The following MAMETesters bugs may now be closed:

• http://mametesters.org/view.php?id=1117
• http://mametesters.org/view.php?id=3395
• http://mametesters.org/view.php?id=2422
• http://mametesters.org/view.php?id=2417

Last but not least, I'd like to give some shout outs to those who made this all possible:

• David "Haze" Haywood
• Charles MacDonald
• Jonathan "Lord Nightmare" Gevaryahu
• Nicola Salmoria
• austere
• #raidenii
• Beer and coffee

As a cute little aside, I've uploaded the work log I kept while doing this. Maybe somebody will find it interesting. There's a lot of content, lots of information on my thought process, a few diagrams and tables, stuff like that. Pretty unfiltered though, so don't be offended by anything I have written. http://daifukkat.su/files/NMK004-WORK-NOTES.txt

Hold on to your hats, this post is gonna be a long one.

Title screen of GunNail

As mentioned in part 1, I revisited the NMK004 dumping project late last month, and made good quick progress. The first thing I did was attach my Saleae Logic logic analyzer to the OPN on my Thunder Dragon board. I started a song and used the logic analyzer to look at a few of the signals. Unfortunately, my model only has 8 probes, so I wasn't able to get a full picture of the interfacing activity. I managed to figure out how to detect a write to the chip, but the lack of probes meant I wouldn't be able to fully grasp the exact activity. Compounded with how many "dummy" writes were constantly being done, I knew I had to move onto writing FPGA code next.

After installing Quartus II, I had to spend a bit of time remembering VHDL's syntax, since I hadn't used it for a while. I spent about an hour writing the initial logging code and making it compile. I'm not as good with VHDL programming as other conventional languages, so I ended up having some broken logic that took me about an hour and a half to fix. Once it worked, I added a filter to make the FPGA only record PSG writes. It turns out that all upkeep pokes that the NMK004 does are exclusively on the FM registers; the PSG registers are only poked when their values need to be changed. This saved a lot of space on my FPGA's relatively small 512KB SRAM. Saving space allows for more bytes to be dumped at once, making the dumping process much faster.

From here, I started probing the possible attack vectors. I knew that the most likely targets would be the note frequency tables and the volume envelope tables for the PSG. I wrote some small testing ROMs to do a quick check, and was dismayed when I found that the volume envelope table could only directly dump 4 bits at a time, and the note frequency table could only directly dump 12 bits. While contemplating what to do, I realized that I could extract an extra 8 bits out of the volume envelope table by simply looking at the timing between writing the new volume. This still essentially only gives me an 8-bit dump per channel that skips every other byte, but it also gives me an additional 4 bits of verification. This particularly came in handy when I realized that the length value ends up being exactly the same for a $00 byte and a $01 byte. Using the 4 bits from the volume value lets me determine exactly which value it is. In addition, since we're just recording values being written to the sound chip, I can now multiplex the three PSG channels to get triple dump speed. Armed with this knowledge, I knew that this was going to be my attack vector, and exactly what I was going to do to make it work.

Title screen of Koutetsu Yousai Strahl

After carefully planning out my attack, I started writing the tools that I would need to make the dump process smooth and easy. With a few hours of work, I had code that would analyze the captured register activity and turn it into a log of activity over time. I also wrote a fair bit of code that would allow querying for when a register changes, finding the next register activity, and such similar things. It ended up being a nicely full featured suite for capture analysis. I still needed some more data from the NMK004 before I could make the tool actually reconstruct the capture into ROM data though. I needed to get the real-time length for the values in the volume envelope table. To accomplish this, I made a bogus ROM that had the length values $00,$01,$02,$04,$08,$10,$20,$40,$80,$FF. I figured this would give me a very good idea of how long each value takes, and it ended up being very spot on. It takes approximately 30250 × value capture ticks for each length value, where $00 is the same as $01. One thing that stood out during this test was that the first tone would last slightly shorter than the rest. 2500 ticks shorter, to be specific. This was an easy fix, but it could have definitely tripped me up if I didn't notice it.

As part of my attack strategy, I utilized an interesting concept in the conversion tool that I call a prioritized usage map. For each incoming nibble -- 2 nibbles for length, 1 nibble for volume register value -- I would store the channel it was captured from, and where that nibble came from. In the case of a volume register value, I gave it the highest priority, 3. For a length nibble, I would give it the lowest priority, 1. If the length byte was determined to be $01, I would use the high nibble and give it medium priority, and ignore the bottom nibble. If a priority value in the usage map is 0, it would mean that the corresponding nibble has not yet been defined for the dump. Because of this system, if two overlapping nibbles were ever different, the tool could print out a warning, and use the value of the higher priority nibble. With this system in place, I felt very confident that the dump would be accurate, or at the very least be extremely obvious when a dump error occurred.

With the data analysis part done, I needed to make the program actually save the results. I decided on making the conversion tool "stackable", in that it takes a working state binary, loads it, performs its task, then saves it back. This way, multiple capture files can combine to create a final ROM image. While I was working on adding this feature, I noticed an interesting behavior in the volume envelope table handling on the NMK004. I originally thought that each table would last forever, but it turns out that it will actually wrap every $100 bytes. I didn't think that this would matter much, as I decided to play it safe and only dump a spread of $C0 bytes per channel per song, but I'd later find that I was dead wrong. It only affected two of the later songs, which I of course hadn't done testing with, so I ignored the behavior.

Considering 3 channels and that each set needs to be run through twice, each song is able to dump approximately $120 full bytes. I performed a test on the first song, and received an all too familiar message: "All Music,Effect Software(C)1990 N M K Corporation". After seeing that message, I decided it was finally time for bed, and put off further work for later.

Title screen of Choujikuu Yousai Macross

After having spent so long on that one day, I took a break for about a week. When I came back, I decided I should start documenting all of my processes and software that I'd developed, as I knew I wanted to release them all along with the ROM itself. I also split up the project into two separate but related projects: OPNCAP and nmk004-trojan. OPNCAP consists of: opncap-fpga, the FPGA code for capturing OPN register activity; reccat, a PC tool to combine capture files; and opncap-pc, a PC library used to parse and use the data in a capture file. nmk004-trojan does what it says on the tin, and consists of: trogen, a tool to generate trojan ROMs; and opnrom, a tool that uses the opncap-pc library to convert a capture file to a ROM image. Once I'd finished splitting stuff out, I continued to spend a good amount of time doing documentation. Writing the documentation was fairly painless and easy, since all the procedures were still pretty ingrained from my marathon work. Doing most of the documentation in one go ended up being a great idea, as it let me work on the project in a different way that I wasn't burned out on, and let me recover from that burn out in a productive manner.

Eventually I felt satisfied with the state of the documentation, and thus decided that I should finally do the dump. I generated a set of ROMs and burned them to 27C512 EPROMs that I had lying around. On my first few tries, I spotted several bugs in opnrom that I had to fix before continuing. Thankfully, I knew I wouldn't have to redump from my PCB because of how simple and fool-proof the actual capturing is. It took me about an hour to fix the opnrom bugs, but once that was done I had absolutely no problems. After cleaning everything up, I proceeded to dump more of the ROM.

Unfortunately, not everything was well. There were two song IDs that caused massive havoc with my capture setup. On the 13th song, the song filled up the SRAM extremely quickly, and I lost all my capture progress from that dumping set. I realized that the cause of this peculiarity was that the section is mostly filled with very low values, causing the dump to write lots of data very quickly. Because each song is a fixed length and the previously mentioned looping behavior of the volume envelope table, it would loop the dump continuously. This combined behavior led to the SRAM filling and then overflowing within a few minutes. My solution to this problem was a bit inelegant, but worked well. When the SRAM was nearly full, I would flip a switch that would pause capturing activity. Then I would copy off the contents, wait for the mischievous track to end, then begin the next song. In this way, I managed to recover the data from the two troublesome tracks and not require any strange hacks in the actual project code.

Once everything was captured, I knew there was only one step left for the dump to be complete: conversion of all capture files into a single ROM image. I ran opnrom over all of the capture files, to produce the final nmk004.bin. The conversion program gave only a single nibble warning, which I have verified to be a non-issue. I was ecstatic. The dump was finally done, I finally had the binary.

Title screen of Acrobat Mission

Once I had the dump complete, I asked the fantastic David "Haze" Haywood for assistance in hooking up the ROM to MAME. Within a day, he had it hooked up and mostly working, though there were some parts that were hacked to work because they weren't understood. I helped out a bit, fixing the 16-bit timer behavior in the TMP90C840 core, and fixing NMI behavior. I had also noticed that the channel balance was very wrong for the newly emulated games, so I spent a while comparing the output from MAME to various PCB recordings, and eventually found values that sound very close. This change actually fixes sound for more than just the NMK004 games, it fixes almost every game in the nmk16.c driver.

There was also a mysterious issue where the emulated NMK004 would reset itself after a set amount of time, and kill all the sound. After tracing pins around my PCB, I deduced that it was attempting to reset the entire system when that happened, and provided Haze with the information necessary to hook that up. According to Haze, the suicide would be prevented if the NMK004 got an NMI within a set amount of time from the last NMI, so I then traced the NMI pin from the NMK004. It led to another ASIC on the board (NMK005), which is the GPIO controller. Eventually, it was deduced that the main CPU pokes the GPIO controller to give the NMK004 an NMI when it needs it. It seems like this setup was done to turn the NMK004 into a watchdog of sorts. Once all that had been hooked up, everything seemed to work very well.

Except for two specific games: Black Heart, by NMK; and Uchuu Senkan Gomorrah, by UPL. These two games would not work with this setup, and I was mystified for a while. Figuring that there must have been some interrupt that was needed that just wasn't firing, I looked at the IRQ implementations for Black Heart. Seeing that IRQ2 was exclusively dedicated to kicking the sound NMI, I realized that was probably the greater issue. Thanks to the wonderful NMK PCB manual located at upl-gravedigger, we were able to massively clean up and improve the accuracy of the IRQ behavior in the NMK driver. This change, much like the sound balance one, affects nearly all games in the nmk16.c driver, so look for that! Sadly, this fixed behavior still didn't fix Gomorrah. In the end, we settled for a game-specific hack that inverts the level of the NMI signal. The reason Gomorrah still wouldn't work is strange, it does kick the sound NMI occasionally, but it often kicks it just slightly too late, and gets reset before the kick happens. We believe this issue may be caused by inaccuracies in the TLCS-90 CPU emulation speed, but it's such a bizarre and difficult issue that we both felt that it wasn't worth trying to fix it at this point in time.

The last change we made is something that might be a bit controversial. Haze mentioned that MAME, as it currently stands, has sprite "wobble" -- indicative of missing sprite buffering -- and asked me if I knew if the sprites were buffered. According to the previously mentioned manual, sprite RAM is indeed DMA'd on each frame, adding 1 frame of sprite delay. However, as part of some investigation I've been doing the past year or so, I also knew that the sprite hardware was framebuffer-based, and double-buffered. This adds a second frame of sprite delay. Therefore, NMK PCBs have 2 frames of sprite delay. We have added this to the MAME driver to preserve accuracy.

The next and final post in the series will be coming very soon. The next post will include the NMK004 ROM, the release of all of my dumping tools, as well as a secret extra bonus that I think will make many people excited. I am also going to give Haze the go-ahead to submit the MAME changes once the next post is up, so everyone will be able to experience the joy that is NMK.

This is post is a sequel to a previous post. I highly recommend reading it before this article, or this one may not make much sense: http://daifukkat.su/blog/archives/2014/09/07/nmk004_rom_dumping_part_2/

In the time since the last post I have finished dumping, reconstructing, and verifying the NMK004 internal ROM. With the help of David "Haze" Haywood, it has been hooked up to MAME. As a result, all of the previously listed games now have perfectly working sound. With the sprite priority fixes I submitted a while ago, I think this pushes the emulation quality of almost all games on NMK's hardware up to "acceptably accurate" at the very least. Expect to see these changes come to MAME within a week or two.

In August 2013, I had submitted fixes to the sprite priority emulation on NMK hardware. Later in the month, I decided to probe the Emuversal shoutbox for opinions on my work. While on the subject, I also mentioned the NMK sound's poor state, and that I hoped it would be fixed soon. Haze replied, and told me that the NMK004 internal ROM was probably only extractable with a decap. There was also a mention that someone speculated that it would be possible to extract the internal ROM by playing it out as sound data, but that the setup would be complex and that nobody had tried to do it yet.

After a bit, I asked for and was given some TLCS-90 datasheets (the same CPU used in the NMK004) as it was assumed that's what it was. Unfortunately, the NMK004 ended up not being the same MCU model as were in any of those datasheets, but I eventually found the correct one much later. The actual model seems to be a TMP90C840, datasheet available at: http://raidenii.net/files/datasheets/cpu/tmp90c840.pdf

Title screen of Vandyke

The next day, I spent some more time probing things and poking people on the Emuversal shoutbox, trying to get a feel for the possibilities for dumping. I concluded that I should try to play out the internal ROM with one of the tables, record the outputs, then analyze the sound files to recover the internal ROM. Within a few hours I had cooked up a sample ROM that attempted to play things out from the internal ROM. I did a short test in MAME just to verify that it would make sound, and it did. From there I made a reference ROM that I could use to compare the various values. By this time I had decided on using PSG note length as my dumping table, due to its long spread and easy reconstruction method.

I had tried to ask people to make a reconstruction tool for me, but this never materialized. I figured it'd be trivial enough, so I wrote my own (extremely ghetto) tool that could analyze the resulting WAV files from recordings. My recording setup involved using an RCA->3.5mm adapter to connect the sound output of my supergun to my PC's microphone port. From there I would open Audacity and let that record until it stopped. When it would stop, I would save the output as a WAV file, perform the conversion, then compress the WAV file down to a good quality OGG and throw away the WAV file. Writing the tool for analysis took a good couple hours, though the tool ended up being pretty solid and worked very well.

After the tool was written, I started dumping the first 256-byte block of the internal ROM. This block was one of the quicker ones to dump, and took five and a half hours for just the first 256 bytes. The next day, I decoded the dump, and got my first validation that everything I was doing was correct and working: "All Music,Effect Software(C)1990 N M K Corporation". You wouldn't believe how excited I was when I saw this message, one that is likely to have never been seen by a single person in literally 20 years, in the converted dump. Unfortunately, the rest of the dump was far from as exciting. Waiting 8 hours for each 256 bytes to dump is absolutely not the best way to keep motivation high, especially when the ROM to dump is 16384 bytes long. Each block takes so long to dump because of the way I'm doing the dumping. Since I was using note lengths, a small 16-bit value would dump extremely quickly, and a high one would dump extremely slowly. For reference, a $FFFF value takes about 7 minutes to dump. Multiply that by 128, and a 256-byte block of $FFFF takes 15 hours to dump. Luckily nothing was that absurd in the blocks that I had dumped.

Eventually I gave in to the length. Sadly, looking back on it, I noticed I was very close to the end of actual data. My last set was $1200-$12FF, and the actual contents of the ROM end at approximately $1400. Everything from $1400 to $1FFF is just an $FF fill, which would have taken ages to dump if I had bothered to do so. It's been about a year since that last excursion, and I'm very glad that I went back and finally got it done. The previous attempt was a serious headache, and I knew that if I made the procedure less of a total headache, I could get it done.

I realize this hasn't been that technical of a write-up, but the lack of notes makes it not too easy for me to write about the technical side and what didn't work. What it essentially came down to, technically, was setting the Shared note length table's pointer to point into the area to dump, then having a bogus song that would play out a note of each length. I seem to recall I also used a padding note between each actual dump note, to help segment them up and make the conversion tool simpler, but I couldn't say for sure anymore.

The next post will be around in a few days, detailing the final attempt all the way down to the nitty gritty. I'm still undecided how I want to release the ROM, but I think I'll figure it out before publishing the next post.