SPC7110 Test Build

Post by **Nach** » Sun Jul 20, 2008 4:05 pm

This is v1.51, with the SPC7110 pack loading code removed (although I didn't remove the GUI stuff for it yet), and real decompression added.

Save states with this version are not compatible with SPC7110 save states of pack loading versions, however SRMs should remain compatible.

Thanks for the code is as follows:
SPC7110:
Research by Dark Force, zsKnight, and The Dumper.
Implementation by zsKnight, Jonas Quinn, and Nach

Decompression:
Research by neviksti, Andreas Naive, and jolly_codger
Implementation by neviksti, Nach, and TRAC
With lots of help from CaitSith2

I spent the better part of two days trying to optimize as much as I could the decompression code, since it was really slow, I was only getting 16 FPS on my 2.5 GHz machine in some cases. Now I get 60/60 and it uses ~20% CPU.

Games are:

Code: Select all

NSRT v3.5 - Nach's SNES ROM Tools

-------------------------Container--------------------------
       File: Far East of Eden Zero (J).gz
   Sub File: Far East of Eden Zero (J)
---------------------Internal ROM Info----------------------
       Name: HU TENGAI MAKYO ZERO     Company: Hudson Soft
     Header: None                        Bank: HiROM
Interleaved: None                         ROM: 40 Mb
       Type: SPC7110 + RTC               SRAM: 64 Kb
  Expansion: None                     Battery: Present
    Country: Japan                      Video: NTSC
  ROM Speed: 120ns (FastROM)         Revision: 1.0
   Checksum: Good 0xDE89            Game Code: Marked, AZRJ
---------------------------Hashes---------------------------
      CRC32: 1E327BD9
        MD5: 29998124A67DEDA111D5ACA8AAF5DCA5
     RIPEMD: 4B4BFFEAC1B74E840F4E717BB19E6CC6475BEE43
      SHA-1: 91FF16DA242D39736AEAD4D360BF6BF7C0AFCE82
    SHA-256: 8620203DA71D32D017BB21F542864C1D90705B87EB67815D06B43F09120318AA
    SHA-512: 9A9D7BD00C1D5B89D04A721D84D1938E103F09BAEB286BDD81762BD68EF6C4C0
             B255C939350EE5A8474E14DD4B08AA6D44DFE1AADD05C765B88958F04F3EDEDC
      Tiger: 7B8878874CE01F2A6945B163CEE43823955A980079FD17A1
  Whirlpool: E441C19D24BB0E70411FFF1D0B42B35937170456206C2127AF91983C81BEAC60
             E7A6F849427E962A09D9E22D55B30004D472A3960D73C91A339834ED35E51C87
--------------------------Database--------------------------
       Name: Far East of Eden Zero
    Country: Japan                   Revision: 1.0
     Port 1: Gamepad                   Port 2: Gamepad
    Genre 1: RPG                      Genre 2: Unknown

-------------------------Container--------------------------
       File: Far East of Eden Zero - Shounen Jump no Shou (J).gz
   Sub File: Far East of Eden Zero - Shounen Jump no Shou (J)
---------------------Internal ROM Info----------------------
       Name: JUMP TENGAIMAKYO ZERO    Company: Hudson Soft
     Header: None                        Bank: HiROM
Interleaved: None                         ROM: 40 Mb
       Type: SPC7110 + RTC               SRAM: 64 Kb
  Expansion: None                     Battery: Present
    Country: Japan                      Video: NTSC
  ROM Speed: 120ns (FastROM)         Revision: 1.0
   Checksum: Good 0x2EE2            Game Code: Marked, AZQJ
---------------------------Hashes---------------------------
      CRC32: BF8FEF7A
        MD5: A7438CD69E608BD2B268DDC0ED05C2CD
     RIPEMD: E283BF95078D166AD748E344ED9827B3374B45D9
      SHA-1: 36E5099128088B95623FF77F8196FCECD3F26731
    SHA-256: 69D06A3F3A4F3BA769541FE94E92B42142E423E9F0924EAB97865B2D826EC82D
    SHA-512: 61129E6CC759C65BCA4208B7C433CF0550B7A6E949E09B2F3E4A08FD4445D308
             BF663C2C19956BEA8348BF2991E538D5FC680D59BC828B8F67B4EC6A7CBEC019
      Tiger: 8412D4B472EC69C9632A64EA87E375FF6DA9D15B147B17B5
  Whirlpool: BF8EA37FCCD8209AA560836D4240FF69A6BBAE6751BBEAB851C36D551E7FBC0B
             48492136A289C66A1109B4A733D062B1608AB260C22A5084CEE35BC205AE7FCE
--------------------------Database--------------------------
       Name: Far East of Eden Zero - Shounen Jump no Shou
    Country: Japan                   Revision: 1.0
     Port 1: Gamepad                   Port 2: Gamepad
    Genre 1: RPG                      Genre 2: Unknown

-------------------------Container--------------------------
       File: Momotarou Dentetsu Happy (J).gz
   Sub File: Momotarou Dentetsu Happy (J)
---------------------Internal ROM Info----------------------
       Name: MOMOTETSU HAPPY          Company: Hudson Soft
     Header: None                        Bank: HiROM
Interleaved: None                         ROM: 24 Mb
       Type: SPC7110                     SRAM: 64 Kb
  Expansion: None                     Battery: Present
    Country: Japan                      Video: NTSC
  ROM Speed: 120ns (FastROM)         Revision: 1.0
   Checksum: Good 0xE28C            Game Code: Marked, AH8J
---------------------------Hashes---------------------------
      CRC32: 0B8D1474
        MD5: 18862A20E311F8CA08D064190C25504F
     RIPEMD: 47BCFF141F8EDD2C39A476E0D16C95E9766D4D57
      SHA-1: A0EEBF715C93751C6423DC277717E365D69C735F
    SHA-256: 9FC7A66464E71D0F056FED2B560F527A5AF69034C96293A2731107479763A9D8
    SHA-512: B354DEBEA0430D7E54B4D4B646FE27FDFBBE9E1BA778C8B388C173B982827A16
             5DA5F588FD4DBCFF5311EA9B1B0E4625B9BDC38635D0B9A5C3985D5D9C6F135C
      Tiger: 2FC55D6A656BF7DA4CABA92A5139EB9DA0CAFE9B716010F1
  Whirlpool: 4D4C68401B267FDA2F2777A770A174F534DFD97245F1B6E530307A20BE819CA9
             673D4BD9BD7F7561E3F2D375C40B4CDA43B9B7C09F7E9567A65529C2A08F1B15
--------------------------Database--------------------------
       Name: Momotarou Dentetsu Happy
    Country: Japan                   Revision: 1.0
     Port 1: Gamepad                   Port 2: Gamepad
    Genre 1: Board Game               Genre 2: Unknown

-------------------------Container--------------------------
       File: Super Power League 4 (J).gz
   Sub File: Super Power League 4 (J)
---------------------Internal ROM Info----------------------
       Name: SUPER POWER LEAG 4       Company: Hudson Soft
     Header: None                        Bank: HiROM
Interleaved: None                         ROM: 16 Mb
       Type: SPC7110                     SRAM: 64 Kb
  Expansion: None                     Battery: Present
    Country: Japan                      Video: NTSC
  ROM Speed: 120ns (FastROM)         Revision: 1.0
   Checksum: Good 0x01AA            Game Code: Marked, AE7J
---------------------------Hashes---------------------------
      CRC32: C07F099D
        MD5: FEBCA3A92D34FFA119891247D9B27D09
     RIPEMD: B62CD87A62ED17AECD7EB0C0D99D97E220B7A705
      SHA-1: 632061AE4892EC6CD75616DA66F7BE5C75E8D7AF
    SHA-256: 0847C8F8021AF9C7C139C98312E4974308650C2CBCB617161BB56C46626258BB
    SHA-512: 26D9FD331BF78D6BD5D1545638C1D5AE80A5399A5A550C50828A439124287931
             8BCB224AC43874FB0F3CE4C6649A54ABC5D9887DD1247EDB705D6891EB687DBF
      Tiger: BE71AE8FA36AAE7DA599905228E7DC287F7A6D34CB877EFA
  Whirlpool: 9EA72BA133C75DBA075507E5536267155040A8934C821AE1F58989CB1EFFE242
             D42D3FE0EB664ED1961E339DD8C57B58109218AA8ACE6EED5A08946EBE198B4B
--------------------------Database--------------------------
       Name: Super Power League 4
    Country: Japan                   Revision: 1.0
     Port 1: Gamepad                   Port 2: Gamepad
    Genre 1: Sports                   Genre 2: Baseball

Report how things work. What kind of FPS do you get. Which CPU do you have? How much CPU does it use?

Also play around the games a bit and see if something isn't decompressing. I'm hoping I didn't cut any important corners, so if a graphic appears missing, or severely garbled, please let me know.

Also test save states and make sure they're working good.

Thanks guys.

http://zsnes.sourceforge.net/zsnesw-spc7110.zip

Obligatory very large expansive set of screenshots to show you something which we were already able to do:

Turambar · Post by **Turambar** » Sun Jul 20, 2008 4:57 pm

I tried all the games mentioned, but the baseball game and Momotarou Dentetsu Happy were so retarded that I focused on Far East of Eden Zero.

Zsnes passed all SPC7110 tests, and graphics and sound were just fine, no corruption at all. My CPU is AMD Athlon-XP 2600+ and the OS used was Windows XP. During the intro in FEoE0 CPU usage was ~30% (60/60 FPS), but when I gained control of the character CPU usage rose to about 75% and while I walked around I got 57-59 FPS. A bit laggy. Inside the houses I got stable 60/60 though. That's probably due to less graphics and such loaded. Yes, and the only time CPU usage hit 100% was when I played that baseball game. When the ball flew over the field, FPS dropped dramatically and the game was unplayable.

Savestates worked flawlessly. I didn't test SRAM.

byuu · Post by **byuu** » Mon Jul 21, 2008 12:50 am

Wow, I can't believe how demanding SPC7110 emulation turned out to be ...

Okay, this is tested on a Pentium IV-M 1.8GHz CPU with FEoEZ.

The rain part of the intro gets 40-55fps in ZSNES, 31-38fps in bsnes. The cart race part gets 30-50fps in ZSNES, 24-37fps in bsnes. On the plus side, most of the intro can maintain 60fps with ZSNES.

Also, I noticed a "bug" in ZSNES, which I've tested against the real cartridge.

On hardware, the first time you run the intro, the maneki neko will attack the statue, which appears to be a claw attack, and the statue jumps back. The caption at the bottom is "gakoun." Every time after you reset and watch the intro again, the neko fires a laser beam whilst the statue fires missles, and then the statue raises some sort of vertical shield to block the beam. The caption here is "dodoun."

Whereas with ZSNES, every time you play, the maneki neko attacks with the laser beam, and the statue fires missles. The statue's vertical shield never appears, and the caption is always "gakoun."

Not sure what's up with that, but it's a relatively minor issue.

zidanax · Post by **zidanax** » Wed Jul 23, 2008 10:19 am

Testing on an Athlon 64 X2 4800+ overclocked to 2.75 Ghz. Most of the games ran 60/60, and with very little CPU usage. The exception is Super Power League 4. Once you're actually on the field, CPU usage goes up, especially when the ball is hit. In that case, CPU usage spikes for a moment, though not quite to 100%. Nevertheless, the game "skips" for a moment.

Post by **Nach** » Thu Jul 24, 2008 7:31 pm

Okay, I got a new build up. Thanks for the feedback everyone.

http://zsnes.sourceforge.net/zsnesw-spc7110-2.zip

I optimized neviksti's decompression code a bit more, but this isn't the good part. I wrote a completely new caching algorithm which is able to cache the complete decompressed data for any game, and search for entries with an upper bound of O(6). What this means, is that once the game already saw a graphic, it'll be able to recall it instantly. Some initial screens may be slow, but if you ever come there again, it should be full speed. So for example, the first time or two watching the FEoEZ intro, I would have it dipping to 55 FPS on occasion, but after that, all repeat loops are full speed. Also, how long it dips for initially should be less as well, since the games keep on re-decompressing graphics multiple times on each screen, now, that's been completely eliminated. Hold down fast forward, you may be surprised. In SPL4, in the field, I'm now able to get 196/60 FPS. I also redid the save state code again. It should now generate save states approximately 64KB smaller than before, which are obviously incompatible with previous build. I also double checked the movie code, so that should work too, unlike the previous build which had some issues there.

Todo:
Finish optimizing neviksti's decompresser so initial loads are even quicker.
Perhaps offer an option to save the cache to disk, so every time you reload a game, it doesn't have to rebuild it. Let me know if you guys want this or not.
Clean out the old pack path stuff from the GUI and the config file.

Let me know how this new build is working for you, if there's any issues or not. Please post your results with FPS, and your CPU, and what you think of it.

Thanks.

Edit:
Er, redownload that, byuu caught a bug that I broke the Press A test if you started a game without an SRM. That's fixed now.

funkyass · Post by **funkyass** » Thu Jul 24, 2008 7:37 pm

yes, make the cache persistent

is it possible to dump all the graphics in one go?

Post by **Nach** » Thu Jul 24, 2008 8:04 pm

funkyass wrote:yes, make the cache persistent

Okay, that's so far 2 votes for.

funkyass wrote: is it possible to dump all the graphics in one go?

Not to my knowledge. Although if CaitSith2 comes up with a new trick, perhaps

sweener2001 · Post by **sweener2001** » Thu Jul 24, 2008 8:57 pm

i would also vote for a persistent cache

creaothceann · Post by **creaothceann** » Thu Jul 24, 2008 10:29 pm

Me too.

(Though wouldn't that be like replicating the graphics packs?

)

caitsith2 · Post by **caitsith2** » Thu Jul 24, 2008 10:40 pm

May well be replicating graphic packs, although, it won't be depending on them anymore. If the cache were deleted, it would just redecompress the data again. The cache would just be a big time speed boost.

Post by **Nach** » Thu Jul 24, 2008 10:43 pm

creaothceann wrote:Me too.

(Though wouldn't that be like replicating the graphics packs? )

I'll probably use the same format, or a very similar one.

The differences now though are that you don't need to get graphics packs, they're generated on the fly. The packs generated this way are also "perfect", no issues of it being an over or underdump. My new caching algorithm also can fix packs when it notices the size is wrong in either direction, something which we couldn't do before.

Also, in overall usability once created, the algorithm I'm using is magnitudes better for caching than what we had in ZSNES before, or any of the 3 options Snes9x provided. All those ways needed more complicated searching to find the right graphic, and possibly needed to load it off the hard drive, now it's all in memory, very quickly indexed, and when applicable, new graphics can be decompressed.

Rashidi · Post by **Rashidi** » Fri Jul 25, 2008 1:22 am

just in case new .ips/.ups maker would works on FEoEZ (or any other SPC7110 roms)..
is there something that patcher (soft or hardpatch otherwise) needs to be aware, about persistent cache?

i do sometimes run the game with or without softpatch interchange-ly, especialy on non-complete/perfect or glithced patch... (FEDA/Shin Megami Tensei)

i also vote for persistent cache

paulguy · Post by **paulguy** » Fri Jul 25, 2008 4:48 am

I imagine the cache doesn't effect ROM data much other than graphics. If you're hacking graphics, though, I imagine you'd just have to delete the cache file(s).

Post by **Nach** » Fri Jul 25, 2008 7:53 am

Rashidi wrote:just in case new .ips/.ups maker would works on FEoEZ (or any other SPC7110 roms)..
is there something that patcher (soft or hardpatch otherwise) needs to be aware, about persistent cache?

No.

I'm making the code flexible enough that it'll work with any new SPC7110 games found or invented. I'm also going to name the cache after the hash instead of the filename, so you can turn on and off patches, and not have cache files clash.

Nightcrawler · Post by **Nightcrawler** » Fri Jul 25, 2008 1:38 pm

Sounds nice Nach. I'd throw in another vote for allowing persistent cache as well. Retaining the playability speed that ZSNES used to have with the graphics packs (more or less) sounds like a nice feature people would appreciate I'm sure.

Post by **Nach** » Fri Jul 25, 2008 3:41 pm

Persistent cache is now implemented, by the overwhelming unanimous vote for it. I'll post another build once I get overall decompression optimized more.

Till then, please report if there's any bugs with the current build, or save states with it.

Deathlike2 · Post by **Deathlike2** » Sat Jul 26, 2008 4:05 am

Nach wrote:Persistent cache is now implemented, by the overwhelming unanimous vote for it. I'll post another build once I get overall decompression optimized more.

Is this persistant cache going to be compressed and/or use a special folder path (or something better, that I probably can't think off the top of my head.

funkyass · Post by **funkyass** » Sat Jul 26, 2008 7:58 am

would this benefit from threading at all?

Post by **grinvader** » Sat Jul 26, 2008 10:41 am

... Stop trying to put threading in damn SNES emulation.

byuu · Post by **byuu** » Sat Jul 26, 2008 4:20 pm

funkyass wrote:would this benefit from threading at all?

No, each bye depends upon the previous byte(s) for context. And we currently give it the decompressed data immediately after it requests it (which may or may not be correct), so you can't do other things for a bit. You can't parallelize the decompression.

Post by **Nach** » Sat Jul 26, 2008 6:59 pm

Deathlike2 wrote:
Nach wrote:Persistent cache is now implemented, by the overwhelming unanimous vote for it. I'll post another build once I get overall decompression optimized more.
Is this persistant cache going to be compressed

I didn't think of that, but perhaps that's not a bad idea.

Here's the size usage just from running for a few moments:

Code: Select all

-rw-r--r-- 1 nach nach  241952 Jul 25 18:56 /home/nach/.zsnes/0B8D1474.gfx
-rw-r--r-- 1 nach nach     390 Jul 25 18:56 /home/nach/.zsnes/0B8D1474.idx
-rw-r--r-- 1 nach nach 1189722 Jul 25 17:46 /home/nach/.zsnes/1E327BD9.gfx
-rw-r--r-- 1 nach nach    2544 Jul 25 17:46 /home/nach/.zsnes/1E327BD9.idx
-rw-r--r-- 1 nach nach  662646 Jul 25 17:47 /home/nach/.zsnes/C07F099D.gfx
-rw-r--r-- 1 nach nach    5064 Jul 25 17:47 /home/nach/.zsnes/C07F099D.idx

The largest I think any .gfx will get is ~12MB, but I'll probably use gzip.

I.S.T. · Post by **I.S.T.** » Sat Jul 26, 2008 7:13 pm

Or possibly Rar?

*runs as far away from the thread as humanly possible*

Rashidi · Post by **Rashidi** » Sat Jul 26, 2008 9:11 pm

Nach wrote:Here's the size usage just from running for a few moments:

Code: Select all

-rw-r--r-- 1 nach nach  241952 Jul 25 18:56 /home/nach/.zsnes/0B8D1474.gfx
-rw-r--r-- 1 nach nach     390 Jul 25 18:56 /home/nach/.zsnes/0B8D1474.idx
-rw-r--r-- 1 nach nach 1189722 Jul 25 17:46 /home/nach/.zsnes/1E327BD9.gfx
-rw-r--r-- 1 nach nach    2544 Jul 25 17:46 /home/nach/.zsnes/1E327BD9.idx
-rw-r--r-- 1 nach nach  662646 Jul 25 17:47 /home/nach/.zsnes/C07F099D.gfx
-rw-r--r-- 1 nach nach    5064 Jul 25 17:47 /home/nach/.zsnes/C07F099D.idx

The largest I think any .gfx will get is ~12MB, but I'll probably use gzip.

looking how good SPC7110 compress data was, i imagine using SPC7110 function on non SPC7110 roms could really benefit patching space problem.
i don't think patched rom with added SPC7110 functionality may properly playable on real cartridge, but that not a problem though at least for me.

Post by **Nach** » Sat Jul 26, 2008 9:53 pm

Rashidi wrote:
Nach wrote:The largest I think any .gfx will get is ~12MB, but I'll probably use gzip.
looking how good SPC7110 compress data was, i imagine using SPC7110 function on non SPC7110 roms could really benefit patching space problem.

You're better off with deflate.
Deflate gets the packs 5-7% smaller than the SPC7110 compression does.

Edit:
gfx files are now gzipped.

Code: Select all

-rw-r--r-- 1 nach nach 120934 Jul 27 00:17 /home/nach/.zsnes/0B8D1474.gfx
-rw-r--r-- 1 nach nach    390 Jul 27 00:17 /home/nach/.zsnes/0B8D1474.idx
-rw-r--r-- 1 nach nach 572960 Jul 27 00:19 /home/nach/.zsnes/1E327BD9.gfx
-rw-r--r-- 1 nach nach   2544 Jul 27 00:19 /home/nach/.zsnes/1E327BD9.idx
-rw-r--r-- 1 nach nach 252561 Jul 27 00:17 /home/nach/.zsnes/C07F099D.gfx
-rw-r--r-- 1 nach nach   5064 Jul 27 00:17 /home/nach/.zsnes/C07F099D.idx

adventure_of_link · Post by **adventure_of_link** » Sat Jul 26, 2008 11:30 pm

I know this is unrelated to the SPC7110 but it needs to be asked:

That cache feature you guys were planning on using for the SPC7110 games, would it help the S-DD1 games speed up as well?