trial to track running code with zsnes

[kentaro-k.21]

hi.

i'm a person who loves to play/research the game Dungeon Master.
and now, i want to extract some information from Dungeon Master ROM data.

once i have extracted gfx/sound data from ROM data.
it costed much time. and it is solved by enough time and patience. now i want to extract another "metrics" data.

about the method how to extract them, it is trial and error. just modifiy zsnes source code to set a trap, and run zsnes to catch the few information by the placed trap. after that, guess next step. set the trap at next possible point. i just repeated these steps for 100 or more times.
this method surely leaded me to the goal, but it costed exorbitant my free time and effort.
so, i decided to code an automated tracker in zsnes. it is to extract some useful info from ROM data thru zsnes.

why automated tracking?
because handwork data extraction is very tough. half or full automated work surely reduces handwork costs.

can automated tracking?
probably. not sure.

how automated tracking can be done?
IMHO it is possible because usual programs use module to separate their problems.
a module is a collection of program. e.g. they are "monster AI", "opening sequence of game", "data decompressor", "data decrypter" and so on.
a problem should be a request. e.g. "write a program to chase the moving player effectively" -> "create a monster AI system"
to solve a problem, a module is built. a module should consist from many amount of program codes.

for example, i write a module to decompress the compressed image. and the module is to decompress the image. and it is not for other purpose (e.g. monster AI).
this means each problem should be separated in module unit.

thus, it is useful to chase the related modules against a target module.
for example, Dungeon viewport renderer will use image decompressor to get creature image. and it is useful to chase who use image decompressor.
at first, you can have a clue to how to use the decompressor. next, you can know a clue where the creature should be rendered ("where" means exact x and y coordination).

MOD overview

currently i'm developing a MOD version of zsnes 1.42.
i'm working to create/attach 2 new features into MOD version.

1) disassembler
2) capture/display CPU stepping graph and call stack in low-level analyze

this MOD should be developed as separated version from main zsnes development.
because their features are not needed for most game players. and their features also decrase the emulation performance.

this version cannot run on MS-DOS/Linux OS. it runs on Windows OS.
because MOD version actively uses MFC and STL/ATL template class technologies to realize these features.
especially MFC and ATL uses Windows API, where MS-DOS/Linux OS don't implement.

MOD progress

1) disassembler

IMHO it works fine. but it is just disassembler, NOT DEBUGGER!
and it does NOT track the program on RAM memory, tracked only ROM memory.

screenshot:
http://homepage3.nifty.com/kkdf/images/006/disasm.png

2) capture/display CPU stepping graph and call stack in low-level analyze

i have problem on this step.
this feature works for the present, but its output isn't good. (currently this feature has vulnerable to cause stack overflow)

point of this feature is to trace the branch/jump/return opecodes.
IMHO ordinal program can be analyzed in this way. but SNES games seems to have a kind of code protection. it prevents this decoder from track sample data to build the graph.

anyway i continue to improve this feature. if you have comments or suggestions, please post here.

screenshot:
http://homepage3.nifty.com/kkdf/images/006/graf.png

DL
win32 only: (early trial version)
http://homepage3.nifty.com/kkdf/zsnes_f ... 051501.rar
src: (zsnes only)
http://homepage3.nifty.com/kkdf/zsnes_fprintf_src.rar

[Nightcrawler]

Here's something that could possibly save you alot of time.

Geiger’s Snes9x Debugger
already does everything you are suggesting. It is on top of SNES9x though rather than ZSNES. However, it works great and is fully featured.

Also ZSNES DOS also already has a trace logger and real time debugger. Much less featured, but usable nontheless.

If both of these options are not what you're looking for than I continue on my friend. It certainly wouldn't be a bad thing for ZSNESW to have some sort of debugging capabilities.

[kentaro-k.21]

hi.

i could understand what i wanna say. and time to talk...

my plan is to create a data mining (or data logger?) tool within zsnes. it is different from SNES debugger.
currently i don't think exactly what is the data for mining. but it should be registers or stack memory around stack register points.

and first problem'll be "where" would you capture data from. (you specify a mining point and mine the data. in this way, first problem is to specify mining point)
in order everyone to easily select the mining point, display a kind of diagram or flow graph.

IMHO it can be displayed, because snes game program uses sub-routine idea to split large program into small ones. i thought it could be done simply by tracking the pair of JSL/RTL and JSR/RTS and so on.
but i couldn't. i continue to search for the good method to do it.

[Nightcrawler]

The data logging feature your suggesting is referred to as trace logging. Geiger's Debugging version or the old Evil Peer's Tracer version of SNES9x will log all the registers, stack pointers, etc.. to a file. It's essentially dissassembly during real time output to a file.
ZSNES DOS can also do this.

I can send you a sample output file if that will help our communication barrier.

I'm pretty sure what you want to do is already done.

[kentaro-k.21]

i knew Geiger's Debugger and its logging feature are enough to work with rom code/data, but it was not all i wanted.

i noticed what i wanted. it is logging feature with custom formatting and freely filter able. but i restrict the objective to analyze the exist program coded in rom data. (e.g. trace the program's behavior by using CPU Logging feature)

what's different between my goal and Geiger's Debugging version's output.

1. custom format?
Geiger's Debugging version generates log in "fixed" format. it is not always enough to analyze the assembler code meaning.

2. freely filter able?
Geiger's Debugging version can generate log in every single step. it is too many to extract required information.


i always modified my zsnes code to gain the objective log which satisfied above 2 criteria whenever i was willing to decode the rom program (e.g. data decompressor).
it would be great if any snes emulators (which can produce customized/filtered log for every person) exist...

thanks.

[Overload]

I've already asked this question on my Discussion Board. I want to add a number of predefined trace formats to the next version of Sleuth. A custom mode isn't out of the question.

[Edit]
I'm sure what you mean about filtered logs?

[Nightcrawler]

i knew Geiger's Debugger and its logging feature are enough to work with rom code/data, but it was not all i wanted.

i noticed what i wanted. it is logging feature with custom formatting and freely filter able. but i restrict the objective to analyze the exist program coded in rom data. (e.g. trace the program's behavior by using CPU Logging feature)

what's different between my goal and Geiger's Debugging version's output.

1. custom format?
Geiger's Debugging version generates log in "fixed" format. it is not always enough to analyze the assembler code meaning.

Geiger's output does not give you enough information to analyze the assembly code meaning? What do you mean? What additional information could you ask for that it does not provide? I have reverse engineered countless routines with it now with no problem.

2. freely filter able?
Geiger's Debugging version can generate log in every single step. it is too many to extract required information.

Geiger's debugger allows you to specify the range of addresses to output. You can tell it what address read/write you would like the trace to start on and how many times to loop, and when to stop. It is very customizable in that aspect. You do not have to trace any uncessecary data.

i always modified my zsnes code to gain the objective log which satisfied above 2 criteria whenever i was willing to decode the rom program (e.g. data decompressor).
it would be great if any snes emulators (which can produce customized/filtered log for every person) exist...

thanks.

I really think Geiger's debugger is already extremely flexible. Perhaps you don't really understand the extents of what it is capable of doing. It really does do alot and even I don't understand all of it's features. I doubt you will see any debugger come out that is much more flexible.

If you really are unhappy with the current tools, you should get started in making one of your own because it seems unlikely someone will come along with something that fits your vision.

[byuu]

Geiger's debugger allows you to specify the range of addresses to output. You can tell it what address read/write you would like the trace to start on and how many times to loop, and when to stop.

Isn't that what breakpoints are for? Or at least, should be for?

What he's saying is that Geiger's (and everyone else's) spits out every single instruction that's been executed. What he is looking for is something where he can tell it what he wants logged, probably something like a file like below that he can feed to the debugger.

Code: Select all

cpu.range = c00000-c08000, c12000-c13ffff ;log these CPU range opcodes
cpu.condition = A.W > 0x8000 && P.C && S <= 0x0180 && S != 0x0140 ;log CPU opcodes when conditions are met
cpu.condition = !P.I !required ;never log anything when inside an interrupt
dram_read.range = c10000-c12000 ;log all reads from these addresses
dram_write.range = 7f0000-7f0200 ;log all writes from here
dram_readwrite.range = [00-3f,80-bf]:[6000-7fff] ;catch chip messages

Or maybe something similar that instead acts as a binary dumper, like:

Code: Select all

start: cpu.PC == 0xc08200 ;start of decompression routine
log: file = "output%d.bin" ;%d auto increments each time a log happens
log: dram.write == 0x2180 ;write all decompression data to file
end: cpu.PC == 0xc09cf0 || cpu.PC == 0xc09cf8 ;end of decompression routine

The performance of such a debugger would no doubt be terrible, though, having to check all of those conditions every time anything at all happens. Not to mention the learning curve would be insane for such a tool.

Personally, it seems like too much work for too little use to me. This is really something that should be written directly into the emulator by people who need it on a case-by-case basis. All of the above could be done in under a minute inside my snes->notify() routine where every event the SNES performs is sent there. If people don't know how to program in c++, but they want super advanced assembly logging tools... well, too bad.

What I always used to use for data decompression was a custom standalone CPU emulator library. Then I would just write in the start and end address in a quick c++ app, and test for whatever conditions I wanted to log data while it was running. Whenever it stopped, I'd modify RAM/registers/whatever, and loop again until I got everything out. I was able to extract the scripts for Bahamut Lagoon, DQ5, the font for DQ5, and ALL of the fonts + graphics for DL this way all in a matter of minutes. This would be impossible even with the above logs as you can't force the CPU to loop through grabbing each and every compressed block this way.
The actual decompression routines very rarely rely on all the subtle features of the SNES being present (interrupts, APU, PPU, etc) anyway.

These are a couple of years old, and I didn't comment much because well, it only needed to be done once, but here's one for DQ3r and one for DL that I wrote. Sorry to mask the offsets in DL, but that work is not public domain.
http://byuu.org/files/scdump.c
http://byuu.org/files/decomp.c

[kentaro-k.21]

Nightcrawler:
in the first post, i had an idea to help to analyze the procedure by displaying code diagram. but it is too difficult for me. and i noticed this idea was wrong.
you suggested me to use exist debugger, also to use logging feature. i support your opinion. it would be the best way to do reverse engineering rom routines.
so i tried both.
breakpoint feature in Geiger's helped so much. it was useful to locate the routine of data decompressor. it is easily done by installing memory read breakpoint.
logging feature also helped as well. it was useful to peep the input/output of data decompressor via CPU registers. i can guess the working of specific rom code (e.g. track the initialize code of data decompressor).

BUT, the log contains many unwanted information if i tweaked the Geiger's emulator to capture CPU tracing log in every emulator's CPU single step.

i just wanted to have automated way to eliminate the unwanted part of log. or having log output with customized format.
i would tweak the customized zsnes source code and run the rom code with it, rather than coding a new tool to extract required information from the very large CPU single stepping log.

If you really are unhappy with the current tools, you should get started in making one of your own because it seems unlikely someone will come along with something that fits your vision.

yes.

byuusan:
thanks for the idea of formatted feeding code.
but i'll use exist script oriented programming language like python.

i'll implement a set of API in customized zsnes. then publish the API to script code. external script code'll call the API to read the emulator's memory, obtain CPU register value, and set a breakpoint. i believe recent script language has enough capacity to write log in customized format, and control the logging feaature...

Overload:
i knew about the great precision-oriented emulator Sleuth today.
i googled Sleuth, visit the site, watched the forum, and found your posts...
i'll continue to implement my wanting feature into my customized zsnes until i know Sleuth implements superior debugging support for logging feature...

thanks.

[kentaro-k.21]

finally i inherited a copy of zsnes source code, and have a new zsnes clone "zisazsnes".

you can get early alpha version and a poor documentaion if you are interested in.
early version: http://homepage3.nifty.com/kkdf/zisazsn ... 101444.rar
documentation: http://members.at.infoseek.co.jp/danmas ... /index.htm

thanks.

[pagefault]

finally i inherited a copy of zsnes source code, and have a new zsnes clone "zisazsnes".

you can get early alpha version and a poor documentaion if you are interested in.
early version: http://homepage3.nifty.com/kkdf/zisazsn ... 101444.rar
documentation: http://members.at.infoseek.co.jp/danmas ... /index.htm

thanks.

This is some interesting work. What are you using to create the log file? Are you reading it from the code in debug.asm? I haven't really had time to look over your code. I was planning to redo the debugger sometime in the future but what you are doing with scripts is a very interesting idea. Would there be any way to make this work in Linux?

[Nach]

Would there be any way to make this work in Linux?

We could use WINElib, or port the code to something standardized.

[Nightcrawler]

kentaro-k.21:

In regards to the unwanted information being dumped. One solution to this is to code a simple log parser. It can be a stand alone utility to open up and go through your log file and trim it down and edit it to your liking. That wouldn't be very hard to do at all.