bsnes v0.033 released

[Nach]

I'd appreciate if you don't make stuff up about me.

Your "software" vs. "real hardware" approach came up multiple times in discussion after byuu released his "state of emulation" rant. It was there that you made such comments.

Such comments as to what? That I prefer not seeing scanlines when playing a game? That I don't like a lot of different consoles when I can have it all on one machine?

I am not just making stuff up.

You're implying that my SNES being broken, and not caring enough to buy a new one means I don't like accuracy. That is made up. You also made up your interpretation. Have you ever known me personally to intentionally hack something up to be something completely different to speed things up?

And if you have discovered a "new" register, please do let us know.

I never claimed to find a new register, only to find out some info about existing ones which was had some question marks in our current doc. The statements were out of context, and you twisted it to say that I'm out to hack SNES emulation when I'm trying to figure stuff out, as per my thread in the dev forum.

I have run tests on the real hardware and it appears that what byuu said is correct "The chip has a length / decrement register, but it does no good since you can read forever from the decompression port. Many times, games do not set the register at all, so you can't use that." The only other registers you need to write to in oder to start a decompression cannot be used to determine a hardware decompression length. So the only option that seems reasonable to me is that you have found some emulation hacks specific to these games (probably DMA related)... and not found a new register. Of course, I could be wrong. New registers are exciting finds... please do share.

I've shared my request for more info on the dev board. My findings show a correlation between what decompression does with a register, and then what the game requests, I can post more in the dev board if you'd prefer my speculation.

[ZH/Franky]

Hey hey now, you're all working on the same thing. Team mates don't argue with each other. Why don't you both lay off each other?

[Dullaron]

Please no flame wars here. Take it to the pm or irc chat. Thank you.

[neviksti]

I am not just making stuff up.

You're implying that my SNES being broken, and not caring enough to buy a new one means I don't like accuracy. That is made up.

The fact that you don't have a snes and prefer it that way is not made up. And yes, that bothers me deeply. It doesn't necessarily mean you don't like accuracy, but it means that you can't possibly know what is accurate.

That doesn't mean I don't appreciate your hard work. For it is hard work, and it is appreciated.

Have you ever known me personally to intentionally hack something up to be something completely different to speed things up?

Since you are a software only person, your approach to things is often to search for patterns in everything ... and then there's this "Nachian" leap, somehow starting to believe these patterns have real independent meaning instead of just convenience. I have always taken issue with that, and is why we usually butt heads.

NSRT is a great tool, but any tool that makes the leap from ROM -> cartridge necessarily is just a database of cartridges. Any patterns of "chip info" (that page you always used to link to), are just patterns and have no real independent meaning. It really is just a database you found patterns to help summarize.

I remember a discussion came up several times about trying to get a snes format that would better represent the cartridge instead of just the ROM. MK even started proposals of formats and real discussions were taking place. You later did come out with a new format, JMA. But you seemed oblivious of the hardware aspects that were begging for a new format and instead focussed on compression. Reading this thread is just crazy to me: http://board.zsnes.com/phpBB2/viewtopic.php?t=1432
Most saw it for what it was, but some bought into your previous comments, and say stuff like:
"You should read this http://nsrt.edgeemu.com/INFO/chipinfo.htm : this is how you know how to map your rom correctly. And it's 100% perfectly working, it's not "heuristical" or anything."


Again, your tools and your work are appreciated. But as a hardware first person, our approaches severely clash when you make claims that seem to raise your found patterns beyond just patterns and represent new information about snes hardware. And comments like this seem disingenuous at best:

<Nach> Muahahaha!
<Nach> I RE'd a new register not covered properly in our existing docs
<Nach> and now I get the proper length to decompress

But now you request:

I never claimed to find a new register, only to find out some info about existing ones which was had some question marks in our current doc. The statements were out of context

So okay. Your statements on reverse engineering were out of context...
beyond that it is just a severe case of differring personal preferences on how to attack a problem. We're not going to agree on that, so let's leave it at that.

EDIT: Just saw previous posts.
I have no hard feelings towards Nach. I just want it to be clear why our approaches clash. I could probably use a reminder of why my hardware first approach should be tempered with a more software approach as well. We all have our preferences. In a perfect world there would be enough developers that they would fill the spectrum of these preferences to cover all bases. I am just pleading to move towards a more middle ground.

[Nach]

I am not just making stuff up.

You're implying that my SNES being broken, and not caring enough to buy a new one means I don't like accuracy. That is made up.

The fact that you don't have a snes and prefer it that way is not made up. And yes, that bothers me deeply.

That bothering you is an opinion, it really shouldn't matter.

It doesn't necessarily mean you don't like accuracy, but it means that you can't possibly know what is accurate.

Do you think perhaps you're biased on that?

Do I really need to own an SNES when others who are more talented than I am can write perfectly good documentation? Some things are also quite obvious. "I can't possibly know what is accurate" is inaccurate. I helped byuu with information on mirroring in bsnes, and it's accurate, yet I worked all the details out without a working SNES, does that mean the code is guaranteed incorrect, or this one case was somehow magical?

That doesn't mean I don't appreciate your hard work. For it is hard work, and it is appreciated.

Thanks.

Have you ever known me personally to intentionally hack something up to be something completely different to speed things up?

Since you are a software only person, your approach to things is often to search for patterns in everything

I'm not sure what you mean by software only, I work with hardware. However, I love compression, and I research things all the times which are pattern based, I love patterns. I don't think there should be something wrong with that.

... and then there's this "Nachian" leap, somehow starting to believe these patterns have real independent meaning instead of just convenience. I have always taken issue with that, and is why we usually butt heads.

I recall you one time using a phrase something like "I think based on the results, it seems that it does matter".

NSRT is a great tool, but any tool that makes the leap from ROM -> cartridge necessarily is just a database of cartridges. Any patterns of "chip info" (that page you always used to link to), are just patterns and have no real independent meaning. It really is just a database you found patterns to help summarize.

It doesn't help that half of them were listen in Nintendo's docs? And the others just happen to be followed? Sure, some of them are random chance, but not always. I found a pattern where 4 letters written in a ROM next to each other, in the exact same place, carry meaning, they even sometimes spell out the meaning I found it to convey, you think it's just some random pattern, that I "Nachinized"? (Thanks for making Nach a verb)

I remember a discussion came up several times about trying to get a snes format that would better represent the cartridge instead of just the ROM. MK even started proposals of formats and real discussions were taking place. You later did come out with a new format, JMA.

Yeah, discussing a new SNES header format is somehow slightly related to a general purpose compressed file format.

But you seemed oblivious of the hardware aspects that were begging for a new format and instead focussed on compression.

So because I work with the SNES Emulation Community on various projects, I'm not allowed to work on other projects? Or if I do, I can't also bring them to the SNES community?

But as a hardware first person, our approaches severely clash when you make claims that seem to raise your found patterns beyond just patterns and represent new information about snes hardware.

This almost sounds like a debate about whether evolution exists or not.

Your statements on reverse engineering were out of context...
beyond that it is just a severe case of differring personal preferences on how to attack a problem. We're not going to agree on that, so let's leave it at that.

Yes, I was quoted out of context, in fact, with a "..." followed by how I sped things up, which was actually due to me optimizing the decompression a bit (which CaitSith2 verified to still be bit for bit accurate to yours), and that I added caching. I find your comment on how I probably just put in hacks completely uncalled for.

[byuu]

NSRT is a great tool, but any tool that makes the leap from ROM -> cartridge necessarily is just a database of cartridges. Any patterns of "chip info" (that page you always used to link to), are just patterns and have no real independent meaning. It really is just a database you found patterns to help summarize.

Though I'm sure Nach agrees with us that they are mere patterns ... I have to strongly agree with this (as well as emphasize that NSRT is indeed an awesome tool). I've tried to convince Nach to store a copy of each game's PCB ID, since he's verifying carts anyway. This would allow me to map ROMs exactly as they appear on hardware. Whereas, what we have now are a large collection of "coincidences". For instance:

Code: Select all

  //research shows only games with very large ROM/RAM sizes require MAD-1 memory mapping of SRAM
  //otherwise, default to safer, larger SRAM address window
  uint16 addr_hi = (memory::cartrom.size() > 0x200000 || memory::cartram.size() > 32 * 1024) ? 0x7fff : 0xffff;
  map(MapLinear, 0x70, 0x7f, 0x0000, addr_hi, memory::cartram);

Sure, that works on all known dumped games, but is it correct? No. Someone hacking the game may attempt to use mirroring that is different on hardware, which would be bad.

And then we have code like this ...

Code: Select all

  //detect presence of BS-X flash cartridge connector (reads extended header information)
  bool has_bsxflash = false;
  if(rom[index - 14] == 'Z') {
    if(rom[index - 11] == 'J') {
      uint8 n13 = rom[index - 13];
      if((n13 >= 'A' && n13 <= 'Z') || (n13 >= '0' && n13 <= '9')) {
        if(company == 0x33 || (rom[index - 10] == 0x00 && rom[index - 4] == 0x00)) {
          has_bsxflash = true;
        }
      }
    }
  }

It works, yes. Beautifully. But I don't see how that has anything to do with hardware emulation. It's barely a step up from making an internal database of cartridge titles to identify hardware present.

And I won't even touch on the header scoring code here ...

I do really wish we could get PCBs, even if Nach doesn't want them. But I simply can't afford to get more than a few dozen carts, let alone getting the rare / expensive ones.

I could probably use a reminder of why my hardware first approach should be tempered with a more software approach as well. We all have our preferences.

I obviously side more with you on testing with hardware over software.

For instance, I remember finding an emulator's workaround for Uniracers writing to OAM mid-frame. He ignored the OAM address selection when writing $200, and only $200. Obviously not right, but it just so happened to mean the address was the expected value. A coincidence that didn't appear to break any other games, which was actually quite lucky since that particular change was global. There was no mention that it was a hack -- it would be really bad if another emu dev without a copier were to see that, and believe it to be correct.
Full disclosure: I have a global hack for Uniracers, myself.

But since you asked, there are many benefits to a software-based approach. There are much lower costs associated wtih software tests (no dragging out the SNES and the copier, hooking it up to the CRT TV, adding headers to your test programs, transferring them via floppy disks, running it through your copier, transferring the SRAM back to the floppy, moving that back to your PC and examining the results -- hoping no errors occured. And your PC is much less likely to fail from wear and tear.) Plus you gain massive speedups in work, and you can hook your emulator to give debugging information.

I usually make a test and verify it does what I want, and then run it on hardware. But yes, that last bit I find 100% vital before claiming anything. To be honest, I'm really uncomfortable with the entirety of the SPC7110 emulation code, except maybe the math unit. I have no hardware proof to base any of it on. The stop and swap trick is going to greatly reduce my hardware life, and greatly add to the complexity of running tests. So Nach getting a best guess on something means we can test that idea first, and potentially save a few swaps, which is definitely a good thing.

For what it's worth, I read "I RE'd a new register not covered properly" to mean the same thing. I figured he found out what $4807 or $4808 did. But then, I often misunderstand him ;)
In this case, it looks like "not covered properly" was the important bit.

[Nach]

And then we have code like this ...

Code: Select all

  //detect presence of BS-X flash cartridge connector (reads extended header information)
  bool has_bsxflash = false;
  if(rom[index - 14] == 'Z') {
    if(rom[index - 11] == 'J') {
      uint8 n13 = rom[index - 13];
      if((n13 >= 'A' && n13 <= 'Z') || (n13 >= '0' && n13 <= '9')) {
        if(company == 0x33 || (rom[index - 10] == 0x00 && rom[index - 4] == 0x00)) {
          has_bsxflash = true;
        }
      }
    }
  }

It works, yes. Beautifully. But I don't see how that has anything to do with hardware emulation. It's barely a step up from making an internal database of cartridge titles to identify hardware present.

See, it looks weird, but you completely miss the point, this isn't just some pattern.
Documentation I have from Nintendo says when (company == 0x33 || (rom[index - 10] == 0x00 && rom[index - 4] == 0x00)) that there is a 2-4 letter word from index - 11 to -14, I didn't just make that up out of nowhere. This (n13 >= 'A' && n13 <= 'Z') || (n13 >= '0' && n13 <= '9') checks to make sure it's a 4 letter codeword and not a 2 letter codeword. As for the codewords, I never found a list of them anywhere, but they do seem to have a standard on the 4 letter codewords. NP Menu ROMs all have the word "MENU" there, Japanese games IIRC always begin with an A and end with a J (why? who knows?), except in a few circumstances, and in every one of those circumstances, the ROM has special hardware, in this case 'Z', signifying it has an expansion slot. Why 'Z'? I have no idea. However I doubt with all the info before you, you can really say it wasn't intentional to use these 4 letter codewords to signify something.
Yes it looks weird without comments, it makes perfect sense if you know the header, and Nintendo's documentation on it.
Can you really say "It works, yes. Beautifully. But I don't see how that has anything to do with hardware emulation."?

[byuu]

Yes, but in the end, it's just bits in a ROM that the real SNES doesn't care about. You could easily patch out those bytes, burn yourself an EPROM, attach it to the cart, and it'd still run.

That emulators cannot say the same is a bigger problem for me than it is for you (and most sane people), I suppose.

[neviksti]

Can you really say "It works, yes. Beautifully. But I don't see how that has anything to do with hardware emulation."?

Yes, because it really truely doesn't have anything to do with hardware emulation. It only works beautifully for the games in your database.
And emulators are forced to do things like this because we only give the emulator a copy of the ROM and not a representation of the entire cartridge contents.

But since you asked, there are many benefits to a software-based approach. There are much lower costs associated wtih software tests (no dragging out the SNES and the copier, hooking it up to the CRT TV, adding headers to your test programs, transferring them via floppy disks, running it through your copier, transferring the SRAM back to the floppy, moving that back to your PC and examining the results -- hoping no errors occured. And your PC is much less likely to fail from wear and tear.) Plus you gain massive speedups in work, and you can hook your emulator to give debugging information.

I usually make a test and verify it does what I want, and then run it on hardware. But yes, that last bit I find 100% vital before claiming anything. To be honest, I'm really uncomfortable with the entirety of the SPC7110 emulation code, except maybe the math unit. I have no hardware proof to base any of it on. The stop and swap trick is going to greatly reduce my hardware life, and greatly add to the complexity of running tests. So Nach getting a best guess on something means we can test that idea first, and potentially save a few swaps, which is definitely a good thing.

Agreed.
Now we've all had our say. So moving on...


Has the swap method been working okay for you?
There were often little glitches with my decompressor setup that required me to reset several times. I could never figure out if it was something bizarre with the FIFO, or if we weren't quite setting up some decompressions correctly. I tested the best I could and it seemed to be a FIFO issue, but I'm really interested to see what you find as I may have missed something.

All the $48xx regions not listed as a register in DarkForce's document gives open bus values. So I think those are really the only registers. (And is probably why he lists that $4808 register.)

EDIT: If you end up writing a nice general purpose memory viewer / editor for the SNES, please share with me a copy. I've ran into limitations of my own memory viewer / editor a couple times but never decided on a better format.

As mentioned, you can load this using your hot-swap method to have a memory viewer running entirely in RAM
http://neviksti.com/SNES/SF2exp3
Just make sure to run it in the mode that has the copier's "save state" features disabled if possible, for the program was written for something else and may trigger a save state and jump to random code. I can upload one with the save state fiddling removed if necessary.

[Nach]

Can you really say "It works, yes. Beautifully. But I don't see how that has anything to do with hardware emulation."?

Yes, because it really truely doesn't have anything to do with hardware emulation. It only works beautifully for the games in your database.

Talking about truly, it'd be nice if you wouldn't use untrue absolutes. It works beautifully for official games not in my database as well. We dump new stuff all the time, and it doesn't disagree. The vast majority of the time, when I notice a pattern with a handful of ROM dumps that I check, it turns out to work with all the ones everyone has.

Sure, we're not guaranteed that it'll work for everything new we come across, but in the end, it does.

Yes the hardware doesn't check it. But that doesn't mean we should throw away standards Nintendo enforced on carts to get a seal of quality, or ignore something which seems to be a rule everyone intentionally followed.

[FitzRoy]

NP Menu ROMs all have the word "MENU" there, Japanese games IIRC always begin with an A and end with a J (why? who knows?), except in a few circumstances, and in every one of those circumstances, the ROM has special hardware, in this case 'Z', signifying it has an expansion slot. Why 'Z'? I have no idea. However I doubt with all the info before you, you can really say it wasn't intentional to use these 4 letter codewords to signify something.

Just to clarify, these are essentially internally stored versions of the middle code of cartridge serials. I can't pinpoint the exact date Nintendo switched over, but it was sometime in the mid 90s that they introduced a beginning and end digit to what was once only the two-digit catalog number. Once it became four digits, the beginning digit would denote the cartridge type and the end digit would become the localization. So if the catalog number is KU, then a code of ZKUJ would mean BS-X slotted cart, KU catalog number, Japanese localization. Not all games have the internal code, but they don't lie when they're there. Keep in mind that there is such a thing as "B" for the first digit as well, which signified that the game was first or only distributed on a Nintendo Power flash cartridge. This means that when you dump game data from an NP cart that first appeared on a standard cart, it's probably still going to have its "A" internal code. Since the data didn't need changing to work on flash carts, they didn't bother changing the code when making it available for the NP distribution.

If you dump hundreds of SNES carts like I do and record the serials, then you'd know how messy it is and why headers gain you only marginal "accuracy" brownie points, but would still be incomplete and selective. The reasoning is:

1. It's been documented that a game can have more than one PCB code yielding the same data. And it's fairly common. And since we can't buy 100 of every game for a decent sample size, we have no idea what "truly produced" codes exist beyond the few we happened to purchase. The best we can do would be to select that one that we buy.
2. Some parts of the codes denote aspects that were interchangeable. I have carts with different memory decoder codes yielding the same data, different ROM split-up codes (I"m sure there's a technical name for this) yielding the same data. So really, there's no way to be sure how many other different codes are out there, because a manufactured version may exist that came on two 512kb chips instead of 1 1MB chip. We have no idea.

So if I understand what Nach is doing, he is constructing the equivalent of an "applicable" code by reading the ROM's internal header attributes. You can do this if you have decoded the PCB serials by looking for patterns in a large sample size of games. That's perfectly acceptable to me from an accuracy standpoint because no one can prove that that code wasn't actually manufactured, and the game behaves exactly the same as any other applicable code, including the ones which were actually manufactured. If it didn't we would have slews of bug reports by now.

[Nach]

So if I understand what Nach is doing, he is constructing the equivalent of an "applicable" code by reading the ROM's internal header attributes. You can do this if you have decoded the PCB serials by looking for patterns in a large sample size of games. That's perfectly acceptable to me from an accuracy standpoint because no one can prove that that code wasn't actually manufactured, and the game behaves exactly the same as any other applicable code, including the ones which were actually manufactured. If it didn't we would have slews of bug reports by now.

That is indeed what I do. I even based some of it by further looking into a list you once posted.

[henke37]

Sounds like we are slowly floating away from the original topic. I am just noting this.

[byuu]

Sure, we're not guaranteed that it'll work for everything new we come across, but in the end, it does.

And that is the crux of our argument. We like guarantees. I don't like the idea that a ROM hacker screwing around with program code at $7fb0 might accidentally change the cartridge mapper, but only in certain emulators. Even if it's more likely that the sun will supernova first.

I want to emulate hardware, period, and hardware does not check those bytes. I don't care about what has been observed in the wild. A ROM can contain any data for any mapping setup, emulators cannot. It's really that simple.

We both understand each other's arguments, we just disagree on the best approach. I just don't see why we can't have it both ways. You're dumping all the carts again anyway, it'd take you guys a few extra seconds to write down ten digit strings along with that.

But then, that info would allow other people do things in a manner with which you disagree, I suppose ...

It's been documented that a game can have more than one PCB code yielding the same data. And it's fairly common.

I really don't see how this is a problem. All we need to do is store a list of observed PCBs. So long as we have one or more for each game, then we know we have correct mapping for a real, true physical copy of that game. Not a Frankenstein mapper that just so happens to work with it. As for which to make the default, that's irrelevant. Use a random one.

[neviksti]

Sure, we're not guaranteed that it'll work for everything new we come across, but in the end, it does.

And that is the crux of our argument. We like guarantees. I don't like the idea that a ROM hacker screwing around with program code at $7fb0 might accidentally change the cartridge mapper, but only in certain emulators. Even if it's more likely that the sun will supernova first.

Actually, in at least one case the "new thing" we came across had to be modified to work with the emulators. So the fact that it works has to some extent become because the emulator dictates what will work.

The example I know of is StarFox 2. It was compiled and given to some emu authors to test before "releasing" it. It had to be modified to have a "proper" internal header to run on the emulators. It is this that was distributed as the "final" rom and not the compiler result which works fine on real hardware of course.

Just an interesting tidbit.



We seem to have gotten into a discussion now of whether some kind of cartridge format (information external to the rom) would be useful or not.

You gain no accuracy for emulation of games, since they are already in the "database". Allowing one to represent the cartridge however, you gain the ability to treat the emulated snes as a platform. And we're not talking fairy tale stuff here. Devices for the snes over a decade old, did their best to represent rom and SRAM mapping in the cartridge. This is a great start. We haven't even caught up to that.

Romhacker needs to extend the rom? Sure, go right ahead, and state where it should be mapped. Such and such should be open bus? Go ahead and mark that.

When I was working with StarOcean I wanted to move the SRAM to a specific location. There was no easy way to do this with most emulators. Sleuth, with rudimentary support for a "cartridge" instead of rom format, did let me.

Actually I eventually worked to make the StarOcean game playable without the SDD1 chip, and the result represented a statically mapped cartridge with 96Mbits of rom. That is how it was presented to the SNES (the hardware can't tell its not really rom, but dram, etc. Although one person wrote to me and appears to be actually making real cartridges of this.) Mapping the rom how I wanted on the real hardware was not a problem. Using my emulator of choice at the time, I could do so for emulation as well.

Everytime someone wants to make modifications that don't easily fit with the current patterns, the current emulators need to be updated with the new rom -> cartridge layout. The only emulator that gave some ability to represent the actual cartridge itself was Sleuth. It was the only one I could use to debug such creations.

The crux of the problem is this:
- The rom -> cartridge internal convertions in emulators are needed. It is great that this has become (as far as I know) perfect for all released games. However, such patterns are not required by the hardware, and if the goal is hardware emulation, then these should not be depended on.

- The real hardware was presented with a cartridge, not soley a rom. Until we come up with a viable alternative, so we can present the emulated snes with a representation of a cartridge instead of solely a rom, we are stuck with emulating games or things that look enough like games to fool the emulators into understanding what we want.

- Considering that historically, there already was some work towards a better format (and this format was probably used by millions), and several discussions were had by emu-authors of how we could expand such formats to handle all known snes hardware (yet for existing roms still be backwards compatible with most existing tools), I find it a shame that this died out.


No format can handle any crazy cartridge people come up with. But handling almost any reasonable mapping of just SRAM and ROM and "nothing" (open bus), would be a great start. And some of those extension proposals would handle almost anything else short of new coprocessors or asics etc. never before seen (although allowed for further extensions to handle such things).

[FitzRoy]

We both understand each other's arguments, we just disagree on the best approach. I just don't see why we can't have it both ways. You're dumping all the carts again anyway, it'd take you guys a few extra seconds to write down ten digit strings along with that.

I write mine down and will make them public when I launch my site. I don't have a problem with it, just so long as you create something similar to the new NES xml format. Anything attached to the ROM is a bad idea, IMO. But also keep in mind that this is an idealist goal and a practical impossibility. There are carts out there that I know I will never get a hold of and I'm not scavenging the rest of my life trying to find them. This is particularly true of many PAL carts and things you can only stumble upon like revisions, because ebay sellers generally ignore you or say they can't find the imprints when you ask them.

But if the main purpose of creating a new header format for the SNES is to help some rom hackers who might accidentally screw up these values, or prevent the modification of prototype material like Starfox 2 which lacked release-quality values, then maybe it's worth doing someday. But I think there will always be more pressing unknowns. And you have to agree that this is a pretty good solution while the NES scene has suffered endlessly from only being able to do exactly as you suggest.

[Nekokabu]

hello^^
I create a alt. Japanese localization file.
http://nekokabu.s7.xrea.com/locale.zip

[tetsuo55]

I've tried to convince Nach to store a copy of each game's PCB ID, since he's verifying carts anyway.

Almost all the No-Intro dumpers not only note the codes but even scan the PCB. You did not want to use their research in the past, maybe things have changed?

- Considering that historically, there already was some work towards a better format (and this format was probably used by millions), and several discussions were had by emu-authors of how we could expand such formats to handle all known snes hardware (yet for existing roms still be backwards compatible with most existing tools), I find it a shame that this died out.


No format can handle any crazy cartridge people come up with. But handling almost any reasonable mapping of just SRAM and ROM and "nothing" (open bus), would be a great start. And some of those extension proposals would handle almost anything else short of new coprocessors or asics etc. never before seen (although allowed for further extensions to handle such things).

Actually work on this is progressing faster than ever, and not just for snes but for ALL media for ALL systems and it fully supports wierd cartmapping:
http://www.bannister.org/forums/ubbthre ... #Post36404

[FitzRoy]

I'm just brainstorming here, but I'm just looking at that thread and shaking my head in confusion. Maybe this isn't such a good plan after all, the idea of distributing a file with each rom just isn't going to work. Nobody's going to change their distribution methods and datters are going to be overwhelmed trying to catalog these new and revisable files. It's bunk.

How about this: every emulator comes with a folder called "pcb". That folder contains a format called ".pcb" which is essentially the mapping code externalized. (You decide if it's system specific, or some kind of universal format decided upon by all. I don't know what's possible.) All known pcb mappings from commercial carts will be included in that folder, each mapping to its own file. Nach's plan becomes the default for any system that is capable, while a user-selectable mapping becomes an option in the emulator's configuration. The option would be to essentially be prompted upon loading a rom and select a pcb file from within that folder to use with that rom or roms. The list of files would not be set, anyone could create their own mapping, place it in the folder, and have it be selectable by the emulator.

Let's keep in mind that the drawbacks involved in this are only drawbacks to the masochists who use it. Is it kind of annoying to have to reference PCBs every time you load a rom? Sure. Is it more annoying to have to put a file in every single rom folder because no one is distributing in this masochist format? Yes.

Somebody poke holes in this before I vent on those forums.

[neviksti]

Three quick things:
- Current emulators, and many tools, already currently ignore headers on the roms.
- Nach has already made and started what sounds like an extendable rom format (although I personally prefer uncompressed formats for rom-hacking reasons, either way it shows that new formats can be introduced without causing problems with old formats).
- Because all games can already be played, for historical reasons, without this extra information ... if the extra stuff is added to the format later and roms don't have it ... emulators don't need to require it.

I'm not suggesting emulators should require roms to have that format. Such a transition would be aweful, as you note yourself.

How about this: every emulator comes with a folder called "pcb". That folder contains a format called ".pcb" which is essentially the mapping code externalized. (You decide if it's system specific, or some kind of universal format decided upon by all. I don't know what's possible.) All known pcb mappings from commercial carts will be included in that folder, each mapping to its own file. Nach's plan becomes the default for any system that is capable, while a user-selectable mapping becomes an option in the emulator's configuration. The option would be to essentially be prompted upon loading a rom and select a pcb file from within that folder to use with that rom or roms. The list of files would not be set, anyone could create their own mapping, place it in the folder, and have it be selectable by the emulator.

I feel that misses the point of what I was hoping for ... a "cartridge" format. This instead seems to still present the emulator not with a cartridge, but a rom, and the user has to remember the rest. I don't like that.

Also, as noted above, if we go this way, let's please please don't have this just be a list of pcb's ... otherwise we're stuck with similar situations as now, we can't just modify the "cartridge" as necessary (that's why I'd like something like an enhanced SF3 header, with a PCB number only for reference "if known/verified").

Because of this great backwards compatibility already offered to us, I see no real drawback to implementation. Those that don't want those features (or even know they exist) won't be affected at all.

That's my take on it at least.
Thanks for sharing your views, as it is nice to start open discussion on this.

[tetsuo55]

You're fears are correct for the short-term.

However MAME+MESS are going to make the move sooner or later, and No-Intro wants to join in on the fun.

That means that there is quite a bit of momentum to get the ball roling.
When the time comes, if i understood correctly Zsnes will support the new format from day 1, if Bsnes joins in it could be a couple of weeks before everyone has upgraded to the new format.

The datting tools are already prepared for the new format, these tools could update the roms on the fly, as the XML files do not contain any copyrighted code and could be hosted anywhere to spread them.

the only way backwards compatibility breaks is if we physically change the data in the roms, like splitting them or byteswapping them, although most modern emulators can handle this quite well.

[FitzRoy]

I feel that misses the point of what I was hoping for ... a "cartridge" format. This instead seems to still present the emulator not with a cartridge, but a rom, and the user has to remember the rest. I don't like that.

See, this is what I don't understand. I don't have a problem with the modularity of emulation. I like keeping game data separate and the only distributable. Look, there are coprocessors on a lot of these cartridges as well, and we place those within the emulator, don't we? I mean, why require people to make a bunch of redundant files for each rom when you can just create one of each on the emulator end and point to them? I see the pcbs as being no different. We need to recognize the difference between binary data and a mechanical pathways which has to be reproduced subjectively in a piece of software. And we further need to recognize the likelihood of a person wanting to write their own coprocessor or pcb code because he finds existing ones insufficient. That's one in a thousand. Why can't that person just modify the source? We know that if byuu does anything, he won't do the externalized files, he hates extra files. He might add the option for selectable pbs if you internalize them, though.

And I look at some of the things they're trying to add in the format in that thread - regions, cue files, checksums, special controllers... That's not a cartridge format, that's a self-contained database.

However MAME+MESS are going to make the move sooner or later, and No-Intro wants to join in on the fun.

I could care less what they do, they're already in their own world. I'm more concerned about stand-alones. Anyone who can stand the interface of a multi-console emulator is beyond saving.

The datting tools are already prepared for the new format, these tools could update the roms on the fly, as the XML files do not contain any copyrighted code and could be hosted anywhere to spread them.

So you download a thousand of 'em and throw em in your rom folder? What happens if the names don't match up? What if you want them zipped with the rom? Then, after spending weeks getting them in that format, what happens when errors are found and they change them? This is a disaster, it incurs practically all the costs and dependencies as headers.

[byuu]

Okay, here's my take on things.

Essentially, the PCB code has all we need to map games. It tells you where all ROM and SRAM data go, what special chips are used, etc. That alone should suffice for mapping of known games.

For mapping future cart layouts, we would want to allow for some sort of flexible custom definition, eg:

Code: Select all

      map(MapShadow, 0x00, 0x3f, 0x8000, 0xffff, memory::cartrom, 0x400000);
      map(MapLinear, 0x40, 0x7f, 0x0000, 0xffff, memory::cartrom, 0x400000);
      map(MapShadow, 0x80, 0xbf, 0x8000, 0xffff, memory::cartrom, 0x000000);
      map(MapLinear, 0xc0, 0xff, 0x0000, 0xffff, memory::cartrom, 0x000000);

Only in a more text-friendly format. Definitely not binary.

The exact mapping for specific PCBs could easily be supported internally by emulators. They are unchanging, and we can limit it to only to commercially mass produced PCB IDs. Eg SHVC-BJ0N-10 can be hard-coded into the emulator. So for a game to use that layout, it would only have to specify that ID to use it, you wouldn't need to describe it.

But we could make custom definitions for them, too, if you want to simplify. And emulators can just use that data if they don't support internal definitions for them. My problem is if there were changes to these files, users wouldn't really know.

Either way, how to let the emulator know about these PCB codes ... I would suggest storing both .sfc and .pcb files inside a zip archive. The .pcb file would have the first line be the PCB ID itself. This would be something like "Custom" if you wanted to make your own. After that, you could describe the mapping of everything on the cart, similar to above. You'd have access to map everything, including special chips.

For mapping of games we don't have PCB data for, or games that the user doesn't have a PCB ID for, we can use the header-based auto-detection and "one size fits all" mappers we have now.

Lastly, and this would be up to individual emulators, we could have some central authority to release a pcb.id file. This file could be a straight-up list of md5sum = PCB ID [, PCB ID 2, PCB ID 3, ...] for all verified dumps.

An emulator could use that when a .pcb file doesn't exist, or if they're opposed to databases as Nach is, they can omit the database.

I do not like XML at all. I think it's a fad trend, and an over-complicated way of storing the same data you already could in plain-text, and it makes it harder, not easier, to edit by hand. It also makes parsing the format much harder without using pre-packaged libraries. But if my other goals were met, I would compromise and agree to XML.

When the time comes, if i understood correctly Zsnes will support the new format from day 1, if Bsnes joins in it could be a couple of weeks before everyone has upgraded to the new format.

You're kidding, right? If it were up to me, the only ROM extension for SNES games would be .SFC for plain carts, .BSC for slotted carts, .BSX for flash carts, .ST for Sufami Turbo carts, and there would be no ROM headers, period.

I can't do that, because I'd have a userbase of zero without the backing of the major players.

It is the SNES96/7/X and ZSNES devs who allowed the current setup of 40 extensions, 10 interleave formats, copier headers and ROM header parsing. Which is most probably because they were competing against each other back then, so being able to do something their competitor wasn't was seen as an advantage.

I would have never supported any of that had I the popularity to do so, and honestly if enough of the core supporters of bsnes here feel I should, I will strip all of that crap anyway. I have in fact removed half of the ROM header extensions thus far.

The reason I have it now is because it's such a de-facto standard. Not doing so means no users. No users means no testing to find new bugs. No new bugs means nothing for me to investigate and improve. Not improving means there's no point in existing.

From what I do know, Nach is adamantly opposed to databases. He also declined to store PCB IDs of games dumped. He's also results-focused rather than theory-focused as myself and neviksti are. That's not wrong, it's different -- nothing more. I also deduce from his efforts with ROM header parsing and NSRT that he's quite happy with the current setup. I think it will be much harder to make such a radical change in ZSNES, which will be required for such a change to catch on and become a new standard. However, I will implement such a change anyway, so long as we can get at least ~90% of games properly cataloged.

[funkyass]

XML is a bit funky. especially when you throw XSLT into the mix. If it is as fancy as they say it is why aren't there OS level services for parsing it?

I say plain text, and lets not limit ourselves to 8.3 filenames(pcb might be taken by another program). At the very least the extension should be .txt.

[Nach]

From what I do know, Nach is adamantly opposed to databases.

Only in emulators, I like them everywhere else. The console didn't need a DB to load a game. Stuff what MAME does for example drives me crazy.

He also declined to store PCB IDs of games dumped.

Because it's not needed. I can generate for you right now a valid possible PCB for every commercial dump we have.

I recognize we need a way for hackers and enthusiasts to define what they want for their hacks/games, or for other really out there stuff. But why make people have to look up a bunch of meaningless PCB IDs? A ROM hacker now has to learn that an ID with a J in the 3rd spot means a particular kind of mapping? We definitely should offer a way for them to define how to map everything, and perhaps some easy snippets to cut and paste, so we can turn a dump into a full blown cartridge, allowing a hacker to rewrite it to his (women don't hack ) heart's content. Also, allowing them to do whatever they want in mapping easily, and not just limiting them to what PCBs we like.

He's also results-focused rather than theory-focused as myself and neviksti are. That's not wrong, it's different -- nothing more.

Yes, I don't believe in spending a ton of extra work to everyone at every level involved just to get exactly the same results we currently are already getting.

I however wholeheartedly support coming up with an easy way for hackers to specify how to map things, and even making NSRT generate these headers on every ROM, so each one becomes a cartridge, and leaving mapping insanity patterns to NSRT. Allowing emulators to view each dump with header as a cartrdige, and not need to do any extra logic.

The thing you're forgetting about me is that I love data driven programming, and despise having to hard code values in. Emulators unfortunately defy the whole no magic number principals, but some concessions we have to make. Although my ideal emulator would probably keep all logic and special numbers in external files.

I also deduce from his efforts with ROM header parsing and NSRT that he's quite happy with the current setup.

I'm not happy with it, the current setup is what led to stuff like MAME, or your initial cart.db, and we simply don't give enough flexability for our hackers. I also desire extra features such as having the emulator know which inputs were used for a particular game, so if desired, the user could have special control setup automated. This is something already implemented in NSRT/ZSNES/Snes9x via NSRT headers.

I think it will be much harder to make such a radical change in ZSNES, which will be required for such a change to catch on and become a new standard.

If you get me on board with any new ideas, I'll have NSRT and ZSNES setup to take care of the situation, and get a standards change in progress. But you'll have to convince me that what we're doing actually helps, and isn't just some meaningless ideal.

However, I will implement such a change anyway, so long as we can get at least ~90% of games properly cataloged.

I'll generate catalogs, but I firmly disagree with collecting info which we know we can generate 100% correctly. I'm also not worried about the "what if" situation of a new dump throwing a wrench in things, because either way then something needs updating, and smart generation can be updated just as easily.