bsnes v0.036 released

byuu · Post by **byuu** » Sun Sep 14, 2008 11:49 pm

byuu.org wrote:2008-09-14 - bsnes v0.036 released

This release fixes a somewhat serious bug introduced in v035, and also vastly improves Windows support for non-ANSI filenames.

The bug was triggered when HDMA would occur during DMA. If the DMA were long enough, subsequent HDMA transfers would be blocked. This caused graphical glitches in Star Ocean, Super Mario Kart, and possible more games. If you noticed any regressions from v034 to v035, this was almost certainly the cause. Once again, we're operating under the assumption that there are no known bugs currently, so please let us know here if you find any.

I've also rewritten the file handling for the emulator. On Windows, attempting to load a file with non-ANSI characters (eg Russian, Japanese, etc) would cause these characters to be removed. This meant that no version of bsnes thus far could load these files. This problem was exacerbated when I ported the user interface to Unicode (UTF-16), this caused even config and locale file loading to crash the emulator.

The root of the problem is that Windows only accepts non-ANSI strings in UTF-16 format, whereas bsnes' UI wrapper converts strings to UTF-8 interally. When passing these file names to the standard file functions (fopen(), std::ifstream, etc), file loading would fail. To fix this, I replaced all file access functions with a new version that would convert the UTF-8 filenames back to UTF-16, and use appropriate access functions (_wfopen(), _wmkdir(), etc.)

... but there is still one limitation to this: ZIP and GZ support use zlib, and JMA support uses libjma. Neither of these libraries convert UTF-8 strings to UTF-16 before attempting to open files. Due to licensing issues, as well as technical issues, I am unable to correct this at this time. What this means is that loading ZIP, GZ and JMA files; on Windows only; and with Unicode characters in the file name only; will cause the image load to fail. Loading uncompressed images (SMC, SFC, etc) will work with or without Unicode on all platforms.

I tried to be as thorough as possible with this fix: command-line arguments (via CommandLineToArvW + GetCommandLineW), user path (via SHGetFolderPathW), real path (via _wfullpath),folder creation (via _wmkdir) and file access/existence checks (via _wfopen) were updated in all cases. I also updated file loading for ROMs (SMC, SFC, etc), save RAM (SRM), real-time clock save (RTC), cheat files (CHT), UPS patches (UPS) and both configuration files (bsnes.cfg and locale.cfg.) Configuration file loading should work even if your username contains non-ANSI characters, and it should also detect config files put in the same folder as the bsnes executable, even if the path to the executable contains non-ANSI characters.

Still, if you spot any bugs, aside from the ZIP/GZ/JMA loading issue, please let me know via e-mail at setsunakun0; at hotmail.

Lastly, I'd like to apologize for the poor support for non-ANSI filenames in the past. Using an English version of Windows didn't expose the problems to me. I'll be more thorough in the future with this.

mudlord · Post by **mudlord** » Mon Sep 15, 2008 2:23 am

Nice job with the hiro fixes.

gllt · Post by **gllt** » Mon Sep 15, 2008 12:24 pm

Fantastic!
Just built last version today in Sid, I enjoy how simple it is to build and install.

wertigon · Post by **wertigon** » Mon Sep 15, 2008 5:41 pm

Good job, as always. So, what's next? Time to simulate those peripherals?

FitzRoy · Post by **FitzRoy** » Mon Sep 15, 2008 5:57 pm

wertigon wrote:Good job, as always. So, what's next? Time to simulate those peripherals?

Byuu posted in the FAQ thread explaining why this is not happening.

Verdauga Greeneyes · Post by **Verdauga Greeneyes** » Mon Sep 15, 2008 6:18 pm

Time for sPPU? Though I imagine you might want to get those div/mul delays and the joypad polling thing emulated first..

Hatsuyuki · Post by **Hatsuyuki** » Mon Sep 15, 2008 8:46 pm

byuu, big thanks for fixing the problem with loading ROMs under non-English Windows

byuu · Post by **byuu** » Mon Sep 15, 2008 9:10 pm

Half of our state is out of power right now, I actually only just managed to post v036 before it happened. They're saying up to seven days. Have intermittent wi-fi access with the laptop, but I'm dead in the water for anything else. Including posting a Hungarian localization I said I would :(

byuu, big thanks for fixing the problem with loading ROMs under non-English Windows

Sure, thanks for being so patient with me on that.

So, aside from ZIP/JMA support, it works as expected? Good to hear. Was afraid I'd miss something. Tons and tons of file access code all over the emulator.

Also used the file class, which has built-in buffering that's missing on Windows w/fgetc(), etc. I guess bsnes is a good place to stress test that library, heheh.

Though I imagine you might want to get those div/mul delays and the joypad polling thing emulated first..

Yeah. Want to get those two first.

byuu · Post by **byuu** » Mon Sep 15, 2008 10:52 pm

I'll carry this discussion over here.

Nach wrote:I could easily add for Windows to use a wchar_t * / wstring open call which wraps to using _wopen() to get an fd to build upon.

The issue is how to make it work on all kinds of other OSs, and having it wrap around other source classes when not using libstdc++.

Given that both Linux and OS X support UTF-8 filenames, I say we just add #ifdef _WIN32 wrappers around each file open call. For fopen(), use MultiByteToWideChar + _wfopen(), for std::ifstream, use the same as fopen(), and pass the handle to the ifstream constructor.

No reason to explicity support Windows' broken Unicode system. If they need to, they can convert UTF-16 to UTF-8 with WideCharToMultiByte before invoking the libraries.

Of course, the thing is we need to update both zlib and libjma to do this, and I have a feeling updating zlib won't be as fun. To use WCTMB and MBTWC we will have to include the Windows header, which will hit compile time quite a bit.

Maybe I should write a simple convertor for UTF-8<>UTF-16. It doesn't appear to be too complicated ... certainly nothing worth including a big time conversion library over.

tetsuo55 · Post by **tetsuo55** » Tue Sep 16, 2008 9:40 am

Who is wrong here
1)zlib and jmalib using suboptimal UTF stuff
OR
2)Bsnes abusing these libraries

In case 1 the libraries need fixing, as this will help everyone, in case 2 bsnes should have the converter built in.

byuu · Post by **byuu** » Tue Sep 16, 2008 11:00 am

The answer is, of course,

3) Microsoft was stupid enough to completely duplicate ~80% of their API to take UTF-16 strings, when they could've used UTF-8 with no changes to the underlying API. I know, they were worried about older multi-lingual OSes using non-Unicode codepages. Still a really bad move, NT would've been a good break to force UTF-8.

The best part is that UTF-16 doesn't even allow direct O(1) indexing, as there are extended symbols requiring more than 16-bits to encode.

---

Anyway, the libraries both take const char* filenames, so there's no way I can encode the names for them to work at this time. But it's really not their fault when const char* is good enough for every other OS out there.

_willow_ · Post by **_willow_** » Tue Sep 16, 2008 12:36 pm

Well, i do use English, German, Japanese and Russian symbols in filenames. Sometimes they got mixed in to single string, esp multimedia files to note the ones. Those UTF-8 nationalist colored programs always works in twists for me.

. Most time i'm forced to use pure English and Russian. Really i don't care of bsnes is non unicode program just because most emulators remains non unicode too. It's not comfortable sometimes but i can live with it, np.

byuu · Post by **byuu** » Tue Sep 16, 2008 1:17 pm

Really i don't care of bsnes is non unicode program just because most emulators remains non unicode too. It's not comfortable sometimes but i can live with it, np.

Yeah, noticed that myself when I made the Japanese test profile. Firefox wouldn't start at all, Winamp just kept looping through the setup, but would never actually open or play music. Winrar flat out ignored anything with Unicode in it, and spit out ???s all over. Even notepad2 wouldn't open any text files.

The only things that worked were the ones that came with Windows. Literally nothing else. It was completely unusable like that.

tetsuo55 · Post by **tetsuo55** » Tue Sep 16, 2008 1:45 pm

byuu wrote:
Really i don't care of bsnes is non unicode program just because most emulators remains non unicode too. It's not comfortable sometimes but i can live with it, np.
Yeah, noticed that myself when I made the Japanese test profile. Firefox wouldn't start at all, Winamp just kept looping through the setup, but would never actually open or play music. Winrar flat out ignored anything with Unicode in it, and spit out ???s all over. Even notepad2 wouldn't open any text files.

The only things that worked were the ones that came with Windows. Literally nothing else. It was completely unusable like that.

Yep thats how it is basically.
I have about 2000 files named with different pagetables and they all show up as "$%^TYG$"

But when i install chinese windows in vmware mount that same directory and browse the files they all work.

Thristian · Post by **Thristian** » Tue Sep 16, 2008 2:33 pm

byuu wrote:The answer is, of course,

3) Microsoft was stupid enough to completely duplicate ~80% of their API to take UTF-16 strings, when they could've used UTF-8 with no changes to the underlying API. I know, they were worried about older multi-lingual OSes using non-Unicode codepages. Still a really bad move, NT would've been a good break to force UTF-8.

NT was their good break to force Unicode instead of having different code-pages everywhere. According to Wikipedia, the first version of Unicode was released in October 1991 and Windows NT 3.1 was released in July 1993. UTF-8 was first publically announced in January 1993, but it wasn't invented by the official Unicode people and wasn't included in the Unicode standard until Unicode 2.0 in 1996.

Microsoft *could* have maybe pushed the release of NT back a year or so if they'd paid attention to the introduction of UTF-8 and realised how much better this third-party encoding would be than any of the official Unicode encodings, but they didn't.

The best part is that UTF-16 doesn't even allow direct O(1) indexing, as there are extended symbols requiring more than 16-bits to encode.

Up until Unicode 3.0 (1999), you *could* fit every Unicode character into a 16-bit integer, so it wasn't such a crazy thing to do in 1991.

tetsuo55 · Post by **tetsuo55** » Tue Sep 16, 2008 2:40 pm

So if we had a choice.

Which version of UTF is the best one?

creaothceann · Post by **creaothceann** » Tue Sep 16, 2008 3:03 pm

UTF-8.

tetsuo55 · Post by **tetsuo55** » Tue Sep 16, 2008 3:25 pm

Does anyone know of a program that can convert "any codepage" to unicode? (So i can fix my thousands of files that display garbage)

byuu · Post by **byuu** » Tue Sep 16, 2008 3:28 pm

Posted a new WIP.

The biggest change was that I rewrote nearly all of the cheat code system, so heavy testing on that would be appreciated.

Someone was mentioning over at Snes9X that it was limited to 300 cheats or something, so someone bumped it to 3,000. Not to be outdone (v036 is limited to 1,024), I vectorized the cheat table, meaning you can have infinite cheats now (limited only to available memory.) Actually cleans up the code quite a bit, too. Removed all the ugly strlcpy() stuff, the limitations on description text length, etc.

Looks like I had a bug with deleting codes, too. I wasn't copying the actual cheat codes. That would corrupt the descriptions on every code after the one you deleted, I think. Strange nobody caught that.

I also cleaned up the OBC-1 code, and added a "Fullscreen" checkbox after "Correct Aspect Ratio". Sorry for the delay with that, FitzRoy. Hopefully the checkbox is good enough for now, as I can't change the text to "Switch to ..." just yet.

tetsuo55 · Post by **tetsuo55** » Tue Sep 16, 2008 3:42 pm

Does the cleanup of the OBC-1 code have any effect on its speed or accuracy?

Cool work on the cheat-engine. I never use cheats so i wouldn't have run into that problem

FitzRoy · Post by **FitzRoy** » Tue Sep 16, 2008 4:19 pm

byuu wrote:Sorry for the delay with that, FitzRoy. Hopefully the checkbox is good enough for now, as I can't change the text to "Switch to ..." just yet.

Thanks, however, I think modes should operate as a checkmark choice with both listed a la PAL/NTSC. I don't like menu actions that don't take checkmarks, because they look exactly like an unchecked item (I consider this an oversight in modern gui standards). And to have to dynamically change what it says based on the mode... seems unnecessary.

By the way, do you think you could make the emulation speed settings universal rather than mode-specific? I'm slightly annoyed at having to enable/disable them in both modes as they're not the kind of things I would want in one but not the other.

FitzRoy · Post by **FitzRoy** » Wed Sep 17, 2008 1:23 am

Thinking about this some more, it doesn't really have anything to do with video settings, it's just the way the program is framing it. This looks like an appropriate time to add a new category:

Display Mode >
[x] Windowed
[ ] Fullscreen
---------------
Video Resolution >
[ ] Scale 1x
...

Everything under Video Mode deals with resolution and how it is stretched, so I think renaming that to avoid two modes would be nicer, too.

King Of Chaos · Post by **King Of Chaos** » Wed Sep 17, 2008 2:20 am

So far so good cheat wise. Testing several games and no problems whatsoever.

ssbastos · Post by **ssbastos** » Wed Sep 17, 2008 8:06 pm

Hello everyone,
Just thought I should mention that the video frameskip option in the windows port does not save when we set it. So that we have to set it every time we start the emu.

tetsuo55 · Post by **tetsuo55** » Thu Sep 18, 2008 8:52 pm

ssbastos wrote:Hello everyone,
Just thought I should mention that the video frameskip option in the windows port does not save when we set it. So that we have to set it every time we start the emu.

Yeah, i didnt report it because it thought it might be on purpose