Niotso

A New Implementation of The Sims Online

  • Overview
  • News
  • Showcase
  • Wiki
  • Code

First videos up

Posted by Fatbag on February 25, 2012
Posted in: Discussion. 6 comments

Three videos have been added to the Niotso YouTube channel, narrated by me.

In the first video, I explained I would scan various items, and also submit the track checksums of the soundtrack CD to freedb and AccurateRip.

I ripped the soundtrack using Exact Audio Copy, configured under the advice of numerous posts on the Internet. The soundtrack unfortunately does not make use of CD-Text; the 3 audio drives I tried it on (two computers, one car), which DO work with another CD, reported nothing. Without this information, we don’t have track names or any identification of the performer/composer.

With that in mind, here are the official CRCs:

Track 01 (5:53) [Ingame: ./music/modes/map/tsomap2_v2.mp3]
    Peak level 99.3 %
    Extraction speed 2.9 X
    Track quality 100.0 %
    Copy CRC 5FB5789F
    Accurately ripped (confidence 1)  [09CEE30D]  (AR v1)
    Copy OK

Track 02 (5:20) [Ingame: ./music/modes/map/tsomap4_v1.mp3]
     Peak level 99.3 %
     Extraction speed 3.2 X
     Track quality 99.9 %
     Copy CRC 75B5EBD7
     Accurately ripped (confidence 1)  [7B497775]  (AR v1)
     Copy OK

Track 03 (5:00) [Ingame: ./music/modes/select/tsosas1_v2.mp3]
     Peak level 99.3 %
     Extraction speed 3.7 X
     Track quality 100.0 %
     Copy CRC 0076F2AB
     Accurately ripped (confidence 1)  [F58BEEE9]  (AR v1)
     Copy OK

Track 04 (5:00) [Ingame: ./music/modes/select/tsosas2_v2.mp3]
     Peak level 99.3 %
     Extraction speed 3.9 X
     Track quality 100.0 %
     Copy CRC 2BE641CE
     Accurately ripped (confidence 1)  [D75C89EC]  (AR v1)
     Copy OK

Track 05 (5:00) [Ingame: ./music/modes/create/tsocas1_v2.mp3]
     Peak level 99.3 %
     Extraction speed 4.2 X
     Track quality 100.0 %
     Copy CRC 48C62922
     Accurately ripped (confidence 1)  [513A2A7C]  (AR v1)
     Copy OK

Track 06 (5:00) (Not found in either game)
     Peak level 99.3 %
     Extraction speed 4.4 X
     Track quality 100.0 %
     Copy CRC 92906A3C
     Accurately ripped (confidence 1)  [6655FDA5]  (AR v1)
     Copy OK

Track 07 (5:45) [Ingame TS1: ./music/modes/map/tsobuild3.mp3]
     Peak level 97.2 %
     Extraction speed 4.6 X
     Track quality 100.0 %
     Copy CRC 6AE3DECE
     Accurately ripped (confidence 1)  [BAC4252F]  (AR v1)
     Copy OK

Track 08 (5:49) [Ingame TS1: ./Music/Modes/Build/build6.mp3]
     Peak level 96.4 %
     Extraction speed 4.8 X
     Track quality 99.9 %
     Copy CRC F7B7E8CB
     Accurately ripped (confidence 1)  [9BB867E6]  (AR v1)
     Copy OK

Track 09 (2:56) [Ingame TS1: ./Music/Stations/Latin/latin3.mp3]
     Peak level 96.3 %
     Extraction speed 5.0 X
     Track quality 100.0 %
     Copy CRC 4562334D
     Accurately ripped (confidence 1)  [413D497B]  (AR v1)
     Copy OK

Track 10 (3:21) [Ingame: ./music/stations/scifi/tsosci3_8a4a80c5.mp3]
     Peak level 99.3 %
     Extraction speed 5.0 X
     Track quality 100.0 %
     Copy CRC C3AF80EF
     Accurately ripped (confidence 1)  [C510CDB2]  (AR v1)
     Copy OK

Track 11 (5:31) [Ingame TS1: ./Music/Stations/Rock/rock5.mp3]
     Peak level 98.0 %
     Extraction speed 5.4 X
     Track quality 100.0 %
     Copy CRC CD20DF35
     Accurately ripped (confidence 1)  [3B8796AD]  (AR v1)
     Copy OK

Track 12 (4:34) [Ingame: ./music/stations/scifi/tsosci4_8a4a80c2.mp3]
     Peak level 99.3 %
     Extraction speed 5.5 X
     Track quality 100.0 %
     Copy CRC A87420B0
     Accurately ripped (confidence 1)  [94C8D9AB]  (AR v1)
     Copy OK

Track 13 (3:17) [Ingame: ./music/stations/oldworld/tsoold7_v1_6a3c88fc.mp3]
     Peak level 99.3 %
     Extraction speed 5.5 X
     Track quality 99.9 %
     Copy CRC 3BF7C275
     Accurately ripped (confidence 1)  [B71EBAB4]  (AR v1)
     Copy OK

Track 14 (3:16) [Ingame: ./music/stations/oldworld/tsoold7_6a3c88fe.mp3]
     Peak level 99.3 %
     Extraction speed 5.8 X
     Track quality 100.0 %
     Copy CRC 02E6FB3D
     Accurately ripped (confidence 1)  [4F7D549A]  (AR v1)
     Copy OK

I am not sharing these files on the Internet.

Scans will be posted here when I get them done.

Character animations done

Posted by Fatbag on February 19, 2012
Posted in: Client, Formats. 5 comments

Sim kiss animation

I expected more. But translations and rotations in animation files are stored completely absolute and do not apply to children. No additions, no subtractions, and in the case of quaternions, no matrix multiplication. In each new frame, you just swap the old values with the new.

So that didn’t take very long. Pictured is a Sim breathing out in solace after a passionate kiss.

What the renderer is still missing, though, is proper time synchronization, which is my current task. In fact, I’d like it to properly interpolate between discrete frames of the animation, so the end result displayed on the screen will be silky smooth on a 60Hz monitor.

Working character renderer

Posted by Fatbag on February 13, 2012
Posted in: Client. 4 comments

We’ve hit a milestone.

libvitaboy can render character models from the game using OpenGL—even robots.

In the above screenshots, green dots are regular bones, the red dot is the root bone, and the yellow dot is the head. The head, body, and hands are swappable.

Pretty nice treat for Valentine’s Day. I hope you have a fantastic one yourself.

Build system switched to CMake

Posted by Fatbag on January 30, 2012
Posted in: Codebase. Leave a Comment

The Niotso codebase previously functioned through custom makefiles. You cd into the zlib folder and type in ‘make’ to compile zlib. You cd into the libpng folder and type in ‘make’ to compile libpng. And the same with each and every package in the codebase, which you *must* perform in sequential order. Anyway, this really is efficient enough for quick experimentation of new code, but now that my packages have started maturing, I’ve been interested in making the end product more proper and functional out of the box.

CMake is the only portable and convenient alternative to this. And it’s not perfect, since many all-too-common features are just left out because I think the developers just forgot about them. You can’t specify different cflags for different packages, the names by which your compilers are invoked are hardcoded into CMake and unchangeable, there is absolutely no support for shell scripting, and the language is just atrocious without basic features like escape characters or ternary operators. But the main concept behind CMake is what matters: abstract build targets into strings so they can be referred to without having the knowledge of their destination file names. This way, you can say, “Package X depends on the shared version of Package Y”, as opposed to, “$(DIST_FOLDER)/PackageX$(EXE): ${DIST_FOLDER}/PackageY$(SO)”; the latter is not only less intuitive, it also (in regular makefiles) requires manually maintaining a list of all built executables and libraries so you can delete them at user request.

While the GNU autotools aims for portability and can on its own take care of dependency and header tracking (what build system can’t?), it requires both a Bourne-compatible shell preinstalled and a Unix environment set up to provide information about your system like hardware and language. Then the generated project is tied to GCC on all platforms. CMake is the only build system generator for all the popular compilers and IDEs (that I’m concerned with, at least), giving the user choice and more flexibility, and at the larger scale inspiring more (friendly) competition between compiler writers.

But being such an idealist, I don’t believe this is enough. There needs to be a single Universal Codebase Definition format (*.ucd) which all IDEs can agree to. Each source file added to a UCD file may override its inherited type (such as “C Source”), cflags, link flags, etc. and specify output directories, dependency targets (which need not correspond to files), and libraries to link with. Variables describing the system are predefined by the UCD parser, such as platform, endianness, compiler, availability of compiler headers, and so on; custom test scripts may be provided inside the UCD file. In addition to this, UCD files shall be fully modular, letting the user detach any subdirectory of one project’s codebase for use in that of another. Of course, this will require all the toplevel definitions (like cflags and make rules) to be redefined by each file, but seeing as we’re talking kilobytes at most here, the convenience is well worth the increase in size. Under this system, imagine opening up your IDE and being given easy access to drag and drop other UCD projects into your own projects.

One concern of my own that I kept revisiting, numerous times over, in picking the right build system for Niotso, is that my codebase consists of many individual packages strewn together into the same repository for convenience. While the GNOME project has hundreds of unique packages which all are large and different enough to warrant its own owner and repository, I have been setting up my code so (to name an example) just the xa reading code is its own library which is actually bundled together with all the other unrelated file reading libraries that I don’t even maintain myself, like libpng, libmpg123, or libexpat, and interacted with as part of the larger FileHandler interface library. Which is linked to not only the Client, but also the FARDive utility. So far, I still haven’t had the time to extract libpq from the entire PostgreSQL source code (powered by the GNU autotools) because it refers to headers from all over the place. With UCD, the IDE could detect where each and every .h and .c file is, and simply move them over for you. Why hasn’t this happened yet? Why aren’t we funding this?

No technical reason, it’s just it’s not implemented yet. Because CMake at least does the job for my specific use case (and many others’), and is also trusted by large open source projects (including Blender, Compiz, KDE, libpng, LLVM/Clang, MySQL, Ogre3D, and Scribus), I have no pressing need to work on UCD, until far in the future when Niotso hits grand release.

So I’ll clarify, Niotso has completely transitioned over to CMake and can now be compiled with one command. I’m now currently initiating work on libvitaboy, the OpenGL character rendering and animation system for Niotso.

UTalk speech codec cracked; over and out!

Posted by Fatbag on January 20, 2012
Posted in: Formats. 10 comments

In mid-December, I discovered a large number of files in the TSOAudio.dat archive that didn’t resemble track data or packing slips or anything tiny like that. These files ranged around 10-50kB, like sound data, but were not the usual WAV, MP3, or XA. This drove my interest in finding out if there were any more codecs of any kind used in the game that we weren’t aware of.

sys/tsoaudio.ini told us the game uses exactly 4 audio codecs: WAV, MP3, XA, and something called UTalk. The header was almost identical to XA, but these files decompressed to noise when using xadecode. By comparing the decompressed size field (from which I can derive sound duration) with the compressed size of the file, I could tell the codec runs at 32kbit/s, while XA runs at 88.2.

I googled and googled Maxis UTalk, UTK (the file extension), and UTM0 (the signature) for 15 minutes but couldn’t find anything. The codec appears to be unique to The Sims Online, so as we had no documentation, my task was to run the game in a disassembler and make sense of what the game does to read these files. Somebody had to do this, and I knew I was the only one in the TSO community with enough experience to do it.

It wasn’t hard to find the file header validation check in the game, since the only instance of “UTM0″ in any of the game binaries appeared in TSOAudioViewD.dll. The game did stop at my breakpoint to load two sounds, but failed both checks. If the game couldn’t load a UTalk sound before the Character Select screen (seeing as we cannot log into the game), it would be impossible to inspect how the game reads these files.

The two sounds it tried to load were the load loop sound and the click sound, both in WAV format. In fact, these were the only sounds used in the log-in screen at all. I hypothesized that the readsound() function in the game’s audio system calls the different readheader functions for these 4 sound formats and decodes it when it finds the right one. And each has a different header (“XA” for XA, “UTM0″ for UTalk, “RIFF” for WAV, and FF FB 90 40 for MP3).

So, in TSOAudio.dat, I replaced the load loop sound (2026960b-29daa4a6-00004f85) with a UTK file that matched the closest to 30 seconds, the duration of the load loop sound. And voila, it worked! The game looped this UTalk sound in place of the old, and (as the name and bitrate imply) it was of course Simlish.

It took about 10 days to gather enough information to find where in memory the UTK file is decompressed into and identify which function does the decompression. At that point, I was able to get the game to decompress any UTK file for us, but we needed more than that: we needed our own code to decompress these files for ourselves. It took about another 2 weeks to fully have all the code interpreted from x86 assembly to C, and another 4 days to analyze exactly how the codec works. The entire process finished on January 18, 2012.

I ran my new utkdecode tool against all 1,200 UTK files in the game, and it decompresses each one of them glitch-free. The code is now being used in Niotso and Afr0’s Project Dollhouse.

I leave you with a sound that I’ve only recalled hearing in The Sims Online. Hope it brings back memories.

Posts navigation
← Older Entries
Newer Entries →
  • Contributors

    • Fatbag
    • Blayer98
    • Propeng
    • Rhys
    • Afr0
    • Zc456
  • Archives

    • February 2014
    • June 2013
    • December 2012
    • July 2012
    • June 2012
    • April 2012
    • March 2012
    • February 2012
    • January 2012
  • Meta

    • Log in
    • Entries RSS
    • Comments RSS
    • WordPress.org
Proudly powered by WordPress Theme: Parament by Automattic.