Niotso

A New Implementation of The Sims Online

  • Overview
  • News
  • Showcase
  • Wiki
  • Code

Summer 2012 status update

Posted by Fatbag on June 16, 2012
Posted in: Client, Formats. 23 comments

To respond to your honest concerns about my absence, the state of affairs for Niotso is fine.

The Sims Online sprite reading is here. This means we can look at any image in the game now. Objects, floors, walls, roofs, catalog thumbnails, menu controls, faces and body textures, the relationship +/- icons and the skilling meter, and recently, the lighting and shadow textures in lightmap.dat.

This extends the fact that we can play any of the game’s sounds and music (WAV, MP3, XA, and UTK files) ingame in Niotso Client and render, and animate, any character models and body accessories.

The payphone introduced in TSO

The pizza maker; the transparent block in the center is visually covered by the oven.

 

IFF chunk format support

As of now, FileHandler’s iff library fully supports all of the game’s chunk formats except for those that are involved with the SimAntics VM. Much of this support comes with Propeng’s analysis of all of the chunks in the game and his generous additions to the wiki.

Chunk    Documentation    Support
BCON         100%           100%
BHAV          90%           100%
BMP_         100%           100%
CATS         100%           100%
CST          100%           100%
CTSS         100%           100%
DGRP          90%           100%
FBMP         100%           100%
FCNS         100%           100%
FWAV         100%           100%
GLOB         100%           100%
OBJD          10%             0%
OBJf         100%           100%
Optn           5%             0%
PALT         100%           100%
Rcon          50%             0%
SLOT           0%             0%
SPR#         100%           100%
SPR2         100%           100%
STR#         100%           100%
TMPL         100%           100%
TPRP          50%             0%
TRCN         100%           100%
TTAB          75%             0%
TTAs         100%           100%

Not only can these chunks can be read; they can be displayed in a web page by iff2html. For a read-only viewer, this has a few benefits over a “proper” GUI. It’s cross-platform, everything can be quickly displayed in tables, you can select and copy both text and images, you can do ctrl+f, and it can all be archived online somewhere if somebody wanted to do that. There is almost no point in coding all of these basic features in a tool or app.

Roadmap

Nearly everything in The Sims Online’s game files but the SimAntics VM, the HIT VM, the city rendering, the texture rendering (yes, grass is rendered), and the EA server (tsomarimba1) configuration can be explored. The remainder of our format research lies in these last tricky subject areas.

That’s not to say that format research is enough to understand how all of these files piece together. That’s actually an interesting puzzle, and thanks to the comments and IFF chunk labels I don’t think it will severely bog us down.

Before hitting any of this stuff, it’s a little more important to set up a primitive lot to supply some visual progress. And I want to clone the user experience leading up to the lot: the log-in screen, create-a-sim tool, and so on, leaving out the city map rendering due to our current lack of understanding of how the game does it.

This involves the following tasks:

  • Set up the audio engine to supply contexts for sounds with an upper barrier of say 16 playable sounds at a time, with some reserved for background music and stinger music. Each game tick, the volumes of all sounds are updated, and if a sound has finished fading out, it will be destroyed and have its handle distributed back into the pool.
  • Set up the OpenGL windowing system. I cannot stress enough how much of a chore this is for OpenGL ES if we want to support smartphones.
  • Set up the UIS parser and start implementing support for all controls; the toughest one is going to be the selectable text box.
  • Transition from the login screen to the select-a-sim screen, and fully replicate the create-a-sim tool, with a rendering of the Sim on the left panel. It is known that the create-a-sim tool in The Sims 1 and The Sims Online does not make use of scene lighting (discovered after altering the light files in The Sims 1 and experimenting).
  • Read all of the necessary data from the original game installation, rather than in the temporary pre-extracted form.
  • Port the frontend to X11 for complete support for Linux. The frontend is very separate from the internals of the game engine, so this should not be an issue. Mac is a different story because it’s hard to set it up in a virtual machine.
  • Release Niotso Technical Preview 1, and transition into the Pre-Alpha development phase (yes, right now we are still in Planning).

This is my overall plan for the rest of this development phase.

Where I’ve been for April 2012

Posted by Fatbag on April 27, 2012
Posted in: Discussion. 8 comments

The loading process depicted by the Niotso engine

I’ve been busy with two things: studying for my 5 AP tests (albeit calculus and computer science are gimmies) and preparing for my Google Summer of Code project—which was just accepted.

FileHandler now supports PNG, JPG, and BMP (24-bit uncompressed, 8-bit uncompressed, and RLE8) for images and WAV, MP3, XA, and UTK for sounds. The login screen has been 100% cloned up until the dialog. I know, not much. I’ve been stuck on attempting a simple fade-out with XAudio2. Sadly, under the API’s constraints, I’ll have to come back to that later. To get the cursor to be clipped within the window border, it needs to be drawn as a texture, so I need a cur parser, which won’t take me more than an hour. This and TGA will both output to BGRA32 (as opposed to BGR24), which means I have to add in the proper functionality to that on the client end. (FileHandler is all set up for that.)

FileHandler now only exports its public functions, so the DLL has shrunk down to 331kB.

This leads me into the UI subsystem. I will not get to participate much on this until after May 7th: the day of the chemistry test. Pic related.

My thoughts on third-party help

Posted by Fatbag on April 6, 2012
Posted in: Codebase. 15 comments

Niotso has always been a personal project with seemingly no end. At last, there’s somebody skilled who actually wants to help: rothn, of TSO Restoration and Vitare.

With simultaneous contributors to the codebase, the days of coming home from school, knowing exactly what my code does and where all of its functionality is derived from, abruptly end. Suppose we get a team of 50 “valuable” members, and together we finish the project in about a year. In that time, I will have worked on 1/50th of what I wanted to work on.

My purpose here with Niotso was always to learn how an entire game is coded from bottom to top, and at the same time I managed to supply a few things: documentation, teasers, and competition. The model I intended was that people can submit new code to a patch tracker (example), where I can review patches and either accept or reject them. This way I still have full knowledge of what everything is doing. But if this model is incompatible with the public interest, maybe the project never should have gone public, and never should have competed with Project Dollhouse, which can already primitively display an ingame lot with a fully functional UI. Or maybe this project is not what I honestly thought it was. If I reject the most immediate form of help, I’m preventing the game from coming back.

And yet all of the programs I make here are open source. Anybody can already take my code and, after learning from it, make improvements to it. But before dropping changes into my own personal workspace, I want the opportunity to review it. Patches don’t make it to the Linux kernel until the meaning and consequence of every last line has been assessed by the inner circle. I’ve set up Niotso Trac to let anybody do just that.

I am not taking away the right for others to assemble and, still, try to bring back the game ASAP. But do remember: I’m still working.

The situation with SimAntics, and why IFF research has been rescheduled for later

Posted by Fatbag on March 16, 2012
Posted in: Client. 5 comments

Having actually felt the water from the waist down, I’ve decided to jump out of IFF chunk research, because cracking SimAntics is one of the biggest challenges we face, and that’s the infrastructure behind the entire IFF chunk structure. For the chunk formats that don’t make some sort of meta-reference to SimAntics which we can’t work out because we don’t quite understand SimAntics yet, all we get is say, a list of sound IDs (FWAV), or pie menu text strings, or other bits and pieces of information which, yeah, are important eventually, but serve no purpose and satisfy nobody until we have a client or actual SimAntics disassembler to make use of them.

So the question then is, why don’t we start figuring out SimAntics? Let me explain the situation.

In both games, we have a file called behavior.iff which holds all the text for Edith, the tool used by Maxis to program their objects. This allows behavior.iff to be translated into other languages and is what allowed Jamie Doornbos to give a label for each instruction (See: “92 [STR#] (008B) – Primitives” in TSO’s behavior.iff). In our case, simply knowing the assembly mnemonic of an instruction does not always help us understand precisely what that instruction does, and regardless, we still have to work out the operand syntax based on no documentation by Maxis.

In The Sims 1, you could guess how an instruction works, change one of its operands, boot into the game (which would generally take less than a minute—about 5-10 seconds on computers today) and immediately test your change and see how the game reacts. If you felt you really needed to, you could run the game in a disassembler and eventually find your object loaded in memory and see what the game does with it. In The Sims Online, we are fettered by the fact that we cannot get past the log-in screen. Hope is not completely lost, however. The situation is just extremely bleak.

In The Sims Online’s credits.iff, Jamie Doornbos is credited as the “Simulation Engineer”, with Patrick J. Barrett III and Don Hopkins as “Engineer”s, who we know played a small role in developing Edith alongside Jamie Doornbos. And judging by the fact that TSOSimulatorClientD.dll contains lots of relevant text strings (take for example, “$Me $Object $MasterObjectName $DynamicObjectName $ObjectCreatorName $DialogStrLocal $DynamicStringLocal $Local $Param $Temp $TempXL $NameLocal $TimeLocal $Attribute $MoneyXL $FixedLocal”), we can assume that the entire SimAntics virtual machine was implemented in that one DLL. This DLL exports exactly one function, like all the others: void * GZDllGetGZCOMDirector(void). This function creates, sets up, and returns the pointer to a C++ object, which the game uses to control the virtual machine. We can successfully interact with it, with some code as follows:

HMODULE dllmodule = LoadLibrary("TSOSimulatorClientD.dll");
void * (__stdcall *GZDllGetGZCOMDirector)(void) =
    (void * (__stdcall *)(void)) GetProcAddress(dllmodule, "GZDllGetGZCOMDirector");
void * value = GZDllGetGZCOMDirector();
printf("%p", value);

The object returned to us does not carry with it any variable or method names. We have to figure out what these functions do by looking at their code in x86 assembly. And we have to figure out what they want as input, and what they give us as output. If you’re interested, I have about, oh, zero percent progress documented on the wiki (link). It’s hard, but eventually doable, perhaps.

This is the icon we will be using for the client.

Let’s look at the upside of things. There are only 68 of these darn instructions, and a bunch have got to exactly parallel the ones in The Sims 1. I also just released a tool called iff2html in the repository, which generates a webpage that lets you view the contents of an IFF file. It only works with the STR#-based chunk formats for now, but it works with them all, so it can read both TSO’s and TS1’s behavior.iff files.

So this is why I am saving this mass of stuff for later. Which means I’m on probably everyone’s favorite task: the client. From this point, I am going to implement all functionality in the client up until city rendering. This means I have to prepare an OpenGL windowing library, which will be interesting as I try to mitigate for the fact that all textures need to have dimensions that are powers-of-2.

And I have just set up a Justin.tv account for Niotso. One of these days I plan to broadcast my programming live and talk with people in the comments, so you see what this process actually involves, not what Ghost and Jonathan set out to make you believe.

IFF chunk research and the new z-buffer calculator

Posted by Fatbag on March 4, 2012
Posted in: Client. 10 comments

The large majority of the new HTML5 z-buffer calculator (on the right) was programmed by RHY3756547. I added alpha-blending and went over the geometry again to fix some minor issues.

The game doesn’t use true isometric projection. The camera’s image is actually vertically compressed a little bit so that each tile’s width is exactly twice its height, instead of height*sqrt(3). This allows each tile to have power-of-2 dimensions for easy uploading to the graphics card.

Link to the calculator: http://www.niotso.org/zbuffer/
(Use the arrow keys)

Meanwhile, I’ve been working on IFF reading support. I’ve just made a basic command-line utility, iffexport, to extract all of the chunks from a given IFF file. My next task is to learn all of the chunk formats and document them on Niotso Wiki.

I haven’t started chunk-parsing yet, so at the moment the only screenshots I can offer are catalog thumbnails, which are stored in plain BMP format. See below:

Posts navigation
← Older Entries
Newer Entries →
  • Contributors

    • Fatbag
    • Blayer98
    • Propeng
    • Rhys
    • Afr0
    • Zc456
  • Archives

    • February 2014
    • June 2013
    • December 2012
    • July 2012
    • June 2012
    • April 2012
    • March 2012
    • February 2012
    • January 2012
  • Meta

    • Log in
    • Entries RSS
    • Comments RSS
    • WordPress.org
Proudly powered by WordPress Theme: Parament by Automattic.