[6.6] Mess with the distance_per_tile again (or still ?)

Topic: [6.6] Mess with the distance_per_tile again (or still ?) (Read 6259 times) previous topic - next topic

[6.6] Mess with the distance_per_tile again (or still ?)

August 30, 2009, 01:30:08 pm

Hello !

I was quite happy that the bug with the distance_per_tile was fixed. But it appears it wasn't the only one, or wasn't fixed fully. My main concern is about loading old savegames, meaning those before scaling was introduced. I have made extensive testing, and found out simply stunning variety of results you could get. For me, it looks like the scaling was not introduced in one, comprehensive way, but in each place it could matter slightly differently, using different methods, thus depending on what exactly you do you get very different results.

Anyway, here's the issue:

Regardless what you do (at least from what I CAN do), you cannot set the scale back to 1km/tile and get the same results as in experimental 3.14.

The zipped excel sheet with all the experimental setup explanation and results can be found here:
http://simutrans-germany.com/files/upload/DPT_test_results.zip

As you can see, the exact numbers never agree between 3.14 and 6.6, regardless what tricks you use. Maybe there are hidden modifiers in the savefile that I couldn't find that would allow to make changes to xml-formatted save to make it play just like 3.14 in regards to distances.
New games started with distance_per_tile (DPT) of 100 are very very close to original 3.14 when it comes to cargo, except for small variation in maintenance cost (no idea why it's there). The bad thing is that you can't get the same numbers when trying to convert old savegame (that has exactly the same setup like the new one you are creating) - when just loading with any DPT setting, the game uses hard-coded default of 30 (except for maintenance cost, this resembles the behavior in 6.4 where current DPT was used for maintenance and DPT from savegame for anything else, now current DPT is used for maintenance conversion, while still default 30 is used for anything else apparently). When "fixing" DPT in savegame, this works half-way, as revenue/operation costs are fixed, but some maintenance costs are not, resulting in discrepancy again.

For the p****engers, the situation is even more complicated, where fixing DPT in savegame drastically reduces p****enger flow, but I'm guessing this is due to time cutoffs (WAD) and not due to wrong DPT calculations. That's why I used cargo as main reference point (and all test games used cargo).

Anyway, it's still impossible to load older savegames and continue playing in 6.6, unless you don't mind basically re-building whole transport network to account for different conditions. Wouldn't it be quite easy actually to change the hard-coded "default DPT" to 100 instead of current 30, so that when loading older savegames, they are sort of converted properly ?

And on top of all that, there's the issue of my old save behaving really odd - as you can see the Revenue depends on CURRENT DPT and not in 100% consistent way. I don't know how it is possible, unless 3.14 saves differently games originally started in earlier versions than ones started in 3.14. While this doesn't seem to affect more recent saves or new games, this might hint to possible weakpoints already present in the code, that might break the game in future. I would think about re-implementing it again in a more robust way. Decide whether it should be game parameter (like station coverage, so always current value from config file is used) or should it be game variable (so the value from save game is always used). But if it's supposed to be variable, then it should be set by user on a game basis, just like climate heights or starting year, instead of being hidden in the config.

Well, all in all, the scaling still turns out to be buggy and very messy when trying to load older savegames.

Well, the main points are:
1. Could the hardcoded default for older saves be changed to 100, so they load properly ?
2. Watch out for the weak points in the code, as it seems the scaling is really unstable in current implementation as all the tests have shown.

Re: [6.6] Mess with the distance_per_tile again (or still ?)

Reply #1 – August 30, 2009, 02:01:34 pm

Dante,

thank you very much for your feedback: it is most helpful. I am not entirely clear from what you wrote above as to precisely what you have found the difficulties to be, however perhaps the following explanation should ****ist your understanding. Please let me know whether what I explain below still leaves anything unaccounted for.

In all versions of Simutrans-Experimental higher than 2.0, there has been a setting for scaling journey times. Before 6.0, when full scaling was introduced, it was called "journey_time_multiplier_percent". This came out of the fact that, if 1 tile was ****umed to be 1km, journey times were far too high to be realistic. In 6.0, this scaling was expanded to include everything that depended on the distance, to make things consistent. The values in the .dat files were ****umed to be per kilometre values, and were divided by the number of kilometres per tile to produce the per tile values used internally by the game. The "journey_time_multiplier_percent" was replaced by the "distance_per_tile" setting. In respect of the original journey times, it functions in exactly the same way as in Simutrans-Experimental versions 2.0 - 5.1, but it also has other functions and a new name.

Thus, if one sets "distance_per_tile" to 100, one does not get the same results as in Simutrans-Experimental 2.0 - 5.1 because those versions would (probably) have had the "journey_time_multiplier_percent" set to something other than 100. The inconsistency of behaviour is present in the old versions of Simutrans-Experimental (in not being consistent about how many tiles made up a kilometre). That inconsistency has been fixed from 6.0 onwards, but the effects of it may linger in saved games made with older versions of Simutrans-Experimental.

Re: [6.6] Mess with the distance_per_tile again (or still ?)

Reply #2 – August 30, 2009, 03:27:08 pm

Ah, I didn't remember that the scaling was started much earlier than 6.0.

Well, this probably explains the behavior of my save game.
****uming:
>I started out before any scaling started (don't remember the version, I think it was 2.4 or something like that)
>When simutrans pre-6.0 loading even older savegame doesn't find the journey distance scaling factor it doesn't write ANY factor into the savegame
>When 6.0+ loads savegame without factor, it doesn't write one EITHER, and uses current DPT from config file for revenue calculation
Then:
>The behavior of my old savegames is explained, where the revenue would depend on current DPT in config, regardless of what's in save game

This requires all 3 ****umptions, and some of them mean that simutrans is acting weird. I mean, the revenue depends on current DPT, but running costs/etc depend on different parameter. This means simutrans is implementing scaling in different ways in different places. And this is dangerous to calculation consistency. It is already broken by loading old savegames, but you never know if a small change in seemingly unrelated place in future wouldn't break the consistency again.

Still, the 6.0+ seems to be using not only ONE parameter for scaling, but has the scaling hidden among several parameters. Otherwise it would be impossible to get some of the results:
>Same revenue and running costs but different maintenance for DPT=30 and DPT=100 loaded 3.14 games
>Same running costs but different maintenance and revenue for DPT=30 and DPT=100 loaded very old games

My point is:
Make ALL the scaling depend on the very same SINGLE number.
Currently it's not (as the test results prove), so the scaling is not robust. New games may not suffer at the moment, but the fact that weakness/problem can't be seen doesn't mean it won't show up later.

[Edit:] Sorry if it sounded like I'm demanding something. I'm just concerned that this may affect the future of experimental and being non-robust may prevent those features to be eventually incorporated into standard. Besides, I just wanted to continue my old game

Re: [6.6] Mess with the distance_per_tile again (or still ?)

Reply #3 – August 30, 2009, 03:43:24 pm

Dante,

there is only one number for scaling, the distance_per_tile value. There is no other number that affects scaling, and never has been. Some of your ****umptions are incorrect:

Quote

>I started out before any scaling started (don't remember the version, I think it was 2.4 or something like that)

The journey_time_multiplier_percent was already in use by 2.0, as stated in the above post, and this was the early form of scaling, but applied only to journey times.

Quote

>When simutrans pre-6.0 loading even older savegame doesn't find the journey distance scaling factor it doesn't write ANY factor into the savegame

I am not sure what you mean here, since loading a saved game does not write anything to any saved game. Do you mean when that saved game is then re-saved in 6.6? It will then write the new values based on the full scaling in 6.6, but using the same scale figure (albeit slightly adjusted) as in < 6.0.

However, note that, if you loaded a game from < 6.0 into > 6.0, but < 6.6, then things like the fixed maintenance of stations, etc., which were brought into the scaling system by 6.6 would not be scaled, even if that game (having been saved in > 6.0 but < 6.6) is then loaded in 6.6.

Quote

>When 6.0+ loads savegame without factor, it doesn't write one EITHER, and uses current DPT from config file for revenue calculation

No, that is not correct. There is only one distance_per_tile variable in the code. If it loads a pre 2.0 saved game, it will use the distance_per_tile value from simuconf.tab.

Re: [6.6] Mess with the distance_per_tile again (or still ?)

Reply #4 – August 30, 2009, 04:40:17 pm

Quote from: jamespetts – on August 30, 2009, 03:43:24 pm

there is only one number for scaling, the distance_per_tile value. There is no other number that affects scaling, and never has been.

Well, then how is it possible that when I load 3.14 game into 6.6 with DPT of 100 I get the same revenue and runnig costs than when I load 3.14 game into 6.6 with DPT of 30, while the maintenance changes ?
And since it's 3.14 game, then these values later on are fixed for these savegames, so they must be saved inside savegame. It doesn't use the value from the config file anymore.

I understand that on a conceptual level there is only one number for scaling, but on the implementation level there are more, or above situation is simply impossible. I'm not saying it's intentional, but the experiment says so.

Quote

I am not sure what you mean here, since loading a saved game does not write anything to any saved game. Do you mean when that saved game is then re-saved in 6.6? It will then write the new values based on the full scaling in 6.6, but using the same scale figure (albeit slightly adjusted) as in < 6.0.

Here, I meant that for example 3.14 loading savegame from 2.3 or maybe even form Simutrans Standard and then re-saving under 3.14.

Quote

No, that is not correct. There is only one distance_per_tile variable in the code. If it loads a pre 2.0 saved game, it will use the distance_per_tile value from simuconf.tab.

I see, so the 30 value that I thought hard-coded was actually sort of transcription of the factor used for journey time for >2.0 <6.0 game. That explains why it's always 30. Still the maintenance costs don't adhere to that, but to the DPT from the config file at the moment the game was first loaded into 6.6. So in the end they get rescaled by different value than revenue and operational costs.

Well, I think I'm beginning to understand the whole thing with sudden revenue etc. changes. The effect of new game with DPT=30 is same as loading 3.14 game with DPT=30 in config. You get the same values. Still, you can never get values as if started with DPT=100 (very close to values present in 3.14) when loading old game. Fine.

Still, there are some problems with the scaling. I'm a scientist IRL. I did the experiment (all the tests) and obtained results. Of course my hypotheses formed on the basis of those results may be wrong, but the results themselves are hard facts.

Well, so the first on the plate is:
Why loading old game with different DPT give different-maintenance but same-revenue/running costs savegames ?

I can upload the savegames I made for testing purposes, but you can get this effect using any pre-6.0 savegame (I think any, for sure 3.14) - just load them with different DPT in config and run on fast forward for a year. You should see the same thing as in my test sheet.

[Edit:] While I can't compile simutrans from sources (I gave up trying to make it work on watcom), I might try someday trace the problem at hand in the source code. Maybe I'll just find the thing ? Don't have enough time for that now, but maybe someday I will.

Re: [6.6] Mess with the distance_per_tile again (or still ?)

Reply #5 – August 30, 2009, 05:37:59 pm

Dante,

thank you very much for your work on this. When I have some time, I shall have to look into this to see whether the inconsistency that you describe is a bug, an unwanted side-effect, or a necessary incident of how things work. Your efforts are much appreciated :-)

Re: [6.6] Mess with the distance_per_tile again (or still ?)

Reply #6 – August 30, 2009, 11:21:57 pm

Well, after incorporating the info that 0.3 scaling factor was present before, just didn't do all it does now, I formulated a working theory and after few calculations it turned out I can explain all maintenance values for new 6.6 and 3.14 games. It sort of explains the revenue and running costs too.

So here's the theory:

Used abbreviations: distance_per_tile value as used in current game = DPT, mysterious building maintenance modifier = MBMM, current distance_per_tile value in config = CDPT, pre-6.0 distance old scaling factor = OSF

Theory:
There are 2 variables that dictate the whole scaling: DPT and MBMM.
DPT affects revenue calculation, running costs and way maintenance.
MBMM affects building costs and maintenance (but not ways).
When you start new game in 6.6, both DPT and MBMM are set to CDPT.
When you load an old game (3.14), DPT is set to OSF=0.3, while MBMM is set to CDPT.
When you modify the DPT in savegame (as I did during testing), it of course affects DPT, but does not affect MBMM at all.

Above theory explains all my test results in 6.6 for new 6.6 games and old 3.14 games (but not 3.14 values in 3.14 games unfortunately). It doesn't touch the my old savegames (I checked, the game contained in them was started in Simutrans Standard, and went through different simutrans experimental stages up to 4.0 I believe), as the behavior of these is still confounding me.

Re: [6.6] Mess with the distance_per_tile again (or still ?)

Reply #7 – August 31, 2009, 08:23:59 am

The difficulty with that theory is that there is no mechanism in the code that could possibly provide for a "MBMM". If there is a bug somewhere, it will be in applying the distance_per_tile value twice or at an incorrect time, or not applying it in certain circumstances. There is no other variable that performs the same or a similar function to distance_per_tile.

Re: [6.6] Mess with the distance_per_tile again (or still ?)

Reply #8 – August 31, 2009, 08:37:47 am

James,

I think I found a bug which will cause the problems experienced by Dante, though I don't know if that is the only one. In the following code from einstellungen_t::rdwr() :

Code: [Select]

			if(file->get_experimental_version() < 6)
			{
				// Scale the costs to match the scale factor.
				cst_multiply_dock *= distance_per_tile;
				cst_multiply_station *= distance_per_tile;
				cst_multiply_roadstop *= distance_per_tile;
				cst_multiply_airterminal *= distance_per_tile;
				cst_multiply_post *= distance_per_tile;
				cst_signal *= distance_per_tile;
				cst_tunnel *= distance_per_tile;
				cst_third_rail *= distance_per_tile;
				cst_buy_land *= distance_per_tile;
				cst_remove_tree *= distance_per_tile;
			}
		}
		else {
			// name of stops
			set_name_language_iso( umgebung_t::language_iso );
 
			// default AIs active
			for(  int i=0;  i<MAX_PLAYER_COUNT;  i++  ) {
				if(  i<2  ) {
					spieler_type[i] = spieler_t::HUMAN;
				}
				else if(  i==3  ) {
					spieler_type[i] = spieler_t::AI_P****ENGER;
				}
				else if(  i<8  ) {
					spieler_type[i] = spieler_t::AI_GOODS;
				}
				else {
					spieler_type[i] = spieler_t::EMPTY;
				}
				automaten[i] = false;
				p****word[i][0] = 0;
			}
		}
 
		if(file->get_version()>101000) {
			file->rdwr_bool( seperate_halt_capacities, "" );
			if(file->get_experimental_version() < 2)
			{
				// Was pay for total distance.
				// Now depracated.
				uint8 dummy;
				file->rdwr_byte( dummy, "" );
			}
 
			file->rdwr_short(starting_month, "");
 
			file->rdwr_short( river_number, "" );
			file->rdwr_short( min_river_length, "" );
			file->rdwr_short( max_river_length, "" );
		}
 
		if(file->get_version()>102000) {
			file->rdwr_bool( avoid_overcrowding, "" );
		}
 
		if(file->get_version()>102001) 
		{
			bool dummy;
			file->rdwr_bool(dummy, "" );
			file->rdwr_bool( with_private_paks, "" );
		}
		if(file->get_experimental_version() >= 1)
		{
			file->rdwr_short(min_bonus_max_distance, "");
			file->rdwr_short(max_bonus_min_distance, "");
			if(file->get_experimental_version() == 1)
			{
				uint16 dummy;
				file->rdwr_short(dummy, "");
			}
			else
			{
				file->rdwr_short(median_bonus_distance, "");
				file->rdwr_short(max_bonus_multiplier_percent, "");
				uint16 distance_per_tile_integer = distance_per_tile * 100.0F;
				file->rdwr_short(distance_per_tile_integer, "");
				if(file->get_experimental_version() < 5)
				{
					// In earlier versions, the default was set to a higher level. This
					// is a problem when the new journey time tolerance features is used.
					distance_per_tile = (distance_per_tile_integer * 2) / 2.5;
				}		
				distance_per_tile = distance_per_tile_integer / 100.0F;
				if(file->get_experimental_version() < 6)
				{
					min_bonus_max_distance /= distance_per_tile;
					max_bonus_min_distance /= distance_per_tile;
				}

distance_per_tile is used even before it is loaded fom the save game. The code for adjusting the various costs are in the toppest portion of the quoted code, but distance_per_tile is not loaded until the bottomest portion. To fix it, please move all the cost adjustments to the last IF block in the quoted code.

Knightly

Re: [6.6] Mess with the distance_per_tile again (or still ?)

Reply #9 – August 31, 2009, 08:54:07 am

That's a great find !

I actually can see this "second factor" for maintenance. Well... even 10 of them

Sure, they are derived from main distance_per_tile, but they are separate variables.

And actually this gives the last piece of the puzzle to us !

We know the identity of MBMM. All those cst_* variables are together MBMM.
If you load pre-6.0 save, because of the bug, the maintenance is scaled by CDPT, while later OSF is read as DPT for savegame.
Thus, this gives results exactly consistent with my theory !

Yay !

Now I'll have to start thinking on the hard part of the problem. How to explain even more crazy behavior of my old, old savegame. It was originally created in Simutrans-Standard and then migrated through several older versions of Experimental (2.x through 3.14). The funny thing it behaves differently than plain 3.14 savegame.

But at least the first part is solved !

[Edit:] Actually, after looking at the posted code again, I see some things that might just be responsible for the old old savegame behavior. Would have to think about that though.

Re: [6.6] Mess with the distance_per_tile again (or still ?)

Reply #10 – August 31, 2009, 05:23:00 pm

James,

I just noticed another bug, which is also from the quoted code above. The relevant portion is reproduced below :

Quote

uint16 distance_per_tile_integer = distance_per_tile * 100.0F;
file->rdwr_short(distance_per_tile_integer, "");
if(file->get_experimental_version() < 5)
{
   // In earlier versions, the default was set to a higher level. This
   // is a problem when the new journey time tolerance features is used.
   distance_per_tile = (distance_per_tile_integer * 2) / 2.5;
}
distance_per_tile = distance_per_tile_integer / 100.0F;

In the above, there are 2 places where distance_per_tile is ****igned a value. The blue statement is, according to your comment, specifically for versions before v5. However, the green statement overrides the variable with another value. ****uming both statements are correct, I guess the code should actually be :

Quote

if(file->get_experimental_version() < 5)
{
   // In earlier versions, the default was set to a higher level. This
   // is a problem when the new journey time tolerance features is used.
   distance_per_tile = (distance_per_tile_integer * 2) / 2.5;
}
else
{
   distance_per_tile = distance_per_tile_integer / 100.0F;
}

If that is not what you intended, then probably the blue statement should be :

Quote

distance_per_tile_integer = (distance_per_tile_integer * 2) / 2.5;

BTW, * 2 / 2.5 can be simplified to * 0.8 .

Knightly