source: tol/doc/performance/TOLCleaningDic05/changes_cleaning_tol.txt @ a2f070d

///////////////////////////////////////////////////////////////////////////
// Proceso de limpieza de TOL previo al desarrollo de OISS
///////////////////////////////////////////////////////////////////////////

El propósito fundamental del presente proceso de limpieza de TOL es dejar una versión de TOL algo más despejada de elementos estériles y confusos así como prepararla para el desarrollo de sistemas OISS que le den una mayor agilidad y robustez a los proyectos desarrollados en TOL, de forma que dé tiempo al equipo de desarrollo a diseñar e implementar un nuevo lenguaje TOL sin bloquear por ello a los analistas de forma excesiva.

En términos del código C++, el objetivo principal era más que nada disponer los templates básicos adecuados para más adelante poder instanciar objetos TOL serializables de forma indexada y probablemente utilizando memoria externa, lo cual se consigue bássicamente virtualizando los métodos principales de los objetos Name, Description, Contens, etc.; y esa ha sido mi principal labor. Sin embargo aprovechando el tirón le he dado un repasito general a TOL haciendo esa limpieza de la que tanto habíamos hablado, eliminando campos superfluos e incorporándolos eventualmente en su lugar correspondiente en la jerarquía de clases. A las clases BText y sobretodo BSet también les he dado un hachazo importante.

Como efecto colateral se consigue que desaparezcan algunos errores en la recompilación, especialmente de funciones TOL de usuario, y una disminución de en torno a un 40% de la memoria usada por TOL y un pequeño aumento de la velocidad que en modo alguno alguno justificarían el trabajo hecho, el cual cobra sentido sólo por el hecho de conseguir una jerarquía de clases más limpia y potente. 

El principal problema de memoria se debe a que los objetos siguen sin borrarse convenientemente como deberían debido al lío que supone el tratamiento de las referencias vía BList y BSet, las cuales llevan bastante mala vida, pues se manipulan en exeso a veces, además de las llamadas directas a InNRefs y DecNRefs. He mejorado parcialmente esto devolvíéndose ahora bastante más memoria que antes por lo que en algunas ocasiones, cuando se hacen procesos repetitivos muy pesados, el ahorro de memoria puede ser brutal.

Sin embargo, en el momento en el que se produzca cualquier pequeño desorden en los procesos de destrucción se empiezan a quedar objetos huérfanos de lista que les pueda ordenar destruirse y por eso pasa lo que pasa. Eso es realmente muy complicado de detectar sin tener casos de código TOL sencillo que reproduzcan y aislen los problemas por lo que no he dedicado apenas tiempo a ello aunque algo sí ha mejorado la situación. En cualquer caso es algo a lo que no merece la pena dedicarle tiempo a no ser que salte algo a la vista en cuaquier momento.

Además este problema quedará salvado, o al menos paliado en gran medida, con el desarrollo de OISS, en cuanto disponga del tiempo necesario para ello.




///////////////////////////////////////////////////////////////////////////
//Basic objects related changes
///////////////////////////////////////////////////////////////////////////
bbasic/tol_batom.h : 
-virtual methods IsDeadObj and System are been downloaded from BSyntaxObject to this level to allow a best memory tracing

bbasic/atm.cpp : 
-Changed memory trace handling

bbasic/tol_barray.h : 
-Constant factor 2 for Add reallocation replaced by ReallocMarginFactor defined as 1.2

bbasic/txt.cpp : 
-BText::defectSize_  =  0 instead of old unused value of 64, due to constant 1 was used;

bbasic/tol_btext.h
bbasic/txt.cpp
bbasic/tol_bdir.h
bbasic/dir.cpp 
bbasic/dirtext.cpp 
bbasic/tol_bopt.h
bbasic/opt.cpp      
bbasic/opttext.cpp
bparser/par.cpp : Now BText doesn't inherites from BAtom. To insert strings in lists BObject will be used.

bbasic/tol_btimer.h
bbasic/timer.cpp : New access functions to miliseconds and string formated of process elapsed time


///////////////////////////////////////////////////////////////////////////
//Math objects related changes
///////////////////////////////////////////////////////////////////////////
bmath/bstat/stat.cpp:
-Frequency(...) must divide in just n parts instead of n+1

///////////////////////////////////////////////////////////////////////////
//Kernel-grammar related changes
///////////////////////////////////////////////////////////////////////////
btol/bgrammar/tol_bstruct.h
btol/bgrammar/str.cpp : BField inherites from BObject instead of BSyntaxObject

btol/bgrammar/tol_bgrammar.h
btol/bgrammar/graacc.cpp     : New IsAutoContens() method of grammars to be used in future

btol/bgrammar/graimp.cpp
Unparsing is used just in Evaluate(BText expression) instead of Evaluate(BTree tree) to save memory and process time.


///////////////////////////////////////////////////////////////////////////
//Syntax-hierarchy related changes
///////////////////////////////////////////////////////////////////////////
btol/bgrammar/tol_bsyntax.h
btol/bgrammar/syn.cpp
btol/bgrammar/tol_bgrammar.h
btol/bgrammar/graacc.cpp
btol/bgrammar/tol_bmethod.h
btol/bgrammar/met.cpp         : 
-Unused class BSyntaxImage and related functions has been eliminated 

btol/bgrammar/tol_bsyntax.h
btol/bgrammar/tol_bgencon.h
btol/bgrammar/tol_bgenobj.h
btol/lang/language.cpp        : 
-Boolean members system_, released_, calculated_, and calculating_ are compacted in a protected bit-struct flags_ member at BSyntaxObject using just a byte all together

tol_bcommon.h,
btol/bgrammar/tol_bsyntax.h
btol/bgrammar/syn.cpp : 
-BSyntaxObject Mode constants are moved to tol_bsyntax.h

btol/bgrammar/tol_bsyntax.h
lang/language.cpp:
-flags_.system_ is set to true for all BSyntaxObjects until TOLHasBeenInitialized() return true, when InitGrammars ends

btol/bgrammar/tol_bsyntax.h
btol/bgrammar/syn.cpp:
-Old BSyntaxObject tolPath_ is restricted to BSetFromFile to be used in new method ParentPath() to access to path of the file where the object was created in.
-New BSynObjOptInfo class to optional storement of name_, description_, localName_ and parent_ in BSyntaxObject just when needed
-Changed memory trace handling

btol/bgrammar/tol_bgentmp.h
btol/bgrammar/tol_bgencon.h
btol/bgrammar/tol_bgenobj.h
lang/language.cpp : 
-class BRenamed is eliminated
-BRenamedGen is renamed as BRefObject
-In BTmpObject args_ member is now a BList* instead of a BSet and a short card_, and BSyntaxObject** array_ are used to accelerate accesses to arguments. 
-New template classes has been inserted in *Contens* template hierarchy
  ·A new template BGraContensBase<Any> without contens_ to save memory when is this member is not needed and allowing virtual access to it
  ·A new template BGraContensFND<Any,Name_,Description_> usefull to returned objects by built-in functions saving memory in each instance
  ·A new temlpate BCopyContens<Any> to use instead of BRenContens<Any> to low size types as Real, Complex, Date, Text, Polyn, Ration. This objects copy the contens of referenced object and try to destroy it. If nos possible destroyer is delayed to the live end of encapsulating object 

btol\matrix_type\matgra.cpp
btol\set_type\setgra.cpp
btol\bgrammar\tol_bgentmp.h:
-Access to BSet BTmpObject<Any>::Args() must be replaced by BList* ArgList() or DupArgList()

btol\bmonte\mc-integrator.cpp
btol\matrix_type\matgra.cpp
btol\real_type\armseval.cpp
btol\real_type\datgrapr.cpp
btol\real_type\datgrast.cpp
btol\real_type\datgrav.cpp
btol\set_type\setgra.cpp
btol\timeset_type\tmsgrav.cpp
btol\bgrammar\tol_bgentmp.h:
lang\language.cpp
-Access to BSet BTmpObject<Any>::Args().Card() must be replaced by NumArgs()

btol\bmonte\mc-integrator.cpp
btol\cseries_type\ctsrgra.cpp
btol\matrix_type\multimin.cpp
btol\serie_type\tsrgra.cpp
btol\set_type\setgra.cpp
btol\bgrammar\tol_bgencon.h
btol\bgrammar\tol_bgenobj.h
btol\bgrammar\tol_bsyntax.h
btol\set_type\tol_bsetgra.h:
lang\language.cpp
-Access to BSyntaxObject or inherited boolean members system_, released_, calculated_, and calculating_ must be changed to flags_.* or BSyntaxObject::Put* functions

btol\matrix_type\matgra.cpp
btol\real_type\datgra.cpp
btol\real_type\datgsrst.cpp
btol\text_type\txtgra.cpp
lang\language.cpp
-BSystem* objects that shouldn't be system constants are changed to BUser* to inherite from BGraContens<Any> instead of undeletable system class BGraConstant<Any>
   
  
///////////////////////////////////////////////////////////////////////////
//TOL Functions and Code-type related changes
///////////////////////////////////////////////////////////////////////////
btol/bgrammar/tol_boper.h
btol/bgrammar/opr.cpp      : 
-New BOperator::AddSystemOperator() to control different creating processes of built-in and user functions that were never deleted internally

btol/bgrammar/met.cpp :
-New bool BuildingMethod() function to special treatment of overloading of method's functions

btol/bgrammar/txthash.cpp : 
-Fixed forbidden accesses to code->Contens() for non BUSerFunction objects
-When BuildingMethod() is false Mode() must be equal to so.Mode() to match

btol/code_type/tol_bcode.h
btol/code_type/cod.cpp      : 
-Internal BCode::operator_ is changed from BUserFunction*  to BStandardOperator*

btol/code_type/tol_bcodgra.h : 
-BUserFunCode::Mode() is changed to BOBJECTMODE instead of BUSERFUNMODE that is reserved to internal operators
-Unused class BCodeCreator is eliminated


///////////////////////////////////////////////////////////////////////////
//Set-type related changes
///////////////////////////////////////////////////////////////////////////
btol\bdb\bdb.cpp
btol\bdb\bdspool.cpp
btol\bgrammar\graimp.cpp
btol\bgrammar\spfuninst.cpp
btol\bmodel\aia.cpp
btol\bmodel\estim.cpp
btol\bmodel\foread.cpp
btol\bmodel\modcalc.cpp
btol\bmodel\model.cpp
btol\bmonte\mc-integrator.cpp
btol\matrix_type\matgra.cpp
btol\matrix_type\multimin.cpp
btol\polynomial_type\polgra.cpp
btol\real_type\datgrast.cpp
btol\set_type\set.cpp
btol\set_type\setgra.cpp
btol\set_type\tol_bset.h
btol\text_type\txtgra.cpp
lang\language.cpp :
-At BSet class BList* element_ is eliminated. When a list is needed a copy is available through ToList()
-Access to BSet BTmpObject<Any>::Args() must be replaced by BList* ArgList() or DupArgList()
-New class BGraContensFNDSet<BSet, Name_, Description_> inherited from BGraContensFND<Any,Name_,Description_> to allow special constructors of bulit-in functions returned sets
-BSet constructor from BList* and operator= to BList* are eliminated to replace in this case by PutElement or RobElement


///////////////////////////////////////////////////////////////////////////
//Real-type related changes
///////////////////////////////////////////////////////////////////////////
btol/real_type/datgra.cpp
bbasic/out.cpp :
-New controls about number of errors and warnings exported to TOL user


///////////////////////////////////////////////////////////////////////////
//Complex-type related changes
///////////////////////////////////////////////////////////////////////////
btol/complex_type/cmpgra.cpp : 
-BCmpConstant* i_  is changed to BParamCmp


///////////////////////////////////////////////////////////////////////////
//Serie-type related changes
///////////////////////////////////////////////////////////////////////////
btol/serie_type/tol_btmser.h
btol/serie_type/tsrgra.cpp : 
-dating_ nRefs_ is not touched by BTimeSerie because is non needed feature.
-unused hashDat_ member has been eliminated

btol\bdb\bdb.cpp
btol\bdb\bdspool.cpp
btol\bmodel\modprev.cpp
btol\serie_type\srg.cpp
btol\serie_type\tsr.cpp:
btol\serie_type\tsrgra.cpp:
btol\serie_type\tsrgrav.cpp:
btol\serie_type\tol_btmser.h:
btol\serie_type\tol_btsrgrp.h
btol\set_type\setgra.cpp:
-BData data_ is now really protected eliminating GetDataBuffer() access 
-New BTimeSerie member functions PutBufDat to change a dat, or Realloc, Alloc and Delete to manage memory
bmath/mathobjects/tol_bdat.h
bmath/mathobjects/dat.cpp:
-New BData class inherited from BArray<BDat> to control Time Series used volume


btol\bdb\bdspool.cpp
btol\bmodel\estim.cpp
btol\bmodel\modcalc.cpp
btol\bmodel\tol_bmodel.h
btol\matrix_type\matgra.cpp
btol\real_type\datgra.cpp
btol\real_type\datgsrst.cpp
btol\serie_type\srg.cpp
btol\serie_type\tol_btsrgrp.h
btol\serie_type\tsrgrai.cpp
btol\text_type\txtgra.cpp
lang\language.cpp
-BSerieGroup and BSerieTable use now a BArray<BUserTimeSerie*> instead of a BSet to store Series in order to avoid secondary effects of references over them
-Now the only one creator for BSerieGroup and BSerieTable has no arguments. 
-Then you can add a Serie, a BList*, a BSet or another BSerieGroup or BSerieTable
-After you can use BSerieGroup to prepare operations between series or test dating and date bounds of a collection of series
 or you can filla BSerieTable with data from collected series.

btol/bmodel/foread.cpp
btol/bmodel/modcalc.cpp
btol/bmodel/model.cpp
btol/bmodel/tol_bmodel.h
-BList* outLst_ and inLst_ are eliminated from BModel to apply new BSerieTable advantages


///////////////////////////////////////////////////////////////////////////
//Console related changes
///////////////////////////////////////////////////////////////////////////
lang/language.cpp : 
-revisited commands and user help for interactive console mode of tol.exe



sizeof(BAtom)=8
sizeof(BSyntaxObject)=16
sizeof(BText)=12
sizeof(BGraContens < BText >)=28
sizeof(BGraConstant< BText >)=28
sizeof(BRenContens < BText >)=20
sizeof(BCopyContens < BText >)=32
sizeof(BTmpContens < BText >)=40
sizeof(BDat)=16
sizeof(BGraContens < BDat >)=32
sizeof(BGraConstant< BDat >)=32
sizeof(BRenContens < BDat >)=20
sizeof(BCopyContens < BDat >)=40
sizeof(BTmpContens < BDat >)=48
sizeof(BDate)=32
sizeof(BGraContens < BDate >)=48
sizeof(BGraConstant< BDate >)=48
sizeof(BRenContens < BDate >)=20
sizeof(BCopyContens < BDate >)=56
sizeof(BTmpContens < BDate >)=64
sizeof(BCode)=4
sizeof(BGraContens < BCode >)=20
sizeof(BGraConstant< BCode >)=20
sizeof(BRenContens < BCode >)=20
sizeof(BCopyContens < BCode >)=24
sizeof(BTmpContens < BCode >)=32
sizeof(BSet)=20
sizeof(BGraContens < BSet >)=36
sizeof(BGraConstant< BSet >)=36
sizeof(BRenContens < BSet >)=20
sizeof(BCopyContens < BSet >)=40
sizeof(BTmpContens < BSet >)=48
sizeof(BMatrix<BDat>)=44
sizeof(BGraContens < BMatrix<BDat> >)=60
sizeof(BGraConstant< BMatrix<BDat> >)=60
sizeof(BRenContens < BMatrix<BDat> >)=20
sizeof(BCopyContens < BMatrix<BDat> >)=64
sizeof(BTmpContens < BMatrix<BDat> >)=72
sizeof(BComplex)=72
sizeof(BGraContens < BComplex >)=88
sizeof(BGraConstant< BComplex >)=88
sizeof(BRenContens < BComplex >)=20
sizeof(BCopyContens < BComplex >)=96
sizeof(BTmpContens < BComplex >)=104
sizeof(BPolyn < BDat >)=12
sizeof(BGraContens < BPolyn < BDat > >)=28
sizeof(BGraConstant< BPolyn < BDat > >)=28
sizeof(BRenContens < BPolyn < BDat > >)=20
sizeof(BCopyContens < BPolyn < BDat > >)=32
sizeof(BTmpContens < BPolyn < BDat > >)=40
sizeof(BRational < BDat >)=24
sizeof(BGraContens < BRational < BDat > >)=40
sizeof(BGraConstant< BRational < BDat > >)=40
sizeof(BRenContens < BRational < BDat > >)=20
sizeof(BCopyContens < BRational < BDat > >)=44
sizeof(BTmpContens < BRational < BDat > >)=52


===================================================================
===================================================================
===================================================================
Revisite



bgrammar/txthash.cpp : 
          if((Mode() == BUSERFUNMODE) && (so.Mode() == BUSERFUNMODE))

          
===================================================================
RCS file: /home/cvsroot/tolp/tol/bbasic/tol_batom.h,v
retrieving revision 1.1.1.1
diff -r1.1.1.1 tol_batom.h
44c44
<     BInt nRefs_; //!< Number of atom references
---
>     unsigned short nRefs_; //!< Number of atom references
57c57


===================================================================
RCS file: /home/cvsroot/tolp/tol/bbasic/tol_butil.h,v
retrieving revision 1.1.1.1
diff -r1.1.1.1 tol_butil.h
153a154,222
> 
> //----------------------------------------------------------------------------
>   class TOL_API BUInt24
> //----------------------------------------------------------------------------


===================================================================
RCS file: /home/cvsroot/tolp/tol/bbasic/txtalgeb.cpp,v
retrieving revision 1.6
diff -r1.6 txtalgeb.cpp
1363c1363,1364
<     buffer_ = new BChar[size_ = maxChars+1];
---
>     size_ = maxChars+1;
>     buffer_ = new BChar[maxChars+1];
1378c1379,1380
<       BChar *newBuf = new BChar[size_=(1+length_*2)];
---
>       size_=(1+length_*2);
>       BChar *newBuf = new BChar[size_];


Index: btol/bgrammar/gra.cpp
===================================================================
RCS file: /home/cvsroot/tolp/tol/btol/bgrammar/gra.cpp,v
retrieving revision 1.6
diff -r1.6 gra.cpp
59a60
>              bool          autoContens,
76a78
>       autoContens_(autoContens),
85c87



===================================================================
Fallo del interfaz en proyectos
[root]
Children=1 2 3 4
Description=Root node
Exclude=0
ExcludeTOL=0
FileName=estimamg.prj
Label=estimamg.prj
Type=ROOT
[1]
Children=
Description=Parámetros de la estimación
Exclude=0
ExcludeTOL=1
FileName=paramicmg.tol
Label=paramicmg.tol
Type=TOL
[2]
Children=
Description=Inicializa los parametros de la estimación, abre la conexión con la BBDD e incluye las funciones
Exclude=0
ExcludeTOL=1
FileName=initestimationmg.tol
Label=initestimationmg.tol
Type=TOL
[3]
Children=
Description=
Exclude=0
ExcludeTOL=0
FileName=estimationmg.tol
Label=estimationmg.tol
Type=TOL
[4]
Children=
Description=Cierra la conexión con la BDD
Exclude=0
ExcludeTOL=1
FileName=../_end.tol
Label=../_end.tol
Type=TOL
[Included TOL Files]
Set estimamg.1=Include("paramicmg.tol");
Set estimamg.2=Include("initestimationmg.tol");
Set estimamg.3=Include("estimationmg.tol");
Set estimamg.4=Include("../_end.tol");



        : BCTmsTemporary() 
        {
    PutName(name);
    PutDescription(description);
    Grammar()->AddObject(this);     


    
  { 
    if(HasName()) {Grammar()->DelObject(this);} 
  }
    
  
  
  
Name: NObject
Value: 21670

Name: NCore
Value: 798573



Name: NObject
Value: 13930

Name: NCore
Value: 389026