Use of global vars in the JModelica run-time

6 posts / 0 new

Fri, 2011-08-05 15:11

Iakov

Offline

Joined: 2011-08-05

Use of global vars in the JModelica run-time

Several tickets on trac have a potential solution that would require us to use global variables. Specifically, for ex:
Error handling (for CPPAD to propagate error code correctly and use jmi->jmi_log for printing messages):
https://trac.jmodelica.org/ticket/257
https://trac.jmodelica.org/ticket/1501
Memory management (reimplementation of malloc/free within FMU):
https://trac.jmodelica.org/ticket/1216
https://trac.jmodelica.org/ticket/1497

In both cases making a “jmi_t*” global would allow to solve the issue. However, global variables are a problem in the general case when several instances of the same JMU/FMU are loaded into a process. To make things even more general we need to assume that the process runs multiple threads concurrently and each thread has its own JMU/FMU.

I see two ways to proceed:

With global vars:
The “current_jmi” pointer should be set by all the JMU/FMU functions that may result in memory allocation/error handling calls, such as ode_derivatives. The normal solution for the multi-threaded case is to use thread local storage. This however presents a portability issue: what thread library should be used?
Pthreads:
pthread_key_create(&key, 0) ; pthread_set_specific(key, current_jmi)
Openmp:
#pragma omp threadprivate(current_jmi)

The best way is probably to have some callback into the calling process, e.g.:
setThreadLocalEnvironment(void*), void* getThreadLocalEnvironment(). If those callbacks are not present then serial execution is assumed. Note that the current standard does not have any provision/recommendations on multithreaded environment that I could find.

If we take this approach than we should propose modification to the FMI standard (additional callbacks).
For error handling with CppAd our implementation of CPPAD_DISCRETE_FUNCTION() would be able to access jmi* for logging error messages.
Not using global vars.
For memory management in FMU we would need to modify Sundials/Superlu/other libs so that jmi pointer is propagated to the routines that do memory allocation. One way is to introduce memory allocation callbacks for the libs.

For error handling with CppAd we can either do some tricks like:
typedef union {
jmi_real_t val;
void* ptr;
} jmi_union_ptr_dbl_t;
And then send in jmi pointer as an argument to the tape. Even if possible – this is not very nice.
An alternative is to request a modification to the CppAd: ability to specify custom streams for output of PrintOp. Since this is a good general feature I plan to ask Brad Bell about it.

My current preference would be:

Use global jmi_t pointer with memory allocation callbacks. Request fmiCallbackFunctions structure to be complemented with fmiReallocateMemory/setThreadLocalEnvironment. / getThreadLocalEnvironment. I’d also propose to add fmiReallocateMemory() to the existing interface. Lack of “reallocate” makes life more difficult when re-implementing all the memory allocation methods since one needs to allocate extra size_t to keep the size of memory block. In Jmodelica this feature should be optional, e.g., controlled by compiler flag. This is since malloc/free do exist on most platforms and an implementation via FMI would make it just slower.
Use custom stream and CppAd PrintOp for error reporting. Note that there is a conditional version of Print in the latest version of CppAD. BTW how often do we update ThirdParty/CppAD? Note that this is only needed for JMU with CppAD (as of today, might change).

Comments?

/Iakov

Wed, 2011-08-10 12:47

jmattsson

Offline

Joined: 2009-10-18

Well...

Hi Iakov

For malloc/free we probably have to use global variables, at least I can't see a way around it. It is pretty easy to set it up so we can detect the situations where it would cause problems, at least if we have these separate from any global jmi struct. I don't really see modifying the libs as a fesiable option.
Your suggestion about setThreadLocalEnvironment/getThreadLocalEnvironment seems circular - to get the fmi_t pointer we need to call getThreadLocalEnvironment(), and to get that we need the fmiCallbackFunctions struct, and to get that we need the fmi_t pointer. This could possibly be solved in the manner described in my first point.
For error handling, I would prefer to solve this by collaborating with Brad Bell. We have the jmi_t pointer when we build the tape, so it should be possible to solve. We need to make sure to think it though before talking with him, though. For one thing, I think a custom stream would be hard to interface with the FMI logging function.

To sum it up, I'd like to avoid both a global jmi_t/fmi_t and thread local storage if we can, and solve the error handling issue in cooperation with Brad.

Anyone else?

Jesper

Wed, 2011-08-10 16:22

Iakov

Offline

Joined: 2011-08-05

Some comments

Hi!

jmattsson wrote:

For malloc/free we probably have to use global variables, at least I can't see a way around it. It is pretty easy to set it up so we can detect the situations where it would cause problems, at least if we have these separate from any global jmi struct. I don't really see modifying the libs as a fesiable option.

I agree that library modification is not feasible.
Just to be clear: malloc implementation will have to call allocateMemory() provided in fmiCallbackFunctions.
The problem with global vars is that several instances of the same FMU can be instantiated within a single process, each in a different thread, each with a different allocateMemory callback.
One way is to follow Dymola and postpone the problem by saying "generated FMU does currently not support inclusion of several instances" or softer: "generated FMU does not currently support concurrent instances". In both cases we can use globals and "reset" them to the currently active instance.
Other way is to follow CppAd and explicitly choose, e.g., OpenMp as supported threaded environment (means -> might work with pthreads but not sure). We might also provide variants depending on the thread lib (quite easy to fix by compile/link flags).
What option would you propose?

jmattsson wrote:

Your suggestion about setThreadLocalEnvironment/getThreadLocalEnvironment seems circular - to get the fmi_t pointer we need to call getThreadLocalEnvironment(), and to get that we need the fmiCallbackFunctions struct, and to get that we need the fmi_t pointer. This could possibly be solved in the manner described in my first point.

Agreed, my bad. To my defence: I can imagine the situation with different memory allocators in different threads. I can not imagine using both openmp and pthreads (or whatever thread lib) in the same code. Bad idea anyway.
I now bend towards different prebuild versions for different thread libs.

jmattsson wrote:

For error handling, I would prefer to solve this by collaborating with Brad Bell. We have the jmi_t pointer when we build the tape, so it should be possible to solve. We need to make sure to think it though before talking with him, though. For one thing, I think a custom stream would be hard to interface with the FMI logging function.

I was thinking sending an strstream (or similar) to Forward in jmi_func_ad_F and then dumping its contents with jmi_log().
Here is Brad's comment: http://list.coin-or.org/pipermail/cppad/2011q3/000215.html
If we would tape jmi_t* (or any 'context' pointer in general) then we indeed need to decide on what we do with it first.

jmattsson wrote:

To sum it up, I'd like to avoid both a global jmi_t/fmi_t and thread local storage if we can

Not sure I follow. What was your proposal for memory allocation in FMU?

/Iakov

Wed, 2011-08-24 09:17

Iakov

Offline

Joined: 2011-08-05

FMI 2.0

With the current FMI 2.0 proposal there are capability flags:

Quote:

canNotUseMemoryManagementFunctions:
If true, the FMU uses its own functions for memory allocation and freeing only. The callback functions allocateMemory and freeMemory given in fmiInstantiateModel are ignored.
canBeInstantiatedOnlyOncePerProcess:
This flag indicates cases (especially for embedded code), where only one instance per FMU is possible
(multiple instantiation is default = false; if multiple instances are needed and this flag = true, the FMUs must be instantiated in different processes).

So we can actually choose what to implement and how. One way is to have a compiler option, e.g., "use_fmi_memory_management". When compiled with this flag a global variable is used to keep a pointer to memory management callbacks. canNotUseMemoryManagementFunctions is false and canBeInstantiatedOnlyOncePerProcess is true. Otherwise we use standard malloc/free and so no global variables are needed. Therefore, canNotUseMemoryManagementFunctions is true and canBeInstantiatedOnlyOncePerProcess is false.

Thu, 2011-09-08 10:14

jmattsson

Offline

Joined: 2009-10-18

Clarification

If we keep global pointers to the malloc/free given by the caller, and a reference count (the number of instantiations that have passed in the same malloc/free pointers), then we can check if the pointers given by one instantiation differs from one already started. This means that we can detect if someone tries to replace the ones we already have. Something along the lines of:

// declarations in global scope
(the typedef for the malloc function pointer) fmi_malloc;
(the typedef for the free function pointer) fmi_free;
volatile int malloc_reference_count;
(a lock from some thread lib)

// in the instantiation function
// new_malloc/new_free = pointers from callback struct
(lock the lock)
if (malloc_reference_count) {
    if (fmi_malloc != new_malloc || fmi_free != new_free)
        (return an error)
} else {
    fmi_malloc = new_malloc;
    fmi_free = new_free;
}
malloc_reference_count++;
(unlock the lock)

// in the FMU-freeing function
(lock the lock)
malloc_reference_count--;
(unlock the lock)

The "current fmu" thread-global var could also work, but I prefer having only what we really need globally.

As a side note, the new capability flags would have been better as positive flags instead of negative ones. (I.e. canUseMemoryManagementFunctions and supportsMultipleInstantiation) :(

Jesper

Fri, 2011-09-09 08:58

Iakov

Offline

Joined: 2011-08-05

Thread safety

Hello!

jmattsson wrote:

If we keep global pointers to the malloc/free given by the caller.

Maybe generalize and have a global structure "jmi_global_data"? The counter can be there as well if we decide to have it.
If we then indicate that FMUs are not thread safe (not a word in the standard on the rules) then we can handle this consistently: switch the jmi_global_data pointer on calls that can use malloc/free (there aren't so many).

jmattsson wrote:

(a lock from some thread lib)

I think that if we do take the troubles with supporting different thread libs (at least win/pthread) then we can use thread local storage and handle the whole thing gracefully.

On a side note: CppAD is not thread safe (uses global vars). Brad is working on thread safety but the code is not stable. Our (Peter's) implementation of AD is not finalized neither. Therefore, I believe it's too early for jmodelica to care about threads too much.

jmattsson wrote:

As a side note, the new capability flags would have been better as positive flags instead of negative ones. (I.e. canUseMemoryManagementFunctions and supportsMultipleInstantiation) :(

I believe the idea is to always have "false" as default (compatible with the previous standard version). Strange names anyway...

/Iakov

JModelica.org

Primary links

Use of global vars in the JModelica run-time