Include path to Python headers for Fortran compilation

9 posts / 0 new
Last post
guitorri
Offline
Joined: 2011-11-30
Include path to Python headers for Fortran compilation

Dear developers,
I was using Assimulo 1.4b2 under Sage for a while. Now I trying to upgrade to Assimulo 2.1 but the compilation/instalation fails.
Sage 4.8 has its own Python 2.6, under its own tree (my case ~/sage-4.8/local/ ).
The C code is compiling OK, but the newly introduced Fortran solvers cannot find the Sage's Python/Numpy headers (?), see below.
I tryed to tweak the setup.py on the Fortran extensions to include the headers, but I don't now what I am doing...
Any idea? How can I fix this issue?
Best regards,
Guilherme
 
=========== Failure ==============
(snip)
building 'assimulo.lib.dopri5' extension
compiling C sources
C compiler: gcc -fno-strict-aliasing -g -O2 -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes

creating build/temp.macosx-10.6-x86_64-2.6/build
creating build/temp.macosx-10.6-x86_64-2.6/build/src.macosx-10.6-x86_64-2.6
creating build/temp.macosx-10.6-x86_64-2.6/build/src.macosx-10.6-x86_64-2.6/assimulo
creating build/temp.macosx-10.6-x86_64-2.6/build/src.macosx-10.6-x86_64-2.6/assimulo/thirdparty
creating build/temp.macosx-10.6-x86_64-2.6/build/src.macosx-10.6-x86_64-2.6/assimulo/thirdparty/hairer
compile options: '-Ibuild/src.macosx-10.6-x86_64-2.6 -I/Users/guilherme/sage-4.8/local/lib/python2.6/site-packages/numpy/core/include -I/Users/guilherme/sage-4.8/local/include/python2.6 -c'
gcc: build/src.macosx-10.6-x86_64-2.6/assimulo/thirdparty/hairer/dopri5module.c
gcc: build/src.macosx-10.6-x86_64-2.6/fortranobject.c
compiling Fortran sources
Fortran f77 compiler: /Users/guilherme/sage-4.8/local/bin/gfortran -Wall -ffixed-form -fno-second-underscore -fPIC -O3 -funroll-loops
Fortran f90 compiler: /Users/guilherme/sage-4.8/local/bin/gfortran -Wall -fno-second-underscore -fPIC -O3 -funroll-loops
Fortran fix compiler: /Users/guilherme/sage-4.8/local/bin/gfortran -Wall -ffixed-form -fno-second-underscore -Wall -fno-second-underscore -fPIC -O3 -funroll-loops
creating build/temp.macosx-10.6-x86_64-2.6/assimulo/thirdparty
creating build/temp.macosx-10.6-x86_64-2.6/assimulo/thirdparty/hairer
compile options: '-Ibuild/src.macosx-10.6-x86_64-2.6 -I/Users/guilherme/sage-4.8/local/lib/python2.6/site-packages/numpy/core/include -I/Users/guilherme/sage-4.8/local/include/python2.6 -c'
gfortran:f77: assimulo/thirdparty/hairer/dopri5.f
assimulo/thirdparty/hairer/dopri5.f:558.35:

      FUNCTION HINIT(N,FCN,X,Y,XEND,POSNEG,F0,F1,Y1,IORD,               
                                  1
Warning: Unused variable xend declared at (1)
assimulo/thirdparty/hairer/dopri5.f: In function ‘contd5’:
assimulo/thirdparty/hairer/dopri5.f:640: warning: ‘__result_contd5’ may be used uninitialized in this function
assimulo/thirdparty/hairer/dopri5.f: In function ‘dopcor’:
assimulo/thirdparty/hairer/dopri5.f:483: warning: ‘nonsti’ may be used uninitialized in this function
gfortran:f77: build/src.macosx-10.6-x86_64-2.6/assimulo/thirdparty/hairer/dopri5-f2pywrappers.f
/Users/guilherme/sage-4.8/local/bin/gfortran -Wall build/temp.macosx-10.6-x86_64-2.6/build/src.macosx-10.6-x86_64-2.6/assimulo/thirdparty/hairer/dopri5module.o build/temp.macosx-10.6-x86_64-2.6/build/src.macosx-10.6-x86_64-2.6/fortranobject.o build/temp.macosx-10.6-x86_64-2.6/assimulo/thirdparty/hairer/dopri5.o build/temp.macosx-10.6-x86_64-2.6/build/src.macosx-10.6-x86_64-2.6/assimulo/thirdparty/hairer/dopri5-f2pywrappers.o -L/Users/guilherme/sage-4.8/local/bin/../lib/gcc/i686-apple-darwin8/4.2.3/x86_64 -lgfortran -o build/lib.macosx-10.6-x86_64-2.6/assimulo/lib/dopri5.so
Undefined symbols:
  "_PyNumber_Int", referenced from:
      _int_from_pyobj in dopri5module.o
  "_PyDict_DelItemString", referenced from:
      _fortran_setattr in fortranobject.o
  "_PyFloat_FromDouble", referenced from:
      _cb_fcn_in_dopcor__user__routines in dopri5module.o
      _cb_solout_in_dopcor__user__routines in dopri5module.o
      _cb_solout_in_dopcor__user__routines in dopri5module.o
      _cb_solout_in_dopcor__user__routines in dopri5module.o
  "_PyErr_NewException", referenced from:
      _initdopri5 in dopri5module.o
  "_PyDict_GetItemString", referenced from:
      _initdopri5 in dopri5module.o
      _fortran_getattr in fortranobject.o
(snip)
=====================

chria
Offline
Joined: 2009-07-29
Hi Guilherme,   I'm unsure at

Hi Guilherme,
 
I'm unsure at the moment on how to solve this issue and will have to dig deeper. I checked the output from an installation on my Ubuntu machine, although running python 2.7 and below you can find the output and as you can see they are very alike regarding the included libraries and headers in the compile/link directives. But It looks like it is not finding correctly libpython. I just did a quick google for the problem and came up with this: http://comments.gmane.org/gmane.os.apple.macports.user/22059 do you think this can be the issue? Either way I'll dig deeper and get back to you as soon as possible.
 
Best
/Christian
 
=========================
building 'assimulo.lib.dopri5' extension
compiling C sources
C compiler: gcc -pthread -fno-strict-aliasing -DNDEBUG -g -fwrapv -O2 -Wall -Wstrict-prototypes -fPIC

creating build/temp.linux-x86_64-2.7/build
creating build/temp.linux-x86_64-2.7/build/src.linux-x86_64-2.7
creating build/temp.linux-x86_64-2.7/build/src.linux-x86_64-2.7/assimulo
creating build/temp.linux-x86_64-2.7/build/src.linux-x86_64-2.7/assimulo/thirdparty
creating build/temp.linux-x86_64-2.7/build/src.linux-x86_64-2.7/assimulo/thirdparty/hairer
compile options: '-Ibuild/src.linux-x86_64-2.7 -I/usr/lib/pymodules/python2.7/numpy/core/include -I/usr/include/python2.7 -c'
gcc: build/src.linux-x86_64-2.7/assimulo/thirdparty/hairer/dopri5module.c
gcc: build/src.linux-x86_64-2.7/fortranobject.c
compiling Fortran sources
Fortran f77 compiler: /usr/bin/gfortran -Wall -ffixed-form -fno-second-underscore -fPIC -O3 -funroll-loops
Fortran f90 compiler: /usr/bin/gfortran -Wall -fno-second-underscore -fPIC -O3 -funroll-loops
Fortran fix compiler: /usr/bin/gfortran -Wall -ffixed-form -fno-second-underscore -Wall -fno-second-underscore -fPIC -O3 -funroll-loops
creating build/temp.linux-x86_64-2.7/assimulo/thirdparty
creating build/temp.linux-x86_64-2.7/assimulo/thirdparty/hairer
compile options: '-Ibuild/src.linux-x86_64-2.7 -I/usr/lib/pymodules/python2.7/numpy/core/include -I/usr/include/python2.7 -c'
gfortran:f77: assimulo/thirdparty/hairer/dopri5.f
assimulo/thirdparty/hairer/dopri5.f:558.35:

      FUNCTION HINIT(N,FCN,X,Y,XEND,POSNEG,F0,F1,Y1,IORD,               
                                   1
Warning: Unused dummy argument 'xend' at (1)
assimulo/thirdparty/hairer/dopri5.f: In function ‘dopcor’:
assimulo/thirdparty/hairer/dopri5.f:483:0: warning: ‘nonsti’ may be used uninitialized in this function
assimulo/thirdparty/hairer/dopri5.f: In function ‘contd5’:
assimulo/thirdparty/hairer/dopri5.f:621:0: warning: ‘__result_contd5’ may be used uninitialized in this function
gfortran:f77: build/src.linux-x86_64-2.7/assimulo/thirdparty/hairer/dopri5-f2pywrappers.f
/usr/bin/gfortran -Wall -Wall -shared build/temp.linux-x86_64-2.7/build/src.linux-x86_64-2.7/assimulo/thirdparty/hairer/dopri5module.o build/temp.linux-x86_64-2.7/build/src.linux-x86_64-2.7/fortranobject.o build/temp.linux-x86_64-2.7/assimulo/thirdparty/hairer/dopri5.o build/temp.linux-x86_64-2.7/build/src.linux-x86_64-2.7/assimulo/thirdparty/hairer/dopri5-f2pywrappers.o -lgfortran -o build/lib.linux-x86_64-2.7/assimulo/lib/dopri5.so
 

guitorri
Offline
Joined: 2011-11-30
Hi Christian. The only

Hi Christian.
The only differences (besides your system wide Python 2.7 and a my local Python 2.6) is that:
- Your C compilation takes the flag -fPIC, mine doesnt.
- Your Fortran linking stage takes the flag -shared, mine doesn't and mine is also linking with the local -lgfortran
Concerning the link you provided, it seems that the LDFLAGS is clean prior to compilation.
How can I put a -shared or --with-pic into the compile/link?
Maybe it is needed to generate a .so (http://cens.ioc.ee/pipermail/f2py-users/2009-June/001832.html)? Will it find the symbols at runtime?
Regards,
Guilherme
 

guitorri
Offline
Joined: 2011-11-30
Hi, Just updating. Now it

Hi,
Just updating. Now it compiles, but doesn't import the extensions...
I added extra_link_args = ['-undefined dynamic_lookup'] to the Fortran config.add_extensions(...).
It should be equivalent to -shared on Linux (http://groups.google.com/group/tango-project/browse_thread/thread/94a7b3c3a98cf178)
It now hangs during Assimulo import, see below.
Can it be 32 / 64 bit related issue? Any ideas?
======
sage: from assimulo.solvers import IDA
---------------------------------------------------------------------------
ImportError                               Traceback (most recent call last)

/Users/guilherme/<ipython console> in <module>()

/Users/guilherme/sage-4.8/local/lib/python2.6/site-packages/assimulo/solvers/__init__.py in <module>()
     20 #Import all the solvers from the different modules
     21 from euler import ExplicitEuler
---> 22 from radau5 import Radau5ODE, Radau5DAE, _Radau5ODE, _Radau5DAE
     23 from sundials import IDA, CVode
     24 from runge_kutta import RungeKutta34, RungeKutta4, Dopri5

/Users/guilherme/sage-4.8/local/lib/python2.6/site-packages/assimulo/solvers/radau5.py in <module>()
     27 from assimulo.lib.radau_core import Radau_Common
     28
---> 29 from assimulo.lib import radau5
     30
     31 class Radau5ODE(Radau_Common,Explicit_ODE):

ImportError: dlopen(/Users/guilherme/sage-4.8/local/lib/python2.6/site-packages/assimulo/lib/radau5.so, 2): no suitable image found.  Did find:
    /Users/guilherme/sage-4.8/local/lib/python2.6/site-packages/assimulo/lib/radau5.so: can't map
sage:
=======
 
I do have the radau5.so:
guilherme$ file radau5.so
radau5.so: Mach-O 64-bit executable x86_64
guilherme$
guilherme$ nm radau5.so | grep -i radau
000000010003cdd8 d _doc_f2py_rout_radau5_contr5
000000010003c5c8 d _doc_f2py_rout_radau5_radau5
0000000100001a20 t _f2py_rout_radau5_contr5
00000001000021d0 t _f2py_rout_radau5_radau5
0000000100003d40 T _initradau5
0000000100036a90 T _radau5_
000000010003d060 b _radau5_error
000000010003d068 b _radau5_module
 
Still looking into it...
Regards,
Guilherme

chria
Offline
Joined: 2009-07-29
Hi Guilherme,   Great! One

Hi Guilherme,
 
Great! One step closer. I'm not sure if it is a 32/ 64 bit issue, is Python in your sage environment 64 bit? On mac I believe you can use the command otool to check the radau5.so file for linked dependencies, could you please try it and post what you find? Also, one of the developer runs on a 64 bit mac without any issues so we at least knows that I can work :)
 
Best
/Christian

guitorri
Offline
Joined: 2011-11-30
Hi Christian,   Python under

Hi Christian,
 
Python under Sage is 64bit (maybe 32 too):
guilherme$ ~/sage-4.8/sage -python
Python 2.6.4 (r264:75706, Jan 30 2012, 14:20:26)
[GCC 4.2.1 (Apple Inc. build 5666) (dot 3)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import platform
>>> print platform.platform()
Darwin-10.8.0-x86_64-i386-64bit
The shared radau.so is 64bit and points to another libs:
guilherme$ pwd
/Users/guilherme/sage-4.8/local/lib/python2.6/site-packages/assimulo/lib

guilherme$ file radau5.so
radau5.so: Mach-O 64-bit executable x86_64

guilherme$ otool -l radau5.so
radau5.so:
Load command 0
      cmd LC_SEGMENT_64
  cmdsize 72
  segname __PAGEZERO
   vmaddr 0x0000000000000000
   vmsize 0x0000000100000000
  fileoff 0
 filesize 0
  maxprot 0x00000000
 initprot 0x00000000
   nsects 0
    flags 0x0
Load command 1
      cmd LC_SEGMENT_64
  cmdsize 632
  segname __TEXT
   vmaddr 0x0000000100000000
   vmsize 0x000000000003b000
  fileoff 0
 filesize 241664
  maxprot 0x00000007
 initprot 0x00000005
   nsects 7
    flags 0x0
Section
  sectname __text
   segname __TEXT
      addr 0x0000000100001100
      size 0x0000000000036f62
    offset 4352
     align 2^4 (16)
    reloff 0
    nreloc 0
     flags 0x80000400
 reserved1 0
 reserved2 0
Section
  sectname __symbol_stub1
   segname __TEXT
      addr 0x0000000100038062
      size 0x00000000000001b6
    offset 229474
     align 2^1 (2)
    reloff 0
    nreloc 0
     flags 0x80000408
 reserved1 0 (index into indirect symbol table)
 reserved2 6 (size of stubs)
Section
  sectname __cstring
   segname __TEXT
      addr 0x0000000100038218
      size 0x0000000000001732
    offset 229912
     align 2^3 (8)
    reloff 0
    nreloc 0
     flags 0x00000002
 reserved1 0
 reserved2 0
Section
  sectname __const
   segname __TEXT
      addr 0x0000000100039950
      size 0x0000000000000460
    offset 235856
     align 2^4 (16)
    reloff 0
    nreloc 0
     flags 0x00000000
 reserved1 0
 reserved2 0
Section
  sectname __stub_helper
   segname __TEXT
      addr 0x0000000100039db0
      size 0x000000000000036c
    offset 236976
     align 2^0 (1)
    reloff 0
    nreloc 0
     flags 0x80000400
 reserved1 0
 reserved2 0
Section
  sectname __unwind_info
   segname __TEXT
      addr 0x000000010003a120
      size 0x00000000000000d4
    offset 237856
     align 2^4 (16)
    reloff 0
    nreloc 0
     flags 0x00000000
 reserved1 0
 reserved2 0
Section
  sectname __eh_frame
   segname __TEXT
      addr 0x000000010003a1f8
      size 0x0000000000000e08
    offset 238072
     align 2^3 (8)
    reloff 0
    nreloc 0
     flags 0x6000000b
 reserved1 0
 reserved2 0
Load command 2
      cmd LC_SEGMENT_64
  cmdsize 632
  segname __DATA
   vmaddr 0x000000010003b000
   vmsize 0x0000000000003000
  fileoff 241664
 filesize 12288
  maxprot 0x00000007
 initprot 0x00000003
   nsects 7
    flags 0x0
Section
  sectname __dyld
   segname __DATA
      addr 0x000000010003b000
      size 0x0000000000000010
    offset 241664
     align 2^3 (8)
    reloff 0
    nreloc 0
     flags 0x00000000
 reserved1 0
 reserved2 0
Section
  sectname __nl_symbol_ptr
   segname __DATA
      addr 0x000000010003b010
      size 0x0000000000000098
    offset 241680
     align 2^2 (4)
    reloff 0
    nreloc 0
     flags 0x00000006
 reserved1 73 (index into indirect symbol table)
 reserved2 0
Section
  sectname __la_symbol_ptr
   segname __DATA
      addr 0x000000010003b0a8
      size 0x0000000000000248
    offset 241832
     align 2^2 (4)
    reloff 0
    nreloc 0
     flags 0x00000007
 reserved1 92 (index into indirect symbol table)
 reserved2 0
Section
  sectname __const
   segname __DATA
      addr 0x000000010003b2f0
      size 0x0000000000000010
    offset 242416
     align 2^3 (8)
    reloff 0
    nreloc 0
     flags 0x00000000
 reserved1 0
 reserved2 0
Section
  sectname __data
   segname __DATA
      addr 0x000000010003b300
      size 0x0000000000001d28
    offset 242432
     align 2^5 (32)
    reloff 0
    nreloc 0
     flags 0x00000000
 reserved1 0
 reserved2 0
Section
  sectname __bss
   segname __DATA
      addr 0x000000010003d040
      size 0x0000000000000068
    offset 0
     align 2^5 (32)
    reloff 0
    nreloc 0
     flags 0x00000001
 reserved1 0
 reserved2 0
Section
  sectname __common
   segname __DATA
      addr 0x000000010003d0c0
      size 0x00000000000002f0
    offset 0
     align 2^6 (64)
    reloff 0
    nreloc 0
     flags 0x00000001
 reserved1 0
 reserved2 0
Load command 3
      cmd LC_SEGMENT_64
  cmdsize 72
  segname __LINKEDIT
   vmaddr 0x000000010003e000
   vmsize 0x0000000000004000
  fileoff 253952
 filesize 12688
  maxprot 0x00000007
 initprot 0x00000001
   nsects 0
    flags 0x0
Load command 4
     cmd LC_SYMTAB
 cmdsize 24
  symoff 253952
   nsyms 375
  stroff 260616
 strsize 6024
Load command 5
            cmd LC_DYSYMTAB
        cmdsize 80
      ilocalsym 0
      nlocalsym 221
     iextdefsym 221
     nextdefsym 63
      iundefsym 284
      nundefsym 91
         tocoff 0
           ntoc 0
      modtaboff 0
        nmodtab 0
   extrefsymoff 0
    nextrefsyms 0
 indirectsymoff 259952
  nindirectsyms 165
      extreloff 0
        nextrel 0
      locreloff 0
        nlocrel 0
Load command 6
          cmd LC_LOAD_DYLINKER
      cmdsize 32
         name /usr/lib/dyld (offset 12)
Load command 7
     cmd LC_UUID
 cmdsize 24
    uuid 96D0D727-A877-0854-DB78-56C5D27FF046
Load command 8
        cmd LC_UNIXTHREAD
    cmdsize 184
     flavor x86_THREAD_STATE64
      count x86_THREAD_STATE64_COUNT
   rax  0x0000000000000000 rbx 0x0000000000000000 rcx  0x0000000000000000
   rdx  0x0000000000000000 rdi 0x0000000000000000 rsi  0x0000000000000000
   rbp  0x0000000000000000 rsp 0x0000000000000000 r8   0x0000000000000000
    r9  0x0000000000000000 r10 0x0000000000000000 r11  0x0000000000000000
   r12  0x0000000000000000 r13 0x0000000000000000 r14  0x0000000000000000
   r15  0x0000000000000000 rip 0x0000000100001100
rflags  0x0000000000000000 cs  0x0000000000000000 fs   0x0000000000000000
    gs  0x0000000000000000
Load command 9
          cmd LC_LOAD_DYLIB
      cmdsize 64
         name /usr/local/lib/libgfortran.2.dylib (offset 24)
   time stamp 2 Thu Jan  1 01:00:02 1970
      current version 3.0.0
compatibility version 3.0.0
Load command 10
          cmd LC_LOAD_DYLIB
      cmdsize 56
         name /usr/local/lib/libgcc_s.1.dylib (offset 24)
   time stamp 2 Thu Jan  1 01:00:02 1970
      current version 1.0.0
compatibility version 1.0.0
Load command 11
          cmd LC_LOAD_DYLIB
      cmdsize 56
         name /usr/lib/libSystem.B.dylib (offset 24)
   time stamp 2 Thu Jan  1 01:00:02 1970
      current version 125.2.11
compatibility version 1.0.0
guilherme$
 
--
Well, the only libgfortran I can find on my system is located under Sage.
guilherme$ find ~/sage-4.8 -name "libgfor*.dylib"
/Users/guilherme/sage-4.8/local/lib/libgfortran.2.0.0.dylib
/Users/guilherme/sage-4.8/local/lib/libgfortran.2.dylib
/Users/guilherme/sage-4.8/local/lib/libgfortran.dylib
/Users/guilherme/sage-4.8/local/lib/ppc64/libgfortran.2.0.0.dylib
/Users/guilherme/sage-4.8/local/lib/ppc64/libgfortran.2.dylib
/Users/guilherme/sage-4.8/local/lib/ppc64/libgfortran.dylib
/Users/guilherme/sage-4.8/local/lib/x86_64/libgfortran.2.0.0.dylib
/Users/guilherme/sage-4.8/local/lib/x86_64/libgfortran.2.dylib
/Users/guilherme/sage-4.8/local/lib/x86_64/libgfortran.dylib
 
The libgcc is probably ok to be the one on the system (Sage used it to compile itself), however it is also present under Sage:
guilherme$ find /usr/ -name "libgcc*.dylib"
/usr//lib/libgcc_s.1.dylib
/usr//lib/libgcc_s.10.4.dylib
/usr//lib/libgcc_s.10.5.dylib
guilherme$ find ~/sage-4.8 -name "libgcc*.dylib"
/Users/guilherme/sage-4.8/local/lib/libgcc_s.1.dylib
/Users/guilherme/sage-4.8/local/lib/libgcc_s.10.4.dylib
/Users/guilherme/sage-4.8/local/lib/libgcc_s.10.5.dylib
/Users/guilherme/sage-4.8/local/lib/libgcc_s_ppc64.1.dylib
/Users/guilherme/sage-4.8/local/lib/libgcc_s_x86_64.1.dylib
 
As you see, I don't have any relevant library under my /usr/local.
Shouldn't radau.so be looking for the libs into the Sage's ~sage-4.8/local/lib?
How can I try this out? Any ideas?
 
Best,
Guilherme
 
 

chria
Offline
Joined: 2009-07-29
Hi Guilherme,   I seems like

Hi Guilherme,
 
I seems like this could be fixed by using the equivalence of the LD_LIBRARY_PATH in mac, "DYLD_LIBRARY_PATH". I guess that if you set that path to ~sage-4.8/local/lib it will look for shared libraries there first. Also it is a comma separate list of multiple paths so try to set the path to sage first.
 
Best
/Christian

guitorri
Offline
Joined: 2011-11-30
Solved

Hi Christian,
Just reportin back to close the issue.
It seems that the problem lies on the Sage's distutils flags for OS X (some open track issues over there).
Using the following link flags Assimulo installs and runs properly...
> env LDFLAGS="-fPIC -undefined dynamic_lookup -bundle -Wall" python setup.py install --sundials-home="$SAGE_LOCAL"
 
Thank you for the support.
Guilherme
 

chria
Offline
Joined: 2009-07-29
Perfect! Glad to hear that it

Perfect! Glad to hear that it is working properly!
 
Best
/Christian

Login or register to post comments