Pastix4Calculix - Build and installation issues

1

Hello,

I was wondering if someone here successfully installed Pastix4Calculix on a CentOS system?
I have explained my issue in detail here (With CUDA-'Threads::Threads' NOT FOUND, Without CUDA - fatal error: pastix_ccores.h · Issue #11 · Dhondtguido/PaStiX4CalculiX · GitHub) .
I am posting the question here again to reach a broader user base.
I have looked in the previous posts, and I did find people facing some similar issues, but their solutions were not quite clear or did not work for me.
PaStix4CalculiX compilation error
Any suggestions?

In addition, during Parsec installation, even though MPI is turned off, some MPI functions were used during the build leading to errors. I could resolve them by just loading the mpich library and include directory paths. But I still find this strange, on why use MPI when I specifically turn it off. Any one faced this too?

Any suggestions would be of great help.

Thanks and Regards,
Hem

2

My 2 cents is CentOS 7 is too old to install Pastix. The GCC is 4.8.5 which is very old . I would first try building Pastix (non-GPU) per these instructions:

https://solverstack.gitlabpages.inria.fr/pastix/md_docs_doxygen_chapters_Pastix_Install.html

My initital attempt at doing so on Centos 7 failed. It created an executable but the simple example problem says the results are wrong…

3

@feacluster Unfortunately I do not have control on the OS. Do you mean Pastix instead of Pastix4Calculix?
I use GCC 8.0.2 for my build. And I already tried Pastix4Calculix without CUDA. It fails to create a ton of header files like pastix_ccores.h and the rest in /build/kernel folder and stops compiling.

4

Correct, yes just Pastix. Usually when I try and install something complex I start with the simplest components to see if they will install. GCC 8 sounds new enough though… Try just Pastix and see if you can get that to work.

BTW, errors about missing include files is usually some path problem…

5

Okay I will try the standard Pastix. To my understanding the missing headers are to be created by the build. At least these headers were not included in the source code.

1 Like
6

Okay, so I was at least able to install Pastix4Calculix without CUDA. I had to turn on the MPI though. Not turning on the MPI lead to conflicting MPI function definitions in nompi.h of Pastix4Calculix and mpi.h used by parsec. Some how parsec could not be compiled without MPI libraries, even though the MPI was set to OFF.

1 Like
7

Interesting ! Can you share your cmake command?

8

The MKL Library and Open MPI 3.2.1 bin and library paths are exported in to PATH and LD_LIBRARY_PATH environment variables Apriori.
Hwloc 2.1 :

#!/bin/bash
### Source Code https://www.open-mpi.org/software/hwloc/v2.1/
umask 022
./configure --prefix=/usr/PaStiX/hwloc_i8 --enable-static=yes CC=gcc CXX=g++
make -j2
make install

Scotch 6.0.8

#!/bin/bash
### Source Code https://gforge.inria.fr/frs/download.php/file/38114/scotch_6.0.8.tar.gz
umask 022
cd src
ln -sf Make.inc/Makefile.inc.x86-64_pc_linux2 Makefile.inc
sed -i '/CFLAGS/ s/$/ -DINTSIZE64/' Makefile.inc

make -j8 scotch
make -j8 esmumps
make prefix=/usr/PaStiX/scotch_i8 install

parsec:

#!/bin/bash
### Source Code - git clone https://bitbucket.org/mfaverge/parsec.git
rm -rf build
if ! [[ -d build ]]; then
mkdir build
fi
cd build
INSTALLPATH="/usr/PaStiX/parsec_i8"
umask 022

# fixes
sed -i '/-1 == cpu/i return cpu;' ../parsec/bindthread.c

cmake \
    -DCMAKE_CXX_COMPILER=g++ \
    -DCMAKE_C_COMPILER=gcc \
    -DCMAKE_Fortran_COMPILER=gfortran \
    -DPARSEC_DIST_WITH_MPI=ON \
    -DCUDA_TOOLKIT_ROOT_DIR=/usr/CUDA/v10.2/cuda-10.2 \
    -DCUDA_DIR=/usr/CUDA/v10.2/cuda-10.2 \
    -DCMAKE_CUDA_HOST_COMPILER=gcc \
    -DPARSEC_GPU_WITH_CUDA=ON \
    -DCUDA_USE_STATIC_CUDA_RUNTIME=ON \
    -DCMAKE_BUILD_TYPE=Release \
    -DCMAKE_INSTALL_PREFIX=${INSTALLPATH} \
    -DPARSEC_WITH_DEVEL_HEADERS=ON \
    -DCMAKE_LIBRARY_PATH=/usr/CUDA/v10.2/cuda-10.2/lib64/stubs \
    -DHWLOC_DIR=/usr/PaStiX/hwloc_i8 \
	-DCMAKE_C_FLAGS="-fopenmp" \
    ..

make -j8
make install

Pastix

#!/bin/bash
### Source Code - git clone https://github.com/Dhondtguido/PaStiX4CalculiX
###
rm -rf build
if ! [[ -d build ]]; then
    mkdir build
fi
cd build
export LD_RUN_PATH=$LD_RUN_PATH:/usr/PaStiX/hwloc_i8/lib:/usr/PaStiX/parsec_i8/lib:/usr/PaStiX/scotch_i8/lib
export INCLUDE_PATH=$INCLUDE_PATH:/usr/PaStiX/hwloc_i8/include:/usr/PaStiX/parsec_i8/include:/usr/PaStiX/scotch_i8/include
export PKG_CONFIG_PATH=$PKG_CONFIG_PATH:/usr/PaStiX/hwloc_i8/lib/pkgconfig:/usr/PaStiX/parsec_i8/lib/pkgconfig
export PATH=$PATH:/usr/PaStiX/scotch_i8/bin:/usr/PaStiX/parsec_i8/bin:/usr/PaStiX/hwloc_i8/bin

cmake \
	-DHWLOC_DIR=/usr/PaStiX/hwloc_i8 \
	-DCUDA_TOOLKIT_ROOT_DIR=/usr/CUDA/v10.2/cuda-10.2 \
	-DCMAKE_INSTALL_PREFIX=/usr/PaStiX/pastix_i8 \
	-DCMAKE_BUILD_TYPE=Release \
	-DPASTIX_WITH_PARSEC=ON \
	-DPARSEC_DIR=/usr/PaStiX/parsec_i8 \
	-DSCOTCH_DIR=/usr/PaStiX/scotch_i8 \
	-DPASTIX_WITH_CUDA=OFF \
	-DCUDA_DIR=/usr/CUDA/v10.2/cuda-10.2 \
	-DPASTIX_ORDERING_SCOTCH=ON \
	-DCMAKE_C_COMPILER=gcc \
	-DCMAKE_CXX_COMPILER=g++ \
	-DCMAKE_Fortran_COMPILER=gfortran \
	-DCMAKE_LIBRARY_PATH=/usr/CUDA/v10.2/cuda-10.2/lib64/stubs \
    -DPASTIX_WITH_MPI=ON \
	-DCMAKE_C_FLAGS="-fopenmp -I/usr/CUDA/v10.2/cuda-10.2/include -I/opt/ohpc/pub/mpi/mpich-gnu8-ohpc/3.2.1/include" \
    ..
make -j8
make install

Makefile for CCX 2.18 with Pastix :

CFLAGS = -Wall -O2 -fopenmp -fpic -I /usr/SPOOLES_2_2 -I$(PASTIX_INCLUDE) -I /opt/ohpc/pub/mpi/mpich-gnu8-ohpc/3.2.1/include -DARCH="Linux" -DSPOOLES -DARPACK -DMATRIXSTORAGE -DLONGLONG -DPASTIX -DPASTIX_FP32 -DUSE_MT=1 -std=c99 -D_POSIX_C_SOURCE=199309L
FFLAGS = -Wall -O2 -fopenmp -fpic -fdefault-integer-8

CC=cc
FC=gfortran

.c.o :
	$(CC) $(CFLAGS) -c $<
.f.o :
	$(FC) $(FFLAGS) -c $<

include Makefile.inc

SCCXMAIN = ccx_2.18.c

OCCXF = $(SCCXF:.f=.o)
OCCXC = $(SCCXC:.c=.o)
OCCXMAIN = $(SCCXMAIN:.c=.o)

DIR=/usr/SPOOLES_2_2
MKL_LIB=/LPP/OPT/x86_64/intel/parallel_studio_xe_2019/mkl/lib/intel64

PASTIX_INCLUDE=/usr/PaStiX/pastix_i8/include
PASTIX_LIBS=/usr/PaStiX/hwloc_i8/lib/libhwloc.so $(MKL_LIB)/libmkl_intel_lp64.so $(MKL_LIB)/libmkl_sequential.so $(MKL_LIB)/libmkl_core.so /lib64/libm.so \
  /usr/PaStiX/pastix_i8/lib/libpastix.a /opt/ohpc/pub/mpi/mpich-gnu8-ohpc/3.2.1/lib/libmpi.a -lpthread -lm /usr/PaStiX/pastix_i8/lib/libspm.a \
  /usr/PaStiX/pastix_i8/lib/libpastix_parsec.a /usr/PaStiX/pastix_i8/lib/libpastix_kernels.a -lrt \
  /usr/PaStiX/parsec_i8/lib/libparsec.so \
  /usr/PaStiX/scotch_i8/lib/libscotch.a /usr/PaStiX/scotch_i8/lib/libscotcherrexit.a -lpthread -lz -lm /usr/lib64/libstdc++.so.6 /usr/lib64/libgomp.so.1

LIBS = \
     $(DIR)/MT/src/spoolesMT.a \
     $(DIR)/spooles.a \
     /usr/ARPACK/libarpack_x86.a \
     $(PASTIX_LIBS) \
     -lpthread -lm -lc
export LDFLAGS="-Wl,--copy-dt-needed-entries"
ccx_2.18_i8: $(OCCXMAIN) ccx_2.18.a $(LIBS)
	./date.pl; $(CC) $(CFLAGS) -c ccx_2.18.c; $(FC) -Wall -O2 -o $@ \
	$(OCCXMAIN) ccx_2.18.a $(LIBS)

ccx_2.18.a: $(OCCXF) $(OCCXC)
	ar vr $@ $?
3 Likes
9

I have tried a few times to install Pastix4Calculix with no luck. Tried on CentOS, Ubuntu, Intel compiler, GNU compiler, without MPI, with MPI…

I seem to have got the furthest with Ubuntu 20, GNU compiler and no MPI. But I get this error when running make_pastix.sh:

-- Configuring done
-- Generating done
-- Build files have been written to: /home/feacluster/PaStiX/pastix_src/build
sopalin/parsec/CMakeFiles/parsec_headers_tgt.dir/build.make:64: *** missing separator.  Stop.
make[1]: *** [CMakeFiles/Makefile2:1717: sopalin/parsec/CMakeFiles/parsec_headers_tgt.dir/all] Error 2
make[1]: *** Waiting for unfinished jobs....
CMakeFiles/bcsc_headers_tgt.dir/build.make:63: *** missing separator.  Stop.
make[1]: *** [CMakeFiles/Makefile2:566: CMakeFiles/bcsc_headers_tgt.dir/all] Error 2
make: *** [Makefile:141: all] Error 2
sopalin/parsec/CMakeFiles/parsec_headers_tgt.dir/build.make:64: *** missing separator.  Stop.
make[1]: *** [CMakeFiles/Makefile2:1717: sopalin/parsec/CMakeFiles/parsec_headers_tgt.dir/all] Error 2
make: *** [Makefile:141: all] Error 2


#!/bin/bash
if ! [[ -d build ]]; then
    mkdir build
fi
cd build

cmake   \
        -DHWLOC_DIR=/home/feacluster/PaStiX/hwloc_i8 \
        -DCMAKE_INSTALL_PREFIX=/home/feacluster/PaStiX/pastix_i8 \
        -DCMAKE_BUILD_TYPE=Release \
        -DPASTIX_WITH_PARSEC=ON \
        -DPARSEC_DIR=/home/feacluster/PaStiX/parsec_i8 \
        -DSCOTCH_DIR=/home/feacluster/PaStiX/scotch_i8 \
        -DPASTIX_WITH_CUDA=ON \
        -DPASTIX_ORDERING_SCOTCH=ON \
        -DCMAKE_C_COMPILER=gcc \
        -DCMAKE_CXX_COMPILER=g++ \
        -DCMAKE_Fortran_COMPILER=gfortran \
        -DCMAKE_C_FLAGS="-fopenmp" \
        ..

make -j2
make install

Here is the contents of my make_pastix.sh file if you spot anything obviously wrong:

#!/bin/bash
if ! [[ -d build ]]; then
    mkdir build
fi
cd build

cmake   \
        -DHWLOC_DIR=/home/feacluster/PaStiX/hwloc_i8 \
        -DCMAKE_INSTALL_PREFIX=/home/feacluster/PaStiX/pastix_i8 \
        -DCMAKE_BUILD_TYPE=Release \
        -DPASTIX_WITH_PARSEC=ON \
        -DPARSEC_DIR=/home/feacluster/PaStiX/parsec_i8 \
        -DSCOTCH_DIR=/home/feacluster/PaStiX/scotch_i8 \
        -DPASTIX_WITH_CUDA=ON \
        -DPASTIX_ORDERING_SCOTCH=ON \
        -DCMAKE_C_COMPILER=gcc \
        -DCMAKE_CXX_COMPILER=g++ \
        -DCMAKE_Fortran_COMPILER=gfortran \
        -DCMAKE_C_FLAGS="-fopenmp" \
        ..

make -j2
make install

And here is the full output from make_pastix.sh

feacluster@gpu:~/PaStiX/pastix_src$ ./make_pastix.sh
./make_pastix.sh: line 7: syntax error near unexpected token `<<<'
./make_pastix.sh: line 7: `<<<<<<< HEAD'
feacluster@gpu:~/PaStiX/pastix_src$ nano make_pastix.sh
feacluster@gpu:~/PaStiX/pastix_src$ ./make_pastix.sh
-- The C compiler identification is GNU 9.3.0
-- The CXX compiler identification is GNU 9.3.0
-- The Fortran compiler identification is GNU 9.3.0
-- Check for working C compiler: /usr/bin/gcc
-- Check for working C compiler: /usr/bin/gcc -- works
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- Detecting C compile features
-- Detecting C compile features - done
-- Check for working CXX compiler: /usr/bin/g++
-- Check for working CXX compiler: /usr/bin/g++ -- works
-- Detecting CXX compiler ABI info
-- Detecting CXX compiler ABI info - done
-- Detecting CXX compile features
-- Detecting CXX compile features - done
-- Check for working Fortran compiler: /usr/bin/gfortran
-- Check for working Fortran compiler: /usr/bin/gfortran  -- works
-- Detecting Fortran compiler ABI info
-- Detecting Fortran compiler ABI info - done
-- Checking whether /usr/bin/gfortran supports Fortran 90
-- Checking whether /usr/bin/gfortran supports Fortran 90 -- yes
-- Found PythonInterp: /usr/bin/python3.8 (found version "3.8.10")
-- Building for target x86_64
-- Found target X86_64
-- Performing Test C_M32or64
-- Performing Test C_M32or64 - Failed
-- Performing Test HAVE_WALL
-- Performing Test HAVE_WALL - Success
-- Performing Test HAVE_WEXTRA
-- Performing Test HAVE_WEXTRA - Success
-- Performing Test HAVE_G3
-- Performing Test HAVE_G3 - Success
-- Performing Test HAVE_ATOMIC_GCC_32_BUILTINS
-- Performing Test HAVE_ATOMIC_GCC_32_BUILTINS - Success
-- Performing Test HAVE_ATOMIC_GCC_64_BUILTINS
-- Performing Test HAVE_ATOMIC_GCC_64_BUILTINS - Success
-- Performing Test HAVE_ATOMIC_GCC_128_BUILTINS
-- Performing Test HAVE_ATOMIC_GCC_128_BUILTINS - Failed
-- Performing Test HAVE_ATOMIC_GCC_128_BUILTINS
-- Performing Test HAVE_ATOMIC_GCC_128_BUILTINS - Success
-- Performing Test HAVE_ATOMIC_XLC_32_BUILTINS
-- Performing Test HAVE_ATOMIC_XLC_32_BUILTINS - Failed
-- Performing Test HAVE_ATOMIC_MIPOSPRO_32_BUILTINS
-- Performing Test HAVE_ATOMIC_MIPOSPRO_32_BUILTINS - Failed
-- Performing Test HAVE_ATOMIC_SUN_32
-- Performing Test HAVE_ATOMIC_SUN_32 - Failed
--       support for 32 bits atomics - found
--       support for 64 bits atomics - found
--       support for 128 bits atomics - found
-- Performing Test HAVE_FALLTHROUGH
-- Performing Test HAVE_FALLTHROUGH - Success
-- Looking for pthread.h
-- Looking for pthread.h - found
-- Performing Test CMAKE_HAVE_LIBC_PTHREAD
-- Performing Test CMAKE_HAVE_LIBC_PTHREAD - Success
-- Found Threads: TRUE
-- Looking for pthread_create
-- Looking for pthread_create - found
-- Looking for sched_setaffinity
-- Looking for sched_setaffinity - found
-- Performing Test HAVE_TIMESPEC_TV_NSEC
-- Performing Test HAVE_TIMESPEC_TV_NSEC - Success
-- Looking for clock_gettime in rt
-- Looking for clock_gettime in rt - found
-- Looking for include file stdarg.h
-- Looking for include file stdarg.h - found
-- Performing Test HAVE_VA_COPY
-- Performing Test HAVE_VA_COPY - Success
-- Looking for asprintf
-- Looking for asprintf - found
-- Looking for vasprintf
-- Looking for vasprintf - found
-- Looking for include file getopt.h
-- Looking for include file getopt.h - found
-- Looking for include file unistd.h
-- Looking for include file unistd.h - found
-- Looking for getopt_long
-- Looking for getopt_long - found
-- Looking for include file errno.h
-- Looking for include file errno.h - found
-- Looking for include file stddef.h
-- Looking for include file stddef.h - found
-- Looking for include file stdbool.h
-- Looking for include file stdbool.h - found
-- Looking for getrusage
-- Looking for getrusage - found
-- Looking for RUSAGE_THREAD
-- Looking for RUSAGE_THREAD - not found
-- Looking for include file limits.h
-- Looking for include file limits.h - found
-- Looking for include file string.h
-- Looking for include file string.h - found
-- Looking for include file libgen.h
-- Looking for include file libgen.h - found
-- Looking for include file complex.h
-- Looking for include file complex.h - found
-- Looking for include file sys/param.h
-- Looking for include file sys/param.h - found
-- Looking for include file sys/types.h
-- Looking for include file sys/types.h - found
-- Looking for include file syslog.h
-- Looking for include file syslog.h - found
-- A cache variable, namely CBLAS_DIR, has been set to specify the install directory of CBLAS
-- A cache variable, namely BLAS_DIR, has been set to specify the install directory of BLAS
-- Found PkgConfig: /usr/bin/pkg-config (found version "0.29.1")
-- Checking for one of the modules 'mkl-dynamic-lp64-iomp'
-- Checking for one of the modules 'mkl-dynamic-lp64-seq'
-- Checking for one of the modules 'openblas'
-- Looking for BLAS - found using PkgConfig
-- Looking for cblas_dscal
-- Looking for cblas_dscal - found
-- Looking for cblas_zgemm3m
-- Looking for cblas_zgemm3m - found
-- Looking for cblas: test with blas succeeds
-- Looking for cblas_dscal
-- Looking for cblas_dscal - found
-- Found CBLAS: /usr/lib/x86_64-linux-gnu/openblas-pthread/libopenblas.so
-- cblas:
-- A cache variable, namely LAPACKE_DIR, has been set to specify the install directory of LAPACKE
-- A cache variable, namely LAPACK_DIR, has been set to specify the install directory of LAPACK
-- Looking for BLAS - found using PkgConfig
-- Looking for Fortran CHEEV
-- Looking for Fortran CHEEV - found
-- Looking for LAPACK in BLAS: found
-- Checking for one of the modules 'mkl-dynamic-lp64-iomp'
-- Checking for one of the modules 'mkl-dynamic-lp64-seq'
-- Checking for one of the modules 'openblas'
-- Looking for LAPACK - found using PkgConfig
-- A library with LAPACK API found.
-- LAPACK_LIBRARIES /usr/lib/x86_64-linux-gnu/openblas-pthread/libopenblas.so
-- Looking for LAPACKE_dgeqrf
-- Looking for LAPACKE_dgeqrf - not found
-- Looking for LAPACKE_dlascl_work
-- Looking for LAPACKE_dlascl_work - not found
-- Looking for lapacke : test with lapack fails
-- Checking for one of the modules 'lapacke'
-- Looking for LAPACKE - found using PkgConfig
-- Looking for LAPACKE_dgeqrf
-- Looking for LAPACKE_dgeqrf - found
-- Looking for LAPACKE_dlascl_work
-- Looking for LAPACKE_dlascl_work - found
-- Found LAPACKE: /usr/lib/x86_64-linux-gnu/liblapacke.so
-- lapacke: /usr/include/x86_64-linux-gnu
-- A cache variable, namely HWLOC_DIR, has been set to specify the install directory of HWLOC
-- Looking for HWLOC - PkgConfig not used
-- Looking for hwloc_topology_init
-- Looking for hwloc_topology_init - found
-- Found HWLOC: /home/feacluster/PaStiX/hwloc_i8/lib/libhwloc.so
-- Performing Test HAVE_HWLOC_PARENT_MEMBER
-- Performing Test HAVE_HWLOC_PARENT_MEMBER - Success
-- Performing Test HAVE_HWLOC_CACHE_ATTR
-- Performing Test HAVE_HWLOC_CACHE_ATTR - Success
-- Performing Test HAVE_HWLOC_OBJ_PU
-- Performing Test HAVE_HWLOC_OBJ_PU - Success
-- Looking for hwloc_bitmap_free in /home/feacluster/PaStiX/hwloc_i8/lib/libhwloc.so
-- Looking for hwloc_bitmap_free in /home/feacluster/PaStiX/hwloc_i8/lib/libhwloc.so - found
-- Found CUDA: /usr/local/cuda-11 (found version "11.5")
-- Looking for cuDeviceCanAccessPeer
-- Looking for cuDeviceCanAccessPeer - not found
-- A cache variable, namely PARSEC_DIR, has been set to specify the install directory of PARSEC
-- Looking for PARSEC - Try to detect pthread
-- Looking for PARSEC - Try to detect libm
-- Looking for PARSEC - Try to detect librt
-- Looking for PARSEC - Try to detect libdl
-- Looking for HWLOC - PkgConfig not used
-- Looking for hwloc_topology_init
-- Looking for hwloc_topology_init - found
-- Looking for parsec -- parsec_config.h not found
-- Looking for parsec -- lib parsec-base not found
-- Looking for parsec -- lib parsec_distribution not found
-- Looking for parsec -- lib parsec_distribution_matrix not found
-- Look for PARSEC - compiler parsec_ptgpp found in /home/feacluster/PaStiX/parsec_i8/bin
-- Looking for parsec_init
-- Looking for parsec_init - found
-- Found PARSEC: /home/feacluster/PaStiX/parsec_i8/lib/libparsec.so
-- A cache variable, namely SCOTCH_DIR, has been set to specify the install directory of SCOTCH
-- Looking for SCOTCH_graphInit
-- Looking for SCOTCH_graphInit - found
-- Performing Test SCOTCH_Num_4
-- Performing Test SCOTCH_Num_4 - Failed
-- Performing Test SCOTCH_Num_8
-- Performing Test SCOTCH_Num_8 - Success
-- Found SCOTCH: /home/feacluster/PaStiX/scotch_i8/lib/libscotch.a;/home/feacluster/PaStiX/scotch_i8/lib/libscotcherrexit.a;/usr/lib/x86_64-linux-gnu/libz.so;/usr/lib/x86_64-linux-gnu/libm.so;/usr/lib/x86_64-linux-gnu/librt.so
-- Scotch inlude dirs: /home/feacluster/PaStiX/scotch_i8/include
-- Checking for one of the modules 'gtg'
-- Building for target x86_64
-- Found target X86_64
-- Looking for BLAS - found using PkgConfig
-- Looking for cblas_dscal
-- Looking for cblas_dscal - found
-- Looking for cblas: test with blas succeeds
-- Looking for cblas_dscal
-- Looking for cblas_dscal - found
-- Looking for BLAS - found using PkgConfig
-- Looking for Fortran CHEEV
-- Looking for Fortran CHEEV - found
-- Looking for LAPACK in BLAS: found
-- Checking for one of the modules 'mkl-dynamic-lp64-iomp'
-- Checking for one of the modules 'mkl-dynamic-lp64-seq'
-- Checking for one of the modules 'openblas'
-- Looking for LAPACK - found using PkgConfig
-- A library with LAPACK API found.
-- LAPACK_LIBRARIES /usr/lib/x86_64-linux-gnu/openblas-pthread/libopenblas.so
-- Looking for LAPACKE_dgeqrf
-- Looking for LAPACKE_dgeqrf - not found
-- Looking for LAPACKE_dlascl_work
-- Looking for LAPACKE_dlascl_work - not found
-- Looking for lapacke : test with lapack fails
-- Looking for LAPACKE - found using PkgConfig
-- Looking for LAPACKE_dgeqrf
-- Looking for LAPACKE_dgeqrf - found
-- Looking for LAPACKE_dlascl_work
-- Looking for LAPACKE_dlascl_work - found
-- Generate precision dependencies in /home/feacluster/PaStiX/pastix_src/spm
/home/feacluster/PaStiX/pastix_src/cmake_modules/morse_cmake/modules/precision_generator/genDependencies.py:22: DeprecationWarning: the imp module is deprecated in favour of importlib; see the module's documentation for alternative uses
  import imp;
-- Generate precision dependencies in /home/feacluster/PaStiX/pastix_src/spm - Done
-- Generate precision dependencies in /home/feacluster/PaStiX/pastix_src/spm
CMake Warning (dev) at cmake_modules/morse_cmake/modules/RulesPrecisions.cmake:185 (if):
  Policy CMP0054 is not set: Only interpret if() arguments as variables or
  keywords when unquoted.  Run "cmake --help-policy CMP0054" for policy
  details.  Use the cmake_policy command to set the policy and suppress this
  warning.

  Quoted variables like "src" will no longer be dereferenced when the policy
  is set to NEW.  Since the policy is not set the OLD behavior will be used.
Call Stack (most recent call first):
  spm/CMakeLists.txt:191 (precisions_rules_py)
This warning is for project developers.  Use -Wno-dev to suppress it.

/home/feacluster/PaStiX/pastix_src/cmake_modules/morse_cmake/modules/precision_generator/genDependencies.py:22: DeprecationWarning: the imp module is deprecated in favour of importlib; see the module's documentation for alternative uses
  import imp;
-- Generate precision dependencies in /home/feacluster/PaStiX/pastix_src/spm - Done
-- Generate precision dependencies in /home/feacluster/PaStiX/pastix_src/spm/tests
/home/feacluster/PaStiX/pastix_src/cmake_modules/morse_cmake/modules/precision_generator/genDependencies.py:22: DeprecationWarning: the imp module is deprecated in favour of importlib; see the module's documentation for alternative uses
  import imp;
-- Generate precision dependencies in /home/feacluster/PaStiX/pastix_src/spm/tests - Done
-- --- Python wrapper is disabled with static libraries
-- Generate precision dependencies in /home/feacluster/PaStiX/pastix_src/kernels
/home/feacluster/PaStiX/pastix_src/cmake_modules/morse_cmake/modules/precision_generator/genDependencies.py:22: DeprecationWarning: the imp module is deprecated in favour of importlib; see the module's documentation for alternative uses
  import imp;
-- Generate precision dependencies in /home/feacluster/PaStiX/pastix_src/kernels - Done
-- Generate precision dependencies in /home/feacluster/PaStiX/pastix_src/kernels
/home/feacluster/PaStiX/pastix_src/cmake_modules/morse_cmake/modules/precision_generator/genDependencies.py:22: DeprecationWarning: the imp module is deprecated in favour of importlib; see the module's documentation for alternative uses
  import imp;
-- Generate precision dependencies in /home/feacluster/PaStiX/pastix_src/kernels - Done
-- Generate precision dependencies in /home/feacluster/PaStiX/pastix_src/kernels
/home/feacluster/PaStiX/pastix_src/cmake_modules/morse_cmake/modules/precision_generator/genDependencies.py:22: DeprecationWarning: the imp module is deprecated in favour of importlib; see the module's documentation for alternative uses
  import imp;
-- Generate precision dependencies in /home/feacluster/PaStiX/pastix_src/kernels - Done
-- Generate precision dependencies in /home/feacluster/PaStiX/pastix_src/kernels/gpus
/home/feacluster/PaStiX/pastix_src/cmake_modules/morse_cmake/modules/precision_generator/genDependencies.py:22: DeprecationWarning: the imp module is deprecated in favour of importlib; see the module's documentation for alternative uses
  import imp;
-- Generate precision dependencies in /home/feacluster/PaStiX/pastix_src/kernels/gpus - Done
/home/feacluster/PaStiX/pastix_src/build/kernels/gpus/pastix_cgemm_vbatched.cu/home/feacluster/PaStiX/pastix_src/build/kernels/gpus/pastix_sgemm_vbatched.cu/home/feacluster/PaStiX/pastix_src/build/kernels/gpus/pastix_dgemm_vbatched.cu/home/feacluster/PaStiX/pastix_src/build/kernels/gpus/pastix_zgemm_vbatched.cu/home/feacluster/PaStiX/pastix_src/kernels/gpus/LightSpMV-1.0/src/cLightSpMV.cu/home/feacluster/PaStiX/pastix_src/kernels/gpus/LightSpMV-1.0/src/Options.cu/home/feacluster/PaStiX/pastix_src/kernels/gpus/LightSpMV-1.0/src/SpMV.cu/home/feacluster/PaStiX/pastix_src/kernels/gpus/LightSpMV-1.0/src/SpMVCSR.cu
-- Generate precision dependencies in /home/feacluster/PaStiX/pastix_src/refinement
/home/feacluster/PaStiX/pastix_src/cmake_modules/morse_cmake/modules/precision_generator/genDependencies.py:22: DeprecationWarning: the imp module is deprecated in favour of importlib; see the module's documentation for alternative uses
  import imp;
-- Generate precision dependencies in /home/feacluster/PaStiX/pastix_src/refinement - Done
-- Generate precision dependencies in /home/feacluster/PaStiX/pastix_src/sopalin/parsec
/home/feacluster/PaStiX/pastix_src/cmake_modules/morse_cmake/modules/precision_generator/genDependencies.py:22: DeprecationWarning: the imp module is deprecated in favour of importlib; see the module's documentation for alternative uses
  import imp;
-- Generate precision dependencies in /home/feacluster/PaStiX/pastix_src/sopalin/parsec - Done
-- Generate precision dependencies in /home/feacluster/PaStiX/pastix_src/sopalin/parsec
/home/feacluster/PaStiX/pastix_src/cmake_modules/morse_cmake/modules/precision_generator/genDependencies.py:22: DeprecationWarning: the imp module is deprecated in favour of importlib; see the module's documentation for alternative uses
  import imp;
-- Generate precision dependencies in /home/feacluster/PaStiX/pastix_src/sopalin/parsec - Done
-- Generate precision dependencies in /home/feacluster/PaStiX/pastix_src/sopalin/parsec
/home/feacluster/PaStiX/pastix_src/cmake_modules/morse_cmake/modules/precision_generator/genDependencies.py:22: DeprecationWarning: the imp module is deprecated in favour of importlib; see the module's documentation for alternative uses
  import imp;
-- Generate precision dependencies in /home/feacluster/PaStiX/pastix_src/sopalin/parsec - Done
-- Generate precision dependencies in /home/feacluster/PaStiX/pastix_src
/home/feacluster/PaStiX/pastix_src/cmake_modules/morse_cmake/modules/precision_generator/genDependencies.py:22: DeprecationWarning: the imp module is deprecated in favour of importlib; see the module's documentation for alternative uses
  import imp;
-- Generate precision dependencies in /home/feacluster/PaStiX/pastix_src - Done
-- Generate precision dependencies in /home/feacluster/PaStiX/pastix_src
/home/feacluster/PaStiX/pastix_src/cmake_modules/morse_cmake/modules/precision_generator/genDependencies.py:22: DeprecationWarning: the imp module is deprecated in favour of importlib; see the module's documentation for alternative uses
  import imp;
-- Generate precision dependencies in /home/feacluster/PaStiX/pastix_src - Done
-- Generate precision dependencies in /home/feacluster/PaStiX/pastix_src
/home/feacluster/PaStiX/pastix_src/cmake_modules/morse_cmake/modules/precision_generator/genDependencies.py:22: DeprecationWarning: the imp module is deprecated in favour of importlib; see the module's documentation for alternative uses
  import imp;
-- Generate precision dependencies in /home/feacluster/PaStiX/pastix_src - Done
-- Generate precision dependencies in /home/feacluster/PaStiX/pastix_src
/home/feacluster/PaStiX/pastix_src/cmake_modules/morse_cmake/modules/precision_generator/genDependencies.py:22: DeprecationWarning: the imp module is deprecated in favour of importlib; see the module's documentation for alternative uses
  import imp;
-- Generate precision dependencies in /home/feacluster/PaStiX/pastix_src - Done
-- Generate precision dependencies in /home/feacluster/PaStiX/pastix_src
/home/feacluster/PaStiX/pastix_src/cmake_modules/morse_cmake/modules/precision_generator/genDependencies.py:22: DeprecationWarning: the imp module is deprecated in favour of importlib; see the module's documentation for alternative uses
  import imp;
-- Generate precision dependencies in /home/feacluster/PaStiX/pastix_src - Done
-- Generate precision dependencies in /home/feacluster/PaStiX/pastix_src
/home/feacluster/PaStiX/pastix_src/cmake_modules/morse_cmake/modules/precision_generator/genDependencies.py:22: DeprecationWarning: the imp module is deprecated in favour of importlib; see the module's documentation for alternative uses
  import imp;
-- Generate precision dependencies in /home/feacluster/PaStiX/pastix_src - Done
-- Generate precision dependencies in /home/feacluster/PaStiX/pastix_src/test
/home/feacluster/PaStiX/pastix_src/cmake_modules/morse_cmake/modules/precision_generator/genDependencies.py:22: DeprecationWarning: the imp module is deprecated in favour of importlib; see the module's documentation for alternative uses
  import imp;
-- Generate precision dependencies in /home/feacluster/PaStiX/pastix_src/test - Done
-- Generate precision dependencies in /home/feacluster/PaStiX/pastix_src/test
/home/feacluster/PaStiX/pastix_src/cmake_modules/morse_cmake/modules/precision_generator/genDependencies.py:22: DeprecationWarning: the imp module is deprecated in favour of importlib; see the module's documentation for alternative uses
  import imp;
-- Generate precision dependencies in /home/feacluster/PaStiX/pastix_src/test - Done
-- A cache variable, namely TMG_DIR, has been set to specify the install directory of TMG
-- Looking for BLAS - found using PkgConfig
-- Looking for Fortran CHEEV
-- Looking for Fortran CHEEV - found
-- Looking for LAPACK in BLAS: found
-- Checking for one of the modules 'mkl-dynamic-lp64-iomp'
-- Checking for one of the modules 'mkl-dynamic-lp64-seq'
-- Checking for one of the modules 'openblas'
-- Looking for LAPACK - found using PkgConfig
-- A library with LAPACK API found.
-- LAPACK_LIBRARIES /usr/lib/x86_64-linux-gnu/openblas-pthread/libopenblas.so
-- Looking for Fortran dlarnv
-- Looking for Fortran dlarnv - found
-- Looking for Fortran dlagsy
-- Looking for Fortran dlagsy - not found
-- Looking for tmg : test with lapack fails
-- Looking for tmg : try to find it elsewhere
-- Looking for Fortran dlarnv
-- Looking for Fortran dlarnv - found
-- Looking for Fortran dlagsy
-- Looking for Fortran dlagsy - found
-- Found TMG: /usr/lib/x86_64-linux-gnu/libtmglib.so
-- Generate precision dependencies in /home/feacluster/PaStiX/pastix_src/test
/home/feacluster/PaStiX/pastix_src/cmake_modules/morse_cmake/modules/precision_generator/genDependencies.py:22: DeprecationWarning: the imp module is deprecated in favour of importlib; see the module's documentation for alternative uses
  import imp;
-- Generate precision dependencies in /home/feacluster/PaStiX/pastix_src/test - Done
-- Generate precision dependencies in /home/feacluster/PaStiX/pastix_src/test
/home/feacluster/PaStiX/pastix_src/cmake_modules/morse_cmake/modules/precision_generator/genDependencies.py:22: DeprecationWarning: the imp module is deprecated in favour of importlib; see the module's documentation for alternative uses
  import imp;
-- Generate precision dependencies in /home/feacluster/PaStiX/pastix_src/test - Done
-- --- Python wrapper is disabled with static libraries
-- Configuring done
-- Generating done
-- Build files have been written to: /home/feacluster/PaStiX/pastix_src/build
sopalin/parsec/CMakeFiles/parsec_headers_tgt.dir/build.make:64: *** missing separator.  Stop.
make[1]: *** [CMakeFiles/Makefile2:1717: sopalin/parsec/CMakeFiles/parsec_headers_tgt.dir/all] Error 2
make[1]: *** Waiting for unfinished jobs....
CMakeFiles/bcsc_headers_tgt.dir/build.make:63: *** missing separator.  Stop.
make[1]: *** [CMakeFiles/Makefile2:566: CMakeFiles/bcsc_headers_tgt.dir/all] Error 2
make: *** [Makefile:141: all] Error 2
sopalin/parsec/CMakeFiles/parsec_headers_tgt.dir/build.make:64: *** missing separator.  Stop.
make[1]: *** [CMakeFiles/Makefile2:1717: sopalin/parsec/CMakeFiles/parsec_headers_tgt.dir/all] Error 2
make: *** [Makefile:141: all] Error 2
10

After a lot of trial and error and emails with the Pastix developers, I got something working. I have v2.18 built on an Ubuntu VM with one Tesla K80 GPU. Using nvidia-smi I can confirm the GPU does appear to be working with Pastix.

Unfortunately, the solve times and memory usage are double what I expect from Pardiso or Spooles. If anyone has it built on a linux machine with gpu, I would like to send you a 1 million dof model to benchmark.

The reason for most of my build errors was due to nvidia drivers and cuda toolkit not installed properly. The mistake I was making was not doing the pre-installation steps before running the cuda_10.1.243_418.87.00_linux.run script. I was under the impression the *.run file would do everything for me.

My next hurdle was then installing standalone pastix. I could not get the older version used by Calculix to install. But I was able to get the latest version to build ( 6.2.1 ).

I modified the simple.c example to read a matrix in ijv format from Calculix. From that I confirmed the GPU was being used in the factorization. It did speed things up by adding around 30-40 gflops/second of computation power. For comparison, the 2 cpus on the machine did total 30 gflops/second . So the GPU kind of performed as though the machine had four->six cpus instead of just two.

Last challenge was then to modify the pastix.c file in the Calculix source to use the newest version of pastix solver. Made some minor edits and was able to run some models. Unfortunately, I noticed the models took longer and used more memory than Pardiso. For some reason the matrices are being defined as “General” instead of “Symmetric” . This is resulting in double the expected memory usage and long solve times. Another strange thing is Pastix implementation solves the models a few times instead of once . It seems to do it first in mixed precision then tries double precision .

I am not sure why they did this. Will need to research further… There are some unsymmetric matrices in Calculix , but I believe they are a minority of cases.

2 Likes
12

Please, could you explain better what do you mean about: pre-installation steps before running the cuda_10.1.243_418.87.00_linux.run script?
Is about days that I’m tring complite CalculiX using PaStiX solver using CUDA but whitout success.