Compiler Diagnostic Test Sets http://ftp.aset.psu.edu/pub/ger/fortran/test/results.txt Arnaud Desitter: http://www.geog.ox.ac.uk/staff/adesitter.html, Oxford University School of Geography and the Environment Herman D. Knoble: http://www.personal.psu.edu/faculty/h/d/hdk/ Penn State Academic Services and Emerging Technologies 14 March 2008 INTRODUCTION This is a Fortran 77/90/95 Test Set. Its purpose is to test run-time diagnostic/debug capability of Windows 9x5/NT/2000 Fortran and Linux Fortran Compilers. The actual source codes were extracted from much larger research projects over 25 years of consulting with research teams. The purpose of these tests is to illustrate problems and how they are diagnosed in concrete situations. Some of the test problems can be diagnosed at compile-time; however since it is not practical to convert large source codes (50,000 lines and larger), these small diagnostic tests illustrate how the same problems might occur and be diagnosed in larger production codes. The main purpose of these Fortran diagnostic benchmarks is to encourage compiler vendors cited to improve their products. It is in this spirit of helping them implement better run-time diagnostics, which in turn encourages a higher integrity in research and mission critical program development, that these tests were compiled and posted here. Regarding the results Tables below, "Yes" means the error was correctly diagnosed; "No" means it was not. For test program source codes please see the test files at: http://ftp.aset.psu.edu/pub/ger/Fortran/test The Polyhedron test site, evaluating these and other compilers is: http://www.polyhedron.com/ We acknowledge tests submitted and/or reviewed: Compaq-Digital CVF for Windows: http://www5.compaq.com/fortran/index.html Open Source g95 for Windows and Linux: http://g95.sourceforge.net/ Intel Fortran for Linux: http://developer.intel.com/software/products/compilers/flin/noncom.htm Lahey-Fujitsu LF95 for Windows: http://www.lahey.com/ Polyhedron: http://www.polyhedron.com (Jon Wallis and John Appleyard). Silverfrost FTN95 for Windows: http://www.silverfrost.com/11/ftn95/overview.asp PathScale PATHF90 for Linux: http://www.pathscale.com/ekopath.html COMPILER VERSIONS and OPTIONS USED Compaq-Digital CVF V6.6C: DFDEBUG.BAT DF %1 /list /check:all /debug:full /traceback /warn:argument_checking /warn:uninitialized /warn:truncated_source /recursive %2 %3 Lahey LF95 V7.10.01: LFDEBUG.BAT lf95 %1 -chk(a,e,s,u,x) -chkglobal -g -pca -stchk -trace -nzero -nsav -trap diou -co -lst -info -w -o0 Silverfrost FTN95 V5.20.0 SALDEBUG.BAT set salfenvar=trap_underflows FTN95 %1 /LIST /DEBUG /CHECK /FULL_UNDEF /FULL_DEBUG /LINK /UNDERFLOW %2 %3 %4 G95 gcc version 4.0.3, G95 June 30 2007 for Windows: Command Line g95 -Wall -pedantic -fbounds-check -ftrace=full -Wuninitialized -Wunset-vars -Wline-truncation %* Environmental variables are set when test program tests these conditions as follows: SET G95_FPU_ZERODIV=1 SET G95_FPU_OVERFLOW=1 SET G95_FPU_UNDERFLOW=1 SET G95_FPU_EXCEPTIONS=1 SET G95_FPU_DENORMAL=1 SET G95_FPU_INVALID=1 Intel IFORT 10.0.23 for Unix: Command Line ifort -FR -check all -O0 -auto -WB -traceback -g -fpe0 -fltconsistency "$@" (See bash scripts ifdebug.sh, comptest.sh, and runtest.sh) PathScale Fortran Version 3.0 for Linux 64-bit AMD Platform. pathf90 -C -O0 -g -msse2 -TENV:simd_imask=OFF fileid (Issue: man eko and search for -TENV: for other options like _omask for overflow). To abort after subscript range error, after compile, run with statement: env F90_BOUNDS_CHECK_ABORT=YES ./a.out LINUX BASH SCRIPTS AND GDB BATCH FILE ifdebug.sh compiles with debug options; called by comptest.sh comptest.sh Batch compiles all tests in Tables 1-4; output file are: *.cout and *.exe runtest.sh Batch runs all tests in Tables 1-4; output files are: *.out segfault.gdb Used in runtest.sh to log but not pause for tests that cause Segmentation Faults. WINDOWS BATCH FILES compdebug.bat compiles with compiler specific debug options; called by comptest.bat comptest.bat Batch compiles all tests in Tables 1-4; output files are: *.cout and *.exe runtest.bat Batch runs all tests in Tables 1-4; output files are *.out TABLE I. Tests to Detect Miscellaneous Compile-time or Run-Time Errors Compiled from a variety of sources by Skip Knoble and Arnaud Desitter Intel Fortran Silverfrost Windows Linux PathScale Source File Compaq CVF Lahey LF95 FTN95 g95 10.0.23 3.0 --------------------------------------------------------------------------------------------------------------------- test1.f Array bound No Yes Yes Yes Yes Yes test2.f Array bound No Yes Yes Yes Yes Yes test3.f String bound No Yes Yes No No No test4.f Uninitialized variable Yes(funder,C) Yes Yes No Yes Yes test5.f Call mismatch Yes(C) Yes (both) Yes (both) Yes Yes No test6.f Call mismatch Yes Yes Yes Yes Yes Yes(bound) test7.f String bound Yes Yes Yes Yes Yes No test8.f String bound Yes Yes Yes Yes Yes No test9.f DO loop index Yes(idbz) Yes(idbz) Yes Yes(WM) Yes No(fpe) test10.f Function mismatch No No Yes No No No test11.f90 Uninitialized variable No Yes Yes No No No test12.f90 FP DIV/0 Yes(Inf) Yes Yes Yes(Inf) Yes Yes(Inf) test13.f90 Array conformance No Yes Yes No No Yes test14.f90 Array conformance No Yes Yes No No Yes test15.f90 DO loop index No Yes Yes No No No test16.f90 Uninitialized variable No Yes Yes No(WC) Yes No test17.f90 String conformance Yes Yes Yes No No No test18.f90 FP Underflow Yes Yes Yes Yes(env) No(fpz) Yes test19.f90 Integer overflow Yes(No) No Yes No No No test20.f90 Pointer usage Yes Yes Yes No No No(SegF) test21.f90 Write on expression Yes(No) Yes(npca) Yes Yes Yes(No) No(SegF) test22.f90 Write on expression Yes(No) Yes(npca) Yes Yes Yes(No) No(SegF) test23.f90 Write on expression Yes(No) Yes(npca) Yes No Yes(No) No(SegF) test24.f90 Write on expression Yes(No) Yes(npca) Yes No Yes(No) No(SegF) test25.f90 Optional Argument Yes(No) Yes Yes Yes(No) Yes(No) No(SegF) test26.f90 Array bound No Yes Yes No No No test27.f90 Dangling pointers No Yes Yes No Yes(No) No(SegF) test28.f90 Dangling pointers No Yes Yes No Yes(No) No(SegF) test29.f Argument conformance No Yes No No No No test30.f Correct Array Reshaping Yes(cor) Yes(cor) Yes(debug) Yes(cor) Yes(cor) Yes test31.f Array bound Yes Yes Yes Yes Yes No test32.f Recursive Write No Yes(-f95) Yes Yes Yes(No) Yes(NoTr) test33.f90 Conditional String Bound No Yes(npca) No No No No test34.f90 Array Statement Bound No Yes No Yes Yes Yes test35.f Modifying Expression #1 No Yes No No No No test36.f Modifying Expression #2 No Yes Yes No No No test37.f Modifying Expression #3 No Yes Yes No No No test38.f90 Inconsistent argument No Yes Yes No No No test39.f90 Call Mismatch No Yes No No No No test40.f90 FP Overflow Yes(inf) Yes Yes(inf) Yes(inf) Yes Yes test41.f90 Illegal Alias No No Yes No No No f2003-1.f90 Array Assignment Extension Yes No No Yes Yes Yes INTENT1.f90 Modifying INTENT(IN) #1 No No Yes No No No INTENT2.f90 Assoc. INTENT(IN/OUT) No No Yes No No No INTENT3.f90 Modifying INTENT(IN) #3 No Yes Yes No No No INTENT4.f90 Expression INTENT(INOUT) No No Yes No No No INTENT5.f90 Expression INTENT(OUT) Yes(No) No Yes No Yes(No) No(SegF) INTENT6.f90 Expression INTENT(IN/OUT) Yes(cor) Yes(cor) Yes(cor) Yes(cor) Yes(cor) Yes(cor) FORMAT1.f90 Format Scale Factor Yes(-stand) Yes(-f95) Yes (/ISO) No No(Yes) No(Yes) FORMAT2.f90 Format Mismatch WRITE Yes(NC) Yes(NC) Yes(NC) Yes Yes No FORMAT3.f90 Format Mismatch READ Yes(NC) Yes(NC) Yes(NC) Yes Yes No FORMAT4.f90 Internal Write Overflow Yes(NC) Yes(NC) Yes(NC) Yes(NC) Yes(NC) Yes(NC) ------------------------------------------------------------------------------------------------------------------- (-f95) -> Use of the LF95 conformance option: -f95 (-iso) -> Use of the FTN95 conformance option: /iso (-stand -> Use of the CVF conformance option: -stand (8No) -> Integer*8 case is not diagnosed at run time. (both) -> Both Compile and Run time message. (bound) -> PathScale flags subscript out of range instead of argument mismatch; traceback is correct. (C) -> Correct Compile time message only if /debug:partial is used. (cor) -> The code is correct; Yes=ran ok; No= gave incorrect error message. (debug) -> Silverfrost FTN95: use only the /debug option instead of /full_debug for this test. (fpe) -> Pathscale flags zero index of Do as: floating point exception with no traceback. (fpz) -> Intel compiled code defaults to set FP underflow result to zero; -no-fpz will not generate a NaN on Opteron systems (using gdb). (funder) -> Floating-point Underflow message given. (gdb) -> Under Linux use: gdb to get error message. (idbz) -> Integer Divide by Zero message given. (Inf) -> Displays result variable value as the word: Infinity. (NaN) -> Sets result variable value to Not-a-Number. (NC) -> Not diagnosed at compile time even though enough info is there. (NoTr) -> Pathscale correct error message but no traceback. (npca) -> Lahey; option pca to "protect constant arguments" is turned off(n). (SegF) -> Pathscale error message: Segmentation Fault with hex location and no traceback. No(Yes) -> Does not diagnose non-conformity but gives correct output. Yes(No) -> Gives a general "Access Violation" or "segmentation fault" rather than specific diagnostic. WM -> problem diagnosed but not with the correct error message. (env) -> g95 environment variable G95_FPU_UNDERFLOW=1 enables correct diagnostic. WC -> Correct Warning given at compile time, but no run-time diagnostic. TABLE II Tests to detect Uninitialized Variables at Run-Time Compiled by Arnaud Desitter Test Test=Uninitialized Variable Compaq Lahey Silverfrost Windows Linux (**) PathScale File(*) For this Type of Array CVF LF95 FTN95 g95 10.0.23 3.0 ---------------------------------------------------------------------------------------------------------- Uin10.f automatic Integer No Yes Yes No No No Uin10-fp.f automatic Real No Yes Yes No No No(NaN) Uin10-dt.f automatic Derived Type No No Yes No No No Uin11.f allocatable Integer No Yes Yes No No No Uin11-fp.f allocatable Real No Yes Yes No No No Uin11-dt.f allocatable Derived Type No No Yes No No No Uin12.f saved Integer No No Yes No No No Uin12-fp.f saved Real No No Yes No No No Uin12-dt.f saved Derived Type No No Yes No No No Uin13.f "common" Integer No Yes Yes No No No Uin13-fp.f "common" Real No Yes Yes No No No Uin13-dt.f "common" Derived Type No No Yes No No No Uin14.f saved Integer declared in modules No Yes Yes No No No Uin14-fp.f saved Real declared in modules No Yes Yes No No No Uin14-dt.f saved Derived Type in modules No No Yes No No No Uin15.f INTENT(OUT) Integer Yes(C) No Yes No No No Uin15-fp.f INTENT(OUT) Real Yes(C) No Yes No No No Uin15-dt.f INTENT(OUT) Derived Type No No Yes No No No Uin16.f pointer Integer No No Yes No No No Uin16-fp.f pointer Real No No Yes No No No Uin16-dt.f pointer Derived Type No No Yes No No No ----------------------------------------------------------------------------------------------------------- (C) means Yes, but warning message at compile-time only. In the case of CVF, the debug option must be /debug:partial to get this warning. (**) Intel Fortran for Linux Version 9.1.023 supports check all to detect, among other things (simple) uninitialized variable values at run-time. In the table above, "Yes" means that the most recent version of the respective compiler correctly diagnosed the uninitialized variable for this type of array (with a correct traceback). "No" indicates that no run-time diagnostic was given, that is, a "wrong answer", namely x(2), was assigned to x(3) and subsequently displayed. (*) Each of these test's INCLUDE 'UNI_AUXI.F' which causes a call to either SUB1(Integer) or SUB1FP (Real) or SUB1DT (Derived Type) which in turn uses an uninitialized (unset) variable array element, x(2) as follows: subroutine sub1(x) integer x(*) x(1) = 1 x(3) = x(2) print * , x(3) end subroutine sub1fp(x) real x(*) x(1) = 1. x(3) = x(2) print * , x(3) end module dt_mod type dt sequence ! necessary for common integer i end type end module subroutine sub1dt(x) use dt_mod type(dt) x(*) x(1) = dt(1) x(3) = x(2) print * , x(3) end TABLE III Tests to detect Uninitialized Variables at Run-Time F2k Compiler Extension(s) Compiled by Arnaud Desitter Test Test detection of uninitialized Compaq Lahey Silverfrost Windows Linux(**) PathScale File(*) variables for an Allocatable array CVF LF95 FTN95 g95 10.0.23 3.0 ----------------------------------------------------------------------------------------------------------- Uin25.f Derived Type - Integer array No(c) No(c) N/A No(c) No(c) No(c) Uin25-fp.f Derived Type - Real array No(c) No(c) N/A No(c) No(c) No(c) Uin25-dt.f Derived Type - Derived Type array No(c) No(c) N/A No(c) No(c) No(c) Uin26.f Dummy argument - Integer array No(c) No(c) N/A No(c) No(c) No(c) Uin26-fp.f Dummy argument - Real array No(c) No(c) N/A No(c) No(c) No(c) Uin26-dt.f Dummy argument - Derived Type array No(c) No(c) N/A No(c) No(c) No(c) Uin27.f Function result - Integer array No(c) No(c) N/A No(c) No(c) No(c) Uin27-fp.f Function result - Real array No(c) No(c) N/A No(c) No(c) No(c) Uin27-dt.f Function result - Derived Type array No(c) No(c) N/A No(c) No(c) No(c) ------------------------------------------------------------------------------------------------------------ (c) - Compiler compiles this code so it can be run; but uninitialized array elements are not detected. (FTN95 does not support this extension). N/A Compiler does not support F2K or F003 extension. TABLE IV Tests to detect Non-Conformant or Non-Allocated Arrays at Run-Time F2k Compiler Extension(s) Compiled by Arnaud Desitter Test Test detection of non-conformant Compaq Lahey Silver Windows Linux PathScale File(*) or non-allocated allocatable ARRAYS CVF LF95 frost g95 10.0.23 3.0 FTN95 ------------------------------------------------------------------------------------------------------------- ALLOC26.f INTENT(OUT) Dummy argument - Integer Yes No N/A Yes Yes(Seg) Yes(Seg) ALLOC26-FP.f INTENT(OUT) Dummy argument - Real Yes No N/A Yes Yes(Seg) Yes(Seg) ALLOC26-DT.f INTENT(OUT) Dummy argument - Derived types Yes No N/A Yes Yes(Seg) Yes(Seg) Con25.f Conformance - Derived Type - Integer Yes Yes N/A Yes Yes Yes Con26.f Conformance - Dummy argument - Integer Yes Yes N/A Yes No Yes Con27.f Conformance - Function result - Integer No Yes N/A Yes No Yes -------------------------------------------------------------------------------------------------------------- Compaq CVF Intel Ifort & Lahey LF95 compile this code so it can be run; and non-conformant arrays are detected with appropriate run-time message with traceback. (FTN95 do not support this extension). Seg -> Ifort Segmentation Fault message given with correct traceback. TABLE V Tests to illustrate that Memory Leaks are not Detected Compiled by Arnaud Desitter Test Compaq Lahey Silverfrost Windows Linux PathScale File Test=Memory Leak Detection CVF LF95 FTN95 g95 10.0.23 3.0 ------------------------------------------------------------------------------------------------------------ LEAK1.f90 pointer assignment with a function No No No Yes No No returning a pointer LEAK2.f90 pointer with ALLOCATE No No No Yes No No ------------------------------------------------------------------------------------------------------------ NOTES: 1) The compilers tested do not detect any memory leaks, but the memory leaks are real. Relative to the Fortran Standards, these compilers don't have to provide such a mechanism. 2) The above compiler run-time environnments do not have any form of memory garbage collection that would reduce or eliminate the leaks. 3) Modern OS's will reclaim the memory allocated upon termination of the process. Windows 9x/Me/NT/2000 do reclaim memory allocated upon termination of these test programs. 4) Memory leaks can cause serious problems during the running life span of a program. -------------------------------------------end--------------------------------------- --