[Folding@home] failed to remove directory/file

[SandStar]

Ik heb momenteel niet genoeg V6 clients draaien om het te verifieren maar als mijn geheugen een keertje meewerkt was/is er al langer een error van de V6 client over het niet kunnen verwijderen van bepaalde files nadat een workunit de 100% heeft bereikt.

Ik ben op zoek naar eventuele logfiles hiervan. Liefst originele files maar een copy paste hier is ook prima.
Het gaat er mij om dat het log voorkomt:
- versie nummer client
- Project (run/clone/gen)
- De error over het niet kunnen verwijderen van een unit.

Zou erg helpen met troubleshooten voor V7

[smoking2000]

Was dat niet het probleem met de nieuwe A3 core die nog files in de work/ directory liet slingeren nadat de -cleanup ronde van de core was voltooid?

Ik heb een hele bende logfiles van allerlei clients, maar geen met logging van de -cleanup:

$ find ~/ -name FAHlog* -print | wc -l
367
$ find ~/projects/qd-tools/testcases/ ~/software/folding* -name FAHlog* -exec grep -i clean {} \; | wc -l
0

[SandStar]

Het gaat idd om de work directory die niet volledig opgeruimd werd.

Ik probeer aan de hand van wat voorbeelden een duidelijk beeld te krijgen wat er nu precies allemaal gebeurd. De V7 client gaat wat anders met work directories om en ik lijk een (zeldzame) bug gevonden te hebben die hiermee samenhangt

[smoking2000]

Volgens mij heb ik op dit moment last van zo'n issue op m'n OS X 10.4 machine:

[12:40:48] Working on queue slot 09 [September 21 12:40:48 UTC]
[12:40:48] + Working ...
[12:40:48] - Calling './FahCore_a3.exe -dir work/ -nice 19 -suffix 09 -np 2 -checkpoint 5 -forceasm -verbose -lifeline 9902 -version 629'

[12:40:49] 
[12:40:49] *------------------------------*
[12:40:49] Folding@Home Gromacs SMP Core
[12:40:49] Version 2.21 (May 7 2010)
[12:40:49] 
[12:40:49] Preparing to commence simulation
[12:40:49] - Ensuring status. Please wait.
[12:40:58] - Assembly optimizations manually forced on.
[12:40:59] - Not checking prior termination.
[12:40:59] - Expanded 1767955 -> 2253305 (decompressed 127.4 percent)
[12:40:59] Called DecompressByteArray: compressed_data_size=1767955 data_size=2253305, decompressed_data_size=2253305 diff=0
[12:40:59] - Digital signature verified
[12:40:59] 
[12:40:59] Project: 6056 (Run 0, Clone 30, Gen 39)
[12:40:59] 
[12:40:59] Assembly optimizations on if available.
[12:40:59] Entering M.D.
[12:41:05] CoreStatus = 0 (0)
[12:41:05] Sending work to server
[12:41:05] Project: 6056 (Run 0, Clone 30, Gen 39)
[12:41:05] - Error: Could not get length of results file work/wuresults_09.dat
[12:41:05] - Error: Could not read unit 09 file. Removing from queue.
[12:41:06] Trying to send all finished work units
[12:41:06] + No unsent completed units remaining.
[12:41:06] + -oneunit flag given and have now finished a unit. Exiting.- Preparing to get new work unit...
[12:41:06] Cleaning up work directory
[12:41:06] ***** Got a SIGTERM signal (15)
[12:41:06] Killing all core threads

Files die in de work/ directory zijn blijven slingeren:

$ ls -l work/*_0{5,6,7,8,9}.*
-rwxr-x---   1 bas  bas      501 Sep 21 14:39 work/logfile_05.txt
-rwxr-x---   1 bas  bas      533 Sep 21 14:39 work/logfile_06.txt
-rwxr-x---   1 bas  bas      533 Sep 21 14:40 work/logfile_07.txt
-rwxr-x---   1 bas  bas      533 Sep 21 14:40 work/logfile_08.txt
-rwxr-x---   1 bas  bas      533 Sep 21 14:41 work/logfile_09.txt
-rw-r--r--   1 bas  bas  1768467 Sep 21 14:39 work/wudata_05.dat
-rw-r--r--   1 bas  bas     7928 Sep 21 14:39 work/wudata_05.log
-rw-r--r--   1 bas  bas  2253305 Sep 21 14:39 work/wudata_05.tpr
-rw-r--r--   1 bas  bas  1768467 Sep 21 14:39 work/wudata_06.dat
-rw-r--r--   1 bas  bas     7928 Sep 21 14:39 work/wudata_06.log
-rw-r--r--   1 bas  bas  2253305 Sep 21 14:39 work/wudata_06.tpr
-rw-r--r--   1 bas  bas  1768467 Sep 21 14:39 work/wudata_07.dat
-rw-r--r--   1 bas  bas     7928 Sep 21 14:40 work/wudata_07.log
-rw-r--r--   1 bas  bas  2253305 Sep 21 14:40 work/wudata_07.tpr
-rw-r--r--   1 bas  bas  1768467 Sep 21 14:40 work/wudata_08.dat
-rw-r--r--   1 bas  bas     7928 Sep 21 14:40 work/wudata_08.log
-rw-r--r--   1 bas  bas  2253305 Sep 21 14:40 work/wudata_08.tpr
-rw-r--r--   1 bas  bas  1768467 Sep 21 14:40 work/wudata_09.dat
-rw-r--r--   1 bas  bas     7928 Sep 21 14:41 work/wudata_09.log
-rw-r--r--   1 bas  bas  2253305 Sep 21 14:40 work/wudata_09.tpr
-rwxr-x---   1 bas  bas      512 Sep 21 14:39 work/wuinfo_05.dat
-rwxr-x---   1 bas  bas      512 Sep 21 14:39 work/wuinfo_06.dat
-rwxr-x---   1 bas  bas      512 Sep 21 14:40 work/wuinfo_07.dat
-rwxr-x---   1 bas  bas      512 Sep 21 14:40 work/wuinfo_08.dat
-rwxr-x---   1 bas  bas      512 Sep 21 14:40 work/wuinfo_09.dat

Volledige dump en log, hij kwam namelijk een paar keer achter elkaar:

Queue Dump

 Index 5: deleted 481.00 pts
   server: 171.64.65.54:8080; project: 6056
   Folding: run 0, clone 30, generation 39; benchmark 0; misc: 721551360, 629, 12 (le)
   Project: 6056 (Run 0, Clone 30, Gen 39)
   issue: Tue Sep 21 14:39:03 2010; begin: Tue Sep 21 14:39:10 2010
   end: Tue Sep 21 14:39:18 2010; due: Mon Sep 27 14:39:10 2010 (6 days)
   preferred: Fri Sep 24 14:39:10 2010 (3 days)
   core URL: http://www.stanford.edu/~pande/OSX/x86_Tiger/Core_a3.fah (V2.21)
   core number: 0xa3; core name: GRO-A3
   CPU: 1,0 x86; OS: 3,0 OSX
   smp cores: 2; cores to use: 2
   tag: P6056R0C30G39
   flops: 1059720278 (1059.720278 megaflops)
   memory: 1024 MB
   client type: 3 Advmethods
   assignment info (le): Tue Sep 21 14:38:27 2010; BF72650A
   CS: 171.67.108.25; P limit: 524286976
   user: [DPC]_Fatal_Error_Group0JMT; team: 92; ID: 4D3190732C0C3E20; mach ID: 1
   work/wudata_05.dat file size: 1768467; WU type: Folding@Home

 Index 6: deleted 481.00 pts
   server: 171.64.65.54:8080; project: 6056
   Folding: run 0, clone 30, generation 39; benchmark 0; misc: 721551360, 629, 12 (le)
   Project: 6056 (Run 0, Clone 30, Gen 39)
   issue: Tue Sep 21 14:39:21 2010; begin: Tue Sep 21 14:39:27 2010
   end: Tue Sep 21 14:39:44 2010; due: Mon Sep 27 14:39:27 2010 (6 days)
   preferred: Fri Sep 24 14:39:27 2010 (3 days)
   core URL: http://www.stanford.edu/~pande/OSX/x86_Tiger/Core_a3.fah (V2.21)
   core number: 0xa3; core name: GRO-A3
   CPU: 1,0 x86; OS: 3,0 OSX
   smp cores: 2; cores to use: 2
   tag: P6056R0C30G39
   flops: 1059720278 (1059.720278 megaflops)
   memory: 1024 MB
   client type: 3 Advmethods
   assignment info (le): Tue Sep 21 14:38:45 2010; BF72651C
   CS: 171.67.108.25; P limit: 524286976
   user: [DPC]_Fatal_Error_Group0JMT; team: 92; ID: 4D3190732C0C3E20; mach ID: 1
   work/wudata_06.dat file size: 1768467; WU type: Folding@Home

 Index 7: deleted 481.00 pts
   server: 171.64.65.54:8080; project: 6056
   Folding: run 0, clone 30, generation 39; benchmark 0; misc: 721551360, 629, 12 (le)
   Project: 6056 (Run 0, Clone 30, Gen 39)
   issue: Tue Sep 21 14:39:47 2010; begin: Tue Sep 21 14:39:54 2010
   end: Tue Sep 21 14:40:11 2010; due: Mon Sep 27 14:39:54 2010 (6 days)
   preferred: Fri Sep 24 14:39:54 2010 (3 days)
   core URL: http://www.stanford.edu/~pande/OSX/x86_Tiger/Core_a3.fah (V2.21)
   core number: 0xa3; core name: GRO-A3
   CPU: 1,0 x86; OS: 3,0 OSX
   smp cores: 2; cores to use: 2
   tag: P6056R0C30G39
   flops: 1059720278 (1059.720278 megaflops)
   memory: 1024 MB
   client type: 3 Advmethods
   assignment info (le): Tue Sep 21 14:39:11 2010; BF726526
   CS: 171.67.108.25; P limit: 524286976
   user: [DPC]_Fatal_Error_Group0smoking2000; team: 92; ID: 4D3190732C0C3E20; mach ID: 1
   work/wudata_07.dat file size: 1768467; WU type: Folding@Home

 Index 8: deleted 481.00 pts
   server: 171.64.65.54:8080; project: 6056
   Folding: run 0, clone 30, generation 39; benchmark 0; misc: 721551360, 629, 12 (le)
   Project: 6056 (Run 0, Clone 30, Gen 39)
   issue: Tue Sep 21 14:40:14 2010; begin: Tue Sep 21 14:40:20 2010
   end: Tue Sep 21 14:40:38 2010; due: Mon Sep 27 14:40:20 2010 (6 days)
   preferred: Fri Sep 24 14:40:20 2010 (3 days)
   core URL: http://www.stanford.edu/~pande/OSX/x86_Tiger/Core_a3.fah (V2.21)
   core number: 0xa3; core name: GRO-A3
   CPU: 1,0 x86; OS: 3,0 OSX
   smp cores: 2; cores to use: 2
   tag: P6056R0C30G39
   flops: 1059720278 (1059.720278 megaflops)
   memory: 1024 MB
   client type: 3 Advmethods
   assignment info (le): Tue Sep 21 14:39:38 2010; BF7262C3
   CS: 171.67.108.25; P limit: 524286976
   user: [DPC]_Fatal_Error_Group0JMT; team: 92; ID: 4D3190732C0C3E20; mach ID: 1
   work/wudata_08.dat file size: 1768467; WU type: Folding@Home

 Index 9: deleted 481.00 pts
   server: 171.64.65.54:8080; project: 6056
   Folding: run 0, clone 30, generation 39; benchmark 0; misc: 721551360, 629, 12 (le)
   Project: 6056 (Run 0, Clone 30, Gen 39)
   issue: Tue Sep 21 14:40:42 2010; begin: Tue Sep 21 14:40:48 2010
   end: Tue Sep 21 14:41:05 2010; due: Mon Sep 27 14:40:48 2010 (6 days)
   preferred: Fri Sep 24 14:40:48 2010 (3 days)
   core URL: http://www.stanford.edu/~pande/OSX/x86_Tiger/Core_a3.fah (V2.21)
   core number: 0xa3; core name: GRO-A3
   CPU: 1,0 x86; OS: 3,0 OSX
   smp cores: 2; cores to use: 2
   tag: P6056R0C30G39
   flops: 1059720278 (1059.720278 megaflops)
   memory: 1024 MB
   client type: 3 Advmethods
   assignment info (le): Tue Sep 21 14:40:06 2010; BF7262EF
   CS: 171.67.108.25; P limit: 524286976
   user: [DPC]_Fatal_Error_Group0smoking2000; team: 92; ID: 4D3190732C0C3E20; mach ID: 1
   work/wudata_09.dat file size: 1768467; WU type: Folding@Home

FAHlog

--- Opening Log file [September 21 12:39:00 UTC] 


# Mac OS X SMP Console Edition ################################################
###############################################################################

                       Folding@Home Client Version 6.29r1

                          http://folding.stanford.edu

###############################################################################
###############################################################################

Launch directory: /Users/bas/software/folding
Executable: ./fah6
Arguments: -local -verbosity 9 -oneunit -smp 2 -forceasm 

[12:39:00] - Ask before connecting: No
[12:39:00] - User name: [DPC]_Fatal_Error_Group0JMT (Team 92)
[12:39:00] - User ID: 203E0C2C7390314C
[12:39:00] - Machine ID: 1
[12:39:00] 
[12:39:00] Loaded queue successfully.
[12:39:00] - Preparing to get new work unit...
[12:39:00] - Autosending finished units... [12:39:00]
[12:39:00] Cleaning up work directory
[12:39:00] Trying to send all finished work units
[12:39:00] + No unsent completed units remaining.
[12:39:00] - Autosend completed
[12:39:01] + Attempting to get work packet
[12:39:01] Passkey found
[12:39:01] - Will indicate memory of 1024 MB
[12:39:01] - Detect CPU. Vendor: GenuineIntel, Family: 6, Model: 14, Stepping: 8
[12:39:01] - Connecting to assignment server
[12:39:01] Connecting to http://assign.stanford.edu:8080/
[12:39:02] Posted data.
[12:39:02] Initial: 40AB; - Successful: assigned to (171.64.65.54).
[12:39:02] + News From Folding@Home: Welcome to Folding@Home
[12:39:02] Loaded queue successfully.
[12:39:02] Sent data
[12:39:02] Connecting to http://171.64.65.54:8080/
[12:39:03] Posted data.
[12:39:03] Initial: 0000; - Receiving payload (expected size: 1768467)
[12:39:10] - Downloaded at ~246 kB/s
[12:39:10] - Averaged speed for that direction ~265 kB/s
[12:39:10] + Received work.
[12:39:10] + Closed connections
[12:39:10] 
[12:39:10] + Processing work unit
[12:39:10] Core required: FahCore_a3.exe
[12:39:10] Core found.
[12:39:10] Working on queue slot 05 [September 21 12:39:10 UTC]
[12:39:10] + Working ...
[12:39:10] - Calling './FahCore_a3.exe -dir work/ -nice 19 -suffix 05 -np 2 -checkpoint 5 -forceasm -verbose -lifeline 9818 -version 629'

[12:39:10] 
[12:39:10] *------------------------------*
[12:39:10] Folding@Home Gromacs SMP Core
[12:39:10] Version 2.21 (May 7 2010)
[12:39:10] 
[12:39:10] Preparing to commence simulation
[12:39:10] - Assembly optimizations manually forced on.
[12:39:10] - Not checking prior termination.
[12:39:11] - Expanded 1767955 -> 2253305 (decompressed 127.4 percent)
[12:39:11] Called DecompressByteArray: compressed_data_size=1767955 data_size=2253305, decompressed_data_size=2253305 diff=0
[12:39:11] - Digital signature verified
[12:39:11] 
[12:39:11] Project: 6056 (Run 0, Clone 30, Gen 39)
[12:39:11] 
[12:39:11] Assembly optimizations on if available.
[12:39:11] Entering M.D.
[12:39:18] CoreStatus = 0 (0)
[12:39:18] Sending work to server
[12:39:18] Project: 6056 (Run 0, Clone 30, Gen 39)
[12:39:18] - Error: Could not get length of results file work/wuresults_05.dat
[12:39:18] - Error: Could not read unit 05 file. Removing from queue.
[12:39:18] Trying to send all finished work units
[12:39:18] + No unsent completed units remaining.
[12:39:18] + -oneunit flag given and have now finished a unit. Exiting.- Preparing to get new work unit...
[12:39:18] Cleaning up work directory
[12:39:18] ***** Got a SIGTERM signal (15)
[12:39:18] Killing all core threads

Folding@Home Client Shutdown.


--- Opening Log file [September 21 12:39:18 UTC] 


# Mac OS X SMP Console Edition ################################################
###############################################################################

                       Folding@Home Client Version 6.29r1

                          http://folding.stanford.edu

###############################################################################
###############################################################################

Launch directory: /Users/bas/software/folding
Executable: ./fah6
Arguments: -local -verbosity 9 -oneunit -smp 2 -forceasm 

[12:39:18] - Ask before connecting: No
[12:39:18] - User name: [DPC]_Fatal_Error_Group0JMT (Team 92)
[12:39:18] - User ID: 203E0C2C7390314C
[12:39:18] - Machine ID: 1
[12:39:18] 
[12:39:18] Loaded queue successfully.
[12:39:18] - Preparing to get new work unit...
[12:39:18] - Autosending finished units... [12:39:18]
[12:39:18] Cleaning up work directory
[12:39:18] Trying to send all finished work units
[12:39:18] + No unsent completed units remaining.
[12:39:18] - Autosend completed
[12:39:19] + Attempting to get work packet
[12:39:19] Passkey found
[12:39:19] - Will indicate memory of 1024 MB
[12:39:19] - Detect CPU. Vendor: GenuineIntel, Family: 6, Model: 14, Stepping: 8
[12:39:19] - Connecting to assignment server
[12:39:19] Connecting to http://assign.stanford.edu:8080/
[12:39:19] Posted data.
[12:39:19] Initial: 40AB; - Successful: assigned to (171.64.65.54).
[12:39:19] + News From Folding@Home: Welcome to Folding@Home
[12:39:20] Loaded queue successfully.
[12:39:20] Sent data
[12:39:20] Connecting to http://171.64.65.54:8080/
[12:39:21] Posted data.
[12:39:21] Initial: 0000; - Receiving payload (expected size: 1768467)
[12:39:27] - Downloaded at ~287 kB/s
[12:39:27] - Averaged speed for that direction ~269 kB/s
[12:39:27] + Received work.
[12:39:27] + Closed connections
[12:39:27] 
[12:39:27] + Processing work unit
[12:39:27] Core required: FahCore_a3.exe
[12:39:27] Core found.
[12:39:27] Working on queue slot 06 [September 21 12:39:27 UTC]
[12:39:27] + Working ...
[12:39:27] - Calling './FahCore_a3.exe -dir work/ -nice 19 -suffix 06 -np 2 -checkpoint 5 -forceasm -verbose -lifeline 9839 -version 629'

[12:39:27] 
[12:39:27] *------------------------------*
[12:39:27] Folding@Home Gromacs SMP Core
[12:39:27] Version 2.21 (May 7 2010)
[12:39:27] 
[12:39:27] Preparing to commence simulation
[12:39:27] - Ensuring status. Please wait.
[12:39:37] - Assembly optimizations manually forced on.
[12:39:37] - Not checking prior termination.
[12:39:37] - Expanded 1767955 -> 2253305 (decompressed 127.4 percent)
[12:39:37] Called DecompressByteArray: compressed_data_size=1767955 data_size=2253305, decompressed_data_size=2253305 diff=0
[12:39:37] - Digital signature verified
[12:39:37] 
[12:39:37] Project: 6056 (Run 0, Clone 30, Gen 39)
[12:39:37] 
[12:39:37] Assembly optimizations on if available.
[12:39:37] Entering M.D.
[12:39:44] CoreStatus = 0 (0)
[12:39:44] Sending work to server
[12:39:44] Project: 6056 (Run 0, Clone 30, Gen 39)
[12:39:44] - Error: Could not get length of results file work/wuresults_06.dat
[12:39:44] - Error: Could not read unit 06 file. Removing from queue.
[12:39:44] Trying to send all finished work units
[12:39:44] + No unsent completed units remaining.
[12:39:44] + -oneunit flag given and have now finished a unit. Exiting.- Preparing to get new work unit...
[12:39:44] Cleaning up work directory
[12:39:44] ***** Got a SIGTERM signal (15)
[12:39:44] Killing all core threads

Folding@Home Client Shutdown.


--- Opening Log file [September 21 12:39:45 UTC] 


# Mac OS X SMP Console Edition ################################################
###############################################################################

                       Folding@Home Client Version 6.29r1

                          http://folding.stanford.edu

###############################################################################
###############################################################################

Launch directory: /Users/bas/software/folding
Executable: ./fah6
Arguments: -local -verbosity 9 -oneunit -smp 2 -forceasm 

[12:39:45] - Ask before connecting: No
[12:39:45] - User name: [DPC]_Fatal_Error_Group0smoking2000 (Team 92)
[12:39:45] - User ID: 203E0C2C7390314C
[12:39:45] - Machine ID: 1
[12:39:45] 
[12:39:45] Loaded queue successfully.
[12:39:45] - Preparing to get new work unit...
[12:39:45] - Autosending finished units... [12:39:45]
[12:39:45] Cleaning up work directory
[12:39:45] Trying to send all finished work units
[12:39:45] + No unsent completed units remaining.
[12:39:45] - Autosend completed
[12:39:45] + Attempting to get work packet
[12:39:45] Passkey found
[12:39:45] - Will indicate memory of 1024 MB
[12:39:45] - Detect CPU. Vendor: GenuineIntel, Family: 6, Model: 14, Stepping: 8
[12:39:45] - Connecting to assignment server
[12:39:45] Connecting to http://assign.stanford.edu:8080/
[12:39:46] Posted data.
[12:39:46] Initial: 40AB; - Successful: assigned to (171.64.65.54).
[12:39:46] + News From Folding@Home: Welcome to Folding@Home
[12:39:46] Loaded queue successfully.
[12:39:46] Sent data
[12:39:46] Connecting to http://171.64.65.54:8080/
[12:39:47] Posted data.
[12:39:47] Initial: 0000; - Receiving payload (expected size: 1768467)
[12:39:54] - Downloaded at ~246 kB/s
[12:39:54] - Averaged speed for that direction ~265 kB/s
[12:39:54] + Received work.
[12:39:54] + Closed connections
[12:39:54] 
[12:39:54] + Processing work unit
[12:39:54] Core required: FahCore_a3.exe
[12:39:54] Core found.
[12:39:54] Working on queue slot 07 [September 21 12:39:54 UTC]
[12:39:54] + Working ...
[12:39:54] - Calling './FahCore_a3.exe -dir work/ -nice 19 -suffix 07 -np 2 -checkpoint 5 -forceasm -verbose -lifeline 9857 -version 629'

[12:39:54] 
[12:39:54] *------------------------------*
[12:39:54] Folding@Home Gromacs SMP Core
[12:39:54] Version 2.21 (May 7 2010)
[12:39:54] 
[12:39:54] Preparing to commence simulation
[12:39:54] - Ensuring status. Please wait.
[12:40:03] - Assembly optimizations manually forced on.
[12:40:03] - Not checking prior termination.
[12:40:04] - Expanded 1767955 -> 2253305 (decompressed 127.4 percent)
[12:40:04] Called DecompressByteArray: compressed_data_size=1767955 data_size=2253305, decompressed_data_size=2253305 diff=0
[12:40:04] - Digital signature verified
[12:40:04] 
[12:40:04] Project: 6056 (Run 0, Clone 30, Gen 39)
[12:40:04] 
[12:40:04] Assembly optimizations on if available.
[12:40:04] Entering M.D.
[12:40:11] CoreStatus = 0 (0)
[12:40:11] Sending work to server
[12:40:11] Project: 6056 (Run 0, Clone 30, Gen 39)
[12:40:11] - Error: Could not get length of results file work/wuresults_07.dat
[12:40:11] - Error: Could not read unit 07 file. Removing from queue.
[12:40:11] Trying to send all finished work units
[12:40:11] + No unsent completed units remaining.
[12:40:11] + -oneunit flag given and have now finished a unit. Exiting.- Preparing to get new work unit...
[12:40:11] Cleaning up work directory
[12:40:11] ***** Got a SIGTERM signal (15)
[12:40:11] Killing all core threads

Folding@Home Client Shutdown.


--- Opening Log file [September 21 12:40:11 UTC] 


# Mac OS X SMP Console Edition ################################################
###############################################################################

                       Folding@Home Client Version 6.29r1

                          http://folding.stanford.edu

###############################################################################
###############################################################################

Launch directory: /Users/bas/software/folding
Executable: ./fah6
Arguments: -local -verbosity 9 -oneunit -smp 2 -forceasm 

[12:40:11] - Ask before connecting: No
[12:40:11] - User name: [DPC]_Fatal_Error_Group0JMT (Team 92)
[12:40:11] - User ID: 203E0C2C7390314C
[12:40:11] - Machine ID: 1
[12:40:11] 
[12:40:11] Loaded queue successfully.
[12:40:11] - Preparing to get new work unit...
[12:40:11] - Autosending finished units... [12:40:11]
[12:40:11] Cleaning up work directory
[12:40:11] Trying to send all finished work units
[12:40:11] + No unsent completed units remaining.
[12:40:11] - Autosend completed
[12:40:12] + Attempting to get work packet
[12:40:12] Passkey found
[12:40:12] - Will indicate memory of 1024 MB
[12:40:12] - Detect CPU. Vendor: GenuineIntel, Family: 6, Model: 14, Stepping: 8
[12:40:12] - Connecting to assignment server
[12:40:12] Connecting to http://assign.stanford.edu:8080/
[12:40:13] Posted data.
[12:40:13] Initial: 40AB; - Successful: assigned to (171.64.65.54).
[12:40:13] + News From Folding@Home: Welcome to Folding@Home
[12:40:13] Loaded queue successfully.
[12:40:13] Sent data
[12:40:13] Connecting to http://171.64.65.54:8080/
[12:40:14] Posted data.
[12:40:14] Initial: 0000; - Receiving payload (expected size: 1768467)
[12:40:20] - Downloaded at ~287 kB/s
[12:40:20] - Averaged speed for that direction ~269 kB/s
[12:40:20] + Received work.
[12:40:20] + Closed connections
[12:40:20] 
[12:40:20] + Processing work unit
[12:40:20] Core required: FahCore_a3.exe
[12:40:20] Core found.
[12:40:20] Working on queue slot 08 [September 21 12:40:20 UTC]
[12:40:20] + Working ...
[12:40:20] - Calling './FahCore_a3.exe -dir work/ -nice 19 -suffix 08 -np 2 -checkpoint 5 -forceasm -verbose -lifeline 9884 -version 629'

[12:40:21] 
[12:40:21] *------------------------------*
[12:40:21] Folding@Home Gromacs SMP Core
[12:40:21] Version 2.21 (May 7 2010)
[12:40:21] 
[12:40:21] Preparing to commence simulation
[12:40:21] - Ensuring status. Please wait.
[12:40:30] - Assembly optimizations manually forced on.
[12:40:30] - Not checking prior termination.
[12:40:30] - Expanded 1767955 -> 2253305 (decompressed 127.4 percent)
[12:40:30] Called DecompressByteArray: compressed_data_size=1767955 data_size=2253305, decompressed_data_size=2253305 diff=0
[12:40:30] - Digital signature verified
[12:40:30] 
[12:40:30] Project: 6056 (Run 0, Clone 30, Gen 39)
[12:40:30] 
[12:40:30] Assembly optimizations on if available.
[12:40:30] Entering M.D.
[12:40:38] CoreStatus = 0 (0)
[12:40:38] Sending work to server
[12:40:38] Project: 6056 (Run 0, Clone 30, Gen 39)
[12:40:38] - Error: Could not get length of results file work/wuresults_08.dat
[12:40:38] - Error: Could not read unit 08 file. Removing from queue.
[12:40:38] Trying to send all finished work units
[12:40:38] + No unsent completed units remaining.
[12:40:38] + -oneunit flag given and have now finished a unit. Exiting.- Preparing to get new work unit...
[12:40:38] Cleaning up work directory
[12:40:38] ***** Got a SIGTERM signal (15)
[12:40:38] Killing all core threads

Folding@Home Client Shutdown.


--- Opening Log file [September 21 12:40:39 UTC] 


# Mac OS X SMP Console Edition ################################################
###############################################################################

                       Folding@Home Client Version 6.29r1

                          http://folding.stanford.edu

###############################################################################
###############################################################################

Launch directory: /Users/bas/software/folding
Executable: ./fah6
Arguments: -local -verbosity 9 -oneunit -smp 2 -forceasm 

[12:40:39] - Ask before connecting: No
[12:40:39] - User name: [DPC]_Fatal_Error_Group0smoking2000 (Team 92)
[12:40:39] - User ID: 203E0C2C7390314C
[12:40:39] - Machine ID: 1
[12:40:39] 
[12:40:39] Loaded queue successfully.
[12:40:39] - Preparing to get new work unit...
[12:40:39] - Autosending finished units... [12:40:39]
[12:40:39] Cleaning up work directory
[12:40:39] Trying to send all finished work units
[12:40:39] + No unsent completed units remaining.
[12:40:39] - Autosend completed
[12:40:39] + Attempting to get work packet
[12:40:39] Passkey found
[12:40:39] - Will indicate memory of 1024 MB
[12:40:39] - Detect CPU. Vendor: GenuineIntel, Family: 6, Model: 14, Stepping: 8
[12:40:39] - Connecting to assignment server
[12:40:39] Connecting to http://assign.stanford.edu:8080/
[12:40:40] Posted data.
[12:40:40] Initial: 40AB; - Successful: assigned to (171.64.65.54).
[12:40:40] + News From Folding@Home: Welcome to Folding@Home
[12:40:40] Loaded queue successfully.
[12:40:40] Sent data
[12:40:40] Connecting to http://171.64.65.54:8080/
[12:40:42] Posted data.
[12:40:42] Initial: 0000; - Receiving payload (expected size: 1768467)
[12:40:48] - Downloaded at ~287 kB/s
[12:40:48] - Averaged speed for that direction ~273 kB/s
[12:40:48] + Received work.
[12:40:48] + Closed connections
[12:40:48] 
[12:40:48] + Processing work unit
[12:40:48] Core required: FahCore_a3.exe
[12:40:48] Core found.
[12:40:48] Working on queue slot 09 [September 21 12:40:48 UTC]
[12:40:48] + Working ...
[12:40:48] - Calling './FahCore_a3.exe -dir work/ -nice 19 -suffix 09 -np 2 -checkpoint 5 -forceasm -verbose -lifeline 9902 -version 629'

[12:40:49] 
[12:40:49] *------------------------------*
[12:40:49] Folding@Home Gromacs SMP Core
[12:40:49] Version 2.21 (May 7 2010)
[12:40:49] 
[12:40:49] Preparing to commence simulation
[12:40:49] - Ensuring status. Please wait.
[12:40:58] - Assembly optimizations manually forced on.
[12:40:59] - Not checking prior termination.
[12:40:59] - Expanded 1767955 -> 2253305 (decompressed 127.4 percent)
[12:40:59] Called DecompressByteArray: compressed_data_size=1767955 data_size=2253305, decompressed_data_size=2253305 diff=0
[12:40:59] - Digital signature verified
[12:40:59] 
[12:40:59] Project: 6056 (Run 0, Clone 30, Gen 39)
[12:40:59] 
[12:40:59] Assembly optimizations on if available.
[12:40:59] Entering M.D.
[12:41:05] CoreStatus = 0 (0)
[12:41:05] Sending work to server
[12:41:05] Project: 6056 (Run 0, Clone 30, Gen 39)
[12:41:05] - Error: Could not get length of results file work/wuresults_09.dat
[12:41:05] - Error: Could not read unit 09 file. Removing from queue.
[12:41:06] Trying to send all finished work units
[12:41:06] + No unsent completed units remaining.
[12:41:06] + -oneunit flag given and have now finished a unit. Exiting.- Preparing to get new work unit...
[12:41:06] Cleaning up work directory
[12:41:06] ***** Got a SIGTERM signal (15)
[12:41:06] Killing all core threads

Folding@Home Client Shutdown.

[ Voor 6% gewijzigd door smoking2000 op 21-09-2010 15:46 . Reden: Files ]

[SandStar]

Thanks!

Het is trouwens een probleem wat wellicht bij meerdere cores voorkomt. Momenteel heb ik het met 78, 79 en AMBER gezien op V7

[SandStar]

Waarschijnlijk fixed nu
Thanks for help!

[smoking2000]

SandStar schreef op dinsdag 28 september 2010 @ 10:45:
Waarschijnlijk fixed nu

Wat is er gefixt? Alleen de v7 client, of is er ook iets aan de cores gewijzigt (die wij niet-v7 users ook draaien)?

Ik heb namelijk nog elke completed WU last van dit issue op m'n OSX Tiger box. Maar ik heb de hoop op een fix alweer verloren. Zie ook het draadje op FF.org

[SandStar]

smoking2000 schreef op dinsdag 28 september 2010 @ 12:03:
[...]

Wat is er gefixt? Alleen de v7 client, of is er ook iets aan de cores gewijzigt (die wij niet-v7 users ook draaien)?

Ik heb namelijk nog elke completed WU last van dit issue op m'n OSX Tiger box. Maar ik heb de hoop op een fix alweer verloren. Zie ook het draadje op FF.org

Alleen aan de v7 client. Bug met file inheriting.

Jouw probleem lijkt aan iets anders te liggen.
Maar je logs zien er wel mega vaag uit.

code:

[09:39:08] + -oneunit flag given and have now finished a unit. Exiting.- Preparing to get new work unit...
[09:39:08] Cleaning up work directory
[09:39:08] + Attempting to get work packet
[09:39:08] Passkey found
[09:39:08] ***** Got a SIGTERM signal (15)
[09:39:08] - Will indicate memory of 1024 MB
[09:39:08] Killing all core threads
[09:39:08] - Connecting to assignment server

Folding@Home Client Shutdown.
[09:39:08] Connecting to http://assign.stanford.edu:8080/

Client stopt en wil tegelijkertijd een nieuwe unit downloaden??
En de restarts van je client lijken ook behoorlijk snel na elkaar te komen.
Nu draai ik zelf geen OSX maar op Linux heeft de A3 core 1min nodig om af te sluiten...

[smoking2000]

SandStar schreef op dinsdag 28 september 2010 @ 13:32:
Jouw probleem lijkt aan iets anders te liggen.
Maar je logs zien er wel mega vaag uit.

[...]

Client stopt en wil tegelijkertijd een nieuwe unit downloaden??
En de restarts van je client lijken ook behoorlijk snel na elkaar te komen.
Nu draai ik zelf geen OSX maar op Linux heeft de A3 core 1min nodig om af te sluiten...

Ik draai al mijn High Performance clients via random-folder, welke de FAH client na elke completed WU herconfigt voor een andere username & team combi. Vandaar dat direct na het afsluiten van een -oneunit sessie een nieuwe word gestart.

Het random-folder script kijkt of alle processen van de gestarte FAH client gestopt zijn, dus niet alleen de fah6 zelf, maar ook de FahCore_XX.exe, pas als deze niet meer draaien zal het de client herconfiggen en opnieuw starten.

De client en cores zijn dus zeker te weten al wel gestopt als random-folder de FAH client aan een nieuwe WU laat beginnen.

Voorbeeld uit mijn logs rond de tijd van de laatste 2 deleted WUs:

[2010-09-27 11:34:25] Rotating logfile if required: ./random-folder.log
[2010-09-27 11:34:25] Current filesize: 792.33 KB
[2010-09-27 11:34:25] Not rotating because size < 1 MB
[2010-09-27 11:34:25]
[2010-09-27 11:34:25] Checking WU status & progress ...
[2010-09-27 11:34:25] Status  : folding now
[2010-09-27 11:34:25] Progress: 99%
[2010-09-27 11:34:25] Expected: Mon Sep 27 11:37:41 2010
[2010-09-27 11:34:25] WU is not finished yet.
[2010-09-27 11:34:25] Waiting for 5 minutes before polling again.

[2010-09-27 11:38:50] Child PID: 13141
[2010-09-27 11:38:50] FAH client exited with status: -1


[2010-09-27 11:38:50] Rotating logfile if required: ./random-folder.log
[2010-09-27 11:38:50] Current filesize: 792.92 KB
[2010-09-27 11:38:50] Not rotating because size < 1 MB
[2010-09-27 11:38:50]
[2010-09-27 11:38:50] Checking WU status & progress ...
[2010-09-27 11:38:50] Status  : finished
[2010-09-27 11:38:50] Finished: Mon Sep 27 11:38:03 2010
[2010-09-27 11:38:50] WU has finished!
[2010-09-27 11:38:50] Checking if process is running: child process 13141
[2010-09-27 11:38:50] No, process is not running
[2010-09-27 11:38:50] Checking if process is running: fah6
[2010-09-27 11:38:51] No, process is not running
[2010-09-27 11:38:51] Checking if process is running: mpiexec
[2010-09-27 11:38:51] No, process is not running
[2010-09-27 11:38:51] Checking if process is running: FahCore_.*.exe
[2010-09-27 11:38:51] No, process is not running
[2010-09-27 11:38:51]
[2010-09-27 11:38:51] Reconfiguring FAH client.
[2010-09-27 11:38:51]
[2010-09-27 11:38:51] Account pool:
[2010-09-27 11:38:51]  ID      : 0
[2010-09-27 11:38:51]  Username: [DPC]_Fatal_Error_Group0smoking2000
[2010-09-27 11:38:51]  Team    : 92
[2010-09-27 11:38:51]  Weight  : 1
[2010-09-27 11:38:51]
[2010-09-27 11:38:51]  ID      : 1
[2010-09-27 11:38:51]  Username: [DPC]_Fatal_Error_Group0JMT
[2010-09-27 11:38:51]  Team    : 92
[2010-09-27 11:38:51]  Weight  : 1
[2010-09-27 11:38:51]
[2010-09-27 11:38:51] Random account: 1
[2010-09-27 11:38:51]       Username: [DPC]_Fatal_Error_Group0JMT
[2010-09-27 11:38:51]           Team: 92
[2010-09-27 11:38:51]
[2010-09-27 11:38:51] Removing account from the pool.
[2010-09-27 11:38:51]
[2010-09-27 11:38:51] Modifying client.cfg
[2010-09-27 11:38:51]
[2010-09-27 11:38:51] Forking...

[2010-09-27 11:38:51] Child PID: 29208
[2010-09-27 11:38:51] Starting FAH client: ./fah6 -local -verbosity 9 -oneunit -smp 2 -forceasm

[2010-09-27 11:38:52] Parent PID: 7632
[2010-09-27 11:38:52] Waiting for 1 minutes before polling again.

[2010-09-27 11:39:08] Child PID: 29208
[2010-09-27 11:39:08] FAH client exited with status: -1


[2010-09-27 11:39:08] Rotating logfile if required: ./random-folder.log
[2010-09-27 11:39:08] Current filesize: 794.93 KB
[2010-09-27 11:39:08] Not rotating because size < 1 MB
[2010-09-27 11:39:08]
[2010-09-27 11:39:08] Checking WU status & progress ...
[2010-09-27 11:39:09] Status  : deleted
[2010-09-27 11:39:09] Finished: Mon Sep 27 11:39:08 2010
[2010-09-27 11:39:09] WU was deleted!
[2010-09-27 11:39:09] Checking if process is running: child process 29208
[2010-09-27 11:39:09] No, process is not running
[2010-09-27 11:39:09] Checking if process is running: fah6
[2010-09-27 11:39:09] No, process is not running
[2010-09-27 11:39:09] Checking if process is running: mpiexec
[2010-09-27 11:39:09] No, process is not running
[2010-09-27 11:39:09] Checking if process is running: FahCore_.*.exe
[2010-09-27 11:39:09] No, process is not running
[2010-09-27 11:39:09]
[2010-09-27 11:39:09] Reconfiguring FAH client.
[2010-09-27 11:39:09]
[2010-09-27 11:39:09] Account pool:
[2010-09-27 11:39:09]  ID      : 0
[2010-09-27 11:39:09]  Username: [DPC]_Fatal_Error_Group0smoking2000
[2010-09-27 11:39:09]  Team    : 92
[2010-09-27 11:39:09]  Weight  : 1
[2010-09-27 11:39:09]
[2010-09-27 11:39:09] Random account: 0
[2010-09-27 11:39:09]       Username: [DPC]_Fatal_Error_Group0smoking2000
[2010-09-27 11:39:09]           Team: 92
[2010-09-27 11:39:09]
[2010-09-27 11:39:09] Removing account from the pool.
[2010-09-27 11:39:09] Account pool is now empty, resetting...
[2010-09-27 11:39:09]
[2010-09-27 11:39:10] Randomly selected account statistics:
[2010-09-27 11:39:10]
[2010-09-27 11:39:10]  [DPC]_Fatal_Error_Group0JMT (92)         : 125 times  41.67%
[2010-09-27 11:39:10]  [DPC]_Fatal_Error_Group0smoking2000 (92) : 121 times  40.33%
[2010-09-27 11:39:10]  [DPC]_Fatal_Error_Group0WolterB (92)     : 54 times  18.00%
[2010-09-27 11:39:10]  ----------------------------------------------------------- +
[2010-09-27 11:39:10]                                     Total : 300 selections
[2010-09-27 11:39:10]
[2010-09-27 11:39:10] Modifying client.cfg
[2010-09-27 11:39:10]
[2010-09-27 11:39:10] Forking...

[2010-09-27 11:39:10] Child PID: 29226
[2010-09-27 11:39:10] Starting FAH client: ./fah6 -local -verbosity 9 -oneunit -smp 2 -forceasm

[2010-09-27 11:39:11] Parent PID: 7632
[2010-09-27 11:39:11] Waiting for 1 minutes before polling again.

[2010-09-27 11:39:35] Child PID: 29226
[2010-09-27 11:39:35] FAH client exited with status: -1


[2010-09-27 11:39:35] Rotating logfile if required: ./random-folder.log
[2010-09-27 11:39:35] Current filesize: 797.33 KB
[2010-09-27 11:39:35] Not rotating because size < 1 MB
[2010-09-27 11:39:35]
[2010-09-27 11:39:35] Checking WU status & progress ...
[2010-09-27 11:39:36] Status  : deleted
[2010-09-27 11:39:36] Finished: Mon Sep 27 11:39:35 2010
[2010-09-27 11:39:36] WU was deleted!
[2010-09-27 11:39:36] Checking if process is running: child process 29226
[2010-09-27 11:39:36] No, process is not running
[2010-09-27 11:39:36] Checking if process is running: fah6
[2010-09-27 11:39:36] No, process is not running
[2010-09-27 11:39:36] Checking if process is running: mpiexec
[2010-09-27 11:39:36] No, process is not running
[2010-09-27 11:39:36] Checking if process is running: FahCore_.*.exe
[2010-09-27 11:39:36] No, process is not running
[2010-09-27 11:39:36]
[2010-09-27 11:39:36] Reconfiguring FAH client.
[2010-09-27 11:39:36]
[2010-09-27 11:39:36] Account pool:
[2010-09-27 11:39:36]  ID      : 0
[2010-09-27 11:39:36]  Username: [DPC]_Fatal_Error_Group0smoking2000
[2010-09-27 11:39:36]  Team    : 92
[2010-09-27 11:39:36]  Weight  : 1
[2010-09-27 11:39:36]
[2010-09-27 11:39:36]  ID      : 1
[2010-09-27 11:39:36]  Username: [DPC]_Fatal_Error_Group0JMT
[2010-09-27 11:39:36]  Team    : 92
[2010-09-27 11:39:36]  Weight  : 1
[2010-09-27 11:39:36]
[2010-09-27 11:39:36] Random account: 0
[2010-09-27 11:39:36]       Username: [DPC]_Fatal_Error_Group0smoking2000
[2010-09-27 11:39:36]           Team: 92
[2010-09-27 11:39:36]
[2010-09-27 11:39:36] Removing account from the pool.
[2010-09-27 11:39:36]
[2010-09-27 11:39:36] Modifying client.cfg
[2010-09-27 11:39:36]
[2010-09-27 11:39:36] Forking...

[2010-09-27 11:39:36] Child PID: 29251
[2010-09-27 11:39:36] Starting FAH client: ./fah6 -local -verbosity 9 -oneunit -smp 2 -forceasm

[2010-09-27 11:39:37] Parent PID: 7632
[2010-09-27 11:39:37] Waiting for 1 minutes before polling again.
[2010-09-27 11:40:37] Rotating logfile if required: ./random-folder.log
[2010-09-27 11:40:37] Current filesize: 799.25 KB
[2010-09-27 11:40:37] Not rotating because size < 1 MB
[2010-09-27 11:40:37]
[2010-09-27 11:40:37] Checking WU status & progress ...
[2010-09-27 11:40:37] Status  : folding now
[2010-09-27 11:40:37] WU is not finished yet.
[2010-09-27 11:40:37] Waiting for 60 minutes before polling again.

[SandStar]

mja, duidelijk een ander probleem. Heb em ook voorbij zien komen in ontwikkeldraad van nieuwe OSX core. Men kwam er alleen niet achter wat nu het probleem is.

Laten we hopen dat V7 wat verbetering kan brengen

[smoking2000]

v7 wordt v6 al over again. Hopen dat de known issues daar eigenlijk mee gefixt worden. Een beetje jammer.

De crash logs geven overigens wel een idee waar het probleem ligt, in de interactie met de OSX libc. Waar hebben we dat eerder gezien?

[SandStar]

smoking2000 schreef op dinsdag 28 september 2010 @ 15:39:
v7 wordt v6 al over again. Hopen dat de known issues daar eigenlijk mee gefixt worden. Een beetje jammer.

De crash logs geven overigens wel een idee waar het probleem ligt, in de interactie met de OSX libc. Waar hebben we dat eerder gezien?

Wat dat die issue met ubuntu systemen?

[smoking2000]

Ik weet dat het eglibc issue inmiddels gefixt is in de v7client, het gaat meer om zaken als de broken service install op Windows, en de "pause if app XYZ is running" feature waar al eeuwen omgevraagt wordt, en natuurlijk dit OSX issues.

Het zou allemaal goed te doen moeten zijn, en hadden eigenlijk tijdens geleden al in v6 gefixt moeten worden, maar de heerlijke prioriteiten van PG weten dat altijd weer te verhinderen.

[SandStar]

ach ja, V7 lijkt iig een leuke verbetering te zijn. En er blijkt best goed te praten te zijn met de dev daarvan.

Ik zal eens OSX in virtualbox installeren en kijken of we wat kunnen reproduceren & fixen.

Nog even terugkomend op dat stukje log wat ik quote: dat speelt zich toch allemaal af in dezelfde seconde en voordat ie afsluit is ie toch echt nieuw werk aan het ophalen.

[smoking2000]

SandStar schreef op woensdag 29 september 2010 @ 09:59:
Nog even terugkomend op dat stukje log wat ik quote: dat speelt zich toch allemaal af in dezelfde seconde en voordat ie afsluit is ie toch echt nieuw werk aan het ophalen.

Hij logt dat hij dat wil gaan doen, ik durf niet met zekerheid te zeggen of hij ook echt die request maakt.

Ik zal eens een tcpdump opzetten om dat te checken.

edit:

Zojuist weer een WU completed, met daarna een paar deleted WUs bij het starten van een nieuwe, maar nergens probeerd de FAH client te connecten met de assignment servers voordat ie stopt vanwege -oneunit

Bij de WU failures gaat de FahCore_a3.exe op zn bek met volgende errors in de crash.log:

Exception:  EXC_BAD_ACCESS (0x0001)
Codes:      KERN_INVALID_ADDRESS (0x0001) at 0x0264a000

[ Voor 31% gewijzigd door smoking2000 op 29-09-2010 10:43 ]