Page 1 of 1
What causes a job to be flagged as errored?
Posted: Fri Jan 27, 2017 11:19 pm
by keyframe
I have a mayatovray job that errors out even though the .vrscene files have all written correctly.
I've checked the mayatovray.py parser, as well as the parser.py parent class, but I really don't feel that it would trip the error based on the output of the logs.
When I attempt to run the command locally, the process returns 0 on exit, and seems to otherwise exit fine.
Is there anywhere else I should be looking?
G
Re: What causes a job to be flagged as errored?
Posted: Sat Jan 28, 2017 12:35 am
by timurhai
Can you copy&paste here task "Log" and "Output".
Re: What causes a job to be flagged as errored?
Posted: Sat Jan 28, 2017 8:28 am
by timurhai
btw
New forum "General" in Afanasy forums created, and this topic moved here.
Re: What causes a job to be flagged as errored?
Posted: Fri Feb 03, 2017 5:07 pm
by keyframe
Here is the output:
Code: Select all
MAYA: /apps/linux/autodesk/maya2016_sp4/bin/maya
MAYA_VERSION: 2016_sp4
Starting "/apps/linux/autodesk/maya2016_sp4/bin/maya"
[2017/Feb/3|12:04:17] V-Ray: V-Ray for Maya version 3.30.01 from Feb 25 2016, 23:02:38
[2017/Feb/3|12:04:17] V-Ray: V-Ray core version is 3.25.01
Initializing V-Ray for Maya
[2017/Feb/3|12:04:17] V-Ray: Loading plugins from "/apps/linux/chaos/3.3.001/maya_vray/vrayplugins/libvray_*.so"
[2017/Feb/3|12:04:17] V-Ray: 73 plugin(s) loaded successfully
[2017/Feb/3|12:04:17] V-Ray: Finished loading plugins.
[2017/Feb/3|12:04:18] V-Ray: Registered dynamic node vrayMultiProjection (id: 1115335/0x001104C7)
[2017/Feb/3|12:04:18] V-Ray: Registered dynamic node VRayCurvature (id: 1115347/0x001104D3)
[2017/Feb/3|12:04:18] V-Ray: Registered dynamic node VRayTriplanar (id: 1115362/0x001104E2)
[2017/Feb/3|12:04:18] V-Ray: Registered dynamic node VRaySkinMtl (id: 1115348/0x001104D4)
[2017/Feb/3|12:04:18] V-Ray: Registered dynamic node VRayPointParticleMtl (id: 1115353/0x001104D9)
[2017/Feb/3|12:04:18] V-Ray: Registered dynamic node VRayStochasticFlakesMtl (id: 1115359/0x001104DF)
Error: file: /Volumes/qumulo/000000_vfx_testing/shots/g16/scenes/vray_2_deferred.ma line 8: line 1: Cannot find procedure "shaderBallRendererMenuUpdate".
File read in 1.3 seconds.
Result: /Volumes/qumulo/000000_vfx_testing/shots/g16/scenes/vray_2_deferred.ma
Warning: file: /apps/linux/autodesk/maya2016_sp4/scripts/others/makeCameraRenderable.mel line 60: Found camera perspShape.
V-Ray: Starting render
[2017/Feb/3|12:04:18] V-Ray: Exporting scene to V-Ray.
[2017/Feb/3|12:04:18] V-Ray: Parsing light links time 0h 0m 0.0s (0.0 s)
[2017/Feb/3|12:04:18] V-Ray: Translating scene geometry for V-Ray
[2017/Feb/3|12:04:18] V-Ray: Total time translating scene for V-Ray 0h 0m 0.0s (0.0 s)
[2017/Feb/3|12:04:18] V-Ray: Rendering.
[2017/Feb/3|12:04:18] V-Ray: Clearing bitmap cache memory...
[2017/Feb/3|12:04:18] V-Ray: Updating frame at time 1.00
[2017/Feb/3|12:04:18] V-Ray: Total time updating frame 0h 0m 0.0s (0.0 s)
[2017/Feb/3|12:04:18] V-Ray: Writing to V-Ray scene file.
[2017/Feb/3|12:04:18] V-Ray: Writing plugins..
[2017/Feb/3|12:04:18] V-Ray: File "/Volumes/qumulo/000000_vfx_testing/shots/g16/vrscenes/vray_2_0001.vrscene" exported
[2017/Feb/3|12:04:18] V-Ray: Total frame time 0h 0m 0.0s (0.0 s)
[2017/Feb/3|12:04:18] V-Ray: Total sequence time 0h 0m 0.0s (0.0 s)
[2017/Feb/3|12:04:18] V-Ray: Render complete
[2017/Feb/3|12:04:18] V-Ray: ========================
[2017/Feb/3|12:04:18] V-Ray: Clearing exporter memory...
[2017/Feb/3|12:04:18] V-Ray: Total time clearing exporter memory 0h 0m 0.0s (0.0 s)
Scene /Volumes/qumulo/000000_vfx_testing/shots/g16/scenes/vray_2_deferred.ma completed.
And the log:
Code: Select all
Fri 03 Feb 12:04.04 : SESSION #1: Starting on "lws13"
Fri 03 Feb 12:04.09 : Killed.
Fri 03 Feb 12:04.09 : Automatically retrying error task1 of 3.
Fri 03 Feb 12:04.09 : SESSION #2: Starting on "lws13"
Fri 03 Feb 12:04.14 : Killed.
Fri 03 Feb 12:04.14 : Automatically retrying error task2 of 3.
Fri 03 Feb 12:04.14 : SESSION #3: Starting on "lws13"
Fri 03 Feb 12:04.19 : Killed.
Fri 03 Feb 12:04.19 : lws13 - AVOIDING HOST !
Fri 03 Feb 12:04.19 : Automatically retrying error task3 of 3.
Re: What causes a job to be flagged as errored?
Posted: Fri Feb 03, 2017 7:54 pm
by keyframe
Upon investigating further, I was wrong my my assumption that it wasn't tripping the parser.
While the mayatovray parser isn't tripped, it seems that the error is flagged by the parent class parser.py.
line 109:
Code: Select all
for string in self.str_error:
if lower.find(string.lower()) != -1:
self.error = True
This will capture every instance of the word error it seems -- whether critical or not.
Will have to think about this...
G
Re: What causes a job to be flagged as errored?
Posted: Sat Feb 04, 2017 3:46 pm
by keyframe
Timur, would you be open to the idea of also checking the process exit code in conjunction with parsing the logs to flag errors?
G
Re: What causes a job to be flagged as errored?
Posted: Sun Feb 05, 2017 11:09 am
by timurhai
Hi.
Afanasy checks task command exit code.
And by default for success finish it should be 0, but can be customized
in a Python service "checkExitStatus" function:
https://github.com/CGRU/cgru/blob/maste ... ce.py#L234
Also by default it search error key stings defined in "self.str_error" array (parser member).
And since 2.2.1 this array is empty by default.
Before 2.2.1 the default array was "self.str_error = ['error']".
This was done some software can output statistics info on task end, for example:
"Errors: 0; Warnings: 0" - means success, but 'error' string exists.
Re: What causes a job to be flagged as errored?
Posted: Tue Feb 07, 2017 2:53 pm
by keyframe
So after all this noise... I was completely off track.
After checking the logs in /var/tmp I realized that what was crashing was the convert post-process (for thumbnails), and not the actual mayatovray job.
I've updated ImageMagick to the current version, and the crashes went away.
What tipped me off was that Nuke jobs were erroring in the same way.
Again, apologies for the noise.
G