What causes a job to be flagged as errored?

General discussions and questions.
Post Reply
keyframe
Posts: 62
Joined: Sat Jan 21, 2017 9:43 pm
Location: Toronto

What causes a job to be flagged as errored?

Post by keyframe »

I have a mayatovray job that errors out even though the .vrscene files have all written correctly.

I've checked the mayatovray.py parser, as well as the parser.py parent class, but I really don't feel that it would trip the error based on the output of the logs.
When I attempt to run the command locally, the process returns 0 on exit, and seems to otherwise exit fine.

Is there anywhere else I should be looking?

G
--
Rocky Linux 8.5, cgru 3.2.1
User avatar
timurhai
Site Admin
Posts: 911
Joined: Sun Jan 15, 2017 8:40 pm
Location: Russia, Korolev
Contact:

Re: What causes a job to be flagged as errored?

Post by timurhai »

Can you copy&paste here task "Log" and "Output".
Timur Hairulin
CGRU 3.3.1, Ubuntu 20.04, 22.04, MS Windows 10 (clients only).
User avatar
timurhai
Site Admin
Posts: 911
Joined: Sun Jan 15, 2017 8:40 pm
Location: Russia, Korolev
Contact:

Re: What causes a job to be flagged as errored?

Post by timurhai »

btw
New forum "General" in Afanasy forums created, and this topic moved here.
Timur Hairulin
CGRU 3.3.1, Ubuntu 20.04, 22.04, MS Windows 10 (clients only).
keyframe
Posts: 62
Joined: Sat Jan 21, 2017 9:43 pm
Location: Toronto

Re: What causes a job to be flagged as errored?

Post by keyframe »

Here is the output:

Code: Select all

MAYA: /apps/linux/autodesk/maya2016_sp4/bin/maya
MAYA_VERSION: 2016_sp4

Starting "/apps/linux/autodesk/maya2016_sp4/bin/maya"

[2017/Feb/3|12:04:17] V-Ray: V-Ray for Maya version 3.30.01 from Feb 25 2016, 23:02:38
[2017/Feb/3|12:04:17] V-Ray: V-Ray core version is 3.25.01
Initializing V-Ray for Maya

[2017/Feb/3|12:04:17] V-Ray: Loading plugins from "/apps/linux/chaos/3.3.001/maya_vray/vrayplugins/libvray_*.so"
[2017/Feb/3|12:04:17] V-Ray: 73 plugin(s) loaded successfully
[2017/Feb/3|12:04:17] V-Ray: Finished loading plugins.
[2017/Feb/3|12:04:18] V-Ray: Registered dynamic node vrayMultiProjection (id: 1115335/0x001104C7)
[2017/Feb/3|12:04:18] V-Ray: Registered dynamic node VRayCurvature (id: 1115347/0x001104D3)
[2017/Feb/3|12:04:18] V-Ray: Registered dynamic node VRayTriplanar (id: 1115362/0x001104E2)
[2017/Feb/3|12:04:18] V-Ray: Registered dynamic node VRaySkinMtl (id: 1115348/0x001104D4)
[2017/Feb/3|12:04:18] V-Ray: Registered dynamic node VRayPointParticleMtl (id: 1115353/0x001104D9)
[2017/Feb/3|12:04:18] V-Ray: Registered dynamic node VRayStochasticFlakesMtl (id: 1115359/0x001104DF)
Error: file: /Volumes/qumulo/000000_vfx_testing/shots/g16/scenes/vray_2_deferred.ma line 8: line 1: Cannot find procedure "shaderBallRendererMenuUpdate".
File read in  1.3 seconds.
Result: /Volumes/qumulo/000000_vfx_testing/shots/g16/scenes/vray_2_deferred.ma
Warning: file: /apps/linux/autodesk/maya2016_sp4/scripts/others/makeCameraRenderable.mel line 60: Found camera perspShape.

V-Ray: Starting render
[2017/Feb/3|12:04:18] V-Ray: Exporting scene to V-Ray.
[2017/Feb/3|12:04:18] V-Ray: Parsing light links time  0h  0m  0.0s (0.0 s)
[2017/Feb/3|12:04:18] V-Ray: Translating scene geometry for V-Ray
[2017/Feb/3|12:04:18] V-Ray: Total time translating scene for V-Ray  0h  0m  0.0s (0.0 s)
[2017/Feb/3|12:04:18] V-Ray: Rendering.
[2017/Feb/3|12:04:18] V-Ray: Clearing bitmap cache memory...
[2017/Feb/3|12:04:18] V-Ray: Updating frame at time 1.00
[2017/Feb/3|12:04:18] V-Ray: Total time updating frame  0h  0m  0.0s (0.0 s)
[2017/Feb/3|12:04:18] V-Ray: Writing to V-Ray scene file.
[2017/Feb/3|12:04:18] V-Ray: Writing plugins..
[2017/Feb/3|12:04:18] V-Ray: File "/Volumes/qumulo/000000_vfx_testing/shots/g16/vrscenes/vray_2_0001.vrscene" exported
[2017/Feb/3|12:04:18] V-Ray: Total frame time  0h  0m  0.0s (0.0 s)
[2017/Feb/3|12:04:18] V-Ray: Total sequence time  0h  0m  0.0s (0.0 s)
[2017/Feb/3|12:04:18] V-Ray: Render complete
[2017/Feb/3|12:04:18] V-Ray: ========================
[2017/Feb/3|12:04:18] V-Ray: Clearing exporter memory...
[2017/Feb/3|12:04:18] V-Ray: Total time clearing exporter memory  0h  0m  0.0s (0.0 s)
Scene /Volumes/qumulo/000000_vfx_testing/shots/g16/scenes/vray_2_deferred.ma completed.
And the log:

Code: Select all

Fri 03 Feb 12:04.04 : SESSION #1: Starting on "lws13"
Fri 03 Feb 12:04.09 : Killed.
Fri 03 Feb 12:04.09 : Automatically retrying error task1 of 3.
Fri 03 Feb 12:04.09 : SESSION #2: Starting on "lws13"
Fri 03 Feb 12:04.14 : Killed.
Fri 03 Feb 12:04.14 : Automatically retrying error task2 of 3.
Fri 03 Feb 12:04.14 : SESSION #3: Starting on "lws13"
Fri 03 Feb 12:04.19 : Killed.
Fri 03 Feb 12:04.19 : lws13 - AVOIDING HOST !
Fri 03 Feb 12:04.19 : Automatically retrying error task3 of 3.
--
Rocky Linux 8.5, cgru 3.2.1
keyframe
Posts: 62
Joined: Sat Jan 21, 2017 9:43 pm
Location: Toronto

Re: What causes a job to be flagged as errored?

Post by keyframe »

Upon investigating further, I was wrong my my assumption that it wasn't tripping the parser.

While the mayatovray parser isn't tripped, it seems that the error is flagged by the parent class parser.py.

line 109:

Code: Select all

        for string in self.str_error:
            if lower.find(string.lower()) != -1:
                self.error = True
This will capture every instance of the word error it seems -- whether critical or not.

Will have to think about this...

G
--
Rocky Linux 8.5, cgru 3.2.1
keyframe
Posts: 62
Joined: Sat Jan 21, 2017 9:43 pm
Location: Toronto

Re: What causes a job to be flagged as errored?

Post by keyframe »

Timur, would you be open to the idea of also checking the process exit code in conjunction with parsing the logs to flag errors?

G
--
Rocky Linux 8.5, cgru 3.2.1
User avatar
timurhai
Site Admin
Posts: 911
Joined: Sun Jan 15, 2017 8:40 pm
Location: Russia, Korolev
Contact:

Re: What causes a job to be flagged as errored?

Post by timurhai »

Hi.
Afanasy checks task command exit code.
And by default for success finish it should be 0, but can be customized
in a Python service "checkExitStatus" function:
https://github.com/CGRU/cgru/blob/maste ... ce.py#L234

Also by default it search error key stings defined in "self.str_error" array (parser member).
And since 2.2.1 this array is empty by default.
Before 2.2.1 the default array was "self.str_error = ['error']".

This was done some software can output statistics info on task end, for example:
"Errors: 0; Warnings: 0" - means success, but 'error' string exists.
Timur Hairulin
CGRU 3.3.1, Ubuntu 20.04, 22.04, MS Windows 10 (clients only).
keyframe
Posts: 62
Joined: Sat Jan 21, 2017 9:43 pm
Location: Toronto

Re: What causes a job to be flagged as errored?

Post by keyframe »

So after all this noise... I was completely off track.

After checking the logs in /var/tmp I realized that what was crashing was the convert post-process (for thumbnails), and not the actual mayatovray job.
I've updated ImageMagick to the current version, and the crashes went away.

What tipped me off was that Nuke jobs were erroring in the same way.

Again, apologies for the noise.

G
--
Rocky Linux 8.5, cgru 3.2.1
Post Reply