Page 1 of 2

Arnold hangs every type of job

Posted: Thu Feb 15, 2018 5:22 pm
by arengorman
Hello!

I'm trying to use arnold with afanasy. Af version is 2.2.2, HtoA is 2.2.2 too, houdini is 16.5.323, OS is linux

As it was done in hou 15, I try to put the following in the setup_houdini.sh script:

Code: Select all

export HOUDINI_PATH="/studio/tools/arnold/arnold_htoa-2.2.2_hou_16.5.323/htoa-2.2.2_ra19eea7_houdini-16.5.323;&"
export solidangle_LICENSE=5053@licserver
After that every type of job, submitted from Houdini, starts to freeze in the very beginning, without any errors or useful output in logs. Just 0% progress forever. No matter if I submitted Mantra rop or Arnold's one.
if I comment the string with HOUDINI_PATH, everything goes OK, but Arnold of course, won't work.
On a local machine these two strings work well in the env setup file, in Afanasy environment they don't.
What am I doing wrong? This method used to work with previous versions of HtoA.

At least any advice on debugging this issue? Verbose levels in ROP settings don't work, jobs seem to freeze before coming up to it.

Re: Arnold hangs every type of job

Posted: Sun Feb 18, 2018 8:34 pm
by timurhai
Hi.
Better to debug such errors w/o afanasy (afrender).
You can run command that afrender run simple copy&paste it in a console.
To see (copy) task command you can double-click job, that double click its task in a GUI.

ps
Better to have and support a single environment.

Re: Arnold hangs every type of job

Posted: Fri Mar 16, 2018 4:53 pm
by arengorman
Hi again.

Thanks for your advice. I did as you said - copied command from task window, launched it in the same environment and everything worked.
I still don't even know what is wrong. And I can't understand also why mantra and bgeo cache tasks behave like arnold tasks (freezes after houdini launch).

I thought it can be a problem of a parser. Nevertheless a quick look at parsers classes in afanasy folder showed me that parsers are OK, it's just functions which look at output and find XX% pattern.

Any more advice? Please...

Re: Arnold hangs every type of job

Posted: Sun Mar 18, 2018 11:56 am
by timurhai
Hi.
Parser can cause a problem, if there is some error in its Python code.
Try to look at afrender process output during such task execution.
To watch afrender output real-time, you can open a console, source cgru environment and run "afrender" command.

Re: Arnold hangs every type of job

Posted: Mon Mar 19, 2018 2:02 pm
by arengorman
Unfortunately, nothing suspicious...

Code: Select all

Mon 19 Mar 16:56.17: INFO    Render registered.
Mon 19 Mar 16:56.35: INFO    Started PID=7676 [houdinitoarnold:1000] rum: arnold2.afanasy1[arnold1][frame 0]
That is what output tab in job details says:

Code: Select all

HOSTNAME=fx08
/studio/tools/hou/hfs16.5
The Houdini 16.5.323 environment has been initialized.
HOUDINI_PATH is set to /studio/tools/arnold/arnold_htoa-2.2.2_hou_16.5.323/htoa-2.2.2_ra19eea7_houdini-16.5.323;&
Houdni otls = '/home/farm/houdini16.5/otls:/studio/tools/af_2.2.2/cgru/plugins/houdini:/studio/tools/hou/hfs16.5/houdini/otls:/studio/tools/arnold/arnold_htoa-2.2.2_hou_16.5.323/htoa-2.2.2_ra19eea7_houdini-16.5.323;&/otls'
LaunchProgram: chdir(): : Permission denied
00:00:00 212MB | log started Mon Mar 19 17:02:09 2018
00:00:00 212MB | Arnold 5.0.2.3 [d2dbe434] linux clang-5.0.0 oiio-1.7.17 osl-1.9.0 vdb-4.0.0 clm-1.0.3.513 rlm-12.2.2 2017/12/21 17:45:45
00:00:00 212MB | running on fx08, pid=7814
00:00:00 212MB | 1 x Intel(R) Core(TM) i7-2600 CPU @ 3.40GHz (4 cores, 8 logical) with 32146MB
00:00:00 212MB | Linux Mint 18.3 (Sylvia), Linux kernel 4.13.0-37-generic
00:00:00 212MB | soft limit for open files raised from 1024 to 1048574
00:00:00 212MB | 
00:00:00 212MB | [vop_shaders] Registering shaders ...
00:00:00 212MB | loading plugins from /studio/tools/arnold/arnold_htoa-2.2.2_hou_16.5.323/htoa-2.2.2_ra19eea7_houdini-16.5.323/arnold/plugins ...
00:00:00 212MB | htoa_mandelbulb_proc.so: mandelbulb uses Arnold 5.0.2.3
00:00:00 212MB | loaded 1 plugins from 1 lib(s) in 0:00.00
00:00:00 212MB | [vop_shaders] Loading metadata file: /studio/tools/arnold/arnold_htoa-2.2.2_hou_16.5.323/htoa-2.2.2_ra19eea7_houdini-16.5.323/arnold/metadata/cameras.mtd
00:00:00 212MB | [vop_shaders] Loading metadata file: /studio/tools/arnold/arnold_htoa-2.2.2_hou_16.5.323/htoa-2.2.2_ra19eea7_houdini-16.5.323/arnold/metadata/lights.mtd
00:00:00 212MB | [vop_shaders] Loading metadata file: /studio/tools/arnold/arnold_htoa-2.2.2_hou_16.5.323/htoa-2.2.2_ra19eea7_houdini-16.5.323/arnold/metadata/options.mtd
00:00:00 212MB | [vop_shaders] Loading metadata file: /studio/tools/arnold/arnold_htoa-2.2.2_hou_16.5.323/htoa-2.2.2_ra19eea7_houdini-16.5.323/arnold/metadata/shaders.mtd
00:00:00 214MB | [vop_shaders] Loading metadata file: /studio/tools/arnold/arnold_htoa-2.2.2_hou_16.5.323/htoa-2.2.2_ra19eea7_houdini-16.5.323/arnold/metadata/shapes.mtd
00:00:00 216MB | [vop_shaders] shader registration done.
00:00:00 216MB | 
00:00:00 216MB | releasing resources
00:00:00 216MB | Arnold shutdown
And this is how it looks, launched without afrender

Code: Select all

farm@fx08 /studio/tools/af_2.2.2/cgru/afanasy $ hrender_af -s 1 -e 1 --by 2 -t "_current_" "/studio/proj/temp/arnold2.hip_arnold2.afanasy1.0319-170758-557.hip" "/out/arnold1"
HOSTNAME=fx08
/studio/tools/hou/hfs16.5
The Houdini 16.5.323 environment has been initialized.
HOUDINI_PATH is set to /studio/tools/arnold/arnold_htoa-2.2.2_hou_16.5.323/htoa-2.2.2_ra19eea7_houdini-16.5.323;&
Houdni otls = '/home/farm/houdini16.5/otls:/studio/tools/af_2.2.2/cgru/plugins/houdini:/studio/tools/hou/hfs16.5/houdini/otls:/studio/tools/arnold/arnold_htoa-2.2.2_hou_16.5.323/htoa-2.2.2_ra19eea7_houdini-16.5.323;&/otls'
00:00:00   214MB         | log started Mon Mar 19 17:08:56 2018
00:00:00   214MB         | Arnold 5.0.2.3 [d2dbe434] linux clang-5.0.0 oiio-1.7.17 osl-1.9.0 vdb-4.0.0 clm-1.0.3.513 rlm-12.2.2 2017/12/21 17:45:45
00:00:00   214MB         | running on fx08, pid=8086
00:00:00   214MB         |  1 x Intel(R) Core(TM) i7-2600 CPU @ 3.40GHz (4 cores, 8 logical) with 32146MB
00:00:00   214MB         |  Linux Mint 18.3 (Sylvia), Linux kernel 4.13.0-37-generic
00:00:00   214MB         |  soft limit for open files raised from 1024 to 1048574
00:00:00   214MB         |  
00:00:00   214MB         | [vop_shaders] Registering shaders ...
00:00:00   214MB         | loading plugins from /studio/tools/arnold/arnold_htoa-2.2.2_hou_16.5.323/htoa-2.2.2_ra19eea7_houdini-16.5.323/arnold/plugins ...
00:00:00   214MB         |  htoa_mandelbulb_proc.so: mandelbulb uses Arnold 5.0.2.3
00:00:00   214MB         | loaded 1 plugins from 1 lib(s) in 0:00.00
00:00:00   214MB         | [vop_shaders] Loading metadata file: /studio/tools/arnold/arnold_htoa-2.2.2_hou_16.5.323/htoa-2.2.2_ra19eea7_houdini-16.5.323/arnold/metadata/cameras.mtd
00:00:00   214MB         | [vop_shaders] Loading metadata file: /studio/tools/arnold/arnold_htoa-2.2.2_hou_16.5.323/htoa-2.2.2_ra19eea7_houdini-16.5.323/arnold/metadata/lights.mtd
00:00:00   214MB         | [vop_shaders] Loading metadata file: /studio/tools/arnold/arnold_htoa-2.2.2_hou_16.5.323/htoa-2.2.2_ra19eea7_houdini-16.5.323/arnold/metadata/options.mtd
00:00:00   214MB         | [vop_shaders] Loading metadata file: /studio/tools/arnold/arnold_htoa-2.2.2_hou_16.5.323/htoa-2.2.2_ra19eea7_houdini-16.5.323/arnold/metadata/shaders.mtd
00:00:00   216MB         | [vop_shaders] Loading metadata file: /studio/tools/arnold/arnold_htoa-2.2.2_hou_16.5.323/htoa-2.2.2_ra19eea7_houdini-16.5.323/arnold/metadata/shapes.mtd
00:00:00   218MB         | [vop_shaders] shader registration done.
00:00:00   218MB         |  
00:00:00   218MB         | releasing resources
00:00:00   218MB         | Arnold shutdown
[htoa] HTOA environment variable already set to /studio/tools/arnold/arnold_htoa-2.2.2_hou_16.5.323/htoa-2.2.2_ra19eea7_houdini-16.5.323;&
HIP set to "/studio/proj/temp"
Trying to set "Right Output Style Progress" on arnold
RopNode.render: frame 1.0
00:00:00    41MB         | log started Mon Mar 19 17:08:58 2018
00:00:00    41MB         | Arnold 5.0.2.3 [d2dbe434] linux clang-5.0.0 oiio-1.7.17 osl-1.9.0 vdb-4.0.0 clm-1.0.3.513 rlm-12.2.2 2017/12/21 17:45:45
00:00:00    41MB         | running on fx08, pid=8135
00:00:00    41MB         |  1 x Intel(R) Core(TM) i7-2600 CPU @ 3.40GHz (4 cores, 8 logical) with 32146MB
00:00:00    41MB         |  Linux Mint 18.3 (Sylvia), Linux kernel 4.13.0-37-generic
00:00:00    41MB         |  soft limit for open files raised from 1048574 to 1048574
00:00:00    41MB         |  
00:00:00    41MB         | [hick] Start
00:00:00    41MB         | [hick] Lowered process priority to 19
00:00:00   281MB         | log started Mon Mar 19 17:08:59 2018
00:00:00   281MB         | Arnold 5.0.2.3 [d2dbe434] linux clang-5.0.0 oiio-1.7.17 osl-1.9.0 vdb-4.0.0 clm-1.0.3.513 rlm-12.2.2 2017/12/21 17:45:45
00:00:00   281MB         | running on fx08, pid=8086
00:00:00   281MB         |  1 x Intel(R) Core(TM) i7-2600 CPU @ 3.40GHz (4 cores, 8 logical) with 32146MB
00:00:00   281MB         |  Linux Mint 18.3 (Sylvia), Linux kernel 4.13.0-37-generic
00:00:00   281MB         |  soft limit for open files raised from 1048574 to 1048574
00:00:00   281MB         |  
00:00:00   281MB         | loading plugins from /studio/tools/arnold/arnold_htoa-2.2.2_hou_16.5.323/htoa-2.2.2_ra19eea7_houdini-16.5.323/arnold/plugins ...
00:00:00   281MB         |  htoa_mandelbulb_proc.so: mandelbulb uses Arnold 5.0.2.3
00:00:00   281MB         | loaded 1 plugins from 1 lib(s) in 0:00.00
00:00:00   300MB WARNING | cannot link a NULL node into skydome_light(shader); if you are trying to remove an existing link, use AiNodeUnlink() instead
00:00:00   300MB WARNING | node entry "standard" is deprecated, and may be removed in the future
00:00:00   300MB ERROR   | [htoa.material] Cannot find shader path /shop/arnold_vopnet1
00:00:00   300MB         | [htoa.texture] Converted 0 textures in 0:00:00.00 (0 skipped, 0 errors)
00:00:00   301MB         | [ass] writing scene to /tmp/houdini_temp/htoa_farm/8086_942f9c1d-ed31-4d14-8f93-cc6a7ad1a711.ass (mask=0xFFFF) ...
00:00:00   301MB         | [ass] wrote 879556 bytes, 10 nodes in 0:00.00
00:00:00   301MB         |  
00:00:00   301MB         | releasing resources
00:00:00   301MB         | Arnold shutdown
00:00:01    41MB         | [hick] Received: log_verbosity: 8135
00:00:01    41MB         | [hick] Received: log_console_enable: 1
00:00:01    42MB         | [ass] loading /tmp/houdini_temp/htoa_farm/8086_942f9c1d-ed31-4d14-8f93-cc6a7ad1a711.ass ...
00:00:01    42MB         |  loading plugins from /studio/tools/arnold/arnold_htoa-2.2.2_hou_16.5.323/htoa-2.2.2_ra19eea7_houdini-16.5.323/arnold/plugins ...
00:00:01    42MB         |   htoa_mandelbulb_proc.so: mandelbulb uses Arnold 5.0.2.3
00:00:01    42MB         |  loaded 1 plugins from 1 lib(s) in 0:00.00
00:00:01    44MB WARNING |  node entry "standard" is deprecated, and may be removed in the future
00:00:01    44MB         | [ass] read 879556 bytes, 10 nodes in 0:00.00
00:00:01    44MB         | 
00:00:01    44MB         | authorizing ...
00:00:01    48MB         | [rlm] authorized for "arnold 21990101" from server licserver
00:00:01    48MB         | 
00:00:01    48MB         | [color_manager] no color manager is active
00:00:01    48MB         | [color_manager] rendering color space is "linear" with declared chromaticities:
00:00:01    48MB         |                 r(0.6400, 0.3300) g(0.3000, 0.6000) b(0.1500, 0.0600) and w(0.3127, 0.3290)
00:00:01    67MB         |  
00:00:01    67MB         | there are 1 light and 3 objects:
00:00:01    67MB         |       1 persp_camera
00:00:01    67MB         |       1 skydome_light
00:00:01    67MB         |       1 utility
00:00:01    67MB         |       2 standard
00:00:01    67MB         |       1 physical_sky
00:00:01    67MB         |       1 driver_exr
00:00:01    67MB         |       1 gaussian_filter
00:00:01    67MB         |       2 polymesh
00:00:01    67MB         |       1 list_aggregate
00:00:01    67MB         |  
00:00:01    67MB         | rendering image at 1280 x 720, 3 AA samples
00:00:01    67MB         |   AA sample clamp   <disabled>
00:00:01    67MB         |   diffuse           samples  2 / depth  1
00:00:01    67MB         |   specular          samples  2 / depth  1
00:00:01    67MB         |   transmission      samples  2 / depth  8
00:00:01    67MB         |   volume indirect   <disabled by depth>
00:00:01    67MB         |   total                          depth 10
00:00:01    67MB         |   bssrdf            samples  2
00:00:01    67MB         |   transparency      depth 10
00:00:01    67MB         |  initializing 12 nodes ...
00:00:01    67MB         |   creating root object list ...
00:00:01    67MB         |  node initialization done in 0:00.00 (single-threaded)
00:00:01    67MB         |  updating 13 nodes ...
00:00:01    67MB         |   scene bounds: (-2 0 -2) -> (2 0.720845699 2)
00:00:01    67MB         |  node update done in 0:00.01 (single-threaded)
00:00:01    67MB         |  [aov] parsing 2 output statements ...
00:00:01    67MB         |  [aov] registered driver: "/out/arnold1:exr:cam1" (driver_exr)
00:00:01    67MB         |  [aov]  * "RGBA" of type RGBA filtered by "/out/arnold1:gaussian_filter" (gaussian_filter)
00:00:01    67MB         |  [aov]  * "diffuse_direct" of type RGB filtered by "/out/arnold1:gaussian_filter" (gaussian_filter)
00:00:01    67MB         |  [aov] done preparing 2 AOVs for 2 outputs to 1 driver (0 deep AOVs)
00:00:02    93MB         |  starting 8 bucket workers of size 64x64 ...
00:00:02   114MB         |   [skydome_light] /obj/arnold_light1: 1000x1000 importance map done in 0:00.03, average energy 0.173573
00:00:03   115MB         |     0% done - 177 rays/pixel
00:00:04   116MB         |     5% done - 156 rays/pixel
00:00:06   116MB         |    10% done - 107 rays/pixel
00:00:06   116MB         |    15% done - 86 rays/pixel
00:00:07   116MB         |    20% done - 54 rays/pixel
00:00:08   116MB         |    25% done - 83 rays/pixel
00:00:08   116MB         |    30% done - 60 rays/pixel
00:00:09   116MB         |    35% done - 23 rays/pixel
00:00:09   116MB         |    40% done - 86 rays/pixel
00:00:10   116MB         |    45% done - 66 rays/pixel
00:00:10   116MB         |    50% done - 23 rays/pixel
00:00:10   116MB         |    55% done - 16 rays/pixel
00:00:10   116MB         |    60% done - 81 rays/pixel
00:00:11   116MB         |    65% done - 19 rays/pixel
00:00:11   116MB         |    70% done - 72 rays/pixel
00:00:11   116MB         |    75% done - 18 rays/pixel
00:00:12   116MB         |    80% done - 64 rays/pixel
00:00:12   116MB         |    85% done - 12 rays/pixel
00:00:12   116MB         |    90% done - 59 rays/pixel
00:00:12   116MB         |    95% done - 12 rays/pixel
00:00:13   116MB         |   100% done - 51 rays/pixel
00:00:13   116MB         |  bucket workers done in 0:11.11
00:00:13   116MB         |  [driver_exr] writing file `/studio/proj/temp/arnold2.hip_arnold2.afanasy1.0319-170758-557.arnold1.0001.exr'
00:00:14   118MB         | render done
00:00:14   118MB         |  
00:00:14   118MB         | releasing resources
00:00:14   119MB         | Arnold shutdown

Re: Arnold hangs every type of job

Posted: Mon Mar 19, 2018 4:10 pm
by timurhai
Hi.
Very strange.
Ensure that use launch command in the console by the same user as afrender service started by.
I see that you tried to launch in the console by "farm" user, default user for afrender is "render".
Look at the output difference:

Code: Select all

LaunchProgram: chdir(): : Permission denied
Who printed it (houdini/arnold/some inhouse plugin)? Why? Try to dig here.

Re: Arnold hangs every type of job

Posted: Mon Mar 19, 2018 4:26 pm
by arengorman
Thanks for replying that quick!

We use "farm" user for afrender processes instead of "render", so it's OK.

PermissionDenied error is written by Houdini, it is written even without Arnold, when everything works perfectly, but since you think it can be important, I'll try to find out what causes it, this may be a key to our problem. If you have any more ideas on what is going on, please share)

Re: Arnold hangs every type of job

Posted: Mon Mar 19, 2018 4:35 pm
by timurhai
To test permissions problem, you can also try to run afrender under the root.
( root always can change working folder )

Re: Arnold hangs every type of job

Posted: Mon Mar 19, 2018 4:48 pm
by arengorman
Yes, I tried to launch afrender under root, situation is absolutely the same, except that error about "chdir" doesn't appear.
It seems that "LaunchProgram: chdir(): : Permission denied" refers to some OTLs, not to renderers

I have an idea to create a generic job with a command from standard job, created by afanasy plugin. I'll post the results here.

Re: Arnold hangs every type of job

Posted: Sat Mar 24, 2018 12:53 pm
by arengorman
Ok, so today I tried to construct generic jobs from this JSON:

Code: Select all

{
    "job":
    {
        "name"                  : "Arnold Generic Job test",
        "user_name"             : "rum",
        "host_name"             : "fx08",
        "blocks":[
        {
            "name"              : "Arnold2",
            "tasks_name"        : "frames @#@-@#@",
            "service"           : "generic",
            "parser"            : "generic",
            "frame_first"       : 1,
            "frame_last"        : 1,
            "frames_per_task"   : 1,
            "frames_inc"        : 1,
            "command"           : "hrender_af @#@",
            "working_directory" : "/studio/proj/temp",
            "files"             : ["/studio/proj/temp/arnold2.arnold1.0001.exr"],
            "tasks":[
            {
                "name"                   : "Arnold",
                "command"                : "-s 1 -e 1 --by 2 -t \"_current_\" \"/studio/proj/temp/arnold2_rum.hip\" \"/out/arnold1\"",
                "files"                  : ["/studio/proj/temp/arnold2_rum.0001.exr"]
            }
            ]
        }
        ]
    }
}
I also tried to use Arnold parser in another json. Unfortunately the situation is absolutely the same.