Page 2 of 5

Re: Afserver crashes

Posted: Fri Feb 24, 2017 12:27 pm
by selsner
We have about 200 dedicated rendernodes and 40 workstations with 25 monitors. But as soon as I add the 40 workstations afserver starts to spit weird glibc double free errors.... We had trouble with afserver crashing before but it was never so obvious.
It's hard to debug in full production right now but I will try.

Re: Afserver crashes

Posted: Wed Mar 01, 2017 11:58 am
by lithorus
Our server just crashed and I got this in 'dmesg':

Code: Select all

afserver[22218]: segfault at 7f88e4ff9d13 ip 00007f894a982e8f sp 00007f89420d8bc0 error 4 in libpthread-2.23.so[7f894a97b000+18000]

Re: Afserver crashes

Posted: Thu Mar 02, 2017 8:08 am
by VisorBNV
Have same trouble with afserver 2.2.1 on openSUSE Leap 42.2(vm on vmware host). Crashes every 2-3 days about.
Libthreads become dead and afserver service unloaded...

Re: Afserver crashes

Posted: Thu Mar 02, 2017 9:33 am
by timurhai
Hi.

I noticed pthread bug on ubuntu16lts (gcc5.4.0) too.
On a test imitating very heady load.
But at work on ubuntu14tls (gcc4.8.4) i can`t reproduce it.

Almost needed threads afserver spawns at start (one thread to write files, one thread to write SQL and so on).
After the start the only reason to raise a thread is to read a socket of each incoming connection.
So it crashed here:
https://github.com/CGRU/cgru/blob/maste ... t.cpp#L222
But for now i do not why.
Also it seems that kernel&gcc versions plays some role (probability).
I will dig this issue deeper.

Re: Afserver crashes

Posted: Thu Mar 02, 2017 6:58 pm
by lithorus
How do you imitate heavy load? I'm trying my best to get afanasy to crash on a test system, without any luck.

I'm trying to connect hundreds of dummy renders while adding jobs in an endless loop.

Re: Afserver crashes

Posted: Fri Mar 03, 2017 9:37 am
by timurhai
Just launch 200 renders on the same host with afserver.
On ubuntu 16lts it craches in a minute.
On ubuntu 14lts i did not wait crash.

Re: Afserver crashes

Posted: Fri Mar 03, 2017 10:36 am
by lithorus
timurhai wrote: Fri Mar 03, 2017 9:37 am Just launch 200 renders on the same host with afserver.
On ubuntu 16lts it craches in a minute.
On ubuntu 14lts i did not wait crash.
I launched about 1400 clients on Ubuntu 16.04 and it didn't crash.
This was using a self compiled version from latest git on my laptop. Did your test machine have SSD?

Re: Afserver crashes

Posted: Fri Mar 03, 2017 12:07 pm
by timurhai
No. My laptop has HDD.

Re: Afserver crashes

Posted: Fri Mar 03, 2017 12:10 pm
by lithorus
timurhai wrote: Fri Mar 03, 2017 12:07 pm No. My laptop has HDD.
So perhaps it's when the thread queue gets too long?
Will try to limit afserver IO on my laptop, to see if it will crash sooner.

Re: Afserver crashes

Posted: Tue Mar 14, 2017 2:17 pm
by selsner
did you get around to dig into this? It is getting annoying.