Afserver crashes

General discussions and questions.
selsner
Posts: 47
Joined: Wed Jan 25, 2017 11:20 am

Re: Afserver crashes

Post by selsner »

We have about 200 dedicated rendernodes and 40 workstations with 25 monitors. But as soon as I add the 40 workstations afserver starts to spit weird glibc double free errors.... We had trouble with afserver crashing before but it was never so obvious.
It's hard to debug in full production right now but I will try.
CGRU 2.3.1 - CentOS 7.7

Sebastian Elsner - Pipeline Technical Director - RISE
www.risefx.com
lithorus
Posts: 28
Joined: Wed Jan 25, 2017 4:14 pm

Re: Afserver crashes

Post by lithorus »

Our server just crashed and I got this in 'dmesg':

Code: Select all

afserver[22218]: segfault at 7f88e4ff9d13 ip 00007f894a982e8f sp 00007f89420d8bc0 error 4 in libpthread-2.23.so[7f894a97b000+18000]
VisorBNV
Posts: 6
Joined: Fri Jan 27, 2017 3:24 pm
Location: Moscow

Re: Afserver crashes

Post by VisorBNV »

Have same trouble with afserver 2.2.1 on openSUSE Leap 42.2(vm on vmware host). Crashes every 2-3 days about.
Libthreads become dead and afserver service unloaded...
User avatar
timurhai
Site Admin
Posts: 911
Joined: Sun Jan 15, 2017 8:40 pm
Location: Russia, Korolev
Contact:

Re: Afserver crashes

Post by timurhai »

Hi.

I noticed pthread bug on ubuntu16lts (gcc5.4.0) too.
On a test imitating very heady load.
But at work on ubuntu14tls (gcc4.8.4) i can`t reproduce it.

Almost needed threads afserver spawns at start (one thread to write files, one thread to write SQL and so on).
After the start the only reason to raise a thread is to read a socket of each incoming connection.
So it crashed here:
https://github.com/CGRU/cgru/blob/maste ... t.cpp#L222
But for now i do not why.
Also it seems that kernel&gcc versions plays some role (probability).
I will dig this issue deeper.
Timur Hairulin
CGRU 3.3.1, Ubuntu 20.04, 22.04, MS Windows 10 (clients only).
lithorus
Posts: 28
Joined: Wed Jan 25, 2017 4:14 pm

Re: Afserver crashes

Post by lithorus »

How do you imitate heavy load? I'm trying my best to get afanasy to crash on a test system, without any luck.

I'm trying to connect hundreds of dummy renders while adding jobs in an endless loop.
User avatar
timurhai
Site Admin
Posts: 911
Joined: Sun Jan 15, 2017 8:40 pm
Location: Russia, Korolev
Contact:

Re: Afserver crashes

Post by timurhai »

Just launch 200 renders on the same host with afserver.
On ubuntu 16lts it craches in a minute.
On ubuntu 14lts i did not wait crash.
Timur Hairulin
CGRU 3.3.1, Ubuntu 20.04, 22.04, MS Windows 10 (clients only).
lithorus
Posts: 28
Joined: Wed Jan 25, 2017 4:14 pm

Re: Afserver crashes

Post by lithorus »

timurhai wrote: Fri Mar 03, 2017 9:37 am Just launch 200 renders on the same host with afserver.
On ubuntu 16lts it craches in a minute.
On ubuntu 14lts i did not wait crash.
I launched about 1400 clients on Ubuntu 16.04 and it didn't crash.
This was using a self compiled version from latest git on my laptop. Did your test machine have SSD?
User avatar
timurhai
Site Admin
Posts: 911
Joined: Sun Jan 15, 2017 8:40 pm
Location: Russia, Korolev
Contact:

Re: Afserver crashes

Post by timurhai »

No. My laptop has HDD.
Timur Hairulin
CGRU 3.3.1, Ubuntu 20.04, 22.04, MS Windows 10 (clients only).
lithorus
Posts: 28
Joined: Wed Jan 25, 2017 4:14 pm

Re: Afserver crashes

Post by lithorus »

timurhai wrote: Fri Mar 03, 2017 12:07 pm No. My laptop has HDD.
So perhaps it's when the thread queue gets too long?
Will try to limit afserver IO on my laptop, to see if it will crash sooner.
selsner
Posts: 47
Joined: Wed Jan 25, 2017 11:20 am

Re: Afserver crashes

Post by selsner »

did you get around to dig into this? It is getting annoying.
CGRU 2.3.1 - CentOS 7.7

Sebastian Elsner - Pipeline Technical Director - RISE
www.risefx.com
Post Reply