Afserver crashes

General discussions and questions.
selsner
Posts: 22
Joined: Wed Jan 25, 2017 11:20 am

Re: Afserver crashes

Post by selsner » Fri Mar 17, 2017 1:16 am

Did you have issues before with afserver and without the new setting? I was not really able to reproduce on centos 6.6 via the afcmd command.....
CGRU 2.2.1 - CentOS 6.6, gcc 4.4.7, Qt 4.6

Sebastian Elsner - Pipeline Technical Director - RISE
www.risefx.com

User avatar
timurhai
Site Admin
Posts: 102
Joined: Sun Jan 15, 2017 8:40 pm
Location: Russia, Korolev
Contact:

Re: Afserver crashes

Post by timurhai » Fri Mar 17, 2017 9:58 am

Hi.

Now new stack size for threads is not committed.
I just checking it on a testing platform.

Yesterday it solved jobs all day long with 400 afrenders with no hung.
So i think that reducing stack size is a solution of this problem.

Now i planning to make stack size parameter in thread class.
And raise only client reading threads with a reduced stack.
As only such threads are always raises while running.
Other threads are launched on only once on start, are more important and can use more stack, so i want to keep large stack for them.

After it i will compile and use such afserver at work.

And after such testing may be a new release will be created.
Timur Hairulin
CGRU 2.2.2 Ubuntu 14.04LTS (gcc4.8.4)

User avatar
timurhai
Site Admin
Posts: 102
Joined: Sun Jan 15, 2017 8:40 pm
Location: Russia, Korolev
Contact:

Re: Afserver crashes

Post by timurhai » Fri Mar 17, 2017 10:08 am

But reducing stack is a temporary (but at seems that working) solution.
It is not good architecture to use blocking sockets and raise a thread for each.
The best way to raise 1 thread once to for all client sockets in a non-blocking manner.

Many thanks to eliemichel as he already developed it here:
https://github.com/eliemichel/cgru/tree/socketpool
This branch shows how to use epoll system call for it.
But it is only for Linux.
I should abstract from epoll and write some class that will hide non-blocking implementation, to use some other methods for windows and mac.

I think that i'll pause my current "render pools" work, and start to think about non-blocking sockets right now.
Timur Hairulin
CGRU 2.2.2 Ubuntu 14.04LTS (gcc4.8.4)

selsner
Posts: 22
Joined: Wed Jan 25, 2017 11:20 am

Re: Afserver crashes

Post by selsner » Fri Mar 17, 2017 10:11 am

I have no good idea on how computation intensive receiving and parsing messages is, but wouldn't it be better to have Thread Pools with more than one thread to do the work?
CGRU 2.2.1 - CentOS 6.6, gcc 4.4.7, Qt 4.6

Sebastian Elsner - Pipeline Technical Director - RISE
www.risefx.com

User avatar
timurhai
Site Admin
Posts: 102
Joined: Sun Jan 15, 2017 8:40 pm
Location: Russia, Korolev
Contact:

Re: Afserver crashes

Post by timurhai » Fri Mar 17, 2017 12:41 pm

We need at least one more solution as epoll is Linux only (not even UNIX).
Later we can add to config what method to choose on Linux.
Anyway the most coding work is to change afserver architecture.
Now it accepts new connection and raises one thread for it.
The thread reads, than do something, that writes an answer.
New architecture assumes that there should be some incoming and outgoing messages (sockets) queue, no matter how they work (epoll or threads).
And in the middle some other thread should react on input information.
This can even be the same thread that solving jobs.
- this way we can get rid of mutex locks everywhere in afanasy code, mutexes will be in queues only.
This will make code more simple, clean and plain (and may be even speed up it ).
( this also was the other idea of eliemichel that makes him to develop socket pools )
Timur Hairulin
CGRU 2.2.2 Ubuntu 14.04LTS (gcc4.8.4)

seven11
Posts: 15
Joined: Tue Feb 21, 2017 8:03 pm

Re: Afserver crashes

Post by seven11 » Fri Mar 17, 2017 3:59 pm

@selsner
I was seeing a large amount of Virtual Memory being consumed then I saw this thread on the forum and decided to stick my nose in it.

Timur,
That looks like a better way of doing it ;-)
You write some nice code!
Scott

User avatar
timurhai
Site Admin
Posts: 102
Joined: Sun Jan 15, 2017 8:40 pm
Location: Russia, Korolev
Contact:

Re: Afserver crashes

Post by timurhai » Sun Mar 19, 2017 8:14 pm

Hi.
Now reduced stack size is committed:
https://github.com/CGRU/cgru/commit/ba1 ... 099cdbc1eb
Tomorrow I will launch such afserver at work.
Timur Hairulin
CGRU 2.2.2 Ubuntu 14.04LTS (gcc4.8.4)

User avatar
timurhai
Site Admin
Posts: 102
Joined: Sun Jan 15, 2017 8:40 pm
Location: Russia, Korolev
Contact:

Re: Afserver crashes

Post by timurhai » Mon Mar 20, 2017 10:11 am

Now server thread to process client connections stack size is configurable, by default is 32768:
https://github.com/CGRU/cgru/commit/dbb ... 9b7098R196
And I also running such afserver at work, to watch stability.
Timur Hairulin
CGRU 2.2.2 Ubuntu 14.04LTS (gcc4.8.4)

selsner
Posts: 22
Joined: Wed Jan 25, 2017 11:20 am

Re: Afserver crashes

Post by selsner » Mon Mar 20, 2017 12:44 pm

running it too now.
CGRU 2.2.1 - CentOS 6.6, gcc 4.4.7, Qt 4.6

Sebastian Elsner - Pipeline Technical Director - RISE
www.risefx.com

selsner
Posts: 22
Joined: Wed Jan 25, 2017 11:20 am

Re: Afserver crashes

Post by selsner » Thu Mar 23, 2017 1:11 pm

server is running quite well now with 80 renders added since the change of stacksize. One crash yesterday after printing:

Code: Select all

AFERROR: JobProgress::JobProgress: invalud number of tasks = 0 (m_job_id=0,block=0) 
But someone send a job via python at the same time, and the submission crashed in the middle.... so this might be not related.
CGRU 2.2.1 - CentOS 6.6, gcc 4.4.7, Qt 4.6

Sebastian Elsner - Pipeline Technical Director - RISE
www.risefx.com

Post Reply

Who is online

Users browsing this forum: No registered users and 1 guest