Page 1 of 1

we want all jobs backup in database not is json data

Posted: Tue Sep 08, 2020 5:37 am
by kamlesh0999
hi
when we submit the job then cgru creating a json data into the temp folder so is this possible the data directly go into the database that means can we remove the the json pipeline data from the jobs

Re: we want all jobs backup in database not is json data

Posted: Tue Sep 08, 2020 6:07 am
by timurhai
Hi.
Much more simple to write some other program that will scan json store and add/update some SQL database.
Afanasy uses JSON protocol for communications and the same functions to store.
Without JSON pipeline it will be some other render manager.

Re: we want all jobs backup in database not is json data

Posted: Tue Sep 08, 2020 11:26 am
by kamlesh0999
Ok thank you so much

Re: we want all jobs backup in database not is json data

Posted: Wed Sep 09, 2020 5:03 am
by kamlesh0999
Hi
can we use MongoDB database for storing Json data and read directly from MongoDB can we use MongoDB database for storing Json data and read directly from MongoDB

Re: we want all jobs backup in database not is json data

Posted: Wed Sep 09, 2020 5:42 am
by timurhai
You can use any DB.
The most interesting questions here are how you will use it and why you need it.

Re: we want all jobs backup in database not is json data

Posted: Thu Sep 10, 2020 7:01 am
by kamlesh0999
ok thanks buddy

Re: we want all jobs backup in database not is json data

Posted: Wed Sep 22, 2021 7:28 pm
by jpisard
we want to use a database to not have jobs definition in RAM

- we test afanasy with 1000 renders and 16G of RAM on the server (debian buster)
- we work on 1000 shots by month
- we have 6 departements. they launch 6 jobs per shot (it's an average)
- we need to keep 2 months of jobs
so we need working with more than 72000 jobs
at 10 000 jobs with several block af_render is long to start (time to put jobs in memory), long to respond on json query. just one core is use at 90-100%.
at 20 000 jobs af_render is killed by out of memory

all other features look good (or very good) but manage jobs in memory limit scalability
(before this probleme we plan to create a web gui or a pyside gui to manage a lot of jobs)
at this time we stop testing afanasy to look on other solution ("la mort dans l'ame")

thanks

Re: we want all jobs backup in database not is json data

Posted: Wed Sep 22, 2021 7:59 pm
by timurhai
Great test!

I guess that by "af_render" you mean "afserver".
My simple notebook, that i coding on, handles 10k jobs easily and takes about 1Gb ram, never tested more amount of jobs. But i guess that a powerful server can handle much more.
If you need to keep 2 month of jobs, huge amount of jobs, some archiving mechanism should be preset in render manager.
Some people periodically asking me for it. Deleting a job should delete data from RAM and put it to an archive. After some time jobs should be deleted from archive. Before that time job can be restored from archive.

But for not Afanasy has no archiving mechanism...

Re: we want all jobs backup in database not is json data

Posted: Thu Sep 23, 2021 7:37 pm
by jpisard
Hi

Yes i mean "afserver"

i have test with a template job who content 6 blocks. So 10K Jobs take more than 5G in ram and server is very long to restart.
In the file system jobs take about 120 M.
With single block in jobs RAM is not the point of failure but cpu go at 100% (CPU E5-2620 v4 @ 2.10GHz virtualized by Xen : not the best for CPU and RAM usage but I/O is very good).
In this case, afserver doesn't seems to use all my CPU core.

Archive jobs look like a workaround, I thinks that jobs in database will be best than RAM/json files

In the past we have collaborate on a render manager (we use it in production). On first version jobs will be in RAM. In database this scale better.
For us the question will be do a choice : Afanasy for freatures but need some works / opencue (we will test that next month) / pipelinefx Qube / recode our farm render to add freatures like multithread, resource management ... (https://github.com/MercenariesEngineering/coalition)

best regards

Re: we want all jobs backup in database not is json data

Posted: Mon Oct 04, 2021 11:01 am
by timurhai
Hi!
I forgot to say, if you need to keep jobs for some statistics, there is a SQL database for it.
It stores just some useful jobs data, and you can't restore jobs from it.

(just now some others asked me the same questions, and i asked what they need to store jobs for)