we want all jobs backup in database not is json data

General discussions and questions.
Post Reply
kamlesh0999
Posts: 11
Joined: Thu Aug 06, 2020 1:19 pm

we want all jobs backup in database not is json data

Post by kamlesh0999 »

hi
when we submit the job then cgru creating a json data into the temp folder so is this possible the data directly go into the database that means can we remove the the json pipeline data from the jobs
User avatar
timurhai
Site Admin
Posts: 768
Joined: Sun Jan 15, 2017 8:40 pm
Location: Russia, Korolev
Contact:

Re: we want all jobs backup in database not is json data

Post by timurhai »

Hi.
Much more simple to write some other program that will scan json store and add/update some SQL database.
Afanasy uses JSON protocol for communications and the same functions to store.
Without JSON pipeline it will be some other render manager.
Timur Hairulin
CGRU 3.2.1, Ubuntu 20.04 LTS, MS Windows 10 (clients only).
kamlesh0999
Posts: 11
Joined: Thu Aug 06, 2020 1:19 pm

Re: we want all jobs backup in database not is json data

Post by kamlesh0999 »

Ok thank you so much
kamlesh0999
Posts: 11
Joined: Thu Aug 06, 2020 1:19 pm

Re: we want all jobs backup in database not is json data

Post by kamlesh0999 »

Hi
can we use MongoDB database for storing Json data and read directly from MongoDB can we use MongoDB database for storing Json data and read directly from MongoDB
User avatar
timurhai
Site Admin
Posts: 768
Joined: Sun Jan 15, 2017 8:40 pm
Location: Russia, Korolev
Contact:

Re: we want all jobs backup in database not is json data

Post by timurhai »

You can use any DB.
The most interesting questions here are how you will use it and why you need it.
Timur Hairulin
CGRU 3.2.1, Ubuntu 20.04 LTS, MS Windows 10 (clients only).
kamlesh0999
Posts: 11
Joined: Thu Aug 06, 2020 1:19 pm

Re: we want all jobs backup in database not is json data

Post by kamlesh0999 »

ok thanks buddy
jpisard
Posts: 2
Joined: Wed Sep 22, 2021 7:11 pm

Re: we want all jobs backup in database not is json data

Post by jpisard »

we want to use a database to not have jobs definition in RAM

- we test afanasy with 1000 renders and 16G of RAM on the server (debian buster)
- we work on 1000 shots by month
- we have 6 departements. they launch 6 jobs per shot (it's an average)
- we need to keep 2 months of jobs
so we need working with more than 72000 jobs
at 10 000 jobs with several block af_render is long to start (time to put jobs in memory), long to respond on json query. just one core is use at 90-100%.
at 20 000 jobs af_render is killed by out of memory

all other features look good (or very good) but manage jobs in memory limit scalability
(before this probleme we plan to create a web gui or a pyside gui to manage a lot of jobs)
at this time we stop testing afanasy to look on other solution ("la mort dans l'ame")

thanks
User avatar
timurhai
Site Admin
Posts: 768
Joined: Sun Jan 15, 2017 8:40 pm
Location: Russia, Korolev
Contact:

Re: we want all jobs backup in database not is json data

Post by timurhai »

Great test!

I guess that by "af_render" you mean "afserver".
My simple notebook, that i coding on, handles 10k jobs easily and takes about 1Gb ram, never tested more amount of jobs. But i guess that a powerful server can handle much more.
If you need to keep 2 month of jobs, huge amount of jobs, some archiving mechanism should be preset in render manager.
Some people periodically asking me for it. Deleting a job should delete data from RAM and put it to an archive. After some time jobs should be deleted from archive. Before that time job can be restored from archive.

But for not Afanasy has no archiving mechanism...
Timur Hairulin
CGRU 3.2.1, Ubuntu 20.04 LTS, MS Windows 10 (clients only).
jpisard
Posts: 2
Joined: Wed Sep 22, 2021 7:11 pm

Re: we want all jobs backup in database not is json data

Post by jpisard »

Hi

Yes i mean "afserver"

i have test with a template job who content 6 blocks. So 10K Jobs take more than 5G in ram and server is very long to restart.
In the file system jobs take about 120 M.
With single block in jobs RAM is not the point of failure but cpu go at 100% (CPU E5-2620 v4 @ 2.10GHz virtualized by Xen : not the best for CPU and RAM usage but I/O is very good).
In this case, afserver doesn't seems to use all my CPU core.

Archive jobs look like a workaround, I thinks that jobs in database will be best than RAM/json files

In the past we have collaborate on a render manager (we use it in production). On first version jobs will be in RAM. In database this scale better.
For us the question will be do a choice : Afanasy for freatures but need some works / opencue (we will test that next month) / pipelinefx Qube / recode our farm render to add freatures like multithread, resource management ... (https://github.com/MercenariesEngineering/coalition)

best regards
User avatar
timurhai
Site Admin
Posts: 768
Joined: Sun Jan 15, 2017 8:40 pm
Location: Russia, Korolev
Contact:

Re: we want all jobs backup in database not is json data

Post by timurhai »

Hi!
I forgot to say, if you need to keep jobs for some statistics, there is a SQL database for it.
It stores just some useful jobs data, and you can't restore jobs from it.

(just now some others asked me the same questions, and i asked what they need to store jobs for)
Timur Hairulin
CGRU 3.2.1, Ubuntu 20.04 LTS, MS Windows 10 (clients only).
Post Reply