Hello guest, if you like this forum, why don't you register? https://forum.fanres.com/member.php?action=register (December 14, 2021) x


Avisynth parallel encoding

27 Replies, 14048 Views

not sure about colourlike but there seems to be x64 versions of median available.

http://ajk.kapsi.fi/projects/avisynth/median-plugin/
(2019-05-11, 08:14 PM)althor1138 Wrote: not sure about colourlike but there seems to be x64 versions of median available.

http://ajk.kapsi.fi/projects/avisynth/median-plugin/

From that folder, take the Median-0.6-RGB64-32+64bit.zip one. It's my modification that's both 32 bit and 64 bit, updated to the newer AviSynth+ headers and also supports RGB64 colorspace for high bit depth Median. Source code is also in the folder in a separate zip.

I actually mentioned this here a while back, but ofc spoRv already forgot. Big Grin
Of course... Happy
Sadly my projects are lost due to an HDD crash... Sad
Fundamental Collection | Vimeo channel | My blog
Encoding Mad Max right now... using MP_Pipeline as a single script, I got 0.9fps... wow... CPU at 5/7%
Code:
MP_Pipeline("""
source1
### ###
source2
filter1
filter2
filter3 #scriptclip
### ###
""")

then, I split the script in seven parts - didn't want to overload the CPU - each 20000 frames long
Code:
MP_Pipeline("""
source1
### ###
source2
filter1
filter2
filter3_1 #scriptclip
trim(000000,019999)
### ###
""")
and so on... I was used to make different scripts (_1, _2 etc) when using scriptclip, as it share the same variables, hence every one has its own variables (var1_1, var2_2 etc) - but I suspect this is not the case with avisynth+, still did not want to risk! Big Grin

Conclusion: now 7 scripts running each at mostly the same speed of single script, for a total of around 6.5x speed - or, it will took 6.5 less time than running a single script - and CPU is running around 35% with "only" 23GB, but I still have A LOT of browser windows opened taking more than 5GB... so, 32bit still working well, let's see if some of them will crush over the night (there is always more than few chances...) - I know that I HAVE to use 64bit, but transition to avisynth+, albeit almost painless, happened few weeks ago, so please gimme a break! Tongue

All in all, a good compromise - probably I could run more scripts at the same time, but even if CPU usage is quite low, there are around 20 threads active, and I prepared my parallel scriptclip up to 8 versions - you know, before I got "only" a 4C/8T CPU, so... Big Grin
Sadly my projects are lost due to an HDD crash... Sad
Fundamental Collection | Vimeo channel | My blog
Just a hint, the problem with 32 bit isn't that you run out of RAM, it's t hat 32 bit applications can only address a maximum of I think 4 GB (or even only 2?). So as long as each single process uses less than 2 GB of RAM, you can run 6 of them simultaneously no problem. If you run out of RAM, Windows will use a pagefile by default, so you're safe in that regard.
Would you be interested in telling us which filters and settings you are using? I'd like to try running this and see what kind of fps I can get on the 1800x.

EDIT: I've been playing around with mp_pipeline today and can't seem to get any kind of performance gain out of it vs running single-threaded. Maybe I'm missing something? I never get more than 10-15% usage out of the cpu while running qtgmc. However, I can switch to avs+ multi-threading and use all physical cpu cores and get it to jump up to 100% cpu usage if I want to.
(This post was last modified: 2019-05-14, 02:22 PM by althor1138.)
I continue to use MP_pipeline for any avisynth script since a lot of time, and I have discovered something in the meanwhile. Should add that upgrading to Avisynth+ was a breeze!

Do note that if you use ScriptClip/GScriptClip (more than once, usually), sometimes it goes awry (read: continue to encode, but in wrong ways) - don't know if it's my fault, though...
And sometimes, with very heavy scripts (the ones I use, even when I split them in successive steps), it crashes.
Said that, it works very well let's say 95% of the times, with A LOT of time spared!

I usually use these kind of scripts:

Code:
MP_Pipeline("""
yoursource
###
filter1
###
filter2
###
filter3

### prefetch: 32, 0
### branch: 32
### ###
""")

I have a CPU with 16 core and 32 threads, so I use 32 for branch; not tested if different numbers for prefetch would increase speed, but I can say that with branch equal to my CPU threads, script goes A LOT faster - and I mean it, like 10 times more...

WARNING: after every ### you lose any variable and get only the last version of the clip, so for example if you work with two clips you can use this:

Code:
MP_Pipeline("""
a=yoursource1
b=yoursource2

a.trim(x,y).filter1+\
b.trim(z,w).filter2+\
a.trim(k,j).filter3
###
filter4 # that will apply to the whole previous clip

### prefetch: 32, 0
### branch: 32
### ###
""")

I invite everyone to test it, firt without using prefetch and branch lines, then replacing them with various numbers, and report your experience here; this would be beneficial for the whole community!

Last thought: if it's 10x faster with a "mere" Threadripper 1950x 16core, think what could do with the 3990x that has 64core and higher IPC!!! Eek
Sadly my projects are lost due to an HDD crash... Sad
Fundamental Collection | Vimeo channel | My blog
Important note: even if I set the branch to 32, it works "only" at 24 - how can I know it? Well, the avisynth status windows shows actual frame number, and the increase step is 24 (while when I set branch at 12 or 16 reflected this number).

Now, don't know if the max branch is: CPU threads/4*3 or CPU threads-12
Sadly my projects are lost due to an HDD crash... Sad
Fundamental Collection | Vimeo channel | My blog

Possibly Related Threads…
Thread Author Replies Views Last Post
  New Tool: Dolby Encoding Engine Wrapper schorman 2 5,074 2022-04-26, 09:12 PM
Last Post: CSchmidlapp
  Flagging 1080p25 as 1080i25 without re-encoding Dr. Cooper 3 495 2021-12-07, 01:14 PM
Last Post: FreaQ
  FFV1 encoding for archival pipefan413 0 925 2021-03-11, 07:14 PM
Last Post: pipefan413
  [Help] Encoding HDR to SDR awsmguy145 3 1,187 2020-09-28, 12:01 PM
Last Post: Chewtobacca
  x264 encoding from Adobe Premiere DoomBot 26 13,004 2019-03-25, 08:26 PM
Last Post: jaminmc
  [Help] Encoding issue with Xvid4psp v5.0.37.8 Beber 7 3,451 2018-12-12, 12:40 AM
Last Post: TomArrow
  Best Anti-Aliasing AVIsynth filter? PDB 3 4,492 2017-08-10, 03:12 AM
Last Post: IcePrick
  Bitrate and video encoding spoRv 0 2,102 2017-05-01, 11:19 AM
Last Post: spoRv
  Banding With Avisynth PDB 5 5,344 2017-03-16, 03:56 PM
Last Post: PDB
  Avisynth: multithreading, 64bit and more spoRv 2 3,464 2017-03-11, 03:31 PM
Last Post: spoRv



Users browsing this thread: 1 Guest(s)