Thread Rating:
  • 2 Vote(s) - 5 Average
  • 1
  • 2
  • 3
  • 4
  • 5
Avisynth parallel encoding
#1
Thumbs Up 
Studing a good way to improve speed with avisynth, I tested some different path...
  • New CPU: obvious... but take in account the complete cpubenchmark score, not only the single thread - if, for example, your old dual core CPU has a single thread score of 500 and complete score of 1000, and the new quad core has a single thread score of 1000 and complete score of 4000, it will go four time faster, more or less!
  • Faster HDD (or better SSD): haven't tried an SSD yet, but I'm pretty sure this will not improve the overall speed.
  • Avisynth 64bit: could indeed improve speed, but it has quite few compatible plugins...
  • MT version and other multithread plugins: could work, but sometimes MT crashes, and other plugins didn't work - probably my fault
  • Parallel encoding: it works well! Follow me...

I tried this path in the latest days; I didn't dare to try it with my poor old dual core, dual threaded CPU, but with the new quad core, eight treaded one, I wanted to give it a try... and, oh boys, it works, very well!

How it works?

Well, just take your script, and "cut" it in several pieces - it seems that four pieces works well here, they run the CPU at around 80%, leaving some breath for surfing and other little things; probably it could work with more pieces, but your mileage can vary.

Before

project1.avs
Code:
avisource("your source.avi")
filter1
filter2
filter3

After

project1_part1.avs
Code:
avisource("your source.avi")
filter1
filter2
filter3
trim(000000,040020)

project1_part2.avs
Code:
avisource("your source.avi")
filter1
filter2
filter3
trim(040000,080020)

project1_part3.avs
Code:
avisource("your source.avi")
filter1
filter2
filter3
trim(080000,120020)

project1_part4.avs
Code:
avisource("your source.avi")
filter1
filter2
filter3
trim(120000,000000)

project1_final.avs
Code:
a=avisource("project1_part1.avi").trim(000000,039999)
b=avisource("project1_part2.avi").trim(000000,039999)
c=avisource("project1_part3.avi").trim(000000,039999)
d=avisource("project1_part4.avi")

a+b+c+d

Encode them at the same time; speed improvement: from 250% up to 400%, depending on how "heavy" are the filters/plugins...

You know, instead of waiting, let's say, two days, it's better to wait "just" 12 hours, don't you think?

Hope this would be of some help.
Sadly my projects are lost due to an HDD crash... Sad
Fundamental Collection | Vimeo channel | My blog
Reply
Thanks given by: TomArrow , born113
#2
I've recently jumped over to avisynth_mt as well now that I have 8 cores. I probably should have done this while on 4 cores too to be honest but I always heard that avisynth_mt was unstable. However, the speed gains are worth it as long as your workflow remains simple enough. With a really complicated script I'd recommend staying with single thread. This is how I initiate my multi-threaded scripts:

Code:
setmemorymax(2048)
setmtmode(3)
avisource(blah.avi)
trim()
setmtmode(2)
temporaldenoiser()
resizer()

The script setup I use for the gout syncing of laserdiscs looks similar to this. It usually encodes at less than 5fps on my old amd and my new ryzen. However, once I enable multi-threading I get 30+ fps during encode. It's like a gamechanger when it comes to rendering out stuff. Instead of a 15 hour encode it's under 2 hours for an entire film.

EDIT: I should mention that this will max out every core and thread to 100% for the duration of the encode. I have a 360mm radiator water-cooled setup now and it struggles to keep the cpu under 60C.
Reply
Thanks given by:
#3
(2017-03-22, 10:16 PM)spoRv Wrote: project1_final.avs
Code:
a=avisource("project1_part1.avi").trim(000000,039999)
b=avisource("project1_part1.avi").trim(000000,039999)
c=avisource("project1_part1.avi").trim(000000,039999)
d=avisource("project1_part1.avi")

a+b+c+d
Won't this give you the same clip repeated 4 times? I take it all of the scripts run simultaneously in virtualdub, might have to try this one day
Reply
Thanks given by:
#4
I remember I used MT some years ago, and *probably* it was faster than non-MT version... still can't remember why I have not used it anymore - probably problems with some of my scripts...

But I would give it a try, sooner or later... AFAIK it works only with certain filters/plugins, though... can you confirm this?

Ryzen: happy to know it works well; you know what? A great review about it would be surely appreciated by many - certainly by me! Big Grin
Sadly my projects are lost due to an HDD crash... Sad
Fundamental Collection | Vimeo channel | My blog
Reply
Thanks given by:
#5
Told you it's great way to boost your script Smile
Reply
Thanks given by:
#6
Ok
Sadly my projects are lost due to an HDD crash... Sad
Fundamental Collection | Vimeo channel | My blog
Reply
Thanks given by:
#7
About Ryzen I can just say that I'm happy with it's capabilities however I'd recommend waiting a few months before pulling the trigger due to the ram instability and constant bios updates. With that being said, I've reached some impressive benchmark numbers with that cinegy app that poita posted. Here is a link to my post with the results:

https://forum.fanres.com/thread-1351-pos...l#pid26928

(2017-03-23, 02:28 AM)spoRv Wrote: Ryzen: happy to know it works well; you know what? A great review about it would be surely appreciated by many - certainly by me! Big Grin

I will continue to update that post and give any relevant information that I come across. Like I said, I recommend waiting a few months for things to stabilize before buying. I couldn't even get POST for the first few days after putting everything together. The last couple bios updates have ironed out a few issues with the ram speeds but I still can't reach 3000 mhz with my ram that is rated at that speed so they have a ways to go imo.
Reply
Thanks given by:
#8
So I figured I'd try out avisynth+ with multi-threading and from my experience the last few hours this is a terrible idea. No matter what it only uses 4 cores at about 10% and averages 3.5 fps. It's really a mess. Uninstalling and getting back to avisynth 2.6mt was also a nightmare. Had to delete registry keys, hidden folders, etc. just to get it to autoload the plugin directory again since avs+ moves all of the pointers in the registry and uninstalling it didn't remove them.

What I'm trying to say here is avoid avs+ especially the multi-threaded variant. If somebody has had success please elaborate lol. I think I might try Vapoursynth here soon just to see if I can figure it out.
Reply
Thanks given by:
#9
Sad to hear that... at least your experience would help someone - me included... Wink
Sadly my projects are lost due to an HDD crash... Sad
Fundamental Collection | Vimeo channel | My blog
Reply
Thanks given by:
#10
(2017-03-23, 02:27 AM)zoidberg Wrote:
(2017-03-22, 10:16 PM)spoRv Wrote: project1_final.avs
Code:
a=avisource("project1_part1.avi").trim(000000,039999)
b=avisource("project1_part1.avi").trim(000000,039999)
c=avisource("project1_part1.avi").trim(000000,039999)
d=avisource("project1_part1.avi")

a+b+c+d
Won't this give you the same clip repeated 4 times? I take it all of the scripts run simultaneously in virtualdub, might have to try this one day

I copied and pasted without changing part numbers... now fixed, thanks!



BlackMirror suggested to use mp_pipeline; I'm trying it right now, and speed increase is around 80%! Not bad at all... I should try it with parallel encoding, it may multiply speed up to 6x, if not more (if I did my math right).
Waiting for him to write a small guide to it!

Edit: without parallel encoding, using one of my slower filter, mp_pipeline increased speed more than 3x!!! Ok
Sadly my projects are lost due to an HDD crash... Sad
Fundamental Collection | Vimeo channel | My blog
Reply
Thanks given by:


Possibly Related Threads…
Thread Author Replies Views Last Post
  [Help] Encoding HDR to SDR awsmguy145 3 130 2020-09-28, 12:01 PM
Last Post: Chewtobacca
  x264 encoding from Adobe Premiere DoomBot 26 8,690 2019-03-25, 08:26 PM
Last Post: jaminmc
  [Help] Encoding issue with Xvid4psp v5.0.37.8 Beber 7 1,913 2018-12-12, 12:40 AM
Last Post: TomArrow
  Best Anti-Aliasing AVIsynth filter? PDB 3 3,261 2017-08-10, 03:12 AM
Last Post: IcePrick
  Bitrate and video encoding spoRv 0 1,477 2017-05-01, 11:19 AM
Last Post: spoRv
  Banding With Avisynth PDB 5 4,062 2017-03-16, 03:56 PM
Last Post: PDB
  Avisynth: multithreading, 64bit and more spoRv 2 2,593 2017-03-11, 03:31 PM
Last Post: spoRv
  Encoding : where to go from here...? Docsap 2 2,121 2017-02-26, 04:42 AM
Last Post: spoRv
  H.265 Encoding PDB 3 2,314 2016-10-18, 08:49 PM
Last Post: Feallan
  Telecide and Decimate in AVIsynth PDB 3 3,997 2016-05-25, 10:50 PM
Last Post: Chewtobacca

Forum Jump:


Users browsing this thread: 1 Guest(s)