【问题标题】:multiprocessing with moviepy使用moviepy进行多处理
【发布时间】:2021-05-15 21:52:37
【问题描述】:

最近我做了一个脚本,需要5分钟的视频剪辑和5个视频剪辑,每个视频1分钟,效果很好,但是对于像我这样的电脑来说太长了,而且我的电脑部分性能非常好:

Intel(R) Core(TM) i7-10700 CPU @ 2.90GHz, 2904 Mhz, 8 Core(s), 16 逻辑处理器

安装的物理内存 (RAM) 16.0 GB

所以我搜索了moviepy的文档“threads”,我在“write_videofile”函数中找到了一些我可以设置我的线程加速的东西,我试过了,但它没有用,我的意思是它有效但它只有它可能更改为更多 2 或 3 it/s。

我还找到了多线程的示例代码,但似乎代码不起作用,因为moviepy.multithreading 不存在于moviepy 库中,请帮助我加快渲染速度, 谢谢

这是我找到的代码:

from moviepy.multithreading import multithread_write_videofile

def concat_clips():
    files = [
        "myclip1.mp4",
        "myclip2.mp4",
        "myclip3.mp4",
        "myclip4.mp4",
    ]
    multithread_write_videofile("output.mp4", get_final_clip, {"files": files})


def get_final_clip(files):
    clips = [VideoFileClip(file) for file in files]
    final = concatenate_videoclips(clips, method="compose")
    return final

这是我的代码:

from moviepy.video.io.ffmpeg_tools import ffmpeg_extract_subclip
from moviepy.editor import *
from numpy import array, true_divide
import cv2
import time


# ffmpeg_extract_subclip("full.mp4", start_seconds, end_seconds, targetname="cut.mp4")




def duration_clip(filename):
    clip = VideoFileClip(filename)
    duration = clip.duration
    return duration


current_time = time.strftime("%Y_%m_%d_%H_%M_%S")


def main():
    global duration
    start = 0
    cut_name_num = 1
    end_seconds = start + 60
    video_duration = duration_clip("video.mp4")
    


    txt = input("Enter Your text please: ") [::-1]
    txt_part = 1

    while start < int(video_duration):
        final_text = f"{str(txt_part)} {txt}"



        try:
            try:
                os.makedirs(f"result_{str(current_time)}/result_edit")
            except FileExistsError:
                pass            
            ffmpeg_extract_subclip("video.mp4", start, end_seconds, targetname=f"result_{str(current_time)}/cut_{str(cut_name_num)}.mp4")

            clip = VideoFileClip(f"result_{str(current_time)}/cut_{str(cut_name_num)}.mp4")

            clip = clip.subclip(0, 60)

            clip = clip.volumex(2)

            txt_clip = TextClip(final_text, font="font/VarelaRound-Regular.ttf", fontsize = 50, color = 'white')

            txt_clip = txt_clip.set_pos(("center","top")).set_duration(60) 

            video = CompositeVideoClip([clip, txt_clip])
            
            clip.write_videofile(f"result_{str(current_time)}/result_edit/cut_{str(cut_name_num)}.mp4")

        except:
            try:
                os.makedirs(f"result_{str(current_time)}/result_edit")
            except FileExistsError:
                pass
            
            ffmpeg_extract_subclip("video.mp4", start, video_duration, targetname=f"result_{str(current_time)}/cut_{str(cut_name_num)}.mp4")

            clip_duration = duration_clip(f"result_{str(current_time)}/cut_{str(cut_name_num)}.mp4")

            clip = VideoFileClip(f"result_{str(current_time)}/cut_{str(cut_name_num)}.mp4")

            clip = clip.subclip(0, clip_duration)

            clip = clip.volumex(2)

            txt_clip = TextClip(final_text, font="font/VarelaRound-Regular.ttf", fontsize = 50, color = 'white')

            txt_clip = txt_clip.set_pos(("center","top")).set_duration(60) 

            video = CompositeVideoClip([clip, txt_clip])

            clip.write_videofile(f"result_{str(current_time)}/result_edit/cut_{str(cut_name_num)}.mp4")

        start += 60
        cut_name_num += 1
        end_seconds = start + 60
        txt_part += 1


if __name__ == "__main__":
    main()

【问题讨论】:

  • 我想知道如何在线程中连接文件。至于我,你不能在写完第一个文件之前开始写第二个文件,所以第二个文件取决于第一个文件,没有地方将它分成两个线程。如果您必须剪切文件,您可以拆分问题,因为创建的文件不会相互依赖。但也许使用多处理它会比使用线程运行得更快。但我想知道如果你直接使用 ffmpeg 或 ffmpeg-python 会有多快。
  • 如何使用多处理来做到这一点?
  • 首先您必须使用可以单独运行的代码创建函数 - 即。您在while-loop 中运行的代码 - 在final_text = ... 之后和start += 60 之前的代码。首先尝试将它作为函数运行——你会看到是否必须使用一些参数来运行这个函数。稍后您可以尝试使用multiprocessing 运行此功能。
  • 您在tryexcept 中重复相同的代码但具有不同的值。如果您使用if/else 来测试值start、video_duration 等,那么代码会更短且更具可读性。顺便说一句:在较新的 Python 中,您可以使用 os.makedirs(..., exist_ok=True),然后您就不需要使用 try/except
  • 如果你使用f-string那么你就不需要str()

标签: python moviepy


【解决方案1】:

使用进程我只将时间缩短了 15-20 秒,因为 ffmpeg 即使在单个进程中也几乎使用了全部 CPU 能力,而我的计算机没有能力更快地运行其他进程。


首先我减少了代码以使其更短。

tryexcept 中的代码具有相似的元素,所以我将它们移到了 try/except 之外。

接下来我用了

        if end > video_duration:
            end = video_duration

我根本不需要try/except

使用os.makedirs(..., exist_ok=True)我不需要在try/except运行它

同时我用 20 秒缩短了时间

        clip = VideoFileClip(filename).subclip(start, end)

而不是

        temp_filename  = f"{base_folder}/cut_{number}.mp4"
        
        fmpeg_extract_subclip(filename, start, end, targetname=temp_filename)
        
        clip = VideoFileClip(temp_filename)

这样我就不用在磁盘上写子剪辑,也不必从磁盘再次读取它。


from moviepy.video.io.ffmpeg_tools import ffmpeg_extract_subclip
from moviepy.editor import *
import time

def main():

    text = input("Enter Your text please: ") [::-1]
    #text = 'Hello World'
    
    base_folder = time.strftime("result_%Y_%m_%d_%H_%M_%S")    
    os.makedirs(f"{base_folder}/result_edit", exist_ok=True)
    
    filename = "video.mp4"
    #filename = "BigBuckBunny.mp4"
    
    video_duration = VideoFileClip(filename).duration

    number = 0 # instead of `cut_name_num` and `txt_part` because both had the same value

    time_start = time.time()

    for start in range(0, int(video_duration), 60):

        end = start + 60
        
        if end > video_duration:
            end = video_duration

        number += 1

        clip_duration = end - start
        print(f'[DEBUG] number: {number:2} | start: {start:6.2f} | end: {end:6.2f} | duration: {clip_duration:.2f}')
       
        final_text = f"{number} {text}"
    
        temp_filename  = f"{base_folder}/cut_{number}.mp4"
        final_filename = f"{base_folder}/result_edit/cut_{number}.mp4"
        
        #ffmpeg_extract_subclip(filename, start, end, targetname=temp_filename)
        
        #clip = VideoFileClip(temp_filename)
        clip = VideoFileClip(filename).subclip(start, end)
        clip = clip.volumex(2)
    
        txt_clip = TextClip(final_text, font="font/VarelaRound-Regular.ttf", fontsize=50, color='white')
        txt_clip = txt_clip.set_pos(("center","top")).set_duration(60)
        
        video = CompositeVideoClip([clip, txt_clip])
    
        video.write_videofile(final_filename)

    # - after loop -
    
    # because I use `number += 1` before loop so now `number` has number of subclips
    print('number of subclips:', number)

    time_end = time.time()
    
    diff = time_end - time_start
    print(f'time: {diff:.2f}s ({diff//60:02.0f}:{diff%60:02.2f})')
    
    
if __name__ == "__main__":
    main()

接下来我将代码移动到带有参数my_process(filename, text, start, end, number, base_folder)的函数

from moviepy.video.io.ffmpeg_tools import ffmpeg_extract_subclip
from moviepy.editor import *
import time

def my_process(filename, text, start, end, number, base_folder):

    clip_duration = end - start
    print(f'[DEBUG] number: {number:2} | start: {start:6.2f} | end: {end:6.2f} | duration: {clip_duration:.2f}')
    
    final_text = f"{number} {text}"

    temp_filename  = f"{base_folder}/cut_{number}.mp4"
    final_filename = f"{base_folder}/result_edit/cut_{number}.mp4"
    
    #print('[DEBUG] ffmpeg_extract_subclip')
    #ffmpeg_extract_subclip(filename, start, end, targetname=temp_filename)
    
    #print('[DEBUG] VideoClip')
    #clip = VideoFileClip(temp_filename)
    clip = VideoFileClip(filename).subclip(start, end)
    clip = clip.volumex(2)

    #print('[DEBUG] TextClip')
    txt_clip = TextClip(final_text, font="font/VarelaRound-Regular.ttf", fontsize=50, color='white')
    txt_clip = txt_clip.set_pos(("center","top")).set_duration(60)
    
    #print('[DEBUG] CompositeVideoClip')
    video = CompositeVideoClip([clip, txt_clip])

    #print('[DEBUG] CompositeVideoClip write')
    video.write_videofile(final_filename)
    #print('[DEBUG] CompositeVideoClip end')
    

def main():

    text = input("Enter Your text please: ") [::-1]
    #text = 'Hello World'
    
    base_folder = time.strftime("result_%Y_%m_%d_%H_%M_%S")    
    os.makedirs(f"{base_folder}/result_edit", exist_ok=True)
    
    filename = "video.mp4"
    #filename = "BigBuckBunny.mp4"
    
    video_duration = VideoFileClip(filename).duration

    number = 0 # instead of `cut_name_num` and `txt_part` because both had the same value

    time_start = time.time()
    
    for start in range(0, int(video_duration), 60):

        end = start + 60
        
        if end > video_duration:
            end = video_duration
        
        number += 1
        
        my_process(filename, text, start, end, number, base_folder)
        
    # - after loop -
    
    # because I use `number += 1` before loop so now `number` has number of subclips
    print('number of subclips:', number)

    time_end = time.time()
    
    diff = time_end - time_start
    print(f'time: {diff:.2f}s ({diff//60:02.0f}:{diff%60:02.2f})')


if __name__ == "__main__":
    main()

现在我可以使用标准模块multiprocessing在分离的进程中运行函数

(或标准模块threadingconcurrent.futures或外部模块JoblibRay等)。

它启动单个进程

# it has to use named arguments`target=`, `args=`

p = multiprocessing.Process(target=my_process, args=(filename, text, start, end, number, base_folder))
p.start()  # start it

但如果我在循环中使用它,那么我将同时启动许多进程。


from moviepy.video.io.ffmpeg_tools import ffmpeg_extract_subclip
from moviepy.editor import *
import time
import multiprocessing

def my_process(filename, text, start, end, number, base_folder):

    clip_duration = end - start
    print(f'[DEBUG] number: {number:2} | start: {start:6.2f} | end: {end:6.2f} | duration: {clip_duration:.2f}')
    
    final_text = f"{number} {text}"

    temp_filename  = f"{base_folder}/cut_{number}.mp4"
    final_filename = f"{base_folder}/result_edit/cut_{number}.mp4"
    
    #print('[DEBUG] ffmpeg_extract_subclip')
    #ffmpeg_extract_subclip(filename, start, end, targetname=temp_filename)
    
    #print('[DEBUG] VideoClip')
    #clip = VideoFileClip(temp_filename)
    clip = VideoFileClip(filename).subclip(start, end)
    clip = clip.volumex(2)

    #print('[DEBUG] TextClip')
    txt_clip = TextClip(final_text, font="font/VarelaRound-Regular.ttf", fontsize=50, color='white')
    txt_clip = txt_clip.set_pos(("center","top")).set_duration(60)
    
    #print('[DEBUG] CompositeVideoClip')
    video = CompositeVideoClip([clip, txt_clip])

    #print('[DEBUG] CompositeVideoClip write')
    video.write_videofile(final_filename)
    #print('[DEBUG] CompositeVideoClip end')
    

def main():

    text = input("Enter Your text please: ") [::-1]
    #text = 'Hello World'
    
    base_folder = time.strftime("result_%Y_%m_%d_%H_%M_%S")    
    os.makedirs(f"{base_folder}/result_edit", exist_ok=True)
    
    filename = "video.mp4"
    #filename = "BigBuckBunny.mp4"
    
    video_duration = VideoFileClip(filename).duration

    number = 0 # instead of `cut_name_num` and `txt_part` because both had the same value

    time_start = time.time()
    
    all_processes = []
    for start in range(0, int(video_duration), 60):

        end = start + 60
        
        if end > video_duration:
            end = video_duration
        
        number += 1
        
        print("add process:", number)
        p = multiprocessing.Process(target=my_process, args=(filename, text, start, end, number, base_folder)) # it has to use `target=`, `args=`
        p.start()  # start it
        all_processes.append(p)  # keep it to use `join()`
            
    # - after loop -
    
    for p in all_processes:
        p.join()  # wait for the end of process
        
    # because I use `number += 1` before loop so now `number` has number of subclips
    print('number of subclips:', number)

    time_end = time.time()
    
    diff = time_end - time_start
    print(f'time: {diff:.2f}s ({diff//60:02.0f}:{diff%60:02.2f})')


if __name__ == "__main__":
    main()

11 个子剪辑的先前版本启动 11 个进程。使用Pool(4),您可以将所有进程放入池中,它将同时运行 4 个进程。当一个进程完成任务时,它将使用新参数启动下一个进程。

这次我使用循环为所有进程创建带有参数的列表

args_for_all_processes = []

for start in range(0, int(video_duration), 60):

    end = start + 60
    
    if end > video_duration:
        end = video_duration
    
    number += 1
    print("add process:", number)
    
    args_for_all_processes.append( (filename, text, start, end, number, base_folder) )

我将这个列表与Pool 一起使用,剩下的事情就交给它了。

# I have 4 CPU so I use Pool(4) - but without value it should automatically use `os.cpu_count()`
with multiprocessing.Pool(4) as pool:      
    results = pool.starmap(my_process, args_for_all_processes)
    #print(results)

Pool 可能会以不同的顺序启动进程,但如果他们使用return 发送一些结果,那么Pool 会以正确的顺序给出结果。

from moviepy.video.io.ffmpeg_tools import ffmpeg_extract_subclip
from moviepy.editor import *
import time
import multiprocessing

def my_process(filename, text, start, end, number, base_folder):

    clip_duration = end - start
    print(f'[DEBUG] number: {number:2} | start: {start:6.2f} | end: {end:6.2f} | duration: {clip_duration:.2f}')
    
    final_text = f"{number} {text}"

    temp_filename  = f"{base_folder}/cut_{number}.mp4"
    final_filename = f"{base_folder}/result_edit/cut_{number}.mp4"
    
    #print('[DEBUG] ffmpeg_extract_subclip')
    #ffmpeg_extract_subclip(filename, start, end, targetname=temp_filename)
    
    #print('[DEBUG] VideoClip')
    #clip = VideoFileClip(temp_filename)
    clip = VideoFileClip(filename).subclip(start, end)
    clip = clip.volumex(2)

    #print('[DEBUG] TextClip')
    txt_clip = TextClip(final_text, font="font/VarelaRound-Regular.ttf", fontsize=50, color='white')
    txt_clip = txt_clip.set_pos(("center","top")).set_duration(60)
    
    #print('[DEBUG] CompositeVideoClip')
    video = CompositeVideoClip([clip, txt_clip])

    #print('[DEBUG] CompositeVideoClip write')
    video.write_videofile(final_filename)
    #print('[DEBUG] CompositeVideoClip end')
    
    # return "OK"  # you can use `return` to send result/information to main process.
    
def main():

    text = input("Enter Your text please: ") [::-1]
    #text = 'Hello World'
    
    base_folder = time.strftime("result_%Y_%m_%d_%H_%M_%S")    
    os.makedirs(f"{base_folder}/result_edit", exist_ok=True)
    
    filename = "video.mp4"
    #filename = "BigBuckBunny.mp4"
    
    video_duration = VideoFileClip(filename).duration

    number = 0 # instead of `cut_name_num` and `txt_part` because both had the same value

    time_start = time.time()
    
    # first create list with arguments for all processes
    
    args_for_all_processes = []
    
    for start in range(0, int(video_duration), 60):

        end = start + 60
        
        if end > video_duration:
            end = video_duration
        
        number += 1
        print("add process:", number)
        
        args_for_all_processes.append( (filename, text, start, end, number, base_folder) )

    # - after loop -            
        
    # next put all processes to pool
        
    with multiprocessing.Pool(4) as pool:  # I have 4 CPU so I use Pool(4) - but it should use `os.cpu_count()` in `Pool()
    
        results = pool.starmap(my_process, args_for_all_processes)
        #print(results)
            
    # - after loop -
    
    # because I use `number += 1` before loop so now `number` has number of subclips
    print('number of subclips:', number)

    time_end = time.time()
    
    diff = time_end - time_start
    print(f'time: {diff:.2f}s ({diff//60:02.0f}:{diff%60:02.2f})')


if __name__ == "__main__":
    main()

【讨论】:

  • 你能解释一下你为什么这样做吗? if end > video_duration: end = video_duration 我有点困惑,因为如果我拍摄的视频可以说是 5:01,那么它会在 5:00 而不是 5:01 结束
  • 最后一个剪辑可以短于 60 秒 - 如果您尝试使用 .subclip(start, end)end 大于 video_duration,则会引发错误。使用if 我得到正确的end 值,我不必使用try/except
  • 哦,现在我明白了,我知道这一点,但我忘记了为什么要尝试,除了你的方法更好,代码更少,谢谢
  • 我知道代码运行良好,但我尝试了我和你的代码,并且没有多处理的代码比其他代码更短。我有一台相当不错的电脑,较短的一台是 330 秒,另一台是 425 秒,你知道为什么吗?
  • 我在一个 49 分钟的文件上试了一下,它的工作速度非常快,对于短视频我应该使用单个进程,我认为它可能比 5 分钟的视频文件做得更快.
猜你喜欢
  • 2017-04-06
  • 2019-11-22
  • 1970-01-01
  • 1970-01-01
  • 1970-01-01
  • 1970-01-01
  • 1970-01-01
  • 2011-04-23
  • 2017-10-09
相关资源
最近更新 更多