在磁盘上更快地保存图像

Question

我开发了一个 Python 多线程程序，一个生产者线程以高 fps（大约 75 fps）获取帧（512 x 640, 16uint），两个消费者线程，一个用于实时可视化，另一个用于保存为 16 位 tiff。对于每个消费者，我使用不同的队列。

实时可视化效果很好，但在视频停止后保存需要很长时间（2 分钟的录制甚至需要 20 秒）。为了保存，我使用了 tifffile 库或 cv2，具有类似的性能。

更新信息

图像是灰度 16 位 numpy 数组，直接放入队列中，不压缩，使用 tifffile.imsave 保存。可视化的第二个队列实时正常工作，因此保存必须是最慢的过程。我需要独立保存每张图像，保存 3D 暂时不是一个选项。使用不同的线程进行保存可能会破坏我的采集顺序。

考虑到我需要以与记录相同的顺序保存它们，有什么方法可以在 Python 和/或 OS (windows10) 中加速该过程？我有一个 SSD 970 EVO 磁盘驱动器

 class VideoGet():
          
            def __init__(self,input_dict,folder,record):
                self.handle=input_dict['handle']
                self.frame_t=input_dict['frametype']
                self.frameSize= input_dict['frameSize']
                # self.buffer = np.zeros(shape=(513,640), dtype=np.uint16)  
                self.record = record
                self.save = False
                self.done=False
                self.counter=0
                self.folder = folder
                
            def displayer(self,q2):
                
                while self.record is True:
                    if q2.empty() is True:
                        pass
                    else:
                        framedisplay = q2.get()
                        cv2.namedWindow("Video", cv2.WINDOW_NORMAL)
                        cv2.imshow("Video", framedisplay[:-1,:])   
    
                        cv2.waitKey(1)
                        q2.task_done()
    
                cv2.destroyAllWindows()         
         
                
            def consumer(self,q):
                data=[]
                 
                while True:
    
                        if q.empty():
                            time.sleep(0.002)
                            pass
    
                        else:
                                                    
                            frame_get = q.get()
                            if frame_get is None:                           
                                print('gone')
                                break
                            imsave(os.path.join(self.folder,(str(self.counter).zfill(5)+'.tiff')), frame_get[:-1,:])
                            if self.counter==0:
                                TS=1.0e-3 *struct.unpack('Q',(frame_get[-1,6:10]).tobytes())[0]
                            entr=[str(self.counter).zfill(5),str(round(1.0e3*(1.0e-3 *struct.unpack('Q',(frame_get[-1,6:10]).tobytes())[0]-TS)))]
                            data.append(entr) 
                                
                            self.counter=self.counter+1
                            q.task_done()                   
                     
                if data:
                      df = pd.DataFrame(data, columns = ['Picture', 'Timestamp'])
                      df.to_csv(os.path.join(self.folder,'timestamps.txt'), header=False, index=False, sep=' ')
                      print('done')
                      self.done=True
    
    
                   
            def producer(self,buffer,q,q2):
                
                while self.record is True:
    
                   buffer=np.empty_like(buffer)            
                   if camera.properties.get_frame(self.handle,self.frame_t,4,buffer,self.frameSize)==0:
                       frame=buffer
                       q2.put(frame)
                       
                       if self.save is True:
                           q.put(frame)
    
                       del frame
    
                print('None')
                q.put(None)
    
            def run(self,buffer,q,q2):
                
                    prod_thread=Thread(target=self.producer,args=(buffer,q,q2,))
                    display_thread=Thread(target=self.displayer,args=(q2,))
                    con_thread= Thread(target=self.consumer, args=(q,))
                    
                    
                    prod_thread.start()
                    display_thread.start()
                    con_thread.start()

Answer 1

当您不显示代码时，很难说出了什么问题。另外，我不清楚为什么如果您有多个作家，收购顺序会改变。

这是一个脚本，用于生成与图像大小相同的合成帧，并将它们按顺序保存为 TIFF 文件。它的速度随着编写器线程的增加呈线性增长：

NFRAMES      NWRITERS       TIME(s)
1000            1             1.48
1000            2             0.78
1000            4             0.48

---

#!/usr/bin/env python3

import time
import numpy as np
import threading, queue
from tifffile import imsave

def writer(q):
    print('[WRITER] Started')
    total = 0
    while True:
        (frameNum, im) = q.get()
        if frameNum < 0:
            break
        # Save as TIFF
        imsave(f'frame-{frameNum}.tif', im)
        total += 1
    print(f'[WRITER] Complete: wrote {total} frames')

if __name__ == "__main__":

    # Edit these to suit
    NFRAMES = 1000
    NWRITERS= 4

    # Create dummy image of correct size
    h, w = 640, 512
    im = np.random.randint(0, 65536, (h,w), dtype=np.uint16)

    # Create a queue to pass frames to writer(s)
    q = queue.Queue(16)

    print('[MAIN] Started')
    start = time.time()

    # Create and start writer thread(s)
    threads = []
    for _ in range(NWRITERS):
        t = threading.Thread(target=writer, args=(q,))
        t.start()
        threads.append(t)

    # Generate a large number of frames to store
    for frameNum in range(NFRAMES):
        # Put a tuple of frameNum and image in queue
        q.put((frameNum, im))

    # Sentinel to tell each writer to exit
    for _ in range(NWRITERS):
        q.put((-1,-1))

    # Wait for our writer thread(s) to exit
    for thread in threads:
       thread.join()

    elapsed = time.time() - start;
    print(f'[MAIN] Complete: {NFRAMES} frames, with {NWRITERS} writers in {elapsed} seconds')

---

样本输出

[MAIN] Started
[WRITER] Started
[WRITER] Started
[WRITER] Started
[WRITER] Started
[WRITER] Complete: wrote 250 frames
[WRITER] Complete: wrote 250 frames
[WRITER] Complete: wrote 250 frames
[WRITER] Complete: wrote 250 frames
[MAIN] Complete: 1000 frames, with 4 writers in 0.4869719505310059 seconds

---

我注意到的一件事是，如果将tifffile.imsave() 替换为：

np.save(f'frame-{frameNum}.npy', im)