要从文件中随机选择一行,您可以使用节省空间的单通道reservoir-sampling algorithm。要删除该行,您可以打印除所选行之外的所有内容:
#!/usr/bin/env python3
import fileinput
with open(filename) as file:
k = select_random_it(enumerate(file), default=[-1])[0]
if k >= 0: # file is not empty
with fileinput.FileInput(filename, inplace=True, backup='.bak') as file:
for i, line in enumerate(file):
if i != k: # keep line
print(line, end='') # stdout is redirected to filename
select_random_it() implements the reservoir-sampling algorithm:
import random
def select_random_it(iterator, default=None, randrange=random.randrange):
"""Return a random element from iterator.
Return default if iterator is empty.
iterator is exhausted.
O(n)-time, O(1)-space algorithm.
"""
# from https://stackoverflow.com/a/1456750/4279
# select 1st item with probability 100% (if input is one item, return it)
# select 2nd item with probability 50% (or 50% the selection stays the 1st)
# select 3rd item with probability 33.(3)%
# select nth item with probability 1/n
selection = default
for i, item in enumerate(iterator, start=1):
if randrange(i) == 0: # random [0..i)
selection = item
return selection
要从文件中打印k 随机行并删除它们:
#!/usr/bin/env python3
import random
import sys
k = 10
filename = 'TestingDeleteLines.txt'
with open(filename) as file:
random_lines = reservoir_sample(file, k) # get k random lines
if not random_lines: # file is empty
sys.exit() # do nothing, exit immediately
print("\n".join(map(str.strip, random_lines))) # print random lines
delete_lines(filename, random_lines) # delete them from the file
其中reservoir_sample() 使用与select_random_it() 相同的算法,但允许选择k 项目而不是一个:
import random
def reservoir_sample(iterable, k,
randrange=random.randrange, shuffle=random.shuffle):
"""Select *k* random elements from *iterable*.
Use O(n) Algorithm R https://en.wikipedia.org/wiki/Reservoir_sampling
If number of items less then *k* then return all items in random order.
"""
it = iter(iterable)
if not (k > 0):
raise ValueError("sample size must be positive")
sample = list(islice(it, k)) # fill the reservoir
shuffle(sample)
for i, item in enumerate(it, start=k+1):
j = randrange(i) # random [0..i)
if j < k:
sample[j] = item # replace item with gradually decreasing probability
return sample
和delete_lines() 实用函数从文件中删除选择的随机行:
import fileinput
import os
def delete_lines(filename, lines):
"""Delete *lines* from *filename*."""
lines = set(lines) # for amortized O(1) lookup
with fileinput.FileInput(filename, inplace=True, backup='.bak') as file:
for line in file:
if line not in lines:
print(line, end='')
os.unlink(filename + '.bak') # remove backup if there is no exception
reservoir_sample()、delete_lines() 函数不会将整个文件加载到内存中,因此它们可以处理任意大文件。