如何合并两个计划执行的程序

Question

我正在尝试合并两个程序或编写第三个程序，将这两个程序作为函数调用。它们应该一个接一个地运行，并在以分钟为单位的一定时间间隔后运行。类似于 make 文件，稍后将包含更多程序。我无法合并它们，也无法将它们放入某种格式，以便我可以在新的main 程序中调用它们。

program_master_id.py 从文件夹位置选择*.csv 文件，计算后将master_ids.csv 文件附加到文件夹的另一个位置。

Program_master_count.py 将count 与相应timeseries 中Ids 的计数相除。

Program_1 master_id.py

import pandas as pd
import numpy as np

# csv file contents
# Need to change to path as the Transition_Data has several *.CSV files

csv_file1 = 'Transition_Data/Test_1.csv' 
csv_file2 = '/Transition_Data/Test_2.csv'

#master file to be appended only

master_csv_file = 'Data_repository/master_lac_Test.csv'

csv_file_all = [csv_file1, csv_file2]

# read csv into df using list comprehension
# I use buffer here, replace stringIO with your file path

df_all = [pd.read_csv(csv_file) for csv_file in csv_file_all]

# processing
# =====================================================
# concat along axis=0, outer join on axis=1
merged = pd.concat(df_all, axis=0, ignore_index=True, join='outer').set_index('Ids')

# custom function to handle/merge duplicates on Ids (axis=0)
def apply_func(group):
    return group.fillna(method='ffill').iloc[-1]

# remove Ids duplicates
merged_unique = merged.groupby(level='Ids').apply(apply_func)

# do the subtraction

df_master = pd.read_csv(master_csv_file, index_col=['Ids']).sort_index()

# select matching records and horizontal concat
df_matched = pd.concat([df_master,merged_unique.reindex(df_master.index)], axis=1)

# use broadcasting
df_matched.iloc[:, 1:] = df_matched.iloc[:, 1:].sub(df_matched.iloc[:, 0], axis=0)

print(df_matched)

Program_2 master_count.py#This does not give any error nor gives any output.

import pandas as pd
import numpy as np

csv_file1 = '/Data_repository/master_lac_Test.csv'
csv_file2 = '/Data_repository/lat_lon_master.csv'

df1 = pd.read_csv(csv_file1).set_index('Ids')

# need to sort index in file 2
df2 = pd.read_csv(csv_file2).set_index('Ids').sort_index()

# df1 and df2 has a duplicated column 00:00:00, use df1 without 1st column
temp = df2.join(df1.iloc[:, 1:])

# do the division by number of occurence of each Ids 
# and add column 00:00:00
def my_func(group):
    num_obs = len(group)
    # process with column name after 00:30:00 (inclusive)
    group.iloc[:,4:] = (group.iloc[:,4:]/num_obs).add(group.iloc[:,3], axis=0)
    return group

result = temp.groupby(level='Ids').apply(my_func)

我正在尝试编写一个main 程序，该程序将首先调用master_ids.py，然后调用master_count.py。它们是一种将两者合并到一个程序中还是将它们编写为函数并在新程序中调用这些函数的方法？请建议。

Answer 1

好吧，假设你有 program1.py：

import pandas as pd
import numpy as np

def main_program1():
    csv_file1 = 'Transition_Data/Test_1.csv' 
    ...
    return df_matched

然后program2.py：

import pandas as pd
import numpy as np

def main_program2():
    csv_file1 = '/Data_repository/master_lac_Test.csv'
    ...
    result = temp.groupby(level='Ids').apply(my_func)
    return result

您现在可以在单独的 python 程序中使用它们，比如 main.py

import time
import program1 # imports program1.py
import program2 # imports program2.py

df_matched = program1.main_program1()
print(df_matched)
# wait
min_wait = 1
time.sleep(60*min_wait)
# call the second one
result = program2.main_program2()

有很多方法可以“改进”这些，但希望这能告诉你要点。我特别推荐您使用What does if __name__ == "__main__": do? 在每个文件中，以便它们可以轻松地从命令行执行或从 python 调用。

另一种选择是 shell 脚本，对于您的 'master_id.py' 和 'master_count.py' 变成（以最简单的形式）

python master_id.py
sleep 60
python master_count.py

保存在'main.sh'中，可以这样执行

sh main.sh