Pyspark没有记录到文件

Question

我正在使用命令 spark-submit 启动一个 pyspark 脚本，将标准输出重定向到文件也使用 tee 来生成日志。

命令如下：

spark-submit test.py  | tee test.xxx

问题是只有用户定义函数UDF内的print只打印在终端上而不是文件tee test.xxx，而所有其他打印都将写入终端和文件。

为了模拟这种行为，我创建了这个最小的完整工作示例：

from pyspark import SparkContext
import pyspark.sql.functions as F #udf, col, count, sum, when, avg, mean, min
from pyspark.sql import SQLContext
from pyspark.sql.types import *
def cutURL(input):
    cutURL.lineNumber += 1
    if input==None or input=="" or not(isinstance(input, str)):
        print("WARNING: not proper string URL: empty or null. Possible line: " + str(cutURL.lineNumber))
        res = "Unknown"
        if input==None: print("string is none")
        elif not(isinstance(input, str)): print("input not a string")
        elif input=="": print("empty string")
        return res
    res = input
    try:
        if (bool(re.search("/devices(.+?)&maxdist=[0-9]+", input))):
            res = re.search("/devices(.+?)&maxdist=[0-9]+", input).group()
        else:
            res = re.sub(r'.*?(/devices/[^/]*_)[^/_]*(/read)', r'\1\2', input)
    except:
        print("WARning in cutURL:")
        print(" not matching regular expression: is the string")
    return res


sc = SparkContext.getOrCreate()
sc.setLogLevel("WARN")
sqlContext = SQLContext(sc)
cutURL.lineNumber = 0
print("This will be printed to both file and terminal")
df = sqlContext.createDataFrame([None, "example",  "others"], "string").toDF("url")
cut_URL_udf =  F.udf(cutURL, StringType())
df2 = df.select(cut_URL_udf("url").alias("cut_URL"))
df2.show()

在这种情况下，字符串WARNING: not proper string URL: empty or null. Possible line: 仅打印在终端上，而不打印到文件中。

如何使 pyspark UDF 中生成的输出也重定向到文件？

编辑 为了更好地解释我的问题，我添加了print("This will be printed to both file and terminal") 行。这将打印到终端并记录到文件，而 udf 中的 print 仅到终端。

Answer 1

编辑：抱歉误读了已经重定向的内容

解决方案是使用正确的日志记录而不是打印：

查找 python 日志记录：

• https://docs.python.org/3/library/logging.html
• https://docs.python.org/3/howto/logging-cookbook.html

登录到控制台和文件的示例：

import logging

# set up logging to file - see previous section for more details
logging.basicConfig(level=logging.DEBUG,
                    format='%(asctime)s %(name)-12s %(levelname)-8s %(message)s',
                    datefmt='%m-%d %H:%M',
                    filename='/temp/myapp.log',
                    filemode='w')
# define a Handler which writes INFO messages or higher to the sys.stderr
console = logging.StreamHandler()
console.setLevel(logging.INFO)
# set a format which is simpler for console use
formatter = logging.Formatter('%(name)-12s: %(levelname)-8s %(message)s')
# tell the handler to use this format
console.setFormatter(formatter)
# add the handler to the root logger
logging.getLogger('').addHandler(console)

# Now, we can log to the root logger, or any other logger. First the root...
logging.info('Jackdaws love my big sphinx of quartz.')