cur.execute() 上的 pymysql 失败

Question

我有一个带有两个表的 MySQL 数据库。 Files 保留先前解析的 CSV 文件的记录 - 我可以成功地将其所有行读入变量

然后我会在指定文件夹中的文件中进行遍历并执行以下操作：

1. 是我感兴趣的文件名格式吗？ 21-05-14.CSV 并且它不在先前解析的文件列表中。如果之前已经解析过或者不是正确的文件类型，请忽略它！
2. 解析感兴趣的文件以提取到变量中，PowerGenList all time:power 值对
3. 对于PowerGenList 中的每一行，提取timestampval 和powerval 并使用它们构造SQL 以将这些值添加到MySQL 数据库表DTP

Example PowerGenList = [[datetime.datetime(2015, 12, 29, 14, 10), '0.012'], [datetime.datetime(2015, 12, 29, 14, 10), '0.012'], [datetime.datetime(2015, 12, 29, 14, 10), '0.012'], [datetime.datetime(2015, 12, 29, 14, 10), '0.012'], [datetime.datetime(2015, 12, 29, 14, 10), '0.012'], [datetime.datetime(2015, 12, 29, 14, 10), '0.012'], [datetime.datetime(2015, 12, 29, 14, 10), '0.012'], [datetime.datetime(2015, 12, 29, 14, 10), '0.012'], [datetime.datetime(2015, 12, 29, 14, 10), '0.012'], [datetime.datetime(2015, 12, 29, 14, 10), '0.012'], [datetime.datetime(2015, 12, 29, 14, 10), '0.012'], [datetime.datetime(2015, 12, 29, 14, 10), '0.012'], [datetime.datetime(2015, 12, 29, 14, 10), '0.012'], [datetime.datetime(2015, 12, 29, 14, 10), '0.012'], [datetime.datetime(2015, 12, 29, 14, 10), '0.012'], [datetime.datetime(2015, 12, 29, 14, 10), '0.012'], [datetime.datetime(2015, 12, 29, 14, 10), '0.012'], [datetime.datetime(2015, 12, 29, 14, 10), '0.012'], [datetime.datetime(2015, 12, 29, 14, 10), '0.012'], [datetime.datetime(2015, 12, 29, 14, 10), '0.012'], [datetime.datetime(2015, 12, 29, 14, 10), '0.012']]

4. 当我在调试模式 (Python3.6) 下通过 VSCode 运行此程序时，我会打印 SQL。 INSERT INTO DTP (DT,PWR) VALUES('2015-12-29 14:10:00','0.012');，我觉得哪个不错？
5. 当我运行 cur.execute(sql) 时，我收到以下错误，并直接跳转到 except: 路径 - （我完全不知道如何分析它失败的原因）

Exception has occurred: InterfaceError
(0, '')
  File "/home/greg/currentGenerated/SBEAM/SunnyCSV2DB-v6.py", line 210, in add_days_power_to_DTP_table
    cur.execute(sql)

During handling of the above exception, another exception occurred:

  File "/home/greg/currentGenerated/SBEAM/SunnyCSV2DB-v6.py", line 215, in add_days_power_to_DTP_table
    addcon.rollback()
  File "/home/greg/currentGenerated/SBEAM/SunnyCSV2DB-v6.py", line 245, in <module>
    add_days_power_to_DTP_table(con, PowerGenList)

功能码为：

def add_days_power_to_DTP_table(addcon, PowerGenList):
    PowerGenList.__len__
    print(PowerGenList)
    table_name = 'DTP'
    timestamp_column  = 'DT'
    power_column = 'PWR'
    cur = addcon.cursor()
    try:
        for row in PowerGenList:
            timestampval    = row[0]
            powerval        = row[1]
            sql             = "INSERT INTO %s (%s,%s) VALUES('%s','%s');" % (table_name, timestamp_column, power_column, timestampval.strftime('%Y-%m-%d %H:%M:%S'), powerval)
            print('DTP SQL statement = ' + sql)
            cur.execute(sql)
            addcon.commit()
    except:
        #print 'DB append failed!'
        syslog.syslog('SunnyData DB append failed')
        addcon.rollback() 

如果相关，我的表格应该是这样的结构：

def recreateDB():  # NOT USED but useful for reference & debug - keep up to date!
  NewFiles = "CREATE TABLE IF NOT EXISTS 'SunnyData2'.'Files' ('idfiles' int(5) NOT NULL AUTO_INCREMENT, 'FN' varchar(15) NOT NULL, 'FD' DATETIME NOT NULL, PRIMARY KEY('idfiles'));"
  NewDTP = "CREATE TABLE IF NOT EXISTS 'SunnyData2'.'DTP' ('idDTP' int(5) NOT NULL AUTO_INCREMENT, 'DT' DATETIME NOT NULL, 'PWR' varchar(6) NOT NULL, PRIMARY KEY('idDTP'));"

Answer 1

我想我找到了错误。查看您最后使用分号的代码。我已将其删除。试一试。我从来没有在执行时使用分号。

sql = "INSERT INTO %s (%s,%s) VALUES('%s','%s') ; " % (table_name, timestamp_column, power_column, timestampval.strftime('%Y-%m-%d %H:%M:%S'), powerval)

另外，不要使用上述格式，而是使用下面提到的 'f' 或 'format' 方法。

sql = f"INSERT INTO {table_name} ({timestamp_column},{power_column}) VALUES('{timestampval.strftime('%Y-%m-%d %H:%M:%S')}','{powerval}')"

更具可读性。

Answer 2

适用于 Sunny Boy Solar 逆变器的完整工作 CSV 解析器

Maria DB：SunnyData2，表格： 文件：（保留以前解析过的文件的记录）

+----------+-------------+------+-----+---------+----------------+
| Field    | Type        | Null | Key | Default | Extra          |
+----------+-------------+------+-----+---------+----------------+
| FN       | varchar(15) | YES  |     | NULL    |                |
| FD       | datetime    | YES  |     | NULL    |                |
| IDXFiles | int(11)     | NO   | PRI | NULL    | auto_increment |
+----------+-------------+------+-----+---------+----------------+

DTP：（所有 CSV 文件汇总的非零功率读数记录）

+--------+------------+------+-----+---------+----------------+
| Field  | Type       | Null | Key | Default | Extra          |
+--------+------------+------+-----+---------+----------------+
| idxDTP | int(11)    | NO   | PRI | NULL    | auto_increment |
| DT     | datetime   | YES  |     | NULL    |                |
| PWR    | varchar(6) | YES  |     | NULL    |                |
+--------+------------+------+-----+---------+----------------+

Python prog 将 SunnyBoy 逆变器 .CSV 文件的文件夹解析为 SunnyData2 DB

#!/usr/bin/python3
# -*- coding: utf-8 -*-

"""
Python >=  version 3 which is meant to work with the pymysql driver

The purpose of this script is to look at each CSV file in a known directory and
parse the time and solar generated power and record that to a MYSQL database

It records the date from line 2, field 10 from the content and creates timestamps, by concatenating the date with the time, 
which is recorded every ten minutes (during the 24 hour day), along with the average power for each slice.

Only values over zero are recorded, as even dark days produce some power and
this keeps the data volume down to the interesting, daytime period.
"""

from os import listdir
from datetime import datetime
from decimal import *
getcontext().prec = 3

import pymysql
import shutil
import syslog
import os, sys

def connect_to_DB():
    """
    Connect to the SunnyData2 DB
    """
    #Database connection parameters
    h  = ""*********""    # host
    u  = ""*********""    # username
    p  = ""*********""    # password
    d  = ""*********""    # database
    #Pt = 3306            # port

    con = pymysql.connect(host  = h, user  = u, password  = p, database  = d)
    print('Connected to ' + str(d))
    return con

def read_status_table():
    """
        Return list of all parsed files, in alphanumeric filename order from DB table of files
    """
    #Query parameters
    table_name  = 'Files'
    filename    = 'FN'
    filedate    = "FD"
    con=connect_to_DB()
    sql1 = "SELECT %s AS fileName, date(%s) AS fileDate FROM %s ORDER BY date(%s) ASC;" % (filename, filedate, table_name, filedate )
    with con:
        cur = con.cursor()
        cur.execute(sql1)
        rows = cur.fetchall()
    return rows

def is_dated_csv(filename):
    """
    Return True if filename matches format YY-MM-DD.csv, otherwise False.
    """
    date_format = '%y-%m-%d.csv'

    try:
        date = datetime.strptime(filename, date_format)
        syslog.syslog('SunnyData file ' + filename + ' is a correctly formatted CSV file')
        return True
    except ValueError: # filename did not match pattern
        syslog.syslog('SunnyData file ' + filename + ' is NOT a correctly formatted CSV file')
        pass #'return' terminates a function
    return False

def parse_for_date(filename):
  """
  Read file for the date - from line 2 field 10
  """
  currentFile = open(filename,'r')
  l1 = currentFile.readline() #ignore first line read
  date_line = currentFile.readline() #read second line
  dateLineArray = date_line.split("|") # use | delimiter to make an array
  day_in_question = dateLineArray[-1] #save the last element ,which is the date
  currentFile.close()
  theDate = normalise_date_to_UTF(day_in_question)
  return theDate

def normalise_date_to_UTF(day_in_question):
  """
  Rather wierdly, some days use YYYY.MM.DD format & others use DD/MM/YYYY
  This function normalises either to UTC with a blank time (midnight)
  """
  if '.' in day_in_question: #it's YYYY.MM.DD
    dateArray = day_in_question.split(".")
    dt = (dateArray[0] + dateArray[1] + dateArray[2].rstrip() + '000000')
  elif '/' in day_in_question: #it's DD/MM/YYYY
    dateArray = day_in_question.split("/")
    dt = (dateArray[2].rstrip() + dateArray[1] + dateArray[0] + '000000')
  theDate = datetime.strptime(dt,'%Y%m%d%H%M%S')
  return theDate #A datetime object

def parse_power_values(path, filename, theDate):
  # timePower = [(),()] #a list that will be assigned two items; the timestamp and the power for that timeslot
  dayPower = [] # A list that will have all the timePowers for the day appended
  currentFile = open(path + filename,'r')
  for i, line in enumerate(currentFile):
    if i <= 7:
      doingSomething = True
      #print 'header' + str(i) + '/ ' + line.rstrip()
    elif ((i > 7) and (i <= 151)):
      lineParts = line.split(';')
      theTime = lineParts[0].split(':')
      theHour = theTime[0]
      theMin = theTime[1]
      timestamp = theDate.replace(hour=int(theHour),minute=int(theMin))
      power = lineParts[1].rstrip()
      if power == '-.---':
        power = 0.000
      if (float(power) > 0):
        dayPower.append([timestamp,power])
    elif i > 151:
     return dayPower
     #break
  currentFile.close()

def add_file_to_Files_table(filenameval, filedateval): # Add the new filename to the DB Files table, as it's now ABOUT TO BE parsed (maybe do this after a successful parse, instead)
    """
        Append the named parsed files to the DB table of files
    """
    #Query parameters
    table_name  = 'Files'
    filename    = 'FN'
    filedate    = 'FD'
    Y           = str(filedateval.year)
    m           = str(filedateval.month)
    d           = str(filedateval.day)
    datestring  = Y+"-"+m+"-"+d
    #D           = datetime.strptime(datestring, "%Y%m%d")
    D           = filedateval.date()
    #filedateval2 = datetime.strptime(str(filedateval), "%Y%m%d")
    sql2 = "INSERT INTO %s (%s, %s) VALUES ('%s', '%s');" % (table_name,filename,filedate,filenameval,filedateval)
    #sql2 = "INSERT INTO %s (%s, %s) VALUES ('%s', '%s');" % (table_name,filename,filedate,filenameval,datestring)
    print('SQL Files statement = ' + sql2)
    logcon = connect_to_DB()
    with logcon:
        logcur = logcon.cursor()
        logcur.execute(sql2) 
        logcon.commit()

def add_days_power_to_DTP_table(PowerGenList):
    PowerGenList.__len__
    table_name = 'DTP'
    timestamp_column  = 'DT'
    power_column = 'PWR'
    addcon = connect_to_DB()
    with addcon:
        addcur = addcon.cursor()
        try:
            for row in PowerGenList:
                timestampval    = row[0]
                powerval        = row[1]
                sql             = "INSERT INTO %s (%s,%s) VALUES('%s','%s')" % (table_name, timestamp_column, power_column, timestampval.strftime('%Y-%m-%d %H:%M:%S'), powerval)
                print('SQL Date-Power statement = ' + sql)
                addcur.execute(sql)
            addcon.commit()
        except:
            #print 'DB append failed!'
            syslog.syslog('SunnyData DB append failed')
            addcon.rollback()
    

#Query parameters - put into a function for query to find average and total power for a given period (based on attribute values)
# StartYear   = "2019"
# EndYear     = "2019"
# StartMonth  = "1"
# EndMonth    = "2"
# StartDay    = "1"
# EndDay      = "31"

# Main start of program  v--v--v--v--v--v--v--v--v--v--v--v--v--v
path = '/SunnyData/SBEAM/'
syslog.syslog('parsing SunnyData CSVs started ------->>>>>>>>>>>>>>>>------')
parsedFiles = read_status_table()
for filename in listdir(path):
    if is_dated_csv(filename): # IS a dated CSV
        if any(filename in s for s in parsedFiles): # but HAS been parsed before
            print("file previously parsed: " + filename) #output current file to StdOut
            pass
        else: # Has NOT been parsed before
            print(filename + ' is a previously unparsed dated CSV') #to StdOut
            theDate = parse_for_date(path + filename) #from the file, extracts the date (line 2 field 10) and returns it normalised to UTF
            print('the date is ' + str(theDate))            
            #try creating powergenlist as a list of all the timestamps and power pairs that need to be added
            PowerGenList = parse_power_values(path, filename, theDate) # Reads in power & time, creates a timestamp and appends value into DTP in DB (maybe a clash between timestamp & datetime)            
            add_file_to_Files_table(filename, theDate) # Add the new filename to the DB Files table, as it's now ABOUT TO BE parsed (maybe do this after a successful parse, instead)
            add_days_power_to_DTP_table(PowerGenList)

            #con.close #Disconnect from the database having parsed each file
    else:
        print(filename + " is not a CSV") #output current file to StdOut
#         pass
#^--^--^--^--^--^--^--^--^--^--^--^--^--^--^--^--^--^--^--^--^--^