长期数据采集

Question

我正在尝试运行用于长期实验室数据采集（约 1 个月）的 Python 脚本。下面的脚本应该从 SQL 数据库中获取数据并从 DAQ 设备中获取数据并将其保存到文本文件中。然后它会实时绘制 1 小时的数据。由于我遇到的内存错误，实时绘图已被注释掉，但这不是我关心的问题。

运行代码 3-4 天会给我以下错误：

Trackback (most recent call last):
 file *** line 105 in <module>
    deltay=float(float(tupline[3]/10-float(tupline2[3])/10)
TypeError: 'NoneType' object has to attribute '__getitem__'

有什么想法可以让这个脚本运行更长的时间吗？

import ctypes
from UniversalLibrary import *
import time
import datetime
from numpy import *
from scipy import *
import MySQLdb as mdb
import matplotlib
matplotlib.use('Agg')
import matplotlib.pyplot as plt
import random



boardNum=0
portNum=10
inputVoltage = 3



BoardNum = 0
Gain = BIP5VOLTS
Chan = 0



spTimes =14*24*60*60     ##here you set the time
timeDuration=spTimes
padsToVisualize =[0,1,2,3,4]  ##pads that will be measured and visualized

#plt.ion()    ##this command is necassary in order to have new points on the plot without erasing the old ones

curTime=time.clock()
movingWinDur=60    ##seconds
print "curtime=",curTime

str_one_line=[]
straverage=[]
while (curTime < timeDuration):
    del str_one_line[:]
    padCounter=0;

    d=datetime.datetime.now()
    str_one_line=[(d.strftime("%Y-%m-%d %H:%M:%S")),',']   ##formatting the date -- this format is chosen based on the requirements for connecting to SQL database

    while (padCounter <len(padsToVisualize)):

        #fig1=plt.figure(padsToVisualize[padCounter])    ## making figures for plotting different pads


        print "pad No.: ", str(padsToVisualize[padCounter])
        #l=cbAOut(BoardNum,Chan,Gain,3019)   ## sets the voltage to 3.0 volts (the number 3019 is based on 12-bit resolution --> DAQ resolution)
        #n=cbDConfigPort(BoardNum,portNum,1)   ##the variable n serves aas a command line where the command is sent to the DAQ
        #b=cbDOut(BoardNum,portNum,(63-padsToVisualize[padCounter])) #####based on Prof. Mayer's solution, the line sends the pad number that we are interested in to the DAQ to get the signals 
        curTime=time.clock()
        d=datetime.datetime.now()
        #########################################################################

        ########## this part will use the datetime to fetch the temperature from the SQL data base
        ########## since the time stamps may not exactly match, the code uses a method of interpolation
        freq = 5
        time.sleep(1.8*freq) ## 1.8 is an arbitrary number ; to make sure there is data on sql database
        searchfor=d   ## the following lines find the temperature at time=searchfor
        print searchfor
        minussearchfor = searchfor-datetime.timedelta(0,freq/1.2)
        STRminussearchfor = minussearchfor.strftime("%Y-%m-%d %H:%M:%S")
        print "STRminussearchfor=", STRminussearchfor
        print "minussearchfor= ", minussearchfor
        plussearchfor =searchfor+datetime.timedelta(0,freq/1.2)
        print "plussearchfor= ", plussearchfor
        STRplussearchfor = plussearchfor.strftime("%Y-%m-%d %H:%M:%S")
        print "STRplussearchfor=", STRplussearchfor
        ##Database connection
        db = mdb.connect("localhost", "root", "qwerty@12345", "bondtest")
        cur = db.cursor()
        cur.execute("SELECT * FROM bondtest_data WHERE DateTime BETWEEN %(date1)s AND %(date2)s ORDER BY DateTime",{'date1':STRminussearchfor,'date2':STRplussearchfor})
##        con=pymssql.connect(host='LAB10-PC\SQLEXPRESS2008R2',user='sa',password='qwerty@12345')
##        cur = con.cursor()
##        cur.execute('SELECT * FROM OVEN11SQL.dbo.OvenLog1 WHERE DateTime BETWEEN %(date1)s AND %(date2)s ORDER BY DateTime',{'date1':STRminussearchfor,'date2':STRplussearchfor})
        tupline1 = cur.fetchone()
##        print 'between1= ',tupline1[1]
        delta = tupline1[1]-searchfor
##        print "delta = " ,delta
        if (int(searchfor.strftime("%S"))-int(tupline1[1].strftime("%S"))>0):
            delta = int(searchfor.strftime("%S"))-int(tupline1[1].strftime("%S"))
        else:
            delta = int(tupline1[1].strftime("%S"))-int(searchfor.strftime("%S"))
##        print 'delta= ',delta
        time1=tupline1[1]-datetime.timedelta(0,freq/1.2)
        STRtime1=time1.strftime("%Y-%m-%d %H:%M:%S")
        time2=tupline1[1]-datetime.timedelta(0,3*freq/1.2)
        STRtime2=time2.strftime("%Y-%m-%d %H:%M:%S")
##        time.sleep(2*freq) ##the program needs to wait for 3*frequency/2 so that the next point is available in SQL data base for interpolation
        cur.execute('SELECT * FROM bondtest_data WHERE DateTime BETWEEN %(date1)s AND %(date2)s ORDER BY DateTime',{'date1':STRtime2,'date2':STRtime1})
        tupline2 = cur.fetchone()   ##next point is fetched in order to find the slope of the line for temperature change
##        print 'between2= ', tupline2[1]
        deltay=float(float(tupline1[3])/10-float(tupline2[3])/10)
        deltax = int(tupline1[1].strftime("%S"))-int(tupline2[1].strftime("%S"))
        deltax = freq
##        print "deltay= ", deltay
##        print "deltax= ", deltax
        slope = deltay/deltax
##        print 'slope= ', slope

        ##in the following if statements depending on whether the temperature is increasing or decreasing the temperature for the desired point will be calculated
        if (tupline2[3]<tupline1[3]):
            tempsearchfor = float(tupline1[3])/10+delta*slope
##            print '+delta*slope= ',delta*slope
        elif (tupline2[3]>tupline1[3]):
            tempsearchfor = float(tupline1[3])/10-delta*slope
##            print '-delta*slope= ',delta*slope
        else:
            tempsearchfor = float(tupline1[3])/10
##        print 'tempserachfor= ',tempsearchfor

        #########################################################################

        strng = [str(int(padsToVisualize[padCounter])),',',(d.strftime("%Y-%m-%d %H:%M:%S")),',',str(round(curTime,4)),',',str(inputVoltage),',']
        str_one_line.extend(strng)         ##str_one_line is the line that contains the values that will be written to the text file , the order is specified in the variables lists excel file
        xyzCounter=Chan
        EngUnits= array(range(50*1),dtype=float).reshape(50,1)    ## constructing the array that will hold the 50 values for each set of signals and is later used for finding the average
        average = array(range(3*1),dtype=float).reshape(3,1)      ## holds the average of the t50 points for x,y,z
##        straverage=array(range(3*1),dtype=str).reshape(3,1)   ##this array will hold the strings for x,y,z signal to be written into txtfile
        del straverage[:]
        while(xyzCounter<3):   ##the way the oven is set up it has x,y,z outputs from channel 0,1,2 accordingly
            #n=cbDConfigPort(BoardNum,portNum,1)
            #b=cbDOut(BoardNum,portNum,(63-padsToVisualize[padCounter]))     #####based on Prof. Mayer's solution, the line sends the pad number that we are interested in to the DAQ to get the signals
            a=0
            time1=time.clock()
            while (a<50):    ## this while loop takes 50 measurements and writes the average of those points in to average array

                #DataValue=cbAIn(BoardNum, xyzCounter, Gain)
                #EngUnits[a,0]=float((cbToEngUnits(BoardNum,Gain,DataValue))/3/100*1000)
                EngUnits[a,0] = random.uniform(0,0.5)
                average[xyzCounter,0]=float(sum(EngUnits))/len(EngUnits)
                a+=1
            time2=time.clock()
            timePerSample=(time2-time1)/50
            print "time per sample= ",timePerSample
            print "samples per second =", (1/timePerSample)   ##measuring the frequency of the emasurements
            tempstr=str(round((average[xyzCounter,0]),3))    ##in order to remove the two brackets temp refers to temporary
            #tempstr=tempstr[1:-1]
            straverage.append(tempstr)
            xyzCounter+=1


        #print average
        temperaturearray=array(range(1*1),dtype=str).reshape(1,1)
        temperaturearray=[str(tempsearchfor)]
        three_sens_signals=array(range(1*5),dtype=str).reshape(1,5)
        three_sens_signals=[str((straverage[0])),',',str((straverage[1])),',',str((straverage[2])),',']
        str_one_line.extend(three_sens_signals)
        str_one_line.extend(temperaturearray)
        str_one_line.extend(',')

        padCounter+=1
        filename='log_simulation.txt'
        f = open(filename,"a")
      ## writing to file section
    print "padcounter=",padCounter,"   str_one_line=", str_one_line
    for item in str_one_line:  
        f.write("%s" % item )
    f.write("\n")
    f.close()

    curTime=time.clock()

Answer 1

简而言之，这段代码有很多地方需要改进/修改（首先编辑所有 import * 语句并像你应该使用的那样使用命名空间）。话虽如此（代码显然大部分都按原样工作），但由于任何原因，存储在 tupline 或 tupline2 中的值都是 None ，因此出现了问题。在您的代码中，这些变量是使用 sql 语句分配的

tupline = cur.fetchone()
tupline2 = cur.fetchone() 

其中一个调用显然运行不顺畅（无论出于何种原因，数据或表已损坏），导致它返回“无”。也许有人正在删除或更换桌子？您可以添加一些语义来检查这种情况，报告并尝试继续。类似于

if tupline == None or tupline2 == None:
    print "Why does tupline = None?"
    continue

您应该弄清楚 sql .fetchone() 方法何时返回 none。确实，您应该做一些更好的日志记录，但这可能会让您进入下一步的调试。

Answer 2

正如 Paul Seeb 所指出的，出现错误消息是因为当代码到达 deltay=... 行时，tupline 或 tupline2 是 None。您的代码（成功）在分配后访问 tupline 的元素，然后到达 deltay 行。所以问题一定出在tupline2。

我的猜测是数据库不包含tupline2 查询的匹配记录；那肯定是要检查的东西。我还注意到您在 tupline2 查询之前注释掉了延迟，这暗示该查询不能可靠地返回记录。我不知道你的程序逻辑，但你也许可以在 tupline2 查询之后用这样的行来解决问题：

if tupline2 is None:
    continue`

或者您可以重试tupline2 查询直到它成功

while True:
    cur.execute(
        """SELECT * FROM bondtest_data 
            WHERE DateTime BETWEEN %(date1)s AND %(date2)s 
            ORDER BY DateTime""",
        {'date1':STRtime2,'date2':STRtime1}
    )
    tupline2 = cur.fetchone()
    if tupline2 is None:
        print "Failed to obtain a record for tupline2; trying again..."
        time.sleep(<some duration>)
    else:
        break

我还注意到，您的代码每次通过循环时都会创建一个新的数据库连接 (db)，并且从未显式关闭它。我建议将db = mdb.connect(...) 命令移到主循环开始的上方，或者在最后一个fetchone() 命令之后的某处添加db.close()。您的代码很可能达到某种连接限制，但这可能会引发不同的异常。当db 被重新分配并且旧连接超出范围时，您更有可能从某种自动关闭连接中受益。