如何让 python Parallel 与本地 mysql 服务器一起工作？答案

【问题标题】：How do I get python Parallel to work with local mysql server?如何让 python Parallel 与本地 mysql 服务器一起工作？
【发布时间】：2015-12-08 16:18:10
【问题描述】：

我正在尝试将一堆插入并行化到 mysql。所有 db 服务器设置看起来都已为大量连接和/或线程做好了准备。下面添加了完整代码和完整错误消息。谷歌搜索最终的错误消息没有返回任何相关信息：

OperationalError: (2006, 'MySQL server has gone away')

Code:
> #!/usr/bin/python
# -*- coding: utf-8 -*-

# price_retrieval.py

from __future__ import print_function

import datetime
import warnings

import MySQLdb as mdb
import requests
from joblib import Parallel, delayed
import multiprocessing

# Obtain a database connection to the MySQL instance
db_host = 'localhost'
db_user = 'sec_user'
db_pass = 'password'
db_name = 'securities_master'
con = mdb.connect(db_host, db_user, db_pass, db_name)


def obtain_list_of_db_tickers():
   """
   Obtains a list of the ticker symbols in the database.
   """
   with con:
       cur = con.cursor()
       cur.execute("SELECT id, ticker FROM symbol")
       data = cur.fetchall()
       return [(d[0], d[1]) for d in data]


def is_db_current():
   # check that database is loaded up to yesterday's details or the last trading day.
   # ??? is there an app for last trading day. # holidays could trip up schedule.
   # nothing is currently designed for same day data.
   sql = "select max(price_date), symbol_id, ticker \
       from securities_master.daily_price left join securities_master.symbol \
       on  securities_master.daily_price.symbol_id =  securities_master.symbol.id \
       group by symbol_id limit 5"
#
   today = datetime.date.today().weekday()
   data = None
   with con:
       cur = con.cursor()
       cur.execute(sql)
       data = cur.fetchall()
   tuples = [(d[0], d[1], d[2]) for d in data]

   # if data is upto yesterday return true
   # elsif data is current up to friday return true
   # so if friday is filled and it's sat or sun or mon return true.
   # Monday = 0 Sunday = 6
   for t in tuples:
       if abs(today - t[0].weekday()) <= 1:
           #good but keep looping.
           continue
       elif today in [5,6,0]  and t[0].weekday() == 4:
           #good but keey looping.
           continue
       else:
           return False
       # made it through checks so
   return True
#        datetime.utcnow().dayofweek

def get_daily_historic_data_yahoo(
       ticker, start_date=(2000,1,1),
       end_date=datetime.date.today().timetuple()[0:3]
   ):
   """
   Obtains data from Yahoo Finance returns and a list of tuples.

   ticker: Yahoo Finance ticker symbol, e.g. "GOOG" for Google, Inc.
   start_date: Start date in (YYYY, M, D) format
   end_date: End date in (YYYY, M, D) format
   """
   # Construct the Yahoo URL with the correct integer query parameters
   # for start and end dates. Note that some parameters are zero-based!
   ticker_tup = (
       ticker, start_date[1]-1, start_date[2],
       start_date[0], end_date[1]-1, end_date[2],
       end_date[0]
   )
   yahoo_url = "http://ichart.finance.yahoo.com/table.csv"
   yahoo_url += "?s=%s&a=%s&b=%s&c=%s&d=%s&e=%s&f=%s"
   yahoo_url = yahoo_url % ticker_tup

   # Try connecting to Yahoo Finance and obtaining the data
   # On failure, print an error message.
   try:
       yf_data = requests.get(yahoo_url).text.split("\n")[1:-1]
       prices = []
       for y in yf_data:
           p = y.strip().split(',')
           prices.append(
               (datetime.datetime.strptime(p[0], '%Y-%m-%d'),
               p[1], p[2], p[3], p[4], p[5], p[6])
           )
   except Exception as e:
       print("Could not download Yahoo data: %s" % e)
   return prices

def get_daily_historid_data_google():
   prices = []
   return prices

def insert_daily_data_into_db(
       data_vendor_id, symbol_id, daily_data
   ):
   """
   Takes a list of tuples of daily data and adds it to the
   MySQL database. Appends the vendor ID and symbol ID to the data.

   daily_data: List of tuples of the OHLC data (with
   adj_close and volume)
   """
   # Create the time now
   now = datetime.datetime.utcnow()

   # Amend the data to include the vendor ID and symbol ID
   daily_data = [
       (data_vendor_id, symbol_id, d[0], now, now,
       d[1], d[2], d[3], d[4], d[5], d[6])
       for d in daily_data
   ]

   # Create the insert strings
   column_str = """data_vendor_id, symbol_id, price_date, created_date,
                last_updated_date, open_price, high_price, low_price,
                close_price, volume, adj_close_price"""
   insert_str = ("%s, " * 11)[:-2]
   final_str = "INSERT IGNORE INTO daily_price (%s) VALUES (%s)" % \
       (column_str, insert_str)

   # Using the MySQL connection, carry out an INSERT INTO for every symbol
   with con:
       cur = con.cursor()
       cur.executemany(final_str, daily_data)

def load_ticker_history(t):
   print( "Adding data for %s" % (t[1]) )
   yf_data = get_daily_historic_data_yahoo(t[1], (2015,01,01))
   insert_daily_data_into_db('1', t[0], yf_data)


if __name__ == "__main__":
   # This ignores the warnings regarding Data Truncation
   # from the Yahoo precision to Decimal(19,4) datatypes
   warnings.filterwarnings('ignore')

   # Loop over the tickers and insert the daily historical
   # data into the database
   tickers = obtain_list_of_db_tickers()
   lentickers = len(tickers)
   num_cores = multiprocessing.cpu_count()
   print("numCores = " + str(num_cores))

   if is_db_current():
       print("db current: True")
   else:
       # not sure why
       Parallel(n_jobs=1)(delayed(load_ticker_history)(t) for t in tickers)
       print("Successfully added Yahoo Finance pricing data to DB.")

错误

输出：

/usr/bin/python /Users/darianhickman/Documents/Quantstart/src/dev/price_retrieval.py numCores = 2 为 A 添加数据为 AA 添加数据为 AAL 添加数据为 AAP 添加数据为 AAPL 添加数据为 ABBV 添加数据 ABC 的数据添加 ABT 的数据添加 ACE 的数据添加数据 ACN 为 ADBE 添加数据为 ADI 添加数据为 ADM 添加数据为 ADP 添加数据为 ADS 添加数据为 ADSK 添加数据用于 ADT 的数据添加用于 AEE 的数据添加用于 AEP 的数据添加用于 AES 为 AET 添加数据为 AFL 添加数据为 AGN 添加数据为 AIG 添加数据为 AIV 添加数据为 AIZ 添加数据为 AKAM 为 ALL 添加数据为 ALLE 添加数据为 ALTR 添加数据 Traceback（最近一次通话最后一次）：文件 "/Users/darianhickman/Documents/Quantstart/src/dev/price_retrieval.py", 第 164 行，在并行（n_jobs = 2）（延迟（load_ticker_history）（t），用于股票代码中的t）文件 “/Library/Python/2.7/site-packages/joblib/parallel.py”，第 660 行，在致电 self.retrieve() 文件“/Library/Python/2.7/site-packages/joblib/parallel.py”，第 543 行，在取回引发异常类型（报告）joblib.my_exceptions.JoblibOperationalError：JoblibOperationalError ___________________________________________________________________________ 多处理异常： ..................................................... ………………………………………………………………………………………………………… /Users/darianhickman/Documents/Quantstart/src/dev/price_retrieval.py 在（） 159 打印（“numCores =” + str（num_cores）） 160 161 如果不是 is_db_current(): 162打印（“分贝电流：真”） 163 其他： --> 164 Parallel(n_jobs=2)(延迟(load_ticker_history)(t) for t in tickers) 165 print("已成功将 Yahoo Finance 定价数据添加到 DB。") 166 167 168

.................................................. ...................... /Library/Python/2.7/site-packages/joblib/parallel.py 在调用(self=Parallel(n_jobs=2), iterable=>) 655 如果 pre_dispatch == "all" 或 n_jobs == 1： 656 # iterable 被上面的 for 循环一次性消耗掉了。 657 # 无需等待异步回调触发 658#消费。第659章 --> 660 self.retrieve() 自我检索 = 661 # 确保我们收到最后一条消息，告诉我们完成了 662 elapsed_time = time.time() - self._start_time 663 self._print('已完成 %3i，共 %3i | 已用：%s 已完成', 664（len（self._output），
---------------------------------------------------------------------------
Sub-process traceback:
---------------------------------------------------------------------------
OperationalError                                   Fri Sep 11 22:42:58 2015 PID: 39551                                   Python
2.7.10：/usr/bin/python ..................... ................................................... /Users/darianhickman/Documents/Quantstart/src/dev/price_retrieval.py 在 load_ticker_history(t=(17L, 'AKAM')) 第141章 142 第143章 144 print("为 %s 添加数据" % (t[1]) ) 145 yf_data = get_daily_historic_data_yahoo(t[1], (2015,01,01)) --> 146 insert_daily_data_into_db('1', t[0], yf_data) t = (17L, 'AKAM') yf_data = [(datetime.datetime(2015, 9, 11, 0, 0), u'74.32', u'75.32', u'73.940002', u'74.669998', u'1368400', u'74.669998'), (datetime.datetime(2015, 9, 10, 0, 0), u'73.279999', u'75.029999', u'73.190002', u'74.550003', u'1408900', u'74.550003'), (datetime.datetime(2015, 9, 9, 0, 0), u'74.879997', u'75.25', u'73.129997', u'73.349998', u'1273300', u'73.349998'), (datetime.datetime(2015, 9, 8, 0, 0), u'73.75', u'74.349998', u'73.059998', u'74.279999', u'1413600', u'74.279999'), (datetime.datetime(2015, 9, 4, 0, 0), u'71.709999', u'73.169998', u'71.709999', u'72.580002', u'1607100', u'72.580002'), (datetime.datetime(2015, 9, 3, 0, 0), u'71.860001', u'74.709999', u'71.540001', u'73.260002', u'2378600', u'73.260002'), (datetime.datetime(2015, 9, 2, 0, 0), u'71.349998', u'71.669998', u'69.669998', u'71.269997', u'1671900', u'71.269997'), (datetime.datetime(2015, 9, 1, 0, 0), u'69.75', u'71.43', u'69.669998', u'70.43', u'2401800', u'70.43'), (datetime.datetime(2015, 8, 31, 0, 0), u'71.660004', u'71.809998', u'70.870003', u'71.309998', u'1764400', u'71.309998'), (datetime.datetime(2015, 8, 28, 0, 0), u'70.589996', u'71.970001', u'70.199997', u'71.900002', u'2371800', u'71.900002'), (datetime.datetime(2015, 8, 27, 0, 0), u'69.089996', u'70.889999', u'68.529999', u'70.50', u'2081800', u'70.50'), (datetime.datetime(2015, 8, 26, 0, 0), u'67.07', u'68.300003', u'65.330002', u'68.169998', u'2238400', u'68.169998'), (datetime.datetime(2015, 8, 25, 0, 0), u'68.239998', u'68.989998', u'65.589996', u'65.629997', u'3112100', u'65.629997'), (datetime.datetime(2015, 8, 24, 0, 0), u'65.010002', u'68.220001', u'63.139999', u'66.519997', u'3779300', u'66.519997'), (datetime.datetime(2015, 8, 21, 0, 0), u'69.190002', u'70.00', u'67.910004', u'68.040001', u'2298300', u'68.040001'), (datetime.datetime(2015, 8, 20, 0, 0), u'72.300003', u'72.57', u'69.769997', u'69.779999', u'2096000', u'69.779999'), (datetime.datetime(2015, 8, 19, 0, 0), u'73.779999', u'73.900002', u'72.290001', u'73.059998', u'881700', u'73.059998'), (datetime.datetime(2015, 8, 18, 0, 0), u'73.959999', u'74.190002', u'73.230003', u'73.849998', u'1220500', u'73.849998'), (datetime.datetime(2015, 8, 17, 0, 0), u'73.010002', u'73.599998', u'72.32', u'73.57', u'1025600', u'73.57'), (datetime.datetime(2015, 8, 14, 0, 0), u'73.379997', u'73.839996', u'72.730003', u'73.370003', u'1328400', u'73.370003'), ...] 147 148 149 如果 name == "main": 150 # 这忽略了关于数据截断的警告

.................................................. ...................... /Users/darianhickman/Documents/Quantstart/src/dev/price_retrieval.py 在 insert_daily_data_into_db(data_vendor_id='1', symbol_id=17L, daily_data=[('1', 17L, datetime.datetime(2015, 9, 11, 0, 0), datetime.datetime(2015, 9, 12, 2, 42, 58, 361925), datetime.datetime(2015, 9, 12, 2, 42, 58, 361925), u'74.32', u'75.32', u'73.940002', u'74.669998', u'1368400', u'74.669998'), ('1', 17L, datetime.datetime(2015, 9, 10, 0, 0), datetime.datetime(2015, 9, 12, 2, 42, 58, 361925), datetime.datetime(2015, 9, 12, 2, 42, 58, 361925), u'73.279999', u'75.029999', u'73.190002', u'74.550003', u'1408900', u'74.550003'), ('1', 17L, datetime.datetime(2015, 9, 9, 0, 0), datetime.datetime(2015, 9, 12, 2, 42, 58, 361925), datetime.datetime(2015, 9, 12, 2, 42, 58, 361925), u'74.879997', u'75.25', u'73.129997', u'73.349998', u'1273300', u'73.349998'), ('1', 17L, datetime.datetime(2015, 9, 8, 0, 0), datetime.datetime(2015, 9, 12, 2, 42, 58, 361925), datetime.datetime(2015, 9, 12, 2, 42, 58, 361925), u'73.75', u'74.349998', u'73.059998', u'74.279999', u'1413600', u'74.279999'), ('1', 17L, datetime.datetime(2015, 9, 4, 0, 0), datetime.datetime(2015, 9, 12, 2, 42, 58, 361925), datetime.datetime(2015, 9, 12, 2, 42, 58, 361925), u'71.709999', u'73.169998', u'71.709999', u'72.580002', u'1607100', u'72.580002'), ('1', 17L, datetime.datetime(2015, 9, 3, 0, 0), datetime.datetime(2015, 9, 12, 2, 42, 58, 361925), datetime.datetime(2015, 9, 12, 2, 42, 58, 361925), u'71.860001', u'74.709999', u'71.540001', u'73.260002', u'2378600', u'73.260002'), ('1', 17L, datetime.datetime(2015, 9, 2, 0, 0), datetime.datetime(2015, 9, 12, 2, 42, 58, 361925), datetime.datetime(2015, 9, 12, 2, 42, 58, 361925), u'71.349998', u'71.669998', u'69.669998', u'71.269997', u'1671900', u'71.269997'), ('1', 17L, datetime.datetime(2015, 9, 1, 0, 0), datetime.datetime(2015, 9, 12, 2, 42, 58, 361925), datetime.datetime(2015, 9, 12, 2, 42, 58, 361925), u'69.75', u'71.43', u'69.669998', u'70.43', u'2401800', u'70.43'), ('1', 17L, datetime.datetime(2015, 8, 31, 0, 0), datetime.datetime(2015, 9, 12, 2, 42, 58, 361925), datetime.datetime(2015, 9, 12, 2, 42, 58, 361925), u'71.660004', u'71.809998', u'70.870003', u'71.309998', u'1764400', u'71.309998'), ('1', 17L, datetime.datetime(2015, 8, 28, 0, 0), datetime.datetime(2015, 9, 12, 2, 42, 58, 361925), datetime.datetime(2015, 9, 12, 2, 42, 58, 361925), u'70.589996', u'71.970001', u'70.199997', u'71.900002', u'2371800', u'71.900002'), ('1', 17L, datetime.datetime(2015, 8, 27, 0, 0), datetime.datetime(2015, 9, 12, 2, 42, 58, 361925), datetime.datetime(2015, 9, 12, 2, 42, 58, 361925), u'69.089996', u'70.889999', u'68.529999', u'70.50', u'2081800', u'70.50'), ('1', 17L, datetime.datetime(2015, 8, 26, 0, 0), datetime.datetime(2015, 9, 12, 2, 42, 58, 361925), datetime.datetime(2015, 9, 12, 2, 42, 58, 361925), u'67.07', u'68.300003', u'65.330002', u'68.169998', u'2238400', u'68.169998'), ('1', 17L, datetime.datetime(2015, 8, 25, 0, 0), datetime.datetime(2015, 9, 12, 2, 42, 58, 361925), datetime.datetime(2015, 9, 12, 2, 42, 58, 361925), u'68.239998', u'68.989998', u'65.589996', u'65.629997', u'3112100', u'65.629997'), ('1', 17L, datetime.datetime(2015, 8, 24, 0, 0), datetime.datetime(2015, 9, 12, 2, 42, 58, 361925), datetime.datetime(2015, 9, 12, 2, 42, 58, 361925), u'65.010002', u'68.220001', u'63.139999', u'66.519997', u'3779300', u'66.519997'), ('1', 17L, datetime.datetime(2015, 8, 21, 0, 0), datetime.datetime(2015, 9, 12, 2, 42, 58, 361925), datetime.datetime(2015, 9, 12, 2, 42, 58, 361925), u'69.190002', u'70.00', u'67.910004', u'68.040001', u'2298300', u'68.040001'), ('1', 17L, datetime.datetime(2015, 8, 20, 0, 0), datetime.datetime(2015, 9, 12, 2, 42, 58, 361925), datetime.datetime(2015, 9, 12, 2, 42, 58, 361925), u'72.300003', u'72.57', u'69.769997', u'69.779999', u'2096000', u'69.779999'), ('1', 17L, datetime.datetime(2015, 8, 19, 0, 0), datetime.datetime(2015, 9, 12, 2, 42, 58, 361925), datetime.datetime(2015, 9, 12, 2, 42, 58, 361925), u'73.779999', u'73.900002', u'72.290001', u'73.059998', u'881700', u'73.059998'), ('1', 17L, datetime.datetime(2015, 8, 18, 0, 0), datetime.datetime(2015, 9, 12, 2, 42, 58, 361925), datetime.datetime(2015, 9, 12, 2, 42, 58, 361925), u'73.959999', u'74.190002', u'73.230003', u'73.849998', u'1220500', u'73.849998'), ('1', 17L, datetime.datetime(2015, 8, 17, 0, 0), datetime.datetime(2015, 9, 12, 2, 42, 58, 361925), datetime.datetime(2015, 9, 12, 2, 42, 58, 361925), u'73.010002', u'73.599998', u'72.32', u'73.57', u'1025600', u'73.57'), ('1', 17L, datetime.datetime(2015, 8, 14, 0, 0), datetime.datetime(2015, 9, 12, 2, 42, 58, 361925), datetime.datetime(2015, 9, 12, 2, 42, 58, 361925), u'73.379997', u'73.839996', u'72.730003', u'73.370003', u'1328400', u'73.370003'), ...]) 136 (column_str, insert_str) 137 138 # 使用 MySQL 连接，对每个符号执行一次 INSERT INTO 139 与缺点： 140 cur = con.cursor() --> 141 cur.executemany(final_str, daily_data) cur.executemany = > final_str = 'INSERT IGNORE INTO daily_price (data_vendor_id, ...LUES (%s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s)' daily_data = [('1', 17L, datetime.datetime(2015, 9, 11, 0, 0), datetime.datetime(2015, 9, 12, 2, 42, 58, 361925), datetime.datetime(2015, 9, 12, 2, 42, 58, 361925), u'74.32', u'75.32', u'73.940002', u'74.669998', u'1368400', u'74.669998'), ('1', 17L, datetime.datetime(2015, 9, 10, 0, 0), datetime.datetime(2015, 9, 12, 2, 42, 58, 361925), datetime.datetime(2015, 9, 12, 2, 42, 58, 361925), u'73.279999', u'75.029999', u'73.190002', u'74.550003', u'1408900', u'74.550003'), ('1', 17L, datetime.datetime(2015, 9, 9, 0, 0), datetime.datetime(2015, 9, 12, 2, 42, 58, 361925), datetime.datetime(2015, 9, 12, 2, 42, 58, 361925), u'74.879997', u'75.25', u'73.129997', u'73.349998', u'1273300', u'73.349998'), ('1', 17L, datetime.datetime(2015, 9, 8, 0, 0), datetime.datetime(2015, 9, 12, 2, 42, 58, 361925), datetime.datetime(2015, 9, 12, 2, 42, 58, 361925), u'73.75', u'74.349998', u'73.059998', u'74.279999', u'1413600', u'74.279999'), ('1', 17L, datetime.datetime(2015, 9, 4, 0, 0), datetime.datetime(2015, 9, 12, 2, 42, 58, 361925), datetime.datetime(2015, 9, 12, 2, 42, 58, 361925), u'71.709999', u'73.169998', u'71.709999', u'72.580002', u'1607100', u'72.580002'), ('1', 17L, datetime.datetime(2015, 9, 3, 0, 0), datetime.datetime(2015, 9, 12, 2, 42, 58, 361925), datetime.datetime(2015, 9, 12, 2, 42, 58, 361925), u'71.860001', u'74.709999', u'71.540001', u'73.260002', u'2378600', u'73.260002'), ('1', 17L, datetime.datetime(2015, 9, 2, 0, 0), datetime.datetime(2015, 9, 12, 2, 42, 58, 361925), datetime.datetime(2015, 9, 12, 2, 42, 58, 361925), u'71.349998', u'71.669998', u'69.669998', u'71.269997', u'1671900', u'71.269997'), ('1', 17L, datetime.datetime(2015, 9, 1, 0, 0), datetime.datetime(2015, 9, 12, 2, 42, 58, 361925), datetime.datetime(2015, 9, 12, 2, 42, 58, 361925), u'69.75', u'71.43', u'69.669998', u'70.43', u'2401800', u'70.43'), ('1', 17L, datetime.datetime(2015, 8, 31, 0, 0), datetime.datetime(2015, 9, 12, 2, 42, 58, 361925), datetime.datetime(2015, 9, 12, 2, 42, 58, 361925), u'71.660004', u'71.809998', u'70.870003', u'71.309998', u'1764400', u'71.309998'), ('1', 17L, datetime.datetime(2015, 8, 28, 0, 0), datetime.datetime(2015, 9, 12, 2, 42, 58, 361925), datetime.datetime(2015, 9, 12, 2, 42, 58, 361925), u'70.589996', u'71.970001', u'70.199997', u'71.900002', u'2371800', u'71.900002'), ('1', 17L, datetime.datetime(2015, 8, 27, 0, 0), datetime.datetime(2015, 9, 12, 2, 42, 58, 361925), datetime.datetime(2015, 9, 12, 2, 42, 58, 361925), u'69.089996', u'70.889999', u'68.529999', u'70.50', u'2081800', u'70.50'), ('1', 17L, datetime.datetime(2015, 8, 26, 0, 0), datetime.datetime(2015, 9, 12, 2, 42, 58, 361925), datetime.datetime(2015, 9, 12, 2, 42, 58, 361925), u'67.07', u'68.300003', u'65.330002', u'68.169998', u'2238400', u'68.169998'), ('1', 17L, datetime.datetime(2015, 8, 25, 0, 0), datetime.datetime(2015, 9, 12, 2, 42, 58, 361925), datetime.datetime(2015, 9, 12, 2, 42, 58, 361925), u'68.239998', u'68.989998', u'65.589996', u'65.629997', u'3112100', u'65.629997'), ('1', 17L, datetime.datetime(2015, 8, 24, 0, 0), datetime.datetime(2015, 9, 12, 2, 42, 58, 361925), datetime.datetime(2015, 9, 12, 2, 42, 58, 361925), u'65.010002', u'68.220001', u'63.139999', u'66.519997', u'3779300', u'66.519997'), ('1', 17L, datetime.datetime(2015, 8, 21, 0, 0), datetime.datetime(2015, 9, 12, 2, 42, 58, 361925), datetime.datetime(2015, 9, 12, 2, 42, 58, 361925), u'69.190002', u'70.00', u'67.910004', u'68.040001', u'2298300', u'68.040001'), ('1', 17L, datetime.datetime(2015, 8, 20, 0, 0), datetime.datetime(2015, 9, 12, 2, 42, 58, 361925), datetime.datetime(2015, 9, 12, 2, 42, 58, 361925), u'72.300003', u'72.57', u'69.769997', u'69.779999', u'2096000', u'69.779999'), ('1', 17L, datetime.datetime(2015, 8, 19, 0, 0), datetime.datetime(2015, 9, 12, 2, 42, 58, 361925), datetime.datetime(2015, 9, 12, 2, 42, 58, 361925), u'73.779999', u'73.900002', u'72.290001', u'73.059998', u'881700', u'73.059998'), ('1', 17L, datetime.datetime(2015, 8, 18, 0, 0), datetime.datetime(2015, 9, 12, 2, 42, 58, 361925), datetime.datetime(2015, 9, 12, 2, 42, 58, 361925), u'73.959999', u'74.190002', u'73.230003', u'73.849998', u'1220500', u'73.849998'), ('1', 17L, datetime.datetime(2015, 8, 17, 0, 0), datetime.datetime(2015, 9, 12, 2, 42, 58, 361925), datetime.datetime(2015, 9, 12, 2, 42, 58, 361925), u'73.010002', u'73.599998', u'72.32', u'73.57', u'1025600', u'73.57'), ('1', 17L, datetime.datetime(2015, 8, 14, 0, 0), datetime.datetime(2015, 9, 12, 2, 42, 58, 361925), datetime.datetime(2015, 9, 12, 2, 42, 58, 361925), u'73.379997', u'73.839996', u'72.730003', u'73.370003', u'1328400', u'73.370003'), ...] 142 第143章 144 print("为 %s 添加数据" % (t[1]) ) 145 yf_data = get_daily_historic_data_yahoo(t[1], (2015,01,01))

.................................................. ...................... /Library/Python/2.7/site-packages/MySQLdb/connections.pyc 在退出(self=<_mysql.connection open to at>, exc=, value=OperationalError(2006, 'MySQL server has gone away'), 待定=）第258章第259章 260 261 def 退出（自我，exc，价值，tb）： 262 如果除外： --> 263 self.rollback() 264 其他：第265章 266 267 def 文字（自我，o）：

OperationalError: (2006, 'MySQL server has gone away')

进程以退出代码 1 结束

【问题讨论】：

已消失表示一段时间不活动，之后守护进程会关闭连接，甚至可能是池管理器
是的，这就是错误非常烦人的原因......当我尝试打开 8 个线程并推送时，绝对不会处于不活动状态。即使我刚刚测试了与工作台的连接性，这也是一个错误。我注意到，当我执行 2 个线程而不是 8 个线程时，该错误需要多几秒钟才能发生。

标签： python mysql parallel-processing

【解决方案1】：

MySQL 连接不是线程安全的。当您有多个进程尝试使用同一连接时，可能会发生此错误。

确保每个并行进程创建和销毁自己与数据库的连接。

来自 MySQL 文档here：

对于派生子进程的应用程序，您也可能会遇到此错误，所有这些子进程都尝试使用与 MySQL 服务器的相同连接。这可以通过为每个子进程使用单独的连接来避免。

【讨论】：

那行得通，我以愚蠢的方式完成了它，并将连接创建放在为每个股票代码执行 mysql 插入的函数中。我猜 2.0 会为每个核心创建一个连接桶，并在它们上循环执行。 Parallel(n_jobs=num_cores)(delayed(load_ticker_history)(t) for t in tickers) 并将 con = mdb.connect(db_host, db_user, db_pass, db_name) 放入 load_ticker_history(t)
Parallel(n_jobs=num_cores)(delayed(load_ticker_history)(t) for t in tickers) 并放置

【解决方案2】：

您应该创建一个连接池，而不是在线程之间共享相同的连接。您几乎可以毫不费力地（在您的连接语句中添加 1 或 2 个参数）使用以下方法来做到这一点：

dbconfig = {
  "database": "test",
  "user":     "joe"
}

cnx = mysql.connector.connect(pool_name = "mypool",
                              pool_size = 3,
                              **dbconfig)

请注意，在此 sn-p 中，您正在隐式创建连接池。您也可以使用以下方式显式创建它：

dbconfig = {
  "database": "test",
  "user":     "joe"
}

cnxpool = mysql.connector.pooling.MySQLConnectionPool(pool_name = "mypool",
                                                      pool_size = 3,
                                                      **dbconfig)

然后：

cnx1 = cnxpool.get_connection()
cnx2 = cnxpool.get_connection()