Macbook pro m1 max 上的 Python 运行错误（在 Tensorflow 上运行）

Question

我正在尝试在我的新 macbook pro max M1 芯片上从 github binary-bot 运行此代码：

金属设备设置为：

Apple M1 Max
systemMemory: 32.00 GB
maxCacheSize: 10.67 GB

我收到以下错误。有什么建议吗？

2021-12-19 17:26:25.248041: I tensorflow/core/common_runtime/pluggable_device/pluggable_device_factory.cc:305] Could not identify NUMA node of platform GPU ID 0, defaulting to 0. Your kernel may not have been built with NUMA support.
2021-12-19 17:26:25.248181: I tensorflow/core/common_runtime/pluggable_device/pluggable_device_factory.cc:271] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 0 MB memory) -> physical PluggableDevice (device: 0, name: METAL, pci bus id: <undefined>)
1 Physical GPUs, 1 Logical GPUs
1 Physical GPUs, 1 Logical GPUs
Trying to connect to IqOption
Successfully Connected!
/Users/abdallahmohamed/Downloads/binary-bot-master/training.py:35: FutureWarning: In a future version of pandas all arguments of DataFrame.drop except for the argument 'labels' will be keyword-only
  df = df.drop("future", 1)
/Users/abdallahmohamed/Downloads/binary-bot-master/training.py:35: FutureWarning: In a future version of pandas all arguments of DataFrame.drop except for the argument 'labels' will be keyword-only
  df = df.drop("future", 1)
train data: 836 validation: 68
sells: 418, buys: 418
VALIDATION sells: 34, buys : 34
0.001-5-SEQ-2-40-16-PRED-1639927591
1 Physical GPUs, 1 Logical GPUs
2021-12-19 17:26:32.262259: W tensorflow/core/platform/profile_utils/cpu_utils.cc:128] Failed to get CPU frequency: 0 Hz
Epoch 1/40
2021-12-19 17:26:33.223096: I tensorflow/core/grappler/optimizers/custom_graph_optimizer_registry.cc:112] Plugin optimizer for device_type GPU is enabled.
2021-12-19 17:26:33.592036: I tensorflow/core/grappler/optimizers/custom_graph_optimizer_registry.cc:112] Plugin optimizer for device_type GPU is enabled.
2021-12-19 17:26:33.646351: I tensorflow/core/grappler/optimizers/custom_graph_optimizer_registry.cc:112] Plugin optimizer for device_type GPU is enabled.
2021-12-19 17:26:33.684523: I tensorflow/core/grappler/optimizers/custom_graph_optimizer_registry.cc:112] Plugin optimizer for device_type GPU is enabled.
2021-12-19 17:26:33.786763: I tensorflow/core/grappler/optimizers/custom_graph_optimizer_registry.cc:112] Plugin optimizer for device_type GPU is enabled.
2021-12-19 17:26:33.866171: I tensorflow/core/grappler/optimizers/custom_graph_optimizer_registry.cc:112] Plugin optimizer for device_type GPU is enabled.
2021-12-19 17:26:33.932667: I tensorflow/core/grappler/optimizers/custom_graph_optimizer_registry.cc:112] Plugin optimizer for device_type GPU is enabled.
53/53 [==============================] - ETA: 0s - loss: 0.8687 - accuracy: 0.52272021-12-19 17:26:35.527128: I tensorflow/core/grappler/optimizers/custom_graph_optimizer_registry.cc:112] Plugin optimizer for device_type GPU is enabled.
2021-12-19 17:26:35.652789: I tensorflow/core/grappler/optimizers/custom_graph_optimizer_registry.cc:112] Plugin optimizer for device_type GPU is enabled.
2021-12-19 17:26:35.681668: I tensorflow/core/grappler/optimizers/custom_graph_optimizer_registry.cc:112] Plugin optimizer for device_type GPU is enabled.
2021-12-19 17:26:35.710022: I tensorflow/core/grappler/optimizers/custom_graph_optimizer_registry.cc:112] Plugin optimizer for device_type GPU is enabled.
WARNING:tensorflow:Can save best model only with val_acc available, skipping.
53/53 [==============================] - 4s 34ms/step - loss: 0.8687 - accuracy: 0.5227 - val_loss: 0.6928 - val_accuracy: 0.5000
Epoch 2/40
53/53 [==============================] - ETA: 0s - loss: 0.7844 - accuracy: 0.5335WARNING:tensorflow:Can save best model only with val_acc available, skipping.
53/53 [==============================] - 1s 24ms/step - loss: 0.7844 - accuracy: 0.5335 - val_loss: 0.6890 - val_accuracy: 0.5000
Epoch 3/40
51/53 [===========================>..] - ETA: 0s - loss: 0.7257 - accuracy: 0.5588WARNING:tensorflow:Can save best model only with val_acc available, skipping.
53/53 [==============================] - 1s 24ms/step - loss: 0.7293 - accuracy: 0.5562 - val_loss: 0.6836 - val_accuracy: 0.5735
Epoch 4/40
52/53 [============================>.] - ETA: 0s - loss: 0.7421 - accuracy: 0.5649WARNING:tensorflow:Can save best model only with val_acc available, skipping.
53/53 [==============================] - 1s 24ms/step - loss: 0.7411 - accuracy: 0.5658 - val_loss: 0.7035 - val_accuracy: 0.4412
Epoch 5/40
52/53 [============================>.] - ETA: 0s - loss: 0.7205 - accuracy: 0.5565WARNING:tensorflow:Can save best model only with val_acc available, skipping.
53/53 [==============================] - 1s 24ms/step - loss: 0.7193 - accuracy: 0.5586 - val_loss: 0.7327 - val_accuracy: 0.4412
Epoch 6/40
52/53 [============================>.] - ETA: 0s - loss: 0.7233 - accuracy: 0.5637WARNING:tensorflow:Can save best model only with val_acc available, skipping.
53/53 [==============================] - 1s 24ms/step - loss: 0.7228 - accuracy: 0.5634 - val_loss: 0.7023 - val_accuracy: 0.5441
Epoch 7/40
51/53 [===========================>..] - ETA: 0s - loss: 0.7192 - accuracy: 0.5588WARNING:tensorflow:Can save best model only with val_acc available, skipping.
53/53 [==============================] - 1s 24ms/step - loss: 0.7187 - accuracy: 0.5586 - val_loss: 0.8523 - val_accuracy: 0.4559
Epoch 8/40
51/53 [===========================>..] - ETA: 0s - loss: 0.7111 - accuracy: 0.5613WARNING:tensorflow:Can save best model only with val_acc available, skipping.
53/53 [==============================] - 1s 24ms/step - loss: 0.7105 - accuracy: 0.5634 - val_loss: 0.7727 - val_accuracy: 0.4559
Epoch 9/40
53/53 [==============================] - ETA: 0s - loss: 0.7151 - accuracy: 0.5514WARNING:tensorflow:Can save best model only with val_acc available, skipping.
53/53 [==============================] - 1s 25ms/step - loss: 0.7151 - accuracy: 0.5514 - val_loss: 0.7105 - val_accuracy: 0.5147
Epoch 10/40
53/53 [==============================] - ETA: 0s - loss: 0.7046 - accuracy: 0.5371WARNING:tensorflow:Can save best model only with val_acc available, skipping.
53/53 [==============================] - 1s 25ms/step - loss: 0.7046 - accuracy: 0.5371 - val_loss: 0.6940 - val_accuracy: 0.5588
Epoch 11/40
53/53 [==============================] - ETA: 0s - loss: 0.7064 - accuracy: 0.5455WARNING:tensorflow:Can save best model only with val_acc available, skipping.
53/53 [==============================] - 1s 25ms/step - loss: 0.7064 - accuracy: 0.5455 - val_loss: 0.7433 - val_accuracy: 0.3971
Epoch 12/40
51/53 [===========================>..] - ETA: 0s - loss: 0.6991 - accuracy: 0.5784WARNING:tensorflow:Can save best model only with val_acc available, skipping.
53/53 [==============================] - 1s 25ms/step - loss: 0.6988 - accuracy: 0.5778 - val_loss: 0.6902 - val_accuracy: 0.5147
Epoch 13/40
52/53 [============================>.] - ETA: 0s - loss: 0.6812 - accuracy: 0.5757WARNING:tensorflow:Can save best model only with val_acc available, skipping.
53/53 [==============================] - 1s 24ms/step - loss: 0.6818 - accuracy: 0.5754 - val_loss: 0.8100 - val_accuracy: 0.4118
Epoch 14/40
52/53 [============================>.] - ETA: 0s - loss: 0.6876 - accuracy: 0.5673WARNING:tensorflow:Can save best model only with val_acc available, skipping.
53/53 [==============================] - 1s 24ms/step - loss: 0.6888 - accuracy: 0.5658 - val_loss: 0.7208 - val_accuracy: 0.5294
Epoch 15/40
52/53 [============================>.] - ETA: 0s - loss: 0.6815 - accuracy: 0.5505WARNING:tensorflow:Can save best model only with val_acc available, skipping.
53/53 [==============================] - 1s 23ms/step - loss: 0.6809 - accuracy: 0.5502 - val_loss: 0.6965 - val_accuracy: 0.5441
Epoch 16/40
51/53 [===========================>..] - ETA: 0s - loss: 0.6886 - accuracy: 0.5711WARNING:tensorflow:Can save best model only with val_acc available, skipping.
53/53 [==============================] - 1s 24ms/step - loss: 0.6900 - accuracy: 0.5670 - val_loss: 0.6529 - val_accuracy: 0.6029
Epoch 17/40
53/53 [==============================] - ETA: 0s - loss: 0.6959 - accuracy: 0.5598WARNING:tensorflow:Can save best model only with val_acc available, skipping.
53/53 [==============================] - 1s 24ms/step - loss: 0.6959 - accuracy: 0.5598 - val_loss: 0.7832 - val_accuracy: 0.4118
Epoch 18/40
52/53 [============================>.] - ETA: 0s - loss: 0.7002 - accuracy: 0.5325WARNING:tensorflow:Can save best model only with val_acc available, skipping.
53/53 [==============================] - 1s 24ms/step - loss: 0.6999 - accuracy: 0.5335 - val_loss: 0.7270 - val_accuracy: 0.3676
WARNING:tensorflow:Layer lstm_3 will not use cuDNN kernels since it doesn't meet the criteria. It will use a generic GPU kernel as fallback when running on GPU.
WARNING:tensorflow:Layer lstm_4 will not use cuDNN kernels since it doesn't meet the criteria. It will use a generic GPU kernel as fallback when running on GPU.
WARNING:tensorflow:Layer lstm_5 will not use cuDNN kernels since it doesn't meet the criteria. It will use a generic GPU kernel as fallback when running on GPU.
Trying to connect to IqOption
Successfully Connected!
2021-12-19 17:27:01.637577: I tensorflow/core/grappler/optimizers/custom_graph_optimizer_registry.cc:112] Plugin optimizer for device_type GPU is enabled.
2021-12-19 17:27:01.792362: F tensorflow/core/framework/tensor.cc:681] Check failed: IsAligned() ptr = 0x2d0efcee0
zsh: abort      /Users/abdallahmohamed/miniforge3/envs/env_tensorflow/bin/python 

Answer 1

在我删除存储的模型并保存一个新模型后它起作用了

Answer 2

我的感觉是这里有两个明显的问题：

1. 使用 Metal 引擎在 M1 机器上安装 Tensorflow 的步骤。
2. 要使用的包的requirements.txt 文件。

背景：

在使用新的 M1 处理器时，由于某些命名约定和版本控制假设往往包含在包 requirement.txt 文件中，您往往会在安装 M1 之前的 Python 包时遇到问题。

另外，需要注意的是，M1 处理器仅支持 Python 3.8 及更高版本。唉，Python 3.7 和所有以前的 Python 版本都是在 M1 处理器发布之前开发的，并且（显然）没有计划向后移植关键补丁。

Tensorflow-macos 和 Tensorflow-metal 安装

安装Tensorflow-macos和TensorFlow-metal的步骤详细here，使用mini-forge可以总结如下：

conda create -n tf python=3.8 -y
conda activate tf
conda install -c apple tensorflow-deps -y
# Navigate the issue with conda environments
# built against pre-macOS 11 SDK use (Just in case)
# this solves the Intel / M1 install issues:
SYSTEM_VERSION_COMPAT=0 pip install tensorflow-macos 
SYSTEM_VERSION_COMPAT=0 pip install tensorflow-metal

所以我们可以测试它是否有效：

#!/usr/bin/env bash

PYCMD=$(cat <<EOF
import tensorflow as tf

tf.config.list_physical_devices()
with tf.device('/GPU'):
  a = tf.random.normal(shape=(2,), dtype=tf.float32)
  b = tf.nn.relu(a)

print(a)
print(b)

EOF
)

python3 -c "$PYCMD" 2>/dev/null

我的测试机器的预期输出：

>./val.sh
Metal device set to: Apple M1

systemMemory: 16.00 GB
maxCacheSize: 5.33 GB

tf.Tensor([-0.4044231   0.08157467], shape=(2,), dtype=float32)
tf.Tensor([0.         0.08157467], shape=(2,), dtype=float32)

好的，一切都（假设）正在工作......

Binary Bot 安装问题说明：

现在我们来看看 Binary Bot...

首先，我们需要通过 git hub 安装

git clone https://github.com/ItamarRocha/binary-bot

好的，现在在安装之前让我们看一下 requirements.txt 文件...

iqoptionapi @ git+git://github.com/Lu-Yi-Hsun/iqoptionapi.git@e96ba2c5b905a139a4765167b08c5df48cf57773
numpy==1.18.0
pandas==1.1.1
tensorflow==2.3.1
scikit-learn==0.23.2
grpcio==1.24.3

重要的是要注意两个关键问题== 和tensorflow。需求文件中的== 意味着与包版本完全匹配。同时我们还没有安装tensorflow，我们已经安装了tensorflow-macos和tensorflow-metal。我们可以检查一下：

>pip show numpy pandas tensorflow scikit-learn grpcio | egrep 'Name:|Version:'
WARNING: Package(s) not found: pandas, scikit-learn, tensorflow
Name: numpy
Version: 1.19.5
Name: grpcio
Version: 1.43.0

如果我们运行：

python3 pip install -r requirements.txt

我们收到大量错误。为什么？好吧，因为requirements.txt 文件需要精确 == 包匹配而不是兼容的~= 包匹配或下载大于或等于>= 的任何版本到规定的版本。

为了解决这个问题，我们可以更改为 >=，这（希望）应该允许我们安装与 Apple M1 芯片兼容的软件包，但如果开发人员依赖软件包中的旧假设，我们可能会破坏兼容性。

此外，requirements.txt 将尝试安装 tensorflow 软件包，而不是 tensorflow-macosx 或 tensorflow-metal 软件包，正如我们看到的那样，当我们运行 requirements.txt 文件不变时会产生许多问题。

那么如何导航呢？

嗯，一种选择是简单地将== 替换为>=，然后将tensorflow 替换或删除为tensorflow-macosx。

sed -i.bak 's/==/>=/g;s/tensorflow/tensorflow-macos/g' requirements.txt

导致：

cat requirements.txt
iqoptionapi @ git+git://github.com/Lu-Yi-Hsun/iqoptionapi.git@e96ba2c5b905a139a4765167b08c5df48cf57773
numpy>=1.18.0
pandas>=1.1.1
tensorflow-macos>=2.3.1
scikit-learn>=0.23.2
grpcio>=1.24.3

现在如果我们运行：

pip install -r requirements.txt

我们安装了最新版本的软件包>=，结果：

pip show numpy pandas tensorflow-macos tensorflow-metal scikit-learn grpcio | egrep 'Name:|Version:'
Name: numpy
Version: 1.22.0
Name: pandas
Version: 1.3.5
Name: tensorflow-macos
Version: 2.7.0
Name: tensorflow-metal
Version: 0.3.0
Name: scikit-learn
Version: 1.0.2
Name: grpcio
Version: 1.43.0

所以，我们现在应该能够测试一切是否正常......

python3 testing.py
Metal device set to: Apple M1

systemMemory: 16.00 GB
maxCacheSize: 5.33 GB

2022-01-08 20:40:40.591430: I tensorflow/core/common_runtime/pluggable_device/pluggable_device_factory.cc:305] Could not identify NUMA node of platform GPU ID 0, defaulting to 0. Your kernel may not have been built with NUMA support.
2022-01-08 20:40:40.591754: I tensorflow/core/common_runtime/pluggable_device/pluggable_device_factory.cc:271] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 0 MB memory) -> physical PluggableDevice (device: 0, name: METAL, pci bus id: <undefined>)
1 Physical GPUs, 1 Logical GPUs
1 Physical GPUs, 1 Logical GPUs
Trying to connect to IqOption
Error when trying to connect
<iqoptionapi.stable_api.IQ_Option object at 0x1031074c0>
Retrying

现在安装包，只需将您的姓名和密码添加到 iq.py 文件，并检查结果是否符合预期。

  19   if iq == None:
  20       print("Trying to connect to IqOption")
  21       iq=IQ_Option('USERNAME','PASSWORD') # YOU HAVE TO ADD YOUR USERNAME AND PASSWORD
  22       iq.connect()

参考“删除模型”这一点，这是公认的答案。我认为这是指binary-bot 目录中的models 目录中的LSTM-best.model。

tree binary-bot                                                  binary-bot
├── README.md
├── __pycache__
│   ├── iq.cpython-38.pyc
│   └── training.cpython-38.pyc
├── imgs
│   ├── mac_forex.jpg
│   └── result_binary.png
├── iq.py
├── models
│   └── LSTM-best.model
├── requirements.txt
├── requirements.txt.bak
├── result_binary.png
├── testing.py
└── training.py

唉，我目前不在一个允许我注册服务 https://iqoption.net/ 以进行全面测试的国家/地区，但我的感觉是事情应该工作到可以根据上述结果验证结果的程度。

希望这能指引人们正确思考使用 TensorFlow 金属插件让 python 包与 Apple-M1 / Apple-Intel 笔记本电脑一起工作所涉及的障碍。

保持安全和健康。