在 HPC 集群中创建 Dask LocalCluster 实例时 SLURM 任务失败

Question

我正在使用命令sbatch和下一个配置排队任务：#SBATCH --job-name=dask-test#SBATCH --ntasks=1#SBATCH --cpus-per-task=10#SBATCH --mem=80G#SBATCH --time=00:30:00#SBATCH --tmp=10G#SBATCH --partition=normal#SBATCH --qos=normalpython ./dask-test.pypython脚本大致如下：import pandas as pdimport dask.dataframe as ddimport numpy as npfrom dask.distributed import Client, LocalClusterprint("Generating LocalCluster...")cluster = LocalCluster()print("Generating Client...")client = Client(cluster, processes=False)print("Scaling client...")client.scale(8)data = dd.read_csv(    BASE_DATA_SOURCE + '/Data-BIGDATFILES-*.csv',    delimiter=';',)def get_min_dt():    min_dt = data.datetime.min().compute()    print("Min is {}".format())print("Getting min dt...")get_min_dt()第一个问题是文本“Generating LocalCluster...”打印了 6 次，这让我怀疑脚本是否同时运行了多次。其次，在几分钟不打印任何内容后，我收到以下消息：/anaconda3/lib/python3.7/site-packages/distributed/node.py:155: UserWarning: Port 8787 is already in use.Perhaps you already have a cluster running?Hosting the HTTP server on port 37396 instead  http_address["port"], self.http_server.port

Answer 1

经过一些研究，我可以得到一个解决方案。不太确定原因，但非常确定它有效。

LocalCluster、Client 及其之后的所有代码（将分发执行的代码）的实例化不得在 Python 脚本的模块级别。相反，此代码必须位于方法中或 __main__ 块内，如下所示：

import pandas as pd

import dask.dataframe as dd

import numpy as np

from dask.distributed import Client, LocalCluster

if __name__ == "__main__":

    print("Generating LocalCluster...")

    cluster = LocalCluster()

    print("Generating Client...")

    client = Client(cluster, processes=False)

    print("Scaling client...")

    client.scale(8)

    data = dd.read_csv(

        BASE_DATA_SOURCE + '/Data-BIGDATFILES-*.csv',

        delimiter=';',

    )

    def get_min_dt():

        min_dt = data.datetime.min().compute()

        print("Min is {}".format())

    print("Getting min dt...")

    get_min_dt()

这个简单的改变带来了不同。在该问题线程中找到了解决方案：https://github.com/dask/distributed/issues/2520#issuecomment-470817810