python 中的无限 while 循环，用 pandas 计算标准差

Question

我们正在尝试删除异常值，但出现了无限循环对于一个学校项目，我们（我和一个朋友）认为创建一个基于数据科学的工具是个好主意。为此，我们开始清理数据库（我不会在这里导入它，因为它太大（xlsx 文件、csv 文件））。我们现在尝试使用“duration_分钟”列的“标准差*3 + 平均值”规则删除异常值。这是我们用来计算标准差和平均值的代码：def calculateSD(database, column):    column = database[[column]]    SD = column.std(axis=None, skipna=None, level=None, ddof=1, numeric_only=None)    return SDdef calculateMean(database, column):    column = database[[column]]    mean = column.mean()    return mean我们认为要做到以下几点：#Now we have to remove the outliers using the code from the SD.py and SDfunction.py filesminutes = trainsData['duration_minutes'].tolist() #takes the column duration_minutes and puts it in a listSD = int(calculateSD(trainsData, 'duration_minutes')) #calculates the SD of the columnmean = int(calculateMean(trainsData, 'duration_minutes'))SDhigh = mean+3*SD上面的代码计算起始值。然后我们启动一个 while 循环来删除异常值。删除异常值后，我们重新计算标准差、均值和 SDhigh。这是 while 循环：while np.any(i >= SDhigh for i in minutes): #used to be >=, it doesnt matter for the outcome    trainsData = trainsData[trainsData['duration_minutes'] < SDhigh] #used to be >=, this caused an infinite loop so I changed it to <=. Then to <    minutes = trainsData['duration_minutes'].tolist()    SD = int(calculateSD(trainsData, 'duration_minutes')) #calculates the SD of the column    mean = int(calculateMean(trainsData, 'duration_minutes'))    SDhigh = mean+3*SD    print(SDhigh) #to see how the values changed and to confirm it is an infinite loop输出如下：611652428354322308300296296296296它继续打印 296，经过几个小时的尝试解决这个问题，我们得出的结论是我们没有我们希望的那么聪明。

Answer 1

你让事情变得比原本应该的更加困难。计算标准差以消除异常值，然后重新计算等等过于复杂（并且统计上不合理）。使用百分位数而不是标准差会更好

import numpy as np

import pandas as pd

# create data

nums = np.random.normal(50, 8, 200)

df = pd.DataFrame(nums, columns=['duration'])

# set threshold based on percentiles

threshold = df['duration'].quantile(.95) * 2

# now only keep rows that are below the threshold

df = df[df['duration']<threshold]