AUC - ROC egri chizig‘i
AUC - ROC egri chizig‘i
Tasniflashda ko‘plab turli baholash ko‘rsatkichlari mavjud. Eng ommabop model qanchalik tez-tez to‘g‘ri ekanligini o‘lchaydigan aniqlikdir. Bu juda yaxshi ko‘rsatkichdir, chunki uni tushunish oson va eng to‘g‘ri taxminlarni olish ko‘pincha orzu qilinadi. Boshqa baholash ko‘rsatkichidan foydalanishni ko‘rib chiqishingiz mumkin bo‘lgan ba’zi holatlar mavjud.
Yana bir keng tarqalgan ko‘rsatkich - bu AUC, qabul qiluvchining ishlash xarakteristikasi (ROC) egri chizig‘i ostidagi maydon. Qabul qiluvchining operatsion xarakteristikasi egri chizig‘i turli tasniflash chegaralarida haqiqiy musbat (TP) tezligini noto‘g‘ri musbat (FP) tezligiga qaratadi. Eshiklar ikkilik tasnifida ikkita classni ajratib turadigan turli xil ehtimollik chegaralaridir. Model classlarni qanchalik yaxshi ajratishini aytish uchun ehtimollikdan foydalanadi.
Balanssiz ma’lumotlar
Aytaylik, bizda nomutanosib ma’lumotlar to‘plami bor, bu yerda ma’lumotlarimizning aksariyati bitta qiymatga ega. Ko‘pchilik sinfini bashorat qilish orqali model uchun yuqori aniqlikni olishimiz mumkin.
Misol
import numpy as np
from sklearn.metrics import accuracy_score, confusion_matrix, roc_auc_score, roc_curve
n = 10000
ratio = .95
n_0 = int((1-ratio) * n)
n_1 = int(ratio * n)
y = np.array([0] * n_0 + [1] * n_1)
# below are the probabilities obtained from a hypothetical model that always predicts the majority class
# probability of predicting class 1 is going to be 100%
y_proba = np.array([1]*n)
y_pred = y_proba > .5
print(f'accuracy score: {accuracy_score(y, y_pred)}')
cf_mat = confusion_matrix(y, y_pred)
print('Confusion matrix')
print(cf_mat)
print(f'class 0 accuracy: {cf_mat[0][0]/n_0}')
print(f'class 1 accuracy: {cf_mat[1][1]/n_1}')
Misolni ishga tushirish »
Garchi biz juda yuqori aniqlikka erishsak ham, model ma’lumotlar haqida hech qanday ma’lumot bermagan, shuning uchun u foydali emas. Biz 1-classni 100% aniq bashorat qilamiz, 0% ni esa noto‘g‘ri taxmin qilamiz. Aniqlik hisobiga ikkita classni biroz ajratib turadigan modelga ega bo‘lish yaxshiroq bo‘lishi mumkin.
Misol
# below are the probabilities obtained from a hypothetical model that doesn't always predict the mode
y_proba_2 = np.array(
np.random.uniform(0, .7, n_0).tolist() +
np.random.uniform(.3, 1, n_1).tolist()
)
y_pred_2 = y_proba_2 > .5
print(f'accuracy score: {accuracy_score(y, y_pred_2)}')
cf_mat = confusion_matrix(y, y_pred_2)
print('Confusion matrix')
print(cf_mat)
print(f'class 0 accuracy: {cf_mat[0][0]/n_0}')
print(f'class 1 accuracy: {cf_mat[1][1]/n_1}')
Misolni ishga tushirish »
Ikkinchi bashorat to‘plami uchun bizda birinchisi kabi yuqori aniqlik balli yo‘q, lekin har bir class uchun aniqlik yanada muvozanatli. Baholash ko‘rsatkichi sifatida aniqlikdan foydalanib, biz birinchi modelni ikkinchisidan yuqoriroq baholaymiz, garchi u bizga ma’lumotlar haqida hech narsa aytmasa ham.
Bunday hollarda, AUC kabi boshqa baholash ko‘rsatkichidan foydalanish afzalroq bo‘ladi.
import matplotlib.pyplot as plt
def plot_roc_curve(true_y, y_prob):
"""
plots the roc curve based of the probabilities
"""
fpr, tpr, thresholds = roc_curve(true_y, y_prob)
plt.plot(fpr, tpr)
plt.xlabel('False Positive Rate')
plt.ylabel('True Positive Rate')
Misol
Model 1:
plot_roc_curve(y, y_proba)
print(f'model 1 AUC score: {roc_auc_score(y, y_proba)}')
Natija
model 1 AUC score: 0.5
Misol
Model 2:
plot_roc_curve(y, y_proba_2)
print(f'model 2 AUC score: {roc_auc_score(y, y_proba_2)}')
Natija
model 2 AUC score: 0.8270551578947367
AUC balli .5 atrofida bo‘lsa, bu model ikki class o‘rtasida farq qila olmasligini anglatadi va egri chiziq 1 qiyalikli chiziqqa o‘xshaydi. AUC ball 1 ga yaqinroq bo‘lsa, bu model ikki classni ajratish qobiliyatiga ega ekanligini va egri chiziq grafikning yuqori chap burchagiga yaqinlashishini bildiradi.
Ehtimollar
AUC class bashoratlarining ehtimollaridan foydalanadigan ko‘rsatkich bo‘lganligi sababli, biz shunga o‘xshash aniqlikka ega bo‘lsa ham, past ballga qaraganda yuqori AUC ballga ega bo‘lgan modelga ko‘proq ishonch hosil qilishimiz mumkin.
Quyidagi ma’lumotlarda bizda gipotetik modellardan ikkita ehtimollik to‘plami mavjud. Birinchisi, ikkita classni bashorat qilishda "ishonchli" bo‘lmagan ehtimollarga ega (ehtimolliklar .5 ga yaqin). Ikkinchisi ikki classni bashorat qilishda ko‘proq "ishonchli" bo‘lgan ehtimollarga ega (ehtimolliklar 0 yoki 1 ekstremallariga yaqin).
Misol
import numpy as np
n = 10000
y = np.array([0] * n + [1] * n)
#
y_prob_1 = np.array(
np.random.uniform(.25, .5, n//2).tolist() +
np.random.uniform(.3, .7, n).tolist() +
np.random.uniform(.5, .75, n//2).tolist()
)
y_prob_2 = np.array(
np.random.uniform(0, .4, n//2).tolist() +
np.random.uniform(.3, .7, n).tolist() +
np.random.uniform(.6, 1, n//2).tolist()
)
print(f'model 1 accuracy score: {accuracy_score(y, y_prob_1>.5)}')
print(f'model 2 accuracy score: {accuracy_score(y, y_prob_2>.5)}')
print(f'model 1 AUC score: {roc_auc_score(y, y_prob_1)}')
print(f'model 2 AUC score: {roc_auc_score(y, y_prob_2)}')
Misolni ishga tushirish »
Misol
Plot model 2:
fpr, tpr, thresholds = roc_curve(y, y_prob_2)
plt.plot(fpr, tpr)
Natija
Ikkala modelning aniqligi o‘xshash bo‘lsa ham, AUC balli yuqori bo‘lgan model ishonchliroq bo‘ladi, chunki u taxmin qilingan ehtimollikni hisobga oladi. Kelajakdagi ma’lumotlarni bashorat qilishda sizga yuqori aniqlik berish ehtimoli ko‘proq.
W3Schools Pathfinder
Yutuqlaringizni kuzating – bu bepul!