Files
General-Homework/01/01.ipynb
T

109 KiB
Raw Blame History

  1. 导入第三方库

导入 pandas 库(简写为 pd)和 numpy 库(简写为 np),并查看当前 pandas 的版本号。

In [1]:
import pandas as pd
import numpy as np

pd.__version__
Out [1]:
'3.0.5'
  1. 检测文件编码

用 chardet 检测 datasets/CRSP_Daily_Market_1926_2025.csv 的编码,并打印检测到的编码名称和置信度。

In [2]:
import chardet


def check_encoding(file):
    with open(file, 'rb') as f:
        result = chardet.detect(f.read())
    return result


file = 'datasets/CRSP_Daily_Market_1926_2025.csv'
result = check_encoding(file)
print(result['encoding'], result['confidence'])
ascii 1.0
  1. 读取 CSV 数据

先读前 5 行确认格式无误,再完整读取,结果赋值给变量 crsp。

In [3]:
crsp = pd.read_csv(file, nrows=5)
crsp
Out [3]:
DlyCalDt vwretd vwretx vwTotVal vwUsdVal vwTotCnt vwUsdCnt ewretd ewretx ewTotVal ewUsdVal ewTotCnt ewUsdCnt sprtrn spindx
0 1925-12-31 NaN NaN 27487487 NaN 500 NaN NaN NaN 27545352 NaN 503 NaN NaN NaN
1 1926-01-02 0.005689 0.005689 27600297 27366756.0 494 491.0 0.009516 0.009516 27658600 27424620.0 497 494.0 NaN NaN
2 1926-01-04 0.000706 0.000706 27577892 27479722.0 499 492.0 0.005780 0.005780 27636515 27538025.0 502 495.0 NaN NaN
3 1926-01-05 -0.004821 -0.004867 27530213 27561757.0 498 496.0 -0.001927 -0.002030 27588552 27620380.0 501 499.0 NaN NaN
4 1926-01-06 -0.000423 -0.000427 27618773 27526838.0 502 497.0 0.001182 0.001155 27676674 27585177.0 505 500.0 NaN NaN

确认格式无误后,完整读取:

In [4]:
crsp = pd.read_csv(file)
crsp
Out [4]:
DlyCalDt vwretd vwretx vwTotVal vwUsdVal vwTotCnt vwUsdCnt ewretd ewretx ewTotVal ewUsdVal ewTotCnt ewUsdCnt sprtrn spindx
0 1925-12-31 NaN NaN 27487487 NaN 500 NaN NaN NaN 27545352 NaN 503 NaN NaN NaN
1 1926-01-02 0.005689 0.005689 27600297 2.736676e+07 494 491.0 0.009516 0.009516 27658600 2.742462e+07 497 494.0 NaN NaN
2 1926-01-04 0.000706 0.000706 27577892 2.747972e+07 499 492.0 0.005780 0.005780 27636515 2.753802e+07 502 495.0 NaN NaN
3 1926-01-05 -0.004821 -0.004867 27530213 2.756176e+07 498 496.0 -0.001927 -0.002030 27588552 2.762038e+07 501 499.0 NaN NaN
4 1926-01-06 -0.000423 -0.000427 27618773 2.752684e+07 502 497.0 0.001182 0.001155 27676674 2.758518e+07 505 500.0 NaN NaN
... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ...
26296 2025-12-24 0.002914 0.002904 88568121481 8.831774e+10 9126 9126.0 0.003368 0.003125 90141105180 8.988752e+10 9498 9498.0 0.003221 6932.05
26297 2025-12-26 -0.000242 -0.000325 88542581122 8.856811e+10 9124 9124.0 -0.002105 -0.002373 90125553062 9.014109e+10 9496 9496.0 -0.000304 6929.94
26298 2025-12-29 -0.004191 -0.004230 88169518843 8.854242e+10 9123 9123.0 -0.008373 -0.008603 89739994612 9.012539e+10 9495 9495.0 -0.003492 6905.74
26299 2025-12-30 -0.001550 -0.001576 88028533719 8.816264e+10 9132 9121.0 -0.002831 -0.003547 89596162338 8.973311e+10 9504 9493.0 -0.001376 6896.24
26300 2025-12-31 -0.007319 -0.007491 87552025121 8.802809e+10 9134 9131.0 -0.005245 -0.005627 89118194675 8.959572e+10 9506 9503.0 -0.007358 6845.50

26301 rows × 15 columns

  1. 初步认识数据

查看 crsp 的行数和列数(shape)、前 5 行(head())和后 5 行(tail()),并打印出所有的列名。

In [5]:
crsp.shape
Out [5]:
(26301, 15)
In [6]:
crsp.head()
Out [6]:
DlyCalDt vwretd vwretx vwTotVal vwUsdVal vwTotCnt vwUsdCnt ewretd ewretx ewTotVal ewUsdVal ewTotCnt ewUsdCnt sprtrn spindx
0 1925-12-31 NaN NaN 27487487 NaN 500 NaN NaN NaN 27545352 NaN 503 NaN NaN NaN
1 1926-01-02 0.005689 0.005689 27600297 27366756.0 494 491.0 0.009516 0.009516 27658600 27424620.0 497 494.0 NaN NaN
2 1926-01-04 0.000706 0.000706 27577892 27479722.0 499 492.0 0.005780 0.005780 27636515 27538025.0 502 495.0 NaN NaN
3 1926-01-05 -0.004821 -0.004867 27530213 27561757.0 498 496.0 -0.001927 -0.002030 27588552 27620380.0 501 499.0 NaN NaN
4 1926-01-06 -0.000423 -0.000427 27618773 27526838.0 502 497.0 0.001182 0.001155 27676674 27585177.0 505 500.0 NaN NaN
In [7]:
crsp.tail()
Out [7]:
DlyCalDt vwretd vwretx vwTotVal vwUsdVal vwTotCnt vwUsdCnt ewretd ewretx ewTotVal ewUsdVal ewTotCnt ewUsdCnt sprtrn spindx
26296 2025-12-24 0.002914 0.002904 88568121481 8.831774e+10 9126 9126.0 0.003368 0.003125 90141105180 8.988752e+10 9498 9498.0 0.003221 6932.05
26297 2025-12-26 -0.000242 -0.000325 88542581122 8.856811e+10 9124 9124.0 -0.002105 -0.002373 90125553062 9.014109e+10 9496 9496.0 -0.000304 6929.94
26298 2025-12-29 -0.004191 -0.004230 88169518843 8.854242e+10 9123 9123.0 -0.008373 -0.008603 89739994612 9.012539e+10 9495 9495.0 -0.003492 6905.74
26299 2025-12-30 -0.001550 -0.001576 88028533719 8.816264e+10 9132 9121.0 -0.002831 -0.003547 89596162338 8.973311e+10 9504 9493.0 -0.001376 6896.24
26300 2025-12-31 -0.007319 -0.007491 87552025121 8.802809e+10 9134 9131.0 -0.005245 -0.005627 89118194675 8.959572e+10 9506 9503.0 -0.007358 6845.50
In [8]:
crsp.columns
Out [8]:
Index(['DlyCalDt', 'vwretd', 'vwretx', 'vwTotVal', 'vwUsdVal', 'vwTotCnt',
       'vwUsdCnt', 'ewretd', 'ewretx', 'ewTotVal', 'ewUsdVal', 'ewTotCnt',
       'ewUsdCnt', 'sprtrn', 'spindx'],
      dtype='str')
  1. 数据的基本信息与统计摘要

用 info() 和 describe() 查看 crsp 的列类型和统计摘要。vwretd 等列名中,vw = value-weighted 市值加权,ew = equal-weighted 等权重;sprtrn、spindx 两列从 1962 年 7 月才开始有数据,所以非缺失值个数明显更少。

In [9]:
crsp.info()
<class 'pandas.DataFrame'>
RangeIndex: 26301 entries, 0 to 26300
Data columns (total 15 columns):
 #   Column    Non-Null Count  Dtype  
---  ------    --------------  -----  
 0   DlyCalDt  26301 non-null  str    
 1   vwretd    26300 non-null  float64
 2   vwretx    26300 non-null  float64
 3   vwTotVal  26301 non-null  int64  
 4   vwUsdVal  26300 non-null  float64
 5   vwTotCnt  26301 non-null  int64  
 6   vwUsdCnt  26300 non-null  float64
 7   ewretd    26300 non-null  float64
 8   ewretx    26300 non-null  float64
 9   ewTotVal  26301 non-null  int64  
 10  ewUsdVal  26300 non-null  float64
 11  ewTotCnt  26301 non-null  int64  
 12  ewUsdCnt  26300 non-null  float64
 13  sprtrn    15981 non-null  float64
 14  spindx    15982 non-null  float64
dtypes: float64(10), int64(4), str(1)
memory usage: 3.0 MB
In [10]:
crsp.describe()
Out [10]:
vwretd vwretx vwTotVal vwUsdVal vwTotCnt vwUsdCnt ewretd ewretx ewTotVal ewUsdVal ewTotCnt ewUsdCnt sprtrn spindx
count 26300.000000 26300.000000 2.630100e+04 2.630000e+04 26301.000000 26300.000000 26300.000000 26300.000000 2.630100e+04 2.630000e+04 26301.000000 26300.000000 15981.000000 15982.000000
mean 0.000421 0.000286 8.463483e+09 8.459652e+09 3935.620661 3930.434487 0.000801 0.000689 8.699635e+09 8.695712e+09 4075.014980 4071.904411 0.000355 1088.306449
std 0.010710 0.010709 1.555537e+10 1.554715e+10 2899.110650 2896.441375 0.010570 0.010572 1.593272e+10 1.592440e+10 3042.168464 3041.323725 0.010357 1371.250067
min -0.171345 -0.171419 1.229773e+07 1.228914e+07 406.000000 404.000000 -0.135468 -0.135510 1.243940e+07 1.243080e+07 408.000000 406.000000 -0.204669 53.490000
25% -0.003854 -0.003980 6.339168e+07 6.330839e+07 944.000000 943.000000 -0.003011 -0.003138 6.372042e+07 6.365784e+07 947.000000 946.000000 -0.004274 106.325000
50% 0.000759 0.000624 7.730611e+08 7.728701e+08 4753.000000 4747.000000 0.001280 0.001172 7.956396e+08 7.954800e+08 4827.000000 4821.000000 0.000485 460.845000
75% 0.005075 0.004952 1.204819e+10 1.204109e+10 6630.000000 6608.000000 0.005063 0.004956 1.237183e+10 1.237003e+10 6876.000000 6872.000000 0.005240 1376.532500
max 0.156838 0.152974 8.856812e+10 8.856811e+10 9166.000000 9163.000000 0.203020 0.201795 9.014111e+10 9.014109e+10 9538.000000 9535.000000 0.115800 6932.050000
  1. 选择列

① 单层中括号取出 sprtrn 列(Series);② 双层中括号同时取出 DlyCalDt 和 sprtrn 两列(DataFrame),并用 type() 验证。

In [11]:
sprtrn_series = crsp['sprtrn']
type(sprtrn_series)
Out [11]:
pandas.Series
In [12]:
crsp_two_cols = crsp[['DlyCalDt', 'sprtrn']]
type(crsp_two_cols)
Out [12]:
pandas.DataFrame
  1. 选择行

① 用 iloc 选出第 0 行到第 4 行、第 0 列到第 2 列;② 用 at 取出第 1 行 vwretd 列的数值;③ 用 loc 选出第 0 行到第 4 行、且只包含 DlyCalDt 和 vwretd 两列。iloc 按整数位置切片(含头不含尾),loc 按标签切片(首尾都包含)。

In [13]:
crsp.iloc[0:5, 0:3]
Out [13]:
DlyCalDt vwretd vwretx
0 1925-12-31 NaN NaN
1 1926-01-02 0.005689 0.005689
2 1926-01-04 0.000706 0.000706
3 1926-01-05 -0.004821 -0.004867
4 1926-01-06 -0.000423 -0.000427
In [14]:
crsp.at[1, 'vwretd']
Out [14]:
0.005689
In [15]:
crsp.loc[0:4, ['DlyCalDt', 'vwretd']]
Out [15]:
DlyCalDt vwretd
0 1925-12-31 NaN
1 1926-01-02 0.005689
2 1926-01-04 0.000706
3 1926-01-05 -0.004821
4 1926-01-06 -0.000423
  1. 日期处理三步曲

① 把 DlyCalDt 列从文本转换为 datetime 格式;② 按 DlyCalDt 升序排序;③ 把 DlyCalDt 设为索引。

In [16]:
crsp['DlyCalDt'] = pd.to_datetime(crsp['DlyCalDt'], format='%Y-%m-%d')
crsp
Out [16]:
DlyCalDt vwretd vwretx vwTotVal vwUsdVal vwTotCnt vwUsdCnt ewretd ewretx ewTotVal ewUsdVal ewTotCnt ewUsdCnt sprtrn spindx
0 1925-12-31 NaN NaN 27487487 NaN 500 NaN NaN NaN 27545352 NaN 503 NaN NaN NaN
1 1926-01-02 0.005689 0.005689 27600297 2.736676e+07 494 491.0 0.009516 0.009516 27658600 2.742462e+07 497 494.0 NaN NaN
2 1926-01-04 0.000706 0.000706 27577892 2.747972e+07 499 492.0 0.005780 0.005780 27636515 2.753802e+07 502 495.0 NaN NaN
3 1926-01-05 -0.004821 -0.004867 27530213 2.756176e+07 498 496.0 -0.001927 -0.002030 27588552 2.762038e+07 501 499.0 NaN NaN
4 1926-01-06 -0.000423 -0.000427 27618773 2.752684e+07 502 497.0 0.001182 0.001155 27676674 2.758518e+07 505 500.0 NaN NaN
... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ...
26296 2025-12-24 0.002914 0.002904 88568121481 8.831774e+10 9126 9126.0 0.003368 0.003125 90141105180 8.988752e+10 9498 9498.0 0.003221 6932.05
26297 2025-12-26 -0.000242 -0.000325 88542581122 8.856811e+10 9124 9124.0 -0.002105 -0.002373 90125553062 9.014109e+10 9496 9496.0 -0.000304 6929.94
26298 2025-12-29 -0.004191 -0.004230 88169518843 8.854242e+10 9123 9123.0 -0.008373 -0.008603 89739994612 9.012539e+10 9495 9495.0 -0.003492 6905.74
26299 2025-12-30 -0.001550 -0.001576 88028533719 8.816264e+10 9132 9121.0 -0.002831 -0.003547 89596162338 8.973311e+10 9504 9493.0 -0.001376 6896.24
26300 2025-12-31 -0.007319 -0.007491 87552025121 8.802809e+10 9134 9131.0 -0.005245 -0.005627 89118194675 8.959572e+10 9506 9503.0 -0.007358 6845.50

26301 rows × 15 columns

In [17]:
crsp = crsp.sort_values('DlyCalDt')
crsp
Out [17]:
DlyCalDt vwretd vwretx vwTotVal vwUsdVal vwTotCnt vwUsdCnt ewretd ewretx ewTotVal ewUsdVal ewTotCnt ewUsdCnt sprtrn spindx
0 1925-12-31 NaN NaN 27487487 NaN 500 NaN NaN NaN 27545352 NaN 503 NaN NaN NaN
1 1926-01-02 0.005689 0.005689 27600297 2.736676e+07 494 491.0 0.009516 0.009516 27658600 2.742462e+07 497 494.0 NaN NaN
2 1926-01-04 0.000706 0.000706 27577892 2.747972e+07 499 492.0 0.005780 0.005780 27636515 2.753802e+07 502 495.0 NaN NaN
3 1926-01-05 -0.004821 -0.004867 27530213 2.756176e+07 498 496.0 -0.001927 -0.002030 27588552 2.762038e+07 501 499.0 NaN NaN
4 1926-01-06 -0.000423 -0.000427 27618773 2.752684e+07 502 497.0 0.001182 0.001155 27676674 2.758518e+07 505 500.0 NaN NaN
... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ...
26296 2025-12-24 0.002914 0.002904 88568121481 8.831774e+10 9126 9126.0 0.003368 0.003125 90141105180 8.988752e+10 9498 9498.0 0.003221 6932.05
26297 2025-12-26 -0.000242 -0.000325 88542581122 8.856811e+10 9124 9124.0 -0.002105 -0.002373 90125553062 9.014109e+10 9496 9496.0 -0.000304 6929.94
26298 2025-12-29 -0.004191 -0.004230 88169518843 8.854242e+10 9123 9123.0 -0.008373 -0.008603 89739994612 9.012539e+10 9495 9495.0 -0.003492 6905.74
26299 2025-12-30 -0.001550 -0.001576 88028533719 8.816264e+10 9132 9121.0 -0.002831 -0.003547 89596162338 8.973311e+10 9504 9493.0 -0.001376 6896.24
26300 2025-12-31 -0.007319 -0.007491 87552025121 8.802809e+10 9134 9131.0 -0.005245 -0.005627 89118194675 8.959572e+10 9506 9503.0 -0.007358 6845.50

26301 rows × 15 columns

In [18]:
crsp = crsp.set_index('DlyCalDt')
crsp
Out [18]:
vwretd vwretx vwTotVal vwUsdVal vwTotCnt vwUsdCnt ewretd ewretx ewTotVal ewUsdVal ewTotCnt ewUsdCnt sprtrn spindx
DlyCalDt
1925-12-31 NaN NaN 27487487 NaN 500 NaN NaN NaN 27545352 NaN 503 NaN NaN NaN
1926-01-02 0.005689 0.005689 27600297 2.736676e+07 494 491.0 0.009516 0.009516 27658600 2.742462e+07 497 494.0 NaN NaN
1926-01-04 0.000706 0.000706 27577892 2.747972e+07 499 492.0 0.005780 0.005780 27636515 2.753802e+07 502 495.0 NaN NaN
1926-01-05 -0.004821 -0.004867 27530213 2.756176e+07 498 496.0 -0.001927 -0.002030 27588552 2.762038e+07 501 499.0 NaN NaN
1926-01-06 -0.000423 -0.000427 27618773 2.752684e+07 502 497.0 0.001182 0.001155 27676674 2.758518e+07 505 500.0 NaN NaN
... ... ... ... ... ... ... ... ... ... ... ... ... ... ...
2025-12-24 0.002914 0.002904 88568121481 8.831774e+10 9126 9126.0 0.003368 0.003125 90141105180 8.988752e+10 9498 9498.0 0.003221 6932.05
2025-12-26 -0.000242 -0.000325 88542581122 8.856811e+10 9124 9124.0 -0.002105 -0.002373 90125553062 9.014109e+10 9496 9496.0 -0.000304 6929.94
2025-12-29 -0.004191 -0.004230 88169518843 8.854242e+10 9123 9123.0 -0.008373 -0.008603 89739994612 9.012539e+10 9495 9495.0 -0.003492 6905.74
2025-12-30 -0.001550 -0.001576 88028533719 8.816264e+10 9132 9121.0 -0.002831 -0.003547 89596162338 8.973311e+10 9504 9493.0 -0.001376 6896.24
2025-12-31 -0.007319 -0.007491 87552025121 8.802809e+10 9134 9131.0 -0.005245 -0.005627 89118194675 8.959572e+10 9506 9503.0 -0.007358 6845.50

26301 rows × 14 columns

  1. 按时间筛选

筛选出 2020 年全年的数据,赋值给 crsp_2020,并检查行数(大约 253 个交易日)。

In [19]:
crsp_2020 = crsp.loc['2020']
print(len(crsp_2020))
crsp_2020
Out [19]:
253
vwretd vwretx vwTotVal vwUsdVal vwTotCnt vwUsdCnt ewretd ewretx ewTotVal ewUsdVal ewTotCnt ewUsdCnt sprtrn spindx
DlyCalDt
2020-01-02 0.007412 0.007316 41464754137 4.115634e+10 6871 6866.0 0.005470 0.005428 42902996443 4.255570e+10 7253 7248.0 0.008379 3257.85
2020-01-03 -0.005828 -0.005900 41221078749 4.146475e+10 6871 6871.0 -0.003057 -0.003072 42645264616 4.290300e+10 7253 7253.0 -0.007060 3234.85
2020-01-06 0.003269 0.003245 41345986288 4.121183e+10 6871 6869.0 0.004548 0.004519 42768514163 4.263602e+10 7253 7251.0 0.003533 3246.28
2020-01-07 -0.002147 -0.002216 41255461753 4.134598e+10 6870 6870.0 -0.000041 -0.000057 42699866212 4.276851e+10 7253 7252.0 -0.002803 3237.18
2020-01-08 0.004028 0.003996 41421675525 4.125546e+10 6871 6870.0 0.000914 0.000903 42870237688 4.269987e+10 7254 7253.0 0.004902 3253.05
... ... ... ... ... ... ... ... ... ... ... ... ... ... ...
2020-12-24 0.002330 0.002195 49379202924 4.929405e+10 7101 7101.0 -0.000277 -0.000458 50948672283 5.091918e+10 7515 7515.0 0.003537 3703.06
2020-12-28 0.003847 0.003836 49569832981 4.937920e+10 7103 7101.0 0.001350 0.001304 51120836691 5.093776e+10 7516 7514.0 0.008723 3735.36
2020-12-29 -0.003652 -0.003660 49387356377 4.956717e+10 7108 7101.0 -0.008483 -0.008656 50978120468 5.111818e+10 7522 7514.0 -0.002227 3727.04
2020-12-30 0.003291 0.003124 49542217608 4.938705e+10 7107 7105.0 0.010155 0.009754 51166821544 5.097782e+10 7521 7519.0 0.001342 3732.04
2020-12-31 0.003558 0.003499 49839515298 4.954190e+10 7111 7105.0 -0.001058 -0.001129 51464416577 5.116650e+10 7525 7519.0 0.006439 3756.07

253 rows × 14 columns

  1. 保存数据

把 crsp_2020 保存为 datasets/CRSP_2020.csv:① 把日期索引一并写入文件(index=True);② 用 encoding='utf-8-sig',Excel 直接打开不出现乱码。保存后重新读回验证。

In [20]:
crsp_2020.to_csv('datasets/CRSP_2020.csv', index=True, encoding='utf-8-sig')
In [21]:
check = pd.read_csv('datasets/CRSP_2020.csv')
check
Out [21]:
DlyCalDt vwretd vwretx vwTotVal vwUsdVal vwTotCnt vwUsdCnt ewretd ewretx ewTotVal ewUsdVal ewTotCnt ewUsdCnt sprtrn spindx
0 2020-01-02 0.007412 0.007316 41464754137 4.115634e+10 6871 6866.0 0.005470 0.005428 42902996443 4.255570e+10 7253 7248.0 0.008379 3257.85
1 2020-01-03 -0.005828 -0.005900 41221078749 4.146475e+10 6871 6871.0 -0.003057 -0.003072 42645264616 4.290300e+10 7253 7253.0 -0.007060 3234.85
2 2020-01-06 0.003269 0.003245 41345986288 4.121183e+10 6871 6869.0 0.004548 0.004519 42768514163 4.263602e+10 7253 7251.0 0.003533 3246.28
3 2020-01-07 -0.002147 -0.002216 41255461753 4.134598e+10 6870 6870.0 -0.000041 -0.000057 42699866212 4.276851e+10 7253 7252.0 -0.002803 3237.18
4 2020-01-08 0.004028 0.003996 41421675525 4.125546e+10 6871 6870.0 0.000914 0.000903 42870237688 4.269987e+10 7254 7253.0 0.004902 3253.05
... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ...
248 2020-12-24 0.002330 0.002195 49379202924 4.929405e+10 7101 7101.0 -0.000277 -0.000458 50948672283 5.091918e+10 7515 7515.0 0.003537 3703.06
249 2020-12-28 0.003847 0.003836 49569832981 4.937920e+10 7103 7101.0 0.001350 0.001304 51120836691 5.093776e+10 7516 7514.0 0.008723 3735.36
250 2020-12-29 -0.003652 -0.003660 49387356377 4.956717e+10 7108 7101.0 -0.008483 -0.008656 50978120468 5.111818e+10 7522 7514.0 -0.002227 3727.04
251 2020-12-30 0.003291 0.003124 49542217608 4.938705e+10 7107 7105.0 0.010155 0.009754 51166821544 5.097782e+10 7521 7519.0 0.001342 3732.04
252 2020-12-31 0.003558 0.003499 49839515298 4.954190e+10 7111 7105.0 -0.001058 -0.001129 51464416577 5.116650e+10 7525 7519.0 0.006439 3756.07

253 rows × 15 columns