您的位置: 首页  >  文章  >  小白入门 Pandas基础

小白入门 Pandas基础

分类: 文章 • 2023-12-03 19:42:04

小白入门 Pandas基础

Pandas基础

一般我们以下面方式导入模块：

In [1]: import pandas as pdIn [2]: import numpy as npIn [3]: import matplotlib.pyplot as plt

Series创建

In [4]: s = pd.Series([1,3,5,np.nan,6,8])In [5]: sOut[5]: 0 1.01 3.02 5.03 NaN4 6.05 8.0dtype: float64

DataFrame创建

  • 通过numpy array 创建数据框：

In [6]: dates = pd.date_range('20130101', periods=6)In [7]: datesOut[7]: DatetimeIndex(['2013-01-01', '2013-01-02', '2013-01-03', '2013-01-04',
 '2013-01-05', '2013-01-06'],
 dtype='datetime64[ns]', freq='D')In [8]: df = pd.DataFrame(np.random.randn(6,4), index=dates, columns=list('ABCD'))In [9]: dfOut[9]:
 A B C D2013-01-01 0.469112 -0.282863 -1.509059 -1.1356322013-01-02 1.212112 -0.173215 0.119209 -1.0442362013-01-03 -0.861849 -2.104569 -0.494929 1.0718042013-01-04 0.721555 -0.706771 -1.039575 0.2718602013-01-05 -0.424972 0.567020 0.276232 -1.0874012013-01-06 -0.673690 0.113648 -1.478427 0.524988

  • 通过字典创建数据框:

In [10]: df2 = pd.DataFrame({ 'A' : 1.,
 ....: 'B' : pd.Timestamp('20130102'),
 ....: 'C' : pd.Series(1,index=list(range(4)),dtype='float32'),
 ....: 'D' : np.array([3] * 4,dtype='int32'),
 ....: 'E' : pd.Categorical(["test","train","test","train"]),
 ....: 'F' : 'foo' })
 ....: In [11]: df2 # 键代表列名Out[11]:
 A B C D E F0 1.0 2013-01-02 1.0 3 test foo1 1.0 2013-01-02 1.0 3 train foo2 1.0 2013-01-02 1.0 3 test foo3 1.0 2013-01-02 1.0 3 train fooIn [12]: df2.dtypes # 类型是不同的Out[12]: A float64B datetime64[ns]C float32D int32E categoryF objectdtype: object

查看数据框

In [14]: df.head() # 数据框的前5行数据Out[14]:
 A B C D2013-01-01 0.469112 -0.282863 -1.509059 -1.1356322013-01-02 1.212112 -0.173215 0.119209 -1.0442362013-01-03 -0.861849 -2.104569 -0.494929 1.0718042013-01-04 0.721555 -0.706771 -1.039575 0.2718602013-01-05 -0.424972 0.567020 0.276232 -1.087401In [15]: df.tail(3) # 数据框的后3行数据Out[15]:
 A B C D2013-01-04 0.721555 -0.706771 -1.039575 0.2718602013-01-05 -0.424972 0.567020 0.276232 -1.0874012013-01-06 -0.673690 0.113648 -1.478427 0.524988

  • 查看索引，列名和numpy数据

In [16]: df.index # 索引Out[16]: DatetimeIndex(['2013-01-01', '2013-01-02', '2013-01-03', '2013-01-04',
 '2013-01-05', '2013-01-06'],
 dtype='datetime64[ns]', freq='D')In [17]: df.columns # 列名Out[17]: Index(['A', 'B', 'C', 'D'], dtype='object')In [18]: df.values # numpy的数据值Out[18]: array([[ 0.4691, -0.2829, -1.5091, -1.1356],
 [ 1.2121, -0.1732, 0.1192, -1.0442],
 [-0.8618, -2.1046, -0.4949, 1.0718],
 [ 0.7216, -0.7068, -1.0396, 0.2719],
 [-0.425 , 0.567 , 0.2762, -1.0874],
 [-0.6737, 0.1136, -1.4784, 0.525 ]])

  • 数据概览

In [19]: df.describe()Out[19]:
 A B C Dcount 6.000000 6.000000 6.000000 6.000000mean 0.073711 -0.431125 -0.687758 -0.233103std 0.843157 0.922818 0.779887 0.973118min -0.861849 -2.104569 -1.509059 -1.13563225% -0.611510 -0.600794 -1.368714 -1.07661050% 0.022070 -0.228039 -0.767252 -0.38618875% 0.658444 0.041933 -0.034326 0.461706max 1.212112 0.567020 0.276232 1.071804
依次是个数，均值，标准差，最小值，上四分位数，中值，下四分位数，最大值。

  • 转置数据框

In [20]: df.TOut[20]:
 2013-01-01 2013-01-02 2013-01-03 2013-01-04 2013-01-05 2013-01-06A 0.469112 1.212112 -0.861849 0.721555 -0.424972 -0.673690B -0.282863 -0.173215 -2.104569 -0.706771 0.567020 0.113648C -1.509059 0.119209 -0.494929 -1.039575 0.276232 -1.478427D -1.135632 -1.044236 1.071804 0.271860 -1.087401 0.524988

  • 通过轴(就是列名&行名)排序

In [21]: df.sort_index(axis=1, ascending=False) # 通过列名排序，按照递减顺序Out[21]:
 D C B A2013-01-01 -1.135632 -1.509059 -0.282863 0.4691122013-01-02 -1.044236 0.119209 -0.173215 1.2121122013-01-03 1.071804 -0.494929 -2.104569 -0.8618492013-01-04 0.271860 -1.039575 -0.706771 0.7215552013-01-05 -1.087401 0.276232 0.567020 -0.4249722013-01-06 0.524988 -1.478427 0.113648 -0.673690

  • 通过值的大小排序

In [22]: df.sort_values(by='B')Out[22]:
 A B C D2013-01-03 -0.861849 -2.104569 -0.494929 1.0718042013-01-04 0.721555 -0.706771 -1.039575 0.2718602013-01-01 0.469112 -0.282863 -1.509059 -1.1356322013-01-02 1.212112 -0.173215 0.119209 -1.0442362013-01-06 -0.673690 0.113648 -1.478427 0.5249882013-01-05 -0.424972 0.567020 0.276232 -1.087401

选择数据框

  • 选取单列，结果是Series类型，等价于df.A

In [23]: df['A']Out[23]: 2013-01-01 0.4691122013-01-02 1.2121122013-01-03 -0.8618492013-01-04 0.7215552013-01-05 -0.4249722013-01-06 -0.673690Freq: D, Name: A, dtype: float64

  • 用[ ]来选取行，按行选取

In [24]: df[0:3] # 选取前三行Out[24]:
 A B C D2013-01-01 0.469112 -0.282863 -1.509059 -1.1356322013-01-02 1.212112 -0.173215 0.119209 -1.0442362013-01-03 -0.861849 -2.104569 -0.494929 1.071804In [25]: df['20130102':'20130104'] # 用序列值(行名)选取前三行Out[25]:
 A B C D2013-01-02 1.212112 -0.173215 0.119209 -1.0442362013-01-03 -0.861849 -2.104569 -0.494929 1.0718042013-01-04 0.721555 -0.706771 -1.039575 0.271860

  • df.loc来选取数据

In [26]: df.loc[dates[0]] # dates是一个时间序列，前面有定义Out[26]: A 0.469112B -0.282863C -1.509059D -1.135632Name: 2013-01-01 00:00:00, dtype: float64# 选取多列 In [27]: df.loc[:,['A','B']]Out[27]:
 A B2013-01-01 0.469112 -0.2828632013-01-02 1.212112 -0.1732152013-01-03 -0.861849 -2.1045692013-01-04 0.721555 -0.7067712013-01-05 -0.424972 0.5670202013-01-06 -0.673690 0.113648 # 指定选取的行数 In [28]: df.loc['20130102':'20130104',['A','B']]Out[28]:
 A B2013-01-02 1.212112 -0.1732152013-01-03 -0.861849 -2.1045692013-01-04 0.721555 -0.706771 # 指定一行就会返回 SeriesIn [29]: df.loc['20130102',['A','B']]Out[29]: A 1.212112B -0.173215Name: 2013-01-02 00:00:00, dtype: float64# 单个数据In [30]: df.loc[dates[0],'A']Out[30]: 0.46911229990718628# 单个数据(equiv to the prior method)In [31]: df.at[dates[0],'A']Out[31]: 0.46911229990718628

  • df.iloc

# 获取第3行数据In [32]: df.iloc[3]Out[32]: A 0.721555B -0.706771C -1.039575D 0.271860Name: 2013-01-04 00:00:00, dtype: float64# 选取一部分数据In [33]: df.iloc[3:5,0:2]Out[33]:
 A B2013-01-04 0.721555 -0.7067712013-01-05 -0.424972 0.567020# 选取指定(特定)数据框,1,2,4行,0,2列In [34]: df.iloc[[1,2,4],[0,2]]Out[34]:
 A C2013-01-02 1.212112 0.1192092013-01-03 -0.861849 -0.4949292013-01-05 -0.424972 0.276232# 只选取行In [35]: df.iloc[1:3,:]Out[35]:
 A B C D2013-01-02 1.212112 -0.173215 0.119209 -1.0442362013-01-03 -0.861849 -2.104569 -0.494929 1.071804# 只选取列In [36]: df.iloc[:,1:3]Out[36]:
 B C2013-01-01 -0.282863 -1.5090592013-01-02 -0.173215 0.1192092013-01-03 -2.104569 -0.4949292013-01-04 -0.706771 -1.0395752013-01-05 0.567020 0.2762322013-01-06 0.113648 -1.478427# 精确地获取指定的一个值In [37]: df.iloc[1,1]Out[37]: -0.17321464905330858# 准确的获取一个值In [38]: df.iat[1,1]Out[38]: -0.17321464905330858

筛选数据

  • 布尔方法选择数据

# 选取A列中大于0的数据框(包含了其他数据)In [39]: df[df.A > 0]Out[39]:
 A B C D2013-01-01 0.469112 -0.282863 -1.509059 -1.1356322013-01-02 1.212112 -0.173215 0.119209 -1.0442362013-01-04 0.721555 -0.706771 -1.039575 0.271860# 筛选整个数据框中大于0的数值，不大于的用NaN代替In [40]: df[df > 0]Out[40]:
 A B C D2013-01-01 0.469112 NaN NaN NaN2013-01-02 1.212112 NaN 0.119209 NaN2013-01-03 NaN NaN NaN 1.0718042013-01-04 0.721555 NaN NaN 0.2718602013-01-05 NaN 0.567020 0.276232 NaN2013-01-06 NaN 0.113648 NaN 0.524988

  • isin()

In [41]: df2 = df.copy()In [42]: df2['E'] = ['one', 'one','two','three','four','three']In [43]: df2Out[43]:
 A B C D E2013-01-01 0.469112 -0.282863 -1.509059 -1.135632 one2013-01-02 1.212112 -0.173215 0.119209 -1.044236 one2013-01-03 -0.861849 -2.104569 -0.494929 1.071804 two2013-01-04 0.721555 -0.706771 -1.039575 0.271860 three2013-01-05 -0.424972 0.567020 0.276232 -1.087401 four2013-01-06 -0.673690 0.113648 -1.478427 0.524988 threeIn [44]: df2[df2['E'].isin(['two','four'])]Out[44]:
 A B C D E2013-01-03 -0.861849 -2.104569 -0.494929 1.071804 two2013-01-05 -0.424972 0.567020 0.276232 -1.087401 four

赋值

其实只要掌握了提取数据的能力，对于赋值很容易理解:

In [45]: s1 = pd.Series([1,2,3,4,5,6], index=pd.date_range('20130102', periods=6))In [46]: s1Out[46]: 2013-01-02 12013-01-03 22013-01-04 32013-01-05 42013-01-06 52013-01-07 6Freq: D, dtype: int64In [47]: df['F'] = s1In [48]: df.at[dates[0],'A'] = 0In [49]: df.iat[0,1] = 0In [50]: df.loc[:,'D'] = np.array([5] * len(df))In [51]: dfOut[51]:
 A B C D F2013-01-01 0.000000 0.000000 -1.509059 5 NaN2013-01-02 1.212112 -0.173215 0.119209 5 1.02013-01-03 -0.861849 -2.104569 -0.494929 5 2.02013-01-04 0.721555 -0.706771 -1.039575 5 3.02013-01-05 -0.424972 0.567020 0.276232 5 4.02013-01-06 -0.673690 0.113648 -1.478427 5 5.0In [52]: df2 = df.copy()In [53]: df2[df2 > 0] = -df2In [54]: df2Out[54]:
 A B C D F2013-01-01 0.000000 0.000000 -1.509059 -5 NaN2013-01-02 -1.212112 -0.173215 -0.119209 -5 -1.02013-01-03 -0.861849 -2.104569 -0.494929 -5 -2.02013-01-04 -0.721555 -0.706771 -1.039575 -5 -3.02013-01-05 -0.424972 -0.567020 -0.276232 -5 -4.02013-01-06 -0.673690 -0.113648 -1.478427 -5 -5.0

缺失值

制作含有缺失值的数据框:

In [55]: df1 = df.reindex(index=dates[0:4], columns=list(df.columns) + ['E'])In [56]: df1.loc[dates[0]:dates[1],'E'] = 1In [57]: df1Out[57]:
 A B C D F E2013-01-01 0.000000 0.000000 -1.509059 5 NaN 1.02013-01-02 1.212112 -0.173215 0.119209 5 1.0 1.02013-01-03 -0.861849 -2.104569 -0.494929 5 2.0 NaN2013-01-04 0.721555 -0.706771 -1.039575 5 3.0 NaN

带有缺失数据的行，全部不要，只要不含缺失数据的行:

In [58]: df1.dropna(how='any')Out[58]:
 A B C D F E2013-01-02 1.212112 -0.173215 0.119209 5 1.0 1.0

填充缺失值:

In [59]: df1.fillna(value=5)Out[59]:
 A B C D F E2013-01-01 0.000000 0.000000 -1.509059 5 5.0 1.02013-01-02 1.212112 -0.173215 0.119209 5 1.0 1.02013-01-03 -0.861849 -2.104569 -0.494929 5 2.0 5.02013-01-04 0.721555 -0.706771 -1.039575 5 3.0 5.0

其他操作

  • 统计

# 按列计算均值In [61]: df.mean()Out[61]: A -0.004474B -0.383981C -0.687758D 5.000000F 3.000000dtype: float64# 按行计算均值In [62]: df.mean(1)Out[62]: 2013-01-01 0.8727352013-01-02 1.4316212013-01-03 0.7077312013-01-04 1.3950422013-01-05 1.8836562013-01-06 1.592306Freq: D, dtype: float64

  • 作用于每个元素

>>> df3 = pd.DataFrame(np.random.randn(3, 3))>>> df3
 0 1 20 -0.029638 1.081563 1.2803001 0.647747 0.831136 -1.5494812 0.513416 -0.884417 0.195343>>> df4 = df3.applymap(lambda x: '%.2f' % x)>>> df4
 0 1 20 -0.03 1.08 1.28 1 0.65 0.83 -1.552 0.51 -0.88 0.20

  • Apply

In [66]: df.apply(np.cumsum)Out[66]:
 A B C D F2013-01-01 0.000000 0.000000 -1.509059 5 NaN2013-01-02 1.212112 -0.173215 -1.389850 10 1.02013-01-03 0.350263 -2.277784 -1.884779 15 3.02013-01-04 1.071818 -2.984555 -2.924354 20 6.02013-01-05 0.646846 -2.417535 -2.648122 25 10.02013-01-06 -0.026844 -2.303886 -4.126549 30 15.0In [67]: df.apply(lambda x: x.max() - x.min()) # 计算每列Out[67]: A 2.073961B 2.671590C 1.785291D 0.000000F 4.000000dtype: float64

  • Histogramming

In [68]: s = pd.Series(np.random.randint(0, 7, size=10))In [69]: sOut[69]: 0 41 22 13 24 65 46 47 68 49 4dtype: int64In [70]: s.value_counts()Out[70]: 4 56 22 21 1dtype: int64

  • String Methods

In [71]: s = pd.Series(['A', 'B', 'C', 'Aaba', 'Baca', np.nan, 'CABA', 'dog', 'cat'])In [72]: s.str.lower()Out[72]: 0 a1 b2 c3 aaba4 baca5 NaN6 caba7 dog8 catdtype: object

Merge

  • Concat

In [1]: df1 = pd.DataFrame({'A': ['A0', 'A1', 'A2', 'A3'],
 ...: 'B': ['B0', 'B1', 'B2', 'B3'],
 ...: 'C': ['C0', 'C1', 'C2', 'C3'],
 ...: 'D': ['D0', 'D1', 'D2', 'D3']},
 ...: index=[0, 1, 2, 3])
 ...: In [2]: df2 = pd.DataFrame({'A': ['A4', 'A5', 'A6', 'A7'],
 ...: 'B': ['B4', 'B5', 'B6', 'B7'],
 ...: 'C': ['C4', 'C5', 'C6', 'C7'],
 ...: 'D': ['D4', 'D5', 'D6', 'D7']},
 ...: index=[4, 5, 6, 7])
 ...: In [3]: df3 = pd.DataFrame({'A': ['A8', 'A9', 'A10', 'A11'],
 ...: 'B': ['B8', 'B9', 'B10', 'B11'],
 ...: 'C': ['C8', 'C9', 'C10', 'C11'],
 ...: 'D': ['D8', 'D9', 'D10', 'D11']},
 ...: index=[8, 9, 10, 11])
 ...: In [4]: frames = [df1, df2, df3]In [5]: result = pd.concat(frames)

小白入门 Pandas基础

  • Join

In [78]: left = pd.DataFrame({'A': ['A0', 'A1', 'A2'],
 ....: 'B': ['B0', 'B1', 'B2']},
 ....: index=['K0', 'K1', 'K2'])
 ....: In [79]: right = pd.DataFrame({'C': ['C0', 'C2', 'C3'],
 ....: 'D': ['D0', 'D2', 'D3']},
 ....: index=['K0', 'K2', 'K3'])
 ....: In [80]: result = left.join(right)

小白入门 Pandas基础

In [81]: result = left.join(right, how='outer')

小白入门 Pandas基础

In [82]: result = left.join(right, how='inner')

小白入门 Pandas基础

In [85]: left = pd.DataFrame({'A': ['A0', 'A1', 'A2', 'A3'],
 ....: 'B': ['B0', 'B1', 'B2', 'B3'],
 ....: 'key': ['K0', 'K1', 'K0', 'K1']})
 ....: In [86]: right = pd.DataFrame({'C': ['C0', 'C1'],
 ....: 'D': ['D0', 'D1']},
 ....: index=['K0', 'K1'])
 ....: In [87]: result = left.join(right, on='key')

小白入门 Pandas基础

  • append

In [87]: df = pd.DataFrame(np.random.randn(8, 4), columns=['A','B','C','D'])In [88]: dfOut[88]:
 A B C D0 1.346061 1.511763 1.627081 -0.9905821 -0.441652 1.211526 0.268520 0.0245802 -1.577585 0.396823 -0.105381 -0.5325323 1.453749 1.208843 -0.080952 -0.2646104 -0.727965 -0.589346 0.339969 -0.6932055 -0.339355 0.593616 0.884345 1.5914316 0.141809 0.220390 0.435589 0.1924517 -0.096701 0.803351 1.715071 -0.708758In [89]: s = df.iloc[3] # 选取第三行In [90]: df.append(s, ignore_index=True) # 追加在数据框末尾Out[90]:
 A B C D0 1.346061 1.511763 1.627081 -0.9905821 -0.441652 1.211526 0.268520 0.0245802 -1.577585 0.396823 -0.105381 -0.5325323 1.453749 1.208843 -0.080952 -0.2646104 -0.727965 -0.589346 0.339969 -0.6932055 -0.339355 0.593616 0.884345 1.5914316 0.141809 0.220390 0.435589 0.1924517 -0.096701 0.803351 1.715071 -0.7087588 1.453749 1.208843 -0.080952 -0.264610

Grouping (分组)

Python里面的分组和R语言里面的data.table包类似，就是按照某几列分组，剩下的列就可以方便参与计算了。例如：

In [91]: df = pd.DataFrame({'A' : ['foo', 'bar', 'foo', 'bar',
 ....: 'foo', 'bar', 'foo', 'foo'],
 ....: 'B' : ['one', 'one', 'two', 'three',
 ....: 'two', 'two', 'one', 'three'],
 ....: 'C' : np.random.randn(8),
 ....: 'D' : np.random.randn(8)})
 ....: In [92]: dfOut[92]:
 A B C D0 foo one -1.202872 -0.0552241 bar one -1.814470 2.3959852 foo two 1.018601 1.5528253 bar three -0.595447 0.1665994 foo two 1.395433 0.0476095 bar two -0.392670 -0.1364736 foo one 0.007207 -0.5617577 foo three 1.928123 -1.623033

df是我们定义de数据框，现在我们要分别计算A列中，foo和bar各自的对应C列和D列的和。可以这样计算：

In [93]: df.groupby('A').sum()Out[93]:
 C DA bar -2.802588 2.42611foo 3.146492 -0.63958

但是，有时候我们还要计算以A&B两列进行分组后的C和D的各自和，这个时候就要进行排列了，因为foo对应的有one,two,three三个值，并且bar对应的也是相同的三个值。所以计算起来应该如下：

In [94]: df.groupby(['A','B']).sum()Out[94]: 
 C DA B bar one -1.814470 2.395985
 three -0.595447 0.166599
 two -0.392670 -0.136473foo one -1.195665 -0.616981
 three 1.928123 -1.623033
 two 2.414034 1.600434

相关推荐