您的位置: 首页  >  文章  >  利用python进入数据分析之usagov_bitly_data数据分析

利用python进入数据分析之usagov_bitly_data数据分析

分类: 文章 • 2024-11-04 18:11:52

获取数据中的时区进行计数统计

In [1]:
path = 'ch02/usagov_bitly_data2012-03-16-1331923249.txt'
In [2]:
open(path).readline()
Out[2]:
'{ "a": "Mozilla\\/5.0 (Windows NT 6.1; WOW64) AppleWebKit\\/535.11 (KHTML, like Gecko) Chrome\\/17.0.963.78 Safari\\/535.11", "c": "US", "nk": 1, "tz": "America\\/New_York", "gr": "MA", "g": "A6qOVH", "h": "wfLQtf", "l": "orofrog", "al": "en-US,en;q=0.8", "hh": "1.usa.gov", "r": "http:\\/\\/www.facebook.com\\/l\\/7AQEFzjSi\\/1.usa.gov\\/wfLQtf", "u": "http:\\/\\/www.ncbi.nlm.nih.gov\\/pubmed\\/22415991", "t": 1331923247, "hc": 1331822918, "cy": "Danvers", "ll": [ 42.576698, -70.954903 ] }\n'

数据转化json格式

In [7]:
import json
path = 'ch02/usagov_bitly_data2012-03-16-1331923249.txt'
records = [json.loads(line) for line in open(path)]  ## 列表推导式
In [4]:
records[0]
Out[4]:
{u'a': u'Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/535.11 (KHTML, like Gecko) Chrome/17.0.963.78 Safari/535.11',
 u'al': u'en-US,en;q=0.8',
 u'c': u'US',
 u'cy': u'Danvers',
 u'g': u'A6qOVH',
 u'gr': u'MA',
 u'h': u'wfLQtf',
 u'hc': 1331822918,
 u'hh': u'1.usa.gov',
 u'l': u'orofrog',
 u'll': [42.576698, -70.954903],
 u'nk': 1,
 u'r': u'http://www.facebook.com/l/7AQEFzjSi/1.usa.gov/wfLQtf',
 u't': 1331923247,
 u'tz': u'America/New_York',
 u'u': u'http://www.ncbi.nlm.nih.gov/pubmed/22415991'}
In [5]:
records[0]['tz']
Out[5]:
u'America/New_York'
In [6]:
print(records[0]['tz'])

America/New_York

用python对时区进行计数

In [11]:
time_zones = [rec['tz'] for rec in records if 'tz' in rec] # 注意有些记录没有时区
In [10]:
time_zones[:10]
Out[10]:
[u'America/New_York',
 u'America/Denver',
 u'America/New_York',
 u'America/Sao_Paulo',
 u'America/New_York',
 u'America/New_York',
 u'Europe/Warsaw',
 u'',
 u'',
 u'']

定义计数函数

In [13]:
def get_counts(sequence):
    counts = {}
    for x in sequence:
        if x in counts:
            counts[x] += 1
        else:
            counts[x] = 1
    return counts
In [15]:
from collections import defaultdict #用标准库函数实现计数

def get_counts2(sequence):
    counts = defaultdict(int) # values will initialize to 0
    for x in sequence:
        counts[x] += 1
    return counts
In [16]:
counts = get_counts(time_zones)
In [17]:
counts['America/New_York']
Out[17]:
1251
In [18]:
len(time_zones)
Out[18]:
3440
In [20]:
#  获取前10个字典的值
def top_counts(count_dict, n=10):
    value_key_pairs = [(count, tz) for tz, count in count_dict.items()]
    value_key_pairs.sort()
    return value_key_pairs[-n:]
In [21]:
top_counts(counts)
Out[21]:
[(33, u'America/Sao_Paulo'),
 (35, u'Europe/Madrid'),
 (36, u'Pacific/Honolulu'),
 (37, u'Asia/Tokyo'),
 (74, u'Europe/London'),
 (191, u'America/Denver'),
 (382, u'America/Los_Angeles'),
 (400, u'America/Chicago'),
 (521, u''),
 (1251, u'America/New_York')]
In [22]:
# 用标准库实现前N个字典的值
from collections import Counter
counts = Counter(time_zones)
counts.most_common(10)
Out[22]:
[(u'America/New_York', 1251),
 (u'', 521),
 (u'America/Chicago', 400),
 (u'America/Los_Angeles', 382),
 (u'America/Denver', 191),
 (u'Europe/London', 74),
 (u'Asia/Tokyo', 37),
 (u'Pacific/Honolulu', 36),
 (u'Europe/Madrid', 35),
 (u'America/Sao_Paulo', 33)]

用pandas对时区进行计数

In [24]:
%matplotlib inline
In [25]:
from __future__ import division
from numpy.random import randn
import numpy as np
import os
import matplotlib.pyplot as plt
import pandas as pd
plt.rc('figure', figsize=(10, 6))
np.set_printoptions(precision=4)
In [26]:
import json
path = 'ch02/usagov_bitly_data2012-03-16-1331923249.txt'
lines = open(path).readlines()
records = [json.loads(line) for line in lines]
In [28]:
from pandas import DataFrame, Series
import pandas as pd

frame = DataFrame(records)  # DataFrame是pandas中基础的数据结构
frame
Out[28]:
  _heartbeat_ a al c cy g gr h hc hh kw l ll nk r t tz u
0 NaN Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKi... en-US,en;q=0.8 US Danvers A6qOVH MA wfLQtf 1.331823e+09 1.usa.gov NaN orofrog [42.576698, -70.954903] 1.0 http://www.facebook.com/l/7AQEFzjSi/1.usa.gov/... 1.331923e+09 America/New_York http://www.ncbi.nlm.nih.gov/pubmed/22415991
1 NaN GoogleMaps/RochesterNY NaN US Provo mwszkS UT mwszkS 1.308262e+09 j.mp NaN bitly [40.218102, -111.613297] 0.0 http://www.AwareMap.com/ 1.331923e+09 America/Denver http://www.monroecounty.gov/etc/911/rss.php
2 NaN Mozilla/4.0 (compatible; MSIE 8.0; Windows NT ... en-US US Washington xxr3Qb DC xxr3Qb 1.331920e+09 1.usa.gov NaN bitly [38.9007, -77.043098] 1.0 http://t.co/03elZC4Q 1.331923e+09 America/New_York http://boxer.senate.gov/en/press/releases/0316...
3 NaN Mozilla/5.0 (Macintosh; Intel Mac OS X 10_6_8)... pt-br BR Braz zCaLwp 27 zUtuOu 1.331923e+09 1.usa.gov NaN alelex88 [-23.549999, -46.616699] 0.0 direct 1.331923e+09 America/Sao_Paulo http://apod.nasa.gov/apod/ap120312.html
4 NaN Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKi... en-US,en;q=0.8 US Shrewsbury 9b6kNl MA 9b6kNl 1.273672e+09 bit.ly NaN bitly [42.286499, -71.714699] 0.0 http://www.shrewsbury-ma.gov/selco/ 1.331923e+09 America/New_York http://www.shrewsbury-ma.gov/egov/gallery/1341...
5 NaN Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKi... en-US,en;q=0.8 US Shrewsbury axNK8c MA axNK8c 1.273673e+09 bit.ly NaN bitly [42.286499, -71.714699] 0.0 http://www.shrewsbury-ma.gov/selco/ 1.331923e+09 America/New_York http://www.shrewsbury-ma.gov/egov/gallery/1341...
6 NaN Mozilla/5.0 (Windows NT 5.1) AppleWebKit/535.1... pl-PL,pl;q=0.8,en-US;q=0.6,en;q=0.4 PL Luban wcndER 77 zkpJBR 1.331923e+09 1.usa.gov NaN bnjacobs [51.116699, 15.2833] 0.0 http://plus.url.google.com/url?sa=z&n=13319232... 1.331923e+09 Europe/Warsaw http://www.nasa.gov/mission_pages/nustar/main/...
7 NaN Mozilla/5.0 (Windows NT 6.1; rv:2.0.1) Gecko/2... bg,en-us;q=0.7,en;q=0.3 None NaN wcndER NaN zkpJBR 1.331923e+09 1.usa.gov NaN bnjacobs NaN 0.0 http://www.facebook.com/ 1.331923e+09   http://www.nasa.gov/mission_pages/nustar/main/...
8 NaN Opera/9.80 (X11; Linux zbov; U; en) Presto/2.1... en-US, en None NaN wcndER NaN zkpJBR 1.331923e+09 1.usa.gov NaN bnjacobs NaN 0.0 http://www.facebook.com/l.php?u=http%3A%2F%2F1... 1.331923e+09   http://www.nasa.gov/mission_pages/nustar/main/...
9 NaN Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKi... pt-BR,pt;q=0.8,en-US;q=0.6,en;q=0.4 None NaN zCaLwp NaN zUtuOu 1.331923e+09 1.usa.gov NaN alelex88 NaN 0.0 http://t.co/o1Pd0WeV 1.331923e+09   http://apod.nasa.gov/apod/ap120312.html
10 NaN Mozilla/5.0 (Windows NT 6.1; WOW64; rv:10.0.2)... en-us,en;q=0.5 US Seattle vNJS4H WA u0uD9q 1.319564e+09 1.usa.gov NaN o_4us71ccioa [47.5951, -122.332603] 1.0 direct 1.331923e+09 America/Los_Angeles https://www.nysdot.gov/rexdesign/design/commun...
11 NaN Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10.4... en-us,en;q=0.5 US Washington wG7OIH DC A0nRz4 1.331816e+09 1.usa.gov NaN darrellissa [38.937599, -77.092796] 0.0 http://t.co/ND7SoPyo 1.331923e+09 America/New_York http://oversight.house.gov/wp-content/uploads/...
12 NaN Mozilla/5.0 (Windows NT 6.1; WOW64; rv:10.0.2)... en-us,en;q=0.5 US Alexandria vNJS4H VA u0uD9q 1.319564e+09 1.usa.gov NaN o_4us71ccioa [38.790901, -77.094704] 1.0 direct 1.331923e+09 America/New_York https://www.nysdot.gov/rexdesign/design/commun...
13 1.331923e+09 NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN
14 NaN Mozilla/5.0 (Windows; U; Windows NT 6.1; en-US... en-us,en;q=0.5 US Marietta 2rOUYc GA 2rOUYc 1.255770e+09 1.usa.gov NaN bitly [33.953201, -84.5177] 1.0 direct 1.331923e+09 America/New_York http://toxtown.nlm.nih.gov/index.php
15 NaN Mozilla/5.0 (Windows NT 6.1) AppleWebKit/535.1... zh-TW,zh;q=0.8,en-US;q=0.6,en;q=0.4 HK Central District nQvgJp 00 rtrrth 1.317318e+09 j.mp NaN walkeryuen [22.2833, 114.150002] 1.0 http://forum2.hkgolden.com/view.aspx?type=BW&m... 1.331923e+09 Asia/Hong_Kong http://www.ssd.noaa.gov/PS/TROP/TCFP/data/curr...
16 NaN Mozilla/5.0 (Windows NT 6.1) AppleWebKit/535.1... zh-TW,zh;q=0.8,en-US;q=0.6,en;q=0.4 HK Central District XdUNr 00 qWkgbq 1.317318e+09 j.mp NaN walkeryuen [22.2833, 114.150002] 1.0 http://forum2.hkgolden.com/view.aspx?type=BW&m... 1.331923e+09 Asia/Hong_Kong http://www.usno.navy.mil/NOOC/nmfc-ph/RSS/jtwc...
17 NaN Mozilla/5.0 (Macintosh; Intel Mac OS X 10.5; r... en-us,en;q=0.5 US Buckfield zH1BFf ME x3jOIv 1.331840e+09 1.usa.gov NaN andyzieminski [44.299702, -70.369797] 0.0 http://t.co/6Cx4ROLs 1.331923e+09 America/New_York http://www.usda.gov/wps/portal/usda/usdahome?c...
18 NaN GoogleMaps/RochesterNY NaN US Provo mwszkS UT mwszkS 1.308262e+09 1.usa.gov NaN bitly [40.218102, -111.613297] 0.0 http://www.AwareMap.com/ 1.331923e+09 America/Denver http://www.monroecounty.gov/etc/911/rss.php
19 NaN Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKi... it-IT,it;q=0.8,en-US;q=0.6,en;q=0.4 IT Venice wcndER 20 zkpJBR 1.331923e+09 1.usa.gov NaN bnjacobs [45.438599, 12.3267] 0.0 http://www.facebook.com/ 1.331923e+09 Europe/Rome http://www.nasa.gov/mission_pages/nustar/main/...
20 NaN Mozilla/5.0 (compatible; MSIE 9.0; Windows NT ... es-ES ES Alcal zQ95Hi 51 ytZYWR 1.331671e+09 bitly.com NaN jplnews [37.516701, -5.9833] 0.0 http://www.facebook.com/ 1.331923e+09 Africa/Ceuta http://voyager.jpl.nasa.gov/imagesvideo/uranus...
21 NaN Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10.6... en-us,en;q=0.5 US Davidsonville wcndER MD zkpJBR 1.331923e+09 1.usa.gov NaN bnjacobs [38.939201, -76.635002] 0.0 http://www.facebook.com/ 1.331923e+09 America/New_York http://www.nasa.gov/mission_pages/nustar/main/...
22 NaN Mozilla/4.0 (compatible; MSIE 8.0; Windows NT ... en-us US Hockessin y3ZImz DE y3ZImz 1.331064e+09 1.usa.gov NaN bitly [39.785, -75.682297] 0.0 direct 1.331923e+09 America/New_York http://portal.hud.gov/hudportal/documents/hudd...
23 NaN Mozilla/5.0 (Macintosh; Intel Mac OS X 10_7_3)... en-us US Lititz wWiOiD PA wWiOiD 1.330218e+09 1.usa.gov NaN bitly [40.174999, -76.3078] 0.0 http://www.facebook.com/l.php?u=http%3A%2F%2F1... 1.331923e+09 America/New_York http://www.tricare.mil/mybenefit/ProfileFilter...
24 NaN Mozilla/5.0 (Windows; U; Windows NT 5.1; es-ES... es-es,es;q=0.8,en-us;q=0.5,en;q=0.3 ES Bilbao wcndER 59 zkpJBR 1.331923e+09 1.usa.gov NaN bnjacobs [43.25, -2.9667] 0.0 http://www.facebook.com/ 1.331923e+09 Europe/Madrid http://www.nasa.gov/mission_pages/nustar/main/...
25 NaN Mozilla/5.0 (Windows NT 6.1) AppleWebKit/535.1... en-GB,en;q=0.8,en-US;q=0.6,en-AU;q=0.4 MY Kuala Lumpur wcndER 14 zkpJBR 1.331923e+09 1.usa.gov NaN bnjacobs [3.1667, 101.699997] 0.0 http://www.facebook.com/ 1.331923e+09 Asia/Kuala_Lumpur http://www.nasa.gov/mission_pages/nustar/main/...
26 NaN Mozilla/5.0 (Windows NT 6.1) AppleWebKit/535.1... ro-RO,ro;q=0.8,en-US;q=0.6,en;q=0.4 CY Nicosia wcndER 04 zkpJBR 1.331923e+09 1.usa.gov NaN bnjacobs [35.166698, 33.366699] 0.0 http://www.facebook.com/?ref=tn_tnmn 1.331923e+09 Asia/Nicosia http://www.nasa.gov/mission_pages/nustar/main/...
27 NaN Mozilla/5.0 (Macintosh; Intel Mac OS X 10_6_8)... en-US,en;q=0.8 BR SPaulo zCaLwp 27 zUtuOu 1.331923e+09 1.usa.gov NaN alelex88 [-23.5333, -46.616699] 0.0 direct 1.331923e+09 America/Sao_Paulo http://apod.nasa.gov/apod/ap120312.html
28 NaN Mozilla/5.0 (iPad; CPU OS 5_0_1 like Mac OS X)... en-us None NaN vNJS4H NaN u0uD9q 1.319564e+09 1.usa.gov NaN o_4us71ccioa NaN 0.0 direct 1.331923e+09   https://www.nysdot.gov/rexdesign/design/commun...
29 NaN Mozilla/5.0 (iPad; U; CPU OS 3_2 like Mac OS X... en-us None NaN FPX0IM NaN FPX0IL 1.331923e+09 1.usa.gov NaN twittershare NaN 1.0 http://t.co/5xlp0B34 1.331923e+09   http://www.ed.gov/news/media-advisories/us-dep...
... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ...
3530 NaN Mozilla/5.0 (Windows NT 6.0) AppleWebKit/535.1... en-US,en;q=0.8 US San Francisco xVZg4P CA wqUkTo 1.331908e+09 go.nasa.gov NaN nasatwitter [37.7645, -122.429398] 0.0 http://www.facebook.com/l.php?u=http%3A%2F%2Fg... 1.331927e+09 America/Los_Angeles http://www.nasa.gov/multimedia/imagegallery/im...
3531 NaN Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10_6... en-US None NaN wcndER NaN zkpJBR 1.331923e+09 1.usa.gov NaN bnjacobs NaN 0.0 direct 1.331927e+09   http://www.nasa.gov/mission_pages/nustar/main/...
3532 NaN Mozilla/5.0 (Windows NT 6.1; WOW64; rv:10.0.2)... en-us,en;q=0.5 US Washington Au3aUS DC A9ct6C 1.331926e+09 1.usa.gov NaN ncsha [38.904202, -77.031998] 1.0 http://www.ncsha.org/ 1.331927e+09 America/New_York http://portal.hud.gov/hudportal/HUD?src=/press...
3533 NaN Mozilla/5.0 (iPad; CPU OS 5_1 like Mac OS X) A... en-us US Jacksonville b2UtUJ FL ieCdgH 1.301393e+09 go.nasa.gov NaN nasatwitter [30.279301, -81.585098] 1.0 direct 1.331927e+09 America/New_York http://apod.nasa.gov/apod/
3534 NaN Mozilla/5.0 (Macintosh; Intel Mac OS X 10_6_8)... en-us US Frisco vNJS4H TX u0uD9q 1.319564e+09 1.usa.gov NaN o_4us71ccioa [33.149899, -96.855499] 1.0 direct 1.331927e+09 America/Chicago https://www.nysdot.gov/rexdesign/design/commun...
3535 NaN Mozilla/5.0 (Windows NT 5.1; rv:10.0.2) Gecko/... en-us US Houston zIgLx8 TX yrPaLt 1.331903e+09 aash.to NaN aashto [29.775499, -95.415199] 1.0 direct 1.331927e+09 America/Chicago http://ntl.bts.gov/lib/44000/44300/44374/FHWA-...
3536 NaN Mozilla/5.0 (BlackBerry; U; BlackBerry 9800; e... en-US,en;q=0.5 None NaN xIcyim NaN yG1TTf 1.331728e+09 go.nasa.gov NaN nasatwitter NaN 0.0 http://t.co/g1VKE8zS 1.331927e+09   http://www.nasa.gov/mission_pages/hurricanes/a...
3537 NaN Mozilla/5.0 (Windows NT 6.1; WOW64; rv:10.0.2)... es-es,es;q=0.8,en-us;q=0.5,en;q=0.3 HN Tegucigalpa zCaLwp 08 w63FZW 1.331547e+09 1.usa.gov NaN bufferapp [14.1, -87.216698] 0.0 http://t.co/A8TJyibE 1.331927e+09 America/Tegucigalpa http://apod.nasa.gov/apod/ap120312.html
3538 NaN Mozilla/5.0 (iPhone; CPU iPhone OS 5_1 like Ma... en-us US Los Angeles qMac9k CA qds1Ge 1.310474e+09 1.usa.gov NaN healthypeople [34.041599, -118.298798] 0.0 direct 1.331927e+09 America/Los_Angeles http://healthypeople.gov/2020/connect/webinars...
3539 NaN Mozilla/5.0 (compatible; Fedora Core 3) FC3 KDE NaN US Bellevue zu2M5o WA zDhdro 1.331586e+09 bit.ly NaN glimtwin [47.615398, -122.210297] 0.0 direct 1.331927e+09 America/Los_Angeles http://www.federalreserve.gov/newsevents/press...
3540 NaN Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKi... en-US,en;q=0.8 US Payson wcndER UT zkpJBR 1.331923e+09 1.usa.gov NaN bnjacobs [40.014198, -111.738899] 0.0 http://www.facebook.com/l.php?u=http%3A%2F%2F1... 1.331927e+09 America/Denver http://www.nasa.gov/mission_pages/nustar/main/...
3541 NaN Mozilla/5.0 (X11; U; OpenVMS AlphaServer_ES40;... NaN US Bellevue zu2M5o WA zDhdro 1.331586e+09 1.usa.gov NaN glimtwin [47.615398, -122.210297] 0.0 direct 1.331927e+09 America/Los_Angeles http://www.federalreserve.gov/newsevents/press...
3542 NaN Mozilla/5.0 (compatible; MSIE 9.0; Windows NT ... en-us US Pittsburg y3reI1 CA y3reI1 1.331926e+09 1.usa.gov NaN bitly [38.0051, -121.838699] 0.0 http://www.facebook.com/l.php?u=http%3A%2F%2F1... 1.331927e+09 America/Los_Angeles http://www.sba.gov/community/blogs/community-b...
3543 1.331927e+09 NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN
3544 NaN Mozilla/5.0 (Windows NT 6.1; WOW64; rv:5.0.1) ... en-us,en;q=0.5 US Wentzville vNJS4H MO u0uD9q 1.319564e+09 1.usa.gov NaN o_4us71ccioa [38.790001, -90.854897] 1.0 direct 1.331927e+09 America/Chicago https://www.nysdot.gov/rexdesign/design/commun...
3545 NaN Mozilla/5.0 (Windows NT 6.1; WOW64; rv:10.0.2)... en-us,en;q=0.5 US Saint Charles vNJS4H IL u0uD9q 1.319564e+09 1.usa.gov NaN o_4us71ccioa [41.9352, -88.290901] 1.0 direct 1.331927e+09 America/Chicago https://www.nysdot.gov/rexdesign/design/commun...
3546 NaN Mozilla/5.0 (iPhone; CPU iPhone OS 5_1 like Ma... en-us US Los Angeles qMac9k CA qds1Ge 1.310474e+09 1.usa.gov NaN healthypeople [34.041599, -118.298798] 1.0 direct 1.331927e+09 America/Los_Angeles http://healthypeople.gov/2020/connect/webinars...
3547 NaN Mozilla/5.0 (Macintosh; Intel Mac OS X 10_6_8)... en-us US Silver Spring y0jYkg MD y0jYkg 1.331852e+09 1.usa.gov NaN bitly [39.052101, -77.014999] 1.0 direct 1.331927e+09 America/New_York http://www.epa.gov/otaq/regs/fuels/additive/e1...
3548 NaN Mozilla/5.0 (iPhone; CPU iPhone OS 5_1 like Ma... en-us US Mcgehee y5rMac AR xANY6O 1.331916e+09 1.usa.gov NaN twitterfeed [33.628399, -91.356903] 1.0 https://twitter.com/fdarecalls/status/18069759... 1.331927e+09 America/Chicago http://www.fda.gov/Safety/Recalls/ucm296326.htm
3549 NaN Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKi... sv-SE,sv;q=0.8,en-US;q=0.6,en;q=0.4 SE Sollefte eH8wu 24 7dtjei 1.260316e+09 1.usa.gov NaN tweetdeckapi [63.166698, 17.266701] 1.0 direct 1.331927e+09 Europe/Stockholm http://www.nasa.gov/mission_pages/WISE/main/in...
3550 NaN Mozilla/4.0 (compatible; MSIE 8.0; Windows NT ... en-us US Conshohocken A00b72 PA yGSwzn 1.331918e+09 1.usa.gov NaN addthis [40.0798, -75.2855] 0.0 http://www.linkedin.com/home?trk=hb_tab_home_top 1.331927e+09 America/New_York http://www.nlm.nih.gov/medlineplus/news/fullst...
3551 NaN Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKi... en-US,en;q=0.8 None NaN wcndER NaN zkpJBR 1.331923e+09 1.usa.gov NaN bnjacobs NaN 0.0 http://plus.url.google.com/url?sa=z&n=13319268... 1.331927e+09   http://www.nasa.gov/mission_pages/nustar/main/...
3552 NaN Mozilla/5.0 (Windows; U; Windows NT 6.1; en-US... NaN US Decatur rqgJuE AL xcz8vt 1.331227e+09 1.usa.gov NaN bootsnall [34.572701, -86.940598] 0.0 direct 1.331927e+09 America/Chicago http://travel.state.gov/passport/passport_5535...
3553 NaN Mozilla/4.0 (compatible; MSIE 7.0; Windows NT ... en-us US Shrewsbury 9b6kNl MA 9b6kNl 1.273672e+09 bit.ly NaN bitly [42.286499, -71.714699] 0.0 http://www.shrewsbury-ma.gov/selco/ 1.331927e+09 America/New_York http://www.shrewsbury-ma.gov/egov/gallery/1341...
3554 NaN Mozilla/4.0 (compatible; MSIE 7.0; Windows NT ... en-us US Shrewsbury axNK8c MA axNK8c 1.273673e+09 bit.ly NaN bitly [42.286499, -71.714699] 0.0 http://www.shrewsbury-ma.gov/selco/ 1.331927e+09 America/New_York http://www.shrewsbury-ma.gov/egov/gallery/1341...
3555 NaN Mozilla/4.0 (compatible; MSIE 9.0; Windows NT ... en US Paramus e5SvKE NJ fqPSr9 1.301298e+09 1.usa.gov NaN tweetdeckapi [40.9445, -74.07] 1.0 direct 1.331927e+09 America/New_York http://www.fda.gov/AdvisoryCommittees/Committe...
3556 NaN Mozilla/5.0 (Windows NT 5.1) AppleWebKit/535.1... en-US,en;q=0.8 US Oklahoma City jQLtP4 OK jQLtP4 1.307530e+09 1.usa.gov NaN bitly [35.4715, -97.518997] 0.0 http://www.facebook.com/l.php?u=http%3A%2F%2F1... 1.331927e+09 America/Chicago http://www.okc.gov/PublicNotificationSystem/Fo...
3557 NaN GoogleMaps/RochesterNY NaN US Provo mwszkS UT mwszkS 1.308262e+09 j.mp NaN bitly [40.218102, -111.613297] 0.0 http://www.AwareMap.com/ 1.331927e+09 America/Denver http://www.monroecounty.gov/etc/911/rss.php
3558 NaN GoogleProducer NaN US Mountain View zjtI4X CA zjtI4X 1.327529e+09 1.usa.gov NaN bitly [37.419201, -122.057404] 0.0 direct 1.331927e+09 America/Los_Angeles http://www.ahrq.gov/qual/qitoolkit/
3559 NaN Mozilla/4.0 (compatible; MSIE 8.0; Windows NT ... en-US US Mc Lean qxKrTK VA qxKrTK 1.312898e+09 1.usa.gov NaN bitly [38.935799, -77.162102] 0.0 http://t.co/OEEEvwjU 1.331927e+09 America/New_York http://herndon-va.gov/Content/public_safety/Pu...

3560 rows × 18 columns

In [29]:
frame['tz'][:10]
Out[29]:
0     America/New_York
1       America/Denver
2     America/New_York
3    America/Sao_Paulo
4     America/New_York
5     America/New_York
6        Europe/Warsaw
7                     
8                     
9                     
Name: tz, dtype: object
In [30]:
tz_counts = frame['tz'].value_counts()
tz_counts[:10]
Out[30]:
America/New_York       1251
                        521
America/Chicago         400
America/Los_Angeles     382
America/Denver          191
Europe/London            74
Asia/Tokyo               37
Pacific/Honolulu         36
Europe/Madrid            35
America/Sao_Paulo        33
Name: tz, dtype: int64
In [31]:
clean_tz = frame['tz'].fillna('Missing')# 替换缺失值
clean_tz[clean_tz == ''] = 'Unknown'# 替换空字符串
tz_counts = clean_tz.value_counts()
tz_counts[:10]
Out[31]:
America/New_York       1251
Unknown                 521
America/Chicago         400
America/Los_Angeles     382
America/Denver          191
Missing                 120
Europe/London            74
Asia/Tokyo               37
Pacific/Honolulu         36
Europe/Madrid            35
Name: tz, dtype: int64
In [32]:
plt.figure(figsize=(10, 4))
Out[32]:
<matplotlib.figure.Figure at 0x44bc230>
<matplotlib.figure.Figure at 0x44bc230>
In [33]:
tz_counts[:10].plot(kind='barh', rot=0)
Out[33]:
<matplotlib.axes._subplots.AxesSubplot at 0x44bcbd0>
利用python进入数据分析之usagov_bitly_data数据分析
利用python进入数据分析之usagov_bitly_data数据分析

按照WINDOWS和非WINDOWS统计最常出现的时区

In [37]:
frame['a'][1]   # 看下浏览器相关信息
Out[37]:
u'GoogleMaps/RochesterNY'
In [36]:
frame['a'][50]
Out[36]:
u'Mozilla/5.0 (Windows NT 5.1; rv:10.0.2) Gecko/20100101 Firefox/10.0.2'
In [39]:
results = Series([x.split()[0] for x in frame.a.dropna()])
results[:5]
Out[39]:
0               Mozilla/5.0
1    GoogleMaps/RochesterNY
2               Mozilla/4.0
3               Mozilla/5.0
4               Mozilla/5.0
dtype: object
In [40]:
results.value_counts()[:8]
Out[40]:
Mozilla/5.0                 2594
Mozilla/4.0                  601
GoogleMaps/RochesterNY       121
Opera/9.80                    34
TEST_INTERNET_AGENT           24
GoogleProducer                21
Mozilla/6.0                    5
BlackBerry8520/5.0.0.681       4
dtype: int64
In [41]:
cframe = frame[frame.a.notnull()]# 移除空值
In [42]:
operating_system = np.where(cframe['a'].str.contains('Windows'),
                            'Windows', 'Not Windows')  #跟进是否包含“Windows”来判断
operating_system[:5]
Out[42]:
array(['Windows', 'Not Windows', 'Windows', 'Not Windows', 'Windows'], 
      dtype='|S11')
In [43]:
by_tz_os = cframe.groupby(['tz', operating_system])
In [44]:
agg_counts = by_tz_os.size().unstack().fillna(0)
agg_counts[:10]
Out[44]:
  Not Windows Windows
tz    
  245.0 276.0
Africa/Cairo 0.0 3.0
Africa/Casablanca 0.0 1.0
Africa/Ceuta 0.0 2.0
Africa/Johannesburg 0.0 1.0
Africa/Lusaka 0.0 1.0
America/Anchorage 4.0 1.0
America/Argentina/Buenos_Aires 1.0 0.0
America/Argentina/Cordoba 0.0 1.0
America/Argentina/Mendoza 0.0 1.0
In [45]:
# Use to sort in ascending order
indexer = agg_counts.sum(1).argsort()
indexer[:10]
Out[45]:
tz
                                  24
Africa/Cairo                      20
Africa/Casablanca                 21
Africa/Ceuta                      92
Africa/Johannesburg               87
Africa/Lusaka                     53
America/Anchorage                 54
America/Argentina/Buenos_Aires    57
America/Argentina/Cordoba         26
America/Argentina/Mendoza         55
dtype: int64
In [46]:
count_subset = agg_counts.take(indexer)[-10:]
count_subset
Out[46]:
  Not Windows Windows
tz    
America/Sao_Paulo 13.0 20.0
Europe/Madrid 16.0 19.0
Pacific/Honolulu 0.0 36.0
Asia/Tokyo 2.0 35.0
Europe/London 43.0 31.0
America/Denver 132.0 59.0
America/Los_Angeles 130.0 252.0
America/Chicago 115.0 285.0
  245.0 276.0
America/New_York 339.0 912.0
In [47]:
plt.figure()
Out[47]:
<matplotlib.figure.Figure at 0x5e69310>
<matplotlib.figure.Figure at 0x5e69310>
In [48]:
count_subset.plot(kind='barh', stacked=True)
Out[48]:
<matplotlib.axes._subplots.AxesSubplot at 0x5e46eb0>
利用python进入数据分析之usagov_bitly_data数据分析
利用python进入数据分析之usagov_bitly_data数据分析
In [49]:
plt.figure()
Out[49]:
<matplotlib.figure.Figure at 0x5c3a2b0>
<matplotlib.figure.Figure at 0x5c3a2b0>
In [50]:
normed_subset = count_subset.div(count_subset.sum(1), axis=0)
normed_subset.plot(kind='barh', stacked=True)
Out[50]:
<matplotlib.axes._subplots.AxesSubplot at 0x5b28d50>
利用python进入数据分析之usagov_bitly_data数据分析
利用python进入数据分析之usagov_bitly_data数据分析

相关推荐