the leap second may crash servers
1.what is leap second
设在巴黎的国际地球自转服务组织宣布,由于地球自转放缓的原因,今年全球将迎来史上第26次闰秒,时间在格林尼治时间6月30日午夜,科学家们将再度给钟表额外增加1秒钟时间,届时原子钟将会在23时59分59秒后停留一秒钟,让原子钟等上地球自转变慢的步伐。
that is:
case 'world time' is slower than atom time
how long a second in scale in this case
2.impacts
yesterday,the leap second was come here at 8'o clock moring,one related thing were occured:
high system cpu,user cpu for all processes lied in that node which run a 'ntp' service
and the kern.log
Jul 1 07:59:59 xxx-01 kernel: [14897543.632649] Clock: inserting leap second 23:59:60 UTC
3.solutions
a.restore time from hardware
first,my fellow checked that but found nothing!then a idea come to his mind:leap second!so he used this command to restore time from haredware
hwclock -s
then the processes in that node all restored to be normal.
b.restart processes
c.add a jump second (experiment,like google)
d.stop ntp at that time
4.root cause
a dead lock occurs when get hardware clock,see [2]
The kernel attempts to print a message when a leap second is inserted or removed. This can cause kernel versions prior to 2.6.29 to hang, due to a deadlock on xtime_lock. See http://lkml.org/lkml/2009/1/2/373 for a trace and explanation.
note:in my case,it only occured in os linux-3.2 which run with 'ntp' service,but for other os linux-3.5 which run with 'ntp' also lucky happened nothing!
ref:
[2]Leap second message can hang the kernel