一些简单常用的shell、awk、sed实例

发表于 | 更新于 | 分类于 ， | 评论数： | 阅读次数：
本文字数： 8.4k | 阅读时长 ≈ 8 分钟

摘要

原创

本文将记录一些简单实用的shell命令、脚本以及一些命令组合。

用ping来做链路监控

1
2
3
4
5
cat ping.sh 
ping xxx.xxx.xxx.xxx  -i 2 | awk '{ print $0"\t" strftime("%D-%H:%M:%S",systime()) } ' 

类似ping 或者tail -f 这种动态输出做过过滤之后的结果因为有buffer 所以不能直接重定向到某个文件中，可以用以下方法去除buffer
unbuffer sh ping.sh  >> ping.log &

多个日志文件合并处理以及日志分析

1
2
3
4
5
6
7
8
9
10
11
12
13
14
# 可以用cat多个文件的方式进行处理
cat log1 log2 log3 |sort -k 4 -t " "

# 可以用sort 讲多个文件合并之后再处理
sort -m log1 log2 -o log_all

# 统计日志不通时间段的访问量

grep ' 15:[0-2][0-9]' haproxy.log | awk '{print $3}' | sed 's/:[0-9]*//' | awk '{++S[$NF]} END {for (a in S) print a, S[a]}'

grep '2015:15:[0-2][0-9]' haproxy.log | awk -F':' '{print $7}' | awk '{++S[$NF]} END {for (a in S) print a, S[a]}'

# 统计用户访问量
grep "android&version_number=2.2.0" 4.log | grep member_id | sed 's/.*member_id%22%3A%22//' | awk -F'%' '{print $1}' | sort -n |uniq -c |sort | wc -l

端口存活、端口扫描

1
2
3
4
5
6
7
8
9
10
11
12
# 可以使用nping来检查某个ip的某个端口的存活状态
yum install nmap
nping 172.18.4.233 --tcp -p 22

# 用zmap来检查某个网段哪些开启了某个端口
zmap  -p 22 10.0.0.0/8 -o b.txt

# 用nc俩检查某个ip开启了哪些端口
nc -z -v -n 192.168.9.2 20-25

# ping整个网段
fping 10.0.0.0/24 -g -a

查看本机开放端口、统计tcp连接数

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
# 查看本机开放端口
netstat -antp 
ss -ant

# 查看本机tcp连接
netstat -antulp
ss -ntlp
ss -ntrlp 

# 统计tcp连接数
netstat -n | awk '/^tcp/ {++S[$NF]} END {for(a in S) print a, S[a]}’
ss -ant | awk '{++S[$1]} END {for(a in S) print a, S[a]}’

LAST_ACK 1
SYN_RECV 14
ESTABLISHED 79
FIN_WAIT1 28
FIN_WAIT2 3
CLOSING 5
TIME_WAIT 1669

#状态：描述
CLOSED：无连接是活动的或正在进行
LISTEN：服务器在等待进入呼叫
SYN_RECV：一个连接请求已经到达，等待确认
SYN_SENT：应用已经开始，打开一个连接
ESTABLISHED：正常数据传输状态
FIN_WAIT1：应用说它已经完成
FIN_WAIT2：另一边已同意释放
ITMED_WAIT：等待所有分组死掉
CLOSING：两边同时尝试关闭
TIME_WAIT：另一边已初始化一个释放
LAST_ACK：等待所有分组死掉


# 其他相关命令
netstat -atpn | perl -nE 'BEGIN{our %stat;};chomp;next if (not /^tcp.*\s+([A-Z]+[A-Z0-9_]+)\s+.*/);if (not exists $stat{$1}){$stat{$1}=1;} else {$stat{$1}++;}END{foreach $k (sort keys %stat) {say "$k: $stat{$k}";}}’

ss -ant | grep ESTAB | awk '{print $5}' | sed 's/:.*//' | awk '{++S[$1]} END {for(a in S) print a, S[a]}'

ss -tan 'sport = :80' | awk '{print $(NF)" "$(NF-1)}' | sed 's/:[^ ]*//g' | sort | uniq -c

查看一个进程启动时间

1
ps -p PID -o lstart

sed

1
2
3
4
5
6
7
8
# sed修改主机名
sed "s/WEBMB/WEB`ifconfig | grep "10.0.0" |awk '{print $2}' | sed 's/.*\.//'`/" -i /etc/sysconfig/network

sed "s/WEBMB/WEB`ifconfig | grep "10.0.0" |awk '{print $2}' | sed 's/.*\.//'`/" /etc/hosts -i

hostname WEB`ifconfig | grep "10.0.0" |awk '{print $2}' | sed 's/.*\.//'`
sed -i "/HOST/s/=.*/`hostname`/" /etc/sysconfig/network
hostname `hostname`.lifec.me

tcpdump抓包

tcpdump抓包会有单独的文章来做记录

1
2
3
4
5
tcpdump -i eth0 

tcpdump -i eth0 tcp and tcp[13]==2 -nn and port 6379

tcpdump -i eth1 -tnn dst port 4730 -c 1000 | awk -F"." '{print $1"."$2"."$3"."$4}' | sort | uniq -c | sort -nr |head -20

反转

1
echo "abcdef" | sed '/\n/!G;s/\(.\)\(.*\n\)/&\2\1/;//D;s/.//'

一个简单的awk问题

1
2
3
4
5
# 题目 要求将下列内容中第一列是a=${num}的 num+1
a=1 x=1
b=2 v=1
a=4 t=121
c=6 a=12
1
2
3
4
5
awk '$1~/a=[0-9]/{split($1,a,"=");a[2]+=1;$1=a[1]"="a[2]}1'


# 另一道题的 题目没了
awk -F'<td>|</td>' '{if(NF%2==0){S[NR-1]=S[NR-1]"\t"$3}else{S[NR]=$1}}END{for(a in S){print S[a]}}’ a

awk处理文本

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
# ques 1
# cat a.ini
[zk]
10.100.0.1
10.100.0.2
[es]
10.0.5.5

awk '{if($1~/\[(.*)\]/){split($1,a,"[");split(a[2],b,"]");t=b[1];c[b[1]]=""}else{t=b[1];c[b[1]]=c[b[1]]$1","}}END{for(i in c){print i,c[i]}}' a.ini
zk 10.100.0.1,10.100.0.2,
es 10.0.5.5,

################################

# cat a.py
#!/usr/bin/env python
# -*- coding: utf-8 -*-

from __future__ import print_function
import re

def getConf(file_name):
    myjson = {}
    with open(file_name,"r") as cfg:
        for i in cfg:
            i = i.strip()
            matchObj = re.match(r'\[(.*)\]',i,re.M|re.I)
            if matchObj:
                keys = matchObj.group(1)
                myjson[keys] = []
                continue
            myjson[keys].append(i)
    return myjson

def main():
    file_name = ["a.ini",]
    for files in file_name:
        print("## file_name : %s" %files)
        config = getConf(files)
        #print(config)
        for k in config:
            print(k + ":" + ",".join(config[k]))

if __name__ == "__main__":
    main()

awk 行转列

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
cat a
a b c d e
1 2 3 4 5
! @ # $ %

awk '{for(i=1;i<=NF;i++)a[NR,i]=$i}END{for(j=1;j<=NF;j++)for(k=1;k<=NR;k++)print a[k,j]}' a
a
1
!
b
2
@
c
3
#
d
4
$
e
5
%
1
awk -vc=10 -vd=20 '/3/{e++;f=1}{if(e==1&&f==1){print "if port"c>FILENAME}else if(e==2&&f==1){print "if port"d>FILENAME}else{print $0>FILENAME};f=0}' a

查看etc目录inode使用量

1
find /etc/ -exec stat -c "%n %i" {} \;|awk -F "[/ ]" '{if(! a[$1-$NF]++) l[$1]++}END{for (i in l) print i,l[i]}'

iperf测试网速

1
2
3
4
# server端开启
iperf -s
# client端打流量
iperf -c 172.16.18.131 -i 1 -p5001

一个awk数据处理

源文件

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
2 192.168.174.2 "GET /group1/M00/00/00/wKiuj1qWeyGAJWsuAAAACXh6ifo038.txt HTTP/1.1"
1 192.168.174.2 "GET /favicon.ico HTTP/1.1"
1 192.168.174.21 "GET /group1/M00/00/00/wKiuj1qWeyGAJWsuAAAACXh6ifo038.txt HTTP/1.1"
1 192.168.174.1 "GET /group1/M00/00/00/wKiuj1qWeyGAJWsuAAAACXh6ifo038.txt HTTP/1.1"
1 192.168.174.1 "GET /favicon.ico HTTP/1.1"
1 192.168.174.19 "GET /group1/M00/00/00/wKiuj1qWeyGAJWsuAAAACXh6ifo038.txt HTTP/1.1"
1 192.168.174.19 "GET /favicon.ico HTTP/1.1"
1 192.168.174.17 "GET /favicon.ico HTTP/1.1"
1 192.168.174.16 "GET /favicon.ico HTTP/1.1"
1 192.168.174.15 "GET /group1/M00/00/00/wKiuj1qWeyGAJWsuAAAACXh6ifo038.txt HTTP/1.1"
1 192.168.174.14 "GET /favicon.ico HTTP/1.1"
1 192.168.174.13 "GET /group1/M00/00/00/wKiuj1qWeyGAJWsuAAAACXh6ifo038.txt HTTP/1.1"
1 192.168.174.13 "GET /favicon.ico HTTP/1.1"
1 192.168.174.12 "GET /group1/M00/00/00/wKiuj1qWeyGAJWsuAAAACXh6ifo038.txt HTTP/1.1"
1 192.168.174.10 "GET /favicon.ico HTTP/1.1"

目标

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
2  "GET /group1/M00/00/00/wKiuj1qWeyGAJWsuAAAACXh6ifo038.txt HTTP/1.1"
   "GET /favicon.ico HTTP/1.1"
1  "GET /group1/M00/00/00/wKiuj1qWeyGAJWsuAAAACXh6ifo038.txt HTTP/1.1"
1  "GET /group1/M00/00/00/wKiuj1qWeyGAJWsuAAAACXh6ifo038.txt HTTP/1.1"
   "GET /favicon.ico HTTP/1.1"
1  "GET /group1/M00/00/00/wKiuj1qWeyGAJWsuAAAACXh6ifo038.txt HTTP/1.1"
   "GET /favicon.ico HTTP/1.1"
1  "GET /favicon.ico HTTP/1.1"
1  "GET /favicon.ico HTTP/1.1"
1  "GET /group1/M00/00/00/wKiuj1qWeyGAJWsuAAAACXh6ifo038.txt HTTP/1.1"
1  "GET /favicon.ico HTTP/1.1"
1  "GET /group1/M00/00/00/wKiuj1qWeyGAJWsuAAAACXh6ifo038.txt HTTP/1.1"
   "GET /favicon.ico HTTP/1.1"
1  "GET /group1/M00/00/00/wKiuj1qWeyGAJWsuAAAACXh6ifo038.txt HTTP/1.1"
1  "GET /favicon.ico HTTP/1.1"
1
awk '{if($2==ip){$1=" "}ip=$2;$2=""}1' 1.txt

shell解析json打印制定key对应的值（伪）

1
2
3
4
5
6
7
8
9
10
11
12
13
echo '{"a":21,"b":60,"c":16,"d":36,"e":6,"f":6,"g":11,"h":6,"i":6,"j":4,"k":16}'
# 要求制定打印key对应的值

# 最简便的方法，但是需要在机器上安装jq
echo '{"a":21,"b":60,"c":16,"d":36,"e":6,"f":6,"g":11,"h":6,"i":6,"j":4,"k":16}' | jq .a

echo '{"a":21,"b":60,"c":16,"d":36,"e":6,"f":6,"g":11,"h":6,"i":6,"j":4,"k":16}' | \
tr -d '"{}' | sed 's/,/\n/g' | \
awk '{split($1,a,":");s[a[1]]=a[2]}END{print s["a"]}'

# 这种方法有点取巧了，待优化
a='c';echo '{"a":21,"b":60,"c":16,"d":36,"e":6,"f":6,"g":11,"h":6,"i":6,"j":4,"k":16}' | \
sed -ne "s/.*$a\":\([0-9][0-9]\).*/\1/p"

Linux四种结束某用户所有进程的方法

1
2
3
4
5
6
7
8
9
10
11
## pkill方式
pkill -u www

## killall方式
killall -u www

## ps方式 ps列出www的pid，然后依次kill掉，比较繁琐。
ps -ef | grep www | awk '{ print $2 }' | xargs kill -9

## pgrep方式 pgrep -u参数查出用户的所有pid，然后依次kill
pgrep -u www | xargs kill -9

shell命令行参数的方式（简）

1
2
3
4
5
6
7
8
9
10
11
# 只有一个参数， 多参数匹配需要用shift，有空再补充
while getopts "c:" opts
do
        case $opts in
                c) c=$OPTARG
                ;;
                ?)
                exit -1
                ;;
        esac
done