我们中有这样的需求就是,当前端和后端通过接口通信以后,我们在日常排查中缺少跟踪接口返回信息和所有头部信息的跟踪,针对于这个需求我们采用nginx+lua+cjson的形式通过lua脚本

准备的模块

  1. LuaJIT

http://luajit.org/download/LuaJIT-2.0.5.tar.gz

  1. lua-nginx-module

https://github.com/openresty/lua-nginx-module/archive/v0.10.9rc7.tar.gz

  1. ngx_devel_kit

https://github.com/simpl/ngx_devel_kit/archive/v0.3.0.tar.gz

  1. lua-cjson

https://www.kyne.com.au/~mark/software/download/lua-cjson-2.1.0.tar.gz

  1. nginx

http://nginx.org/download/nginx-1.14.1.tar.gz

软件安装

  1. LuaJit安装
wget http://luajit.org/download/LuaJIT-2.0.5.tar.gz
tar -zxvf LuaJIT-2.0.5.tar.gz
cd LuaJIT-2.0.5
make
make install PREFIX=/usr/local/luajit
# 环境变量必须配置!
vi /etc/profile
# 加入下面两个环境变量
export LUAJIT_LIB=/usr/local/luajit/lib
export LUAJIT_INC=/usr/local/luajit/include/luajit-2.0
  1. ngx_devel_kit和lua-nginx-module

自己选择安装位置,最好公共目录,我放在/usr/local下

wget https://github.com/simpl/ngx_devel_kit/archive/v0.3.0.tar.gz
wget https://github.com/openresty/lua-nginx-module/archive/v0.10.9rc7.tar.gz
tar -zxvf v0.3.0.tar.gz
tar -zxvf v0.10.9rc7.tar.gz
cp -R lua-nginx-module/ /usr/local/
cp -R ngx_devel_kit/ /usr/local/
  1. 安装nginx编译nginx
wget http://nginx.org/download/nginx-1.14.1.tar.gz
tar -zxvf nginx-1.14.1.tar.gz
cd nginx-1.14.1
# 注:报错gcc需要安装,可以执行
yum install -y gcc g++ gcc-c++
#依赖报错,可以执行
yum -y install zlib zlib-devel openssl openssl--devel pcre pcre-devel
./configure --with-ld-opt="-Wl,-rpath,$LUAJIT_LIB" --conf-path=/etc/nginx/nginx.conf --add-module=/usr/local/ngx_devel_kit --add-module=/usr/local/lua-nginx-module
make -j 4 && make install
# 加载lua库
echo "/usr/local/luajit/lib" >> /etc/ld.so.conf
ldconfig
  1. lua-cjson安装
wget https://www.kyne.com.au/~mark/software/download/lua-cjson-2.1.0.tar.gz
tar zxvf lua-cjson-2.1.0.tar.gz
cd lua-cjson-2.1.0
vi Makefile
# 修改前缀(路径后不要留空格!!)
PREFIX = /usr/local/luajit
LUA_INCLUDE_DIR = $(PREFIX)/include/luajit-2.0
make install
# lua中引用cjson库时必须指定位置
package.path = "/usr/local/luajit/mylua/?.lua;"
package.cpath = "/usr/local/luajit/lib/lua/5.1/?.so;"
local cjson = require("cjson");

网站配置

日志手机依旧依靠nginx log的形式我们针对于需要的内容进行收集和填充到日志中去,依赖原有的阿里云日志服务进行收集

日志服务

  1. 定义日志格式
log_format json_combined escape=json
'{'
'"time_local":"$time_local",'
'"remote_addr":"$remote_addr",'
'"remote_user":"$remote_user",'
'"request":"$request",'
'"req_headers":"$req_headers",'
'"status": "$status",'
'"http_host": "$http_host",'
'"body_bytes_sent":"$body_bytes_sent",'
'"request_time":"$request_time",'
'"http_referrer":"$http_referer",'
'"content_type":"$content_type",'
'"req_body":"$req_body",'
'"resp_body":"$resp_body",'
'"ua":"$http_user_agent",'
'"cookie_u":"$cookie_u",'
'"referer":"$http_referer",'
'"xff":"$http_x_forwarded_for",'
'"ups_status":"$upstream_status",'
'"ups_addr":"$upstream_addr",'
'"resp_content_type":"$resp_content_type",'
'"ups_time":"$upstream_response_time"'
'}';
  1. 定义req_header和resp_body收集
# 开启lua记录请求体
lua_need_request_body on;
set $resp_body "";
set $resp_content_type "";
set $req_body "";
set $req_headers "";
header_filter_by_lua '
package.path = "/usr/local/luajit/mylua/?.lua;"
package.cpath = "/usr/local/luajit/lib/lua/5.1/?.so;"
local cjson = require("cjson");
local headers, err = ngx.req.get_headers()
local json = {};
for k, v in pairs(headers) do
json[k] = v;
end
ngx.var.req_headers = cjson.encode(json);
';
body_filter_by_lua '
ngx.var.resp_content_type = ngx.resp.get_headers()["Content-Type"]
local resp_body = ngx.arg[1]
ngx.ctx.buffered = (ngx.ctx.buffered or "") .. resp_body
if ngx.arg[2] and string.find(ngx.var.resp_content_type, "application/json") then
ngx.var.resp_body = ngx.ctx.buffered
ngx.var.req_body = ngx.var.request_body
end
';
  1. odps收集表结构
CREATE TABLE `access_json` (
`ip` string,
`remote_user` string,
`req_headers` string,
`status` bigint,
`http_host` string,
`body_bytes_sent` bigint,
`request_time` string,
`http_referer` string,
`content_type` string,
`req_body` string,
`resp_body` string,
`http_user_agent` string,
`cookie_u` string,
`http_x_forwarded_for` string,
`upstream_status` string,
`upstream_addr` string,
`resp_content_type` string,
`upstream_response_time` string,
`request` string
)
PARTITIONED BY (dt string);

AB压力测试

yum -y install httpd-tools
[root@zookeeperslave2 ~]# ab -c 1000 -n 10000 -r http://192.168.1.169/aaa
This is ApacheBench, Version 2.3 <$Revision: 1430300 $>
Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/
Licensed to The Apache Software Foundation, http://www.apache.org/
Benchmarking 192.168.1.169 (be patient)
Completed 1000 requests
Completed 2000 requests
Completed 3000 requests
Completed 4000 requests
Completed 5000 requests
Completed 6000 requests
Completed 7000 requests
Completed 8000 requests
Completed 9000 requests
Completed 10000 requests
Finished 10000 requests
Server Software: nginx/1.14.1
Server Hostname: 192.168.1.169
Server Port: 80
Document Path: /aaa
Document Length: 17 bytes
Concurrency Level: 1000
Time taken for tests: 1.508 seconds
Complete requests: 10000
Failed requests: 0
Write errors: 0
Total transferred: 1660000 bytes
HTML transferred: 170000 bytes
Requests per second: 6632.41 [#/sec] (mean)
Time per request: 150.775 [ms] (mean)
Time per request: 0.151 [ms] (mean, across all concurrent requests)
Transfer rate: 1075.18 [Kbytes/sec] received
Connection Times (ms)
min mean[+/-sd] median max
Connect: 0 52 169.9 18 1040
Processing: 11 50 29.4 41 273
Waiting: 0 48 29.3 39 270
Total: 30 102 171.4 65 1292
Percentage of the requests served within a certain time (ms)
50% 65
66% 73
75% 82
80% 90
90% 150
95% 156
98% 1059
99% 1065
100% 1292 (longest request)

问题集结

  1. AIO找不到的问题
./configure: no supported file AIO was found
Currently file AIO is supported on FreeBSD 4.3+ and Linux 2.6.22+ only

安装开发者工具

yum groupinstall 'Development Tools'
  1. 缺少libxml2/libxslt模块
./configure: error: the HTTP XSLT module requires the libxml2/libxslt
libraries. You can either do not enable the module or install the libraries.

安装缺失库

yum -y install libxml2 libxml2-dev
yum -y install libxslt-devel
  1. 缺少GD库
./configure: error: the HTTP image filter module requires the GD library.
You can either do not enable the module or install the libraries.

安装GD库

yum -y install gd-devel
  1. 缺少ExtUtils
./configure: error: perl module ExtUtils::Embed is required

安装ExtUtils

yum -y install perl-devel perl-ExtUtils-Embed
  1. 缺少GeoIP库
./configure: error: the GeoIP module requires the GeoIP library.
You can either do not enable the module or install the library.

安装缺失库

yum -y install GeoIP GeoIP-devel GeoIP-data

6、 缺少Google Perftools (编译环境我是不装这个的,从编译选线去除这个依赖就行)

./configure: error: the Google perftools module requires the Google perftools
library. You can either do not enable the module or install the library.
# 安装libunwind
wget http://download.savannah.gnu.org/releases/libunwind/libunwind-0.99-beta.tar.gz
tar zxvf libunwind-0.99-beta.tar.gz
cd libunwind-0.99-beta
./configure
make
make install
# 安装google-perftools
wget https://github.com/gperftools/gperftools/releases/download/gperftools-2.5/gperftools-2.5.tar.gz
tar zxvf gperftools-2.5.tar.gz
cd gperftools-2.5
./configure --prefix=/usr/local/gperftools
make && make install
vi /etc/ld.so.conf.d/usr_local_lib.conf
#加入
/usr/local/lib
ldconfig
# 添加环境
vi /etc/profile
export LD_PRELOAD=/usr/local/gperftools/lib/libtcmalloc.so
export HEAPPROFILE=/usr/bin/gzip

8.模块版本不对

[emerg] module "/usr/lib64/nginx/modules/ngx_http_geoip_module.so" version 1012002 instead of 1...ip.conf:1

这个错误是因为之前nginx是yum安装的他的版本是1.12.的 我这次安装的nginx是最新版本1.14.的所以我需要对这个模块进行升级

vi /etc/yum.repos.d/nginx.repo
[nginx]
name=nginx repo
baseurl=http://nginx.org/packages/centos/7/$basearch/
gpgcheck=0
enabled=1
yum remove nginx-mod*
yum install nginx-module-*