从零编译nginx+lua+cjson模块的nginx服务实现日志全程监控

我们中有这样的需求就是，当前端和后端通过接口通信以后，我们在日常排查中缺少跟踪接口返回信息和所有头部信息的跟踪，针对于这个需求我们采用nginx+lua+cjson的形式通过lua脚本

准备的模块

LuaJIT

http://luajit.org/download/LuaJIT-2.0.5.tar.gz

lua-nginx-module

https://github.com/openresty/lua-nginx-module/archive/v0.10.9rc7.tar.gz

ngx_devel_kit

https://github.com/simpl/ngx_devel_kit/archive/v0.3.0.tar.gz

lua-cjson

https://www.kyne.com.au/~mark/software/download/lua-cjson-2.1.0.tar.gz

nginx

http://nginx.org/download/nginx-1.14.1.tar.gz

软件安装

LuaJit安装

wget http://luajit.org/download/LuaJIT-2.0.5.tar.gz

tar -zxvf LuaJIT-2.0.5.tar.gz

cd LuaJIT-2.0.5

make

make install PREFIX=/usr/local/luajit

# 环境变量必须配置！
vi /etc/profile

# 加入下面两个环境变量
export LUAJIT_LIB=/usr/local/luajit/lib
export LUAJIT_INC=/usr/local/luajit/include/luajit-2.0

ngx_devel_kit和lua-nginx-module

自己选择安装位置，最好公共目录,我放在/usr/local下

wget https://github.com/simpl/ngx_devel_kit/archive/v0.3.0.tar.gz

wget https://github.com/openresty/lua-nginx-module/archive/v0.10.9rc7.tar.gz

tar -zxvf v0.3.0.tar.gz

tar -zxvf v0.10.9rc7.tar.gz

cp -R lua-nginx-module/ /usr/local/
cp -R ngx_devel_kit/ /usr/local/

安装nginx编译nginx


wget http://nginx.org/download/nginx-1.14.1.tar.gz

tar -zxvf nginx-1.14.1.tar.gz

cd nginx-1.14.1

# 注：报错gcc需要安装，可以执行

yum install -y gcc g++ gcc-c++

#依赖报错，可以执行

yum -y install zlib zlib-devel openssl openssl--devel pcre pcre-devel

./configure --with-ld-opt="-Wl,-rpath,$LUAJIT_LIB" --conf-path=/etc/nginx/nginx.conf  --add-module=/usr/local/ngx_devel_kit --add-module=/usr/local/lua-nginx-module

make -j 4 && make install

# 加载lua库
echo "/usr/local/luajit/lib" >> /etc/ld.so.conf
ldconfig

lua-cjson安装


wget https://www.kyne.com.au/~mark/software/download/lua-cjson-2.1.0.tar.gz

tar zxvf lua-cjson-2.1.0.tar.gz

cd lua-cjson-2.1.0

vi Makefile

# 修改前缀(路径后不要留空格！！)
PREFIX =            /usr/local/luajit
LUA_INCLUDE_DIR =   $(PREFIX)/include/luajit-2.0

make install

# lua中引用cjson库时必须指定位置
package.path = "/usr/local/luajit/mylua/?.lua;"
package.cpath = "/usr/local/luajit/lib/lua/5.1/?.so;"
local cjson = require("cjson");

网站配置

日志手机依旧依靠nginx log的形式我们针对于需要的内容进行收集和填充到日志中去，依赖原有的阿里云日志服务进行收集

日志服务

定义日志格式

log_format json_combined escape=json
                '{'
                    '"time_local":"$time_local",'
                    '"remote_addr":"$remote_addr",'
                    '"remote_user":"$remote_user",'
                    '"request":"$request",'
                    '"req_headers":"$req_headers",'
                    '"status": "$status",'
                    '"http_host": "$http_host",'
                    '"body_bytes_sent":"$body_bytes_sent",'
                    '"request_time":"$request_time",'
                    '"http_referrer":"$http_referer",'
                    '"content_type":"$content_type",'
                    '"req_body":"$req_body",'
                    '"resp_body":"$resp_body",'
                    '"ua":"$http_user_agent",'
                    '"cookie_u":"$cookie_u",'
                    '"referer":"$http_referer",'
                    '"xff":"$http_x_forwarded_for",'
                    '"ups_status":"$upstream_status",'
                    '"ups_addr":"$upstream_addr",'
                    '"resp_content_type":"$resp_content_type",'
                    '"ups_time":"$upstream_response_time"'
                '}';

定义req_header和resp_body收集

# 开启lua记录请求体
lua_need_request_body on;
set $resp_body "";
set $resp_content_type "";
set $req_body "";
set $req_headers "";
header_filter_by_lua '
             package.path = "/usr/local/luajit/mylua/?.lua;"
             package.cpath = "/usr/local/luajit/lib/lua/5.1/?.so;"
             local cjson = require("cjson");
             local headers, err = ngx.req.get_headers()
             local json = {};
             for k, v in pairs(headers) do
                 json[k] = v;
             end
             ngx.var.req_headers = cjson.encode(json);
';
body_filter_by_lua '
             ngx.var.resp_content_type = ngx.resp.get_headers()["Content-Type"]
             local resp_body = ngx.arg[1]
             ngx.ctx.buffered = (ngx.ctx.buffered or "") .. resp_body
             if ngx.arg[2] and string.find(ngx.var.resp_content_type, "application/json") then
                     ngx.var.resp_body = ngx.ctx.buffered
                     ngx.var.req_body = ngx.var.request_body
             end
';

odps收集表结构

CREATE TABLE `access_json` (
	`ip` string,
	`remote_user` string,
	`req_headers` string,
	`status` bigint,
	`http_host` string,
	`body_bytes_sent` bigint,
	`request_time` string,
	`http_referer` string,
	`content_type` string,
	`req_body` string,
	`resp_body` string,
	`http_user_agent` string,
	`cookie_u` string,
	`http_x_forwarded_for` string,
	`upstream_status` string,
	`upstream_addr` string,
	`resp_content_type` string,
	`upstream_response_time` string,
	`request` string
) 
 PARTITIONED BY (dt string);

AB压力测试

 yum -y install httpd-tools
[root@zookeeperslave2 ~]# ab -c 1000 -n 10000 -r  http://192.168.1.169/aaa
This is ApacheBench, Version 2.3 <$Revision: 1430300 $>
Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/
Licensed to The Apache Software Foundation, http://www.apache.org/

Benchmarking 192.168.1.169 (be patient)
Completed 1000 requests
Completed 2000 requests
Completed 3000 requests
Completed 4000 requests
Completed 5000 requests
Completed 6000 requests
Completed 7000 requests
Completed 8000 requests
Completed 9000 requests
Completed 10000 requests
Finished 10000 requests


Server Software:        nginx/1.14.1
Server Hostname:        192.168.1.169
Server Port:            80

Document Path:          /aaa
Document Length:        17 bytes

Concurrency Level:      1000
Time taken for tests:   1.508 seconds
Complete requests:      10000
Failed requests:        0
Write errors:           0
Total transferred:      1660000 bytes
HTML transferred:       170000 bytes
Requests per second:    6632.41 [#/sec] (mean)
Time per request:       150.775 [ms] (mean)
Time per request:       0.151 [ms] (mean, across all concurrent requests)
Transfer rate:          1075.18 [Kbytes/sec] received

Connection Times (ms)
              min  mean[+/-sd] median   max
Connect:        0   52 169.9     18    1040
Processing:    11   50  29.4     41     273
Waiting:        0   48  29.3     39     270
Total:         30  102 171.4     65    1292

Percentage of the requests served within a certain time (ms)
  50%     65
  66%     73
  75%     82
  80%     90
  90%    150
  95%    156
  98%   1059
  99%   1065
 100%   1292 (longest request)

问题集结

AIO找不到的问题

1 2	./configure: no supported file AIO was found Currently file AIO is supported on FreeBSD 4.3+ and Linux 2.6.22+ only

安装开发者工具

1	yum groupinstall 'Development Tools'

缺少libxml2/libxslt模块

1 2	./configure: error: the HTTP XSLT module requires the libxml2/libxslt libraries. You can either do not enable the module or install the libraries.

安装缺失库

1 2	yum -y install libxml2 libxml2-dev yum -y install libxslt-devel

缺少GD库

1 2	./configure: error: the HTTP image filter module requires the GD library. You can either do not enable the module or install the libraries.

安装GD库

1	yum -y install gd-devel

缺少ExtUtils

1	./configure: error: perl module ExtUtils::Embed is required

安装ExtUtils

1	yum -y install perl-devel perl-ExtUtils-Embed

缺少GeoIP库

1 2	./configure: error: the GeoIP module requires the GeoIP library. You can either do not enable the module or install the library.

安装缺失库

1	yum -y install GeoIP GeoIP-devel GeoIP-data

6、缺少Google Perftools （编译环境我是不装这个的，从编译选线去除这个依赖就行）

1 2	./configure: error: the Google perftools module requires the Google perftools library. You can either do not enable the module or install the library.

# 安装libunwind
wget http://download.savannah.gnu.org/releases/libunwind/libunwind-0.99-beta.tar.gz

tar zxvf libunwind-0.99-beta.tar.gz
cd libunwind-0.99-beta
./configure
make
make install

# 安装google-perftools
wget https://github.com/gperftools/gperftools/releases/download/gperftools-2.5/gperftools-2.5.tar.gz

tar zxvf gperftools-2.5.tar.gz
cd gperftools-2.5
./configure --prefix=/usr/local/gperftools
make && make install
vi /etc/ld.so.conf.d/usr_local_lib.conf
#加入
/usr/local/lib
ldconfig

# 添加环境
vi /etc/profile
export LD_PRELOAD=/usr/local/gperftools/lib/libtcmalloc.so
export HEAPPROFILE=/usr/bin/gzip

8.模块版本不对

1	[emerg] module "/usr/lib64/nginx/modules/ngx_http_geoip_module.so" version 1012002 instead of 1...ip.conf:1

这个错误是因为之前nginx是yum安装的他的版本是1.12.*的我这次安装的nginx是最新版本1.14.*的所以我需要对这个模块进行升级

vi /etc/yum.repos.d/nginx.repo

[nginx]
name=nginx repo
baseurl=http://nginx.org/packages/centos/7/$basearch/
gpgcheck=0
enabled=1

yum remove nginx-mod*
yum install nginx-module-*