Use ulog to monitor port logs (by quqi99)

Author: Zhang Hua, published in: 2019-08-14
Copyright Statement: It can be reproduced arbitrarily. When reproducing, please indicate the original source of the article, the author's information and the Copyright Statement in the form of hyperlinks.( https://zhhuabj.blog.csdn.net)

problem

For security audit requirements, it may be necessary to record the connections between neutron security group and FWaaS and all ACCEPT and DROP in LBaaS. There are many methods, such as sg-logging, ovs IPFIX, ulog and so on. This paper focuses on the ulog method. The other two methods are posted but not debugged.

ulog

ulog is handled by the kernel through the ulogd process that sends logs directly to the user state through netlink.
sudo apt install ulogd2 ulogd2-json

# cat /etc/ulogd.conf
# Example configuration for ulogd
# Adapted to Debian by Achilleas Kotsis <achille@debian.gr>

[global]
######################################################################
# GLOBAL OPTIONS
######################################################################


# logfile for status messages
logfile="syslog"

# loglevel: debug(1), info(3), notice(5), error(7) or fatal(8) (default 5)
loglevel=3

######################################################################
# PLUGIN OPTIONS
######################################################################

# We have to configure and load all the plugins we want to use

# general rules:
# 1. load the plugins _first_ from the global section
# 2. options for each plugin in seperate section below


plugin="/usr/lib/x86_64-linux-gnu/ulogd/ulogd_inppkt_NFLOG.so"
#plugin="/usr/lib/x86_64-linux-gnu/ulogd/ulogd_inppkt_ULOG.so"
#plugin="/usr/lib/x86_64-linux-gnu/ulogd/ulogd_inppkt_UNIXSOCK.so"
plugin="/usr/lib/x86_64-linux-gnu/ulogd/ulogd_inpflow_NFCT.so"
plugin="/usr/lib/x86_64-linux-gnu/ulogd/ulogd_filter_IFINDEX.so"
plugin="/usr/lib/x86_64-linux-gnu/ulogd/ulogd_filter_IP2STR.so"
plugin="/usr/lib/x86_64-linux-gnu/ulogd/ulogd_filter_IP2BIN.so"
#plugin="/usr/lib/x86_64-linux-gnu/ulogd/ulogd_filter_IP2HBIN.so"
plugin="/usr/lib/x86_64-linux-gnu/ulogd/ulogd_filter_PRINTPKT.so"
plugin="/usr/lib/x86_64-linux-gnu/ulogd/ulogd_filter_HWHDR.so"
plugin="/usr/lib/x86_64-linux-gnu/ulogd/ulogd_filter_PRINTFLOW.so"
#plugin="/usr/lib/x86_64-linux-gnu/ulogd/ulogd_filter_MARK.so"
plugin="/usr/lib/x86_64-linux-gnu/ulogd/ulogd_output_LOGEMU.so"
plugin="/usr/lib/x86_64-linux-gnu/ulogd/ulogd_output_SYSLOG.so"
plugin="/usr/lib/x86_64-linux-gnu/ulogd/ulogd_output_XML.so"
#plugin="/usr/lib/x86_64-linux-gnu/ulogd/ulogd_output_SQLITE3.so"
plugin="/usr/lib/x86_64-linux-gnu/ulogd/ulogd_output_GPRINT.so"
#plugin="/usr/lib/x86_64-linux-gnu/ulogd/ulogd_output_NACCT.so"
#plugin="/usr/lib/x86_64-linux-gnu/ulogd/ulogd_output_PCAP.so"
#plugin="/usr/lib/x86_64-linux-gnu/ulogd/ulogd_output_PGSQL.so"
#plugin="/usr/lib/x86_64-linux-gnu/ulogd/ulogd_output_MYSQL.so"
#plugin="/usr/lib/x86_64-linux-gnu/ulogd/ulogd_output_DBI.so"
plugin="/usr/lib/x86_64-linux-gnu/ulogd/ulogd_raw2packet_BASE.so"
plugin="/usr/lib/x86_64-linux-gnu/ulogd/ulogd_inpflow_NFACCT.so"
plugin="/usr/lib/x86_64-linux-gnu/ulogd/ulogd_output_GRAPHITE.so"
plugin="/usr/lib/x86_64-linux-gnu/ulogd/ulogd_output_JSON.so"

# this is a stack for flow-based logging via LOGEMU
stack=ct1:NFCT,ip2str1:IP2STR,print1:PRINTFLOW,emu1:LOGEMU
stack=ct2:NFCT,ip2str1:IP2STR,json1:JSON

[ct1]
#netlink_socket_buffer_size=217088
#netlink_socket_buffer_maxsize=1085440
#netlink_resync_timeout=60 # seconds to wait to perform resynchronization
#pollinterval=10 # use poll-based logging instead of event-driven
# If pollinterval is not set, NFCT plugin will work in event mode
# In this case, you can use the following filters on events:
#accept_src_filter=192.168.1.0/24,1:2::/64 # source ip of connection must belong to these networks
#accept_dst_filter=192.168.1.0/24 # destination ip of connection must belong to these networks
#accept_proto_filter=tcp,sctp # layer 4 proto of connections
event_mask=0x00000001
hash_enable=0

[emu1]
file="/var/log/ulog/syslogemu.log"
sync=1

[ct2]
hash_enable=0
event_mask=0x00000004
#accept_src_filter=10.211.55.0/24

[emu2]
file="/var/log/ulog/syslogemu2.log"
sync=1

[json1]
sync=1
file="/var/log/ulog/ulogd.json"

To run the ulogd service under the namespace of neutron-gateway:

sudo ip netns exec qrouter-82ff3e71-e021-4ce2-aa92-5ec6cb6d7c3d /usr/sbin/ulogd -d -p /run/ulogd-test.pid -u ulog

Then create a virtual machine, allocate floating IP, and ping it. Observe the log as follows:

$ ping 10.5.150.82
PING 10.5.150.82 (10.5.150.82) 56(84) bytes of data.
64 bytes from 10.5.150.82: icmp_seq=1 ttl=63 time=5.71 ms

$ tail -f /var/log/ulog/syslogemu.log
Aug 14 05:38:20 juju-da14dc-stein-5 [NEW] ORIG: SRC=10.5.0.8 DST=10.5.150.82 PROTO=ICMP TYPE=0 CODE=8 PKTS=0 BYTES=0 , REPLY: SRC=192.168.21.73 DST=10.5.0.8 PROTO=ICMP TYPE=0 CODE=8 PKTS=0 BYTES=0

Or json format:

# tail -f /var/log/ulog/ulogd.json
{"timestamp": "2019-08-14T06:10:55", "dvc": "Netfilter", "orig.ip.protocol": 17, "orig.l4.sport": 34308, "orig.l4.dport": 123, "orig.raw.pktlen": 0, "orig.raw.pktcount": 0, "reply.ip.protocol": 17, "reply.l4.sport": 123, "reply.l4.dport": 34308, "reply.raw.pktlen": 0, "reply.raw.pktcount": 0, "ct.mark": 0, "ct.id": 2009647936, "ct.event": 4, "flow.end.sec": 1565763055, "flow.end.usec": 77990, "oob.family": 2, "oob.protocol": 0, "src_ip": "10.5.0.25", "dest_ip": "91.189.94.4", "reply.ip.saddr.str": "91.189.94.4", "reply.ip.daddr.str": "10.5.0.25"}

The meaning of event_mask in the configuration file is (for a more complete example, see: https://gist.github.com/chagridsada/20fde58e49205af0d33a ):

NF_NETLINK_CONNTRACK_NEW: 0×00000001
NF_NETLINK_CONNTRACK_UPDATE: 0×00000002
NF_NETLINK_CONNTRACK_DESTROY: 0×00000004

The above usage can also be broadcasted. The following configuration of log1 uses broadcasting group 32, so it needs to be specified in iptables by the'- nflog-group'parameter (iptables --nflog-group)

plugin="/usr/lib/x86_64-linux-gnu/ulogd/ulogd_output_LOGEMU.so"

stack=log1:NFLOG,base1:BASE,ifi1:IFINDEX,ip2str1:IP2STR,print1:PRINTPKT,emu1:LOGEMU
 
# Logging of system packet through NFLOG
[log1]
# netlink multicast group (the same as the iptables --nflog-group param)
# Group O is used by the kernel to log connection tracking invalid message
group=32

[emu1]
file="/var/log/iptables.log"
#sync=1

nc -v 10.0.8.203 8080  #10.0.8.203 is FIP of VM
iptables -I FORWARD -p tcp --dport 8080 -j ULOG --nflog-group 32 --log-prefix "dbg: FORWARD "
iptables -I neutron-openvswi-FORWARD -p tcp --dport 8080 -j ULOG --nflog-group 32 --log-prefix "dbg: n-o-FORWARD "
iptables -I neutron-openvswi-sg-chain -p tcp --dport 8080 -j ULOG --nflog-group 32 --log-prefix "dbg: n-o-sg-chain "
iptables -I neutron-openvswi-ic42f8a9b-6 -p tcp --dport 8080 -j ULOG --nflog-group 32 --log-prefix "dbg: n-o-ic42f8a9b-6 "
iptables -I neutron-openvswi-sg-fallback -p tcp --dport 8080 -j ULOG --nflog-group 32 --log-prefix "dbg: n-o-sg-fallback "

Note that ULOG in the above iptables can also be replaced by LOG and NFLOG. It takes a long time for LOG to print logs. NFLOG can use the netlink kernel and user mode in both directions. ULOG can only transfer logs from the kernel to user mode through the netlink-log protocol.

Appendix - Other Methods - sg-logging

Neutron has this feature. The disadvantage is that it can cause certain MQ performance problems, which will lead to rpc_timeout of components. If heat is used with dhcp-agent, the problem will be more serious, because heat will create many ports concurrently, while dhcp-agent can only process ports one after another, which will result in rpc-timeout of dhcp-agent end and cause f-timeout. If there are some retry debugging such as anti-aninity on the heat side and another batch of ports, it will be a vicious circle. In addition, sg-logging does not currently support FIP mapping to fixed-ip, such as SNAT/DNA T and LBaaS VIP.
Packet logging service is designed as a Neutron plug-in that captures network packets for relevant resources (e.g. security group or firewall group) when the registered events occur. see https://docs.openstack.org/neutron/rocky/admin/config-logging.html

#./generate-bundle.sh -s bionic -r stein --num-compute 13 --heat --neutron-sg-logging
#juju deploy ./b/openstack.yaml --overlay ./b/o/neutron-gateway.yaml --overlay ./b/o/heat.yaml --overlay ./b/o/neutron-sg-logging.yaml
./generate-bundle.sh -s bionic -r stein --num-compute 13
juju add-model stein
juju deploy ./b/openstack.yaml --overlay ./b/o/neutron-gateway.yaml
juju deploy heat
juju add-relation heat mysql
juju add-relation heat keystone
juju add-relation heat rabbitmq-server
./configure

#source ~/stsstack-bundles/openstack/novarc
#http_proxy=http://squid.internal:3128 wget http://download.cirros-cloud.net/0.4.0/cirros-0.4.0-x86_64-disk.img
#openstack image create --disk-format=qcow2 --container-format=bare --public cirros --file ./cirros-0.4.0-x86_64-disk.img

#juju config neutron-openvswitch firewall-driver=openvswitch
juju config neutron-api enable-security-group-logging=True
#juju config neutron-api enable-firewall-group-logging=True
juju config neutron-openvswitch security-group-log-burst-limit 25
#juju config neutron-openvswitch security-group-log-rate-limit unlimited


openstack network loggable resources list
openstack network log create mylog --resource-type security_group --description "test" --event ALL
#openstack network log create mylog --resource-type security_group --resource sg1
#openstack network log create mylog --resource-type security_group --target portA
#openstack network log create mylog --resource-type security_group --target portB --resource sg1
openstack network log set --disable mylog

# fix error 'Quota exceeded for resources: ['security_group'].'
neutron quota-show 
openstack quota set --secgroup-rules 10000 --secgroups 1000 5275d0bc16fe42c39b96469b9019f7ce

openstack stack list
openstack stack create -e green.test.nonprd.yaml  -t juju_k8s_stack.yaml mystack

Annex - Other Methods - ovs IPFIX

ovs can output packages as IPFIX, but may only support the output of ACCEPT packages, not DROP packages.

#https://community.riverbed.com/s/article/DOC-5783
sudo apt install libfixbuf3 libfixbuf3-dev libpcap-dev
wget https://sourceforge.net/projects/libipfix/files/latest/download && tar -xf download
cd libipfix_110224 && ./configure && make && sudo make install
sudo ipfix_collector -p 4740
#juju config neutron-openvswitch ipfix-target=10.5.0.8:4740
ovs-vsctl set Bridge br-int ipfix=@i -- --id=@i create IPFIX targets=\"10.5.0.8:4740\" sampling=64 cache_active_timeout=60 cache_max_flows=128
ovs-vsctl list-br| xargs -l -I{} ovs-vsctl set IPFIX {} sampling=1 
ovs-vsctl list Bridge br-int |grep ipfix
ovs-vsctl list ipfix <ipfix-id>
#ovs-vsctl clear Bridge br-int ipfix

for node in {0,1}; do juju ssh neutron-gateway/$node sudo 'ovs-vsctl list-br| xargs -l -I{} ovs-vsctl set IPFIX {} sampling=1'; done 
for node in {0,1}; do juju ssh neutron-gateway/$node sudo 'ovs-vsctl list ipfix'; done 
for node in {0..30}; do juju ssh nova-compute-kvm/$node sudo 'ovs-vsctl list-br| sudo xargs -l -I{} ovs-vsctl set IPFIX {} sampling=1'; done 
for node in {0..30}; do juju ssh nova-compute-kvm/$node sudo 'ovs-vsctl list ipfix'; done 
#Turn on IPFIX with juju 
juju config neutron-gateway ipfix-target=172.30.140.47:5500 
juju config neutron-openvswitch ipfix-target=172.30.140.47:5500 
#Turn off ipfix with the command below if we need to revert 
# juju config neutron-gateway --reset ipfix-target 
# juju config neutron-openvswitch --reset ipfix-target

Tags: Linux OpenStack sudo iptables

Posted on Wed, 14 Aug 2019 00:17:59 -0700 by eves