Troubleshooting

Once SPAN Agents show in active state in the cluster Monitoring/Agent Overview page, no action is needed on the ERSPAN Virtual Machine, user does not need to log into it. If that is not happening or if the flows are not reported to the cluster, following information will help pinpoint deployment problems.

In normal conditions, on the VM:

• systemctl status tet_vm_setup reports an inactive service with SUCCESS exit status;

• systemctl status tet-nic-driver reports an active service;

• docker network ls reports five networks: host, none and three erspan-<iface name>;

• ip link only reports the loopback interface;

• docker ps reports three running containers;

• docker logs <cid> for each container contains the message:INFO success: tet-sensor entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)

• docker exec <cid> ifconfig reports only one interface, besides the loopback;

• docker exec <cid> route -n reports the default gateway;

• docker exec <cid> iptables -t raw -S PREROUTING reports the rule -A PREROUTING -p gre -j DROP;

If any of the above does not hold true, check the deployment script logs in /local/tetration/logs/ tet_vm_setup.log for the reason why the SPAN agent containers deployment failed.

Any other agent registration/connectivity issue can be troubleshooted the same way it is done for agents running on a host via the docker exec command:

• docker exec <cid> ps -ef reports the two tet-engine, tet-engine check_conf instances and two /usr/local/tet/tet-sensor -f /usr/local/tet/conf/.sensor_config instances, one with root user and one with tet-sensor user, along with the process manager /usr/bin/ python /usr/bin/supervisord -c /etc/supervisord.conf -n instance.

• docker exec <cid> cat /usr/local/tet/log/tet-sensor.log shows the agent’s logs;

• docker exec <cid> cat /usr/local/tet/log/fetch_sensor_id.log shows the agent’s registration logs;

• docker exec <cid> cat /usr/local/tet/log/check_conf_update.log shows the configuration update polling logs;

  If necessary, traffic to/from the container can be monitored with tcpdump after setting into the container’s network namespace:

  1. Retrieve the container’s network namespace (SandboxKey) via docker inspect <cid> | grep SandboxKey;

  2. Set into the container’s network namespace nsenter --net=/var/run/docker/netns/...;

  3. Monitor eth0 traffic tcpdump -i eth0 -n.