Advanced Problem Determination

I posted previously about finding suspiciously short TCP connection to find leads for problem determination.    You can get the same TCP statistics from tshark, when file sizes are too unruly for transfer or loading in a GUI. tshark -r foo.cap -q -z conv,tcp The output is super-wide and not blogger.com friendly, but it includes TCP addresses, ports, bytes sent, and duration.

Hypothetical scenario is that a user has disabled TLS < 1.2 and has sporadic SSL handshake failures.  The suspected culprit is just a non-TLS 1.2 client, but people can be simultaneously skeptical and lazy. # Show TLSv1.2 client hellos tshark -Y "ssl.handshake.version == 0x303" -r ssl.cap  # Show TLSv1.1 client hellos tshark -Y "ssl.handshake.version == 0x302" -r ssl.cap  # Show TLSv1.0 client hellos tshark -Y "ssl.handshake.version == 0x301" -r ssl.cap  Caution!  It's easy to glance at wireshark/tshark output and be confused that all clients speak TLSv1.2.  TLS has versioning at the outter record layer and also in the inner handshake messages.  99% of the time, the inner  versioning is the important one.  But wireshark will often display TLSv1.2 on the outter layer even when it's transmitted as 0x30 x00 because of some early ambiguity in specs.  Net, don't look at the version in the Record twistie or the en...

On many systems, the backtick or $() operator causes a temporary file to be used to capture the output of the command before shoving it into the variable.  Often times these are by design only intending to capture and process one line (or even one word!) of output. Many of these uses can be refactored into reading from a pipeline: NEW_PATH=`foo|grep bar | ...`  ... $NEW_PATH ...  ... $NEW_PATH ...  ... $NEW_PATH ... becomes foo|grep bar | while read NEW_PATH; do    ... $NEW_PATH ...    ... $NEW_PATH ...    ... $NEW_PATH ... done On a degenerate, but not at all staged shell script that I looked at, this cut the execution time by 3x (which was important because it took in excess of 90 seconds to run)

There are few things more frustrating then waiting for large wireshark captures to download or filters to execute.  tshark, a utility included with wireshark, can help you very quickly slice and dice large captures to share with friends & family after you've found something interesting. For example, if you know the ephemeral port involved, create a new packet capture: tshark -r big.cap -Y "tcp.port eq 35979"  -w filtered.cap Or if you know that things really got interesting after some specific time (as copied from the time value in an Ethernet frame) tshark -Y '(frame.time >= "Oct 15, 2012 16:00:00") &&  f rame.time <= "Oct 15, 2012 17:00:00")'  -r big.cap -w filtered.cap A similar tool named editcap can be used to split a large capture into multiple time-based intervals: editcap -i 60 big.cap split.cap Note: When dealing with captures in a non-native format, I've found you have to explicitly add -F to sp...

If you have a problem where TCP connections fail or are timing out, and you don't know the source port, you can use Wiresharks statistics menu to quickly identify the suspiciously "short" TCP streams. After loading up your capture, select "statistics" from the top level menu then "conversations".  Click the "TCP" tab and you'll be able to sort by number of packets or bytes in the TCP conversation.   Once you have your list of streams with a low # of bytes or packets, you can right-click the row and apply as a filter to your normal capture window.   Don't be too discriminating about the streams you look deeper into via the filtered window! Because of retransmits, the conversation view may show 3 or 4 packets, and hundreds of bytes, for a single retransmitted SYN! Speaking of retransmits, another trick for debugging connection problems is directly looking for connection attempts that lead to retransmits.  This i...