Advanced Problem Determination

This is the for-dummies cheatsheet for creating a basic WLP systemd unit. Note that the paths here use a separate WLP_USER_DIR directory specified in etc/server.env Step 1: Create /etc/systemd/system/wlp.service: [Unit] Description=wlp After=network-online.target Wants=network-online.target [Service] ExecStart=/home/covener/data/apps/wlp-bin/bin/server start ExecStop=/home/covener/data/apps/wlp-bin/bin/server stop User=covener Environment=JAVA_HOME=/home/covener/java Type=forking Restart = always PIDFile=/home/covener/data/apps/wlp-usr/servers/.pid/defaultServer.pid [Install] WantedBy=multi-user.target Step 2: Stop your server manually (systemd gets confused by a define service that was started outside of systemd) Step 3: Get systemctl to pick up the new unit: systemctl daemon-reload Step 4: Start server with systemctl: systemctl start wlp Step 5: Enable for starting on reboot: systemd enable wlp Other Hints: systemctl --no-...

I was staring at some tprof output today, which is not something I often look at.  It took me quite a while to realize that CPU usage in the "shared" column (contrast with "Kernel" and "User") simply meant CPU usage accumulated in shared library  code.

Large packet captures, with little corroborating information like a time or ephemeral port of interest, can be annoying to work with.  One old trick is to add a column that shows "http.time" which allows you to quickly look at the quickest and slowest TTFB transactions. One shortcoming to this sort is that it doesn't show you data for requests that had no HTTP response captured. This can either be due to a prolonged hang, or bad luck at the end of a capture.  To check for these directly, you can filter on ""http.request && !http.response_in" which will show you requests that never got a response (http.response_in is used internally by the dissector when piecing together http.time values)

In a recent adventure, I became interested in measuring CPU usage on AIX.  I don't have the benefit of professional help or third-party monitoring.  I also rarely trust either of these things when they're available, because either one can have trouble describing what's actually being measured. On any system, there are lots of pitfalls to consider when a laymen starts wondering about CPU usage. On AIX, there are some unique pitfalls. What's being measured? To determine the health of a system, you might quickly check the overall CPU usage to make sure it's within some tolerance. But once a system is unhealthy, you'll often quickly need to focus on individual process usage. Sometimes, the process-level statistics are presented in ways very different from the system-wide. Sometimes this is simply because different tools are used -- whether the calculations, meanings of fields, or just the sampling intervals are different. Something as simple as ...

I posted previously about finding suspiciously short TCP connection to find leads for problem determination.    You can get the same TCP statistics from tshark, when file sizes are too unruly for transfer or loading in a GUI. tshark -r foo.cap -q -z conv,tcp The output is super-wide and not blogger.com friendly, but it includes TCP addresses, ports, bytes sent, and duration.

Hypothetical scenario is that a user has disabled TLS < 1.2 and has sporadic SSL handshake failures.  The suspected culprit is just a non-TLS 1.2 client, but people can be simultaneously skeptical and lazy. # Show TLSv1.2 client hellos tshark -Y "ssl.handshake.version == 0x303" -r ssl.cap  # Show TLSv1.1 client hellos tshark -Y "ssl.handshake.version == 0x302" -r ssl.cap  # Show TLSv1.0 client hellos tshark -Y "ssl.handshake.version == 0x301" -r ssl.cap  Caution!  It's easy to glance at wireshark/tshark output and be confused that all clients speak TLSv1.2.  TLS has versioning at the outter record layer and also in the inner handshake messages.  99% of the time, the inner  versioning is the important one.  But wireshark will often display TLSv1.2 on the outter layer even when it's transmitted as 0x30 x00 because of some early ambiguity in specs.  Net, don't look at the version in the Record twistie or the en...

On many systems, the backtick or $() operator causes a temporary file to be used to capture the output of the command before shoving it into the variable.  Often times these are by design only intending to capture and process one line (or even one word!) of output. Many of these uses can be refactored into reading from a pipeline: NEW_PATH=`foo|grep bar | ...`  ... $NEW_PATH ...  ... $NEW_PATH ...  ... $NEW_PATH ... becomes foo|grep bar | while read NEW_PATH; do    ... $NEW_PATH ...    ... $NEW_PATH ...    ... $NEW_PATH ... done On a degenerate, but not at all staged shell script that I looked at, this cut the execution time by 3x (which was important because it took in excess of 90 seconds to run)