Advanced Problem Determination

After an OS upgrade, my IBM Directory Server was hanging waiting for its underlying database to come up. db2start was returning the error above, which implied some other manager process should be started -- which is a red herring. A one-time db2ftok made db2start magically work. Maybe there was a usable error message somewhere? Maybe not.

This is the for-dummies cheatsheet for creating a basic WLP systemd unit. Note that the paths here use a separate WLP_USER_DIR directory specified in etc/server.env Step 1: Create /etc/systemd/system/wlp.service: [Unit] Description=wlp After=network-online.target Wants=network-online.target [Service] ExecStart=/home/covener/data/apps/wlp-bin/bin/server start ExecStop=/home/covener/data/apps/wlp-bin/bin/server stop User=covener Environment=JAVA_HOME=/home/covener/java Type=forking Restart = always PIDFile=/home/covener/data/apps/wlp-usr/servers/.pid/defaultServer.pid [Install] WantedBy=multi-user.target Step 2: Stop your server manually (systemd gets confused by a define service that was started outside of systemd) Step 3: Get systemctl to pick up the new unit: systemctl daemon-reload Step 4: Start server with systemctl: systemctl start wlp Step 5: Enable for starting on reboot: systemd enable wlp Other Hints: systemctl --no-...

I was staring at some tprof output today, which is not something I often look at.  It took me quite a while to realize that CPU usage in the "shared" column (contrast with "Kernel" and "User") simply meant CPU usage accumulated in shared library  code.

Large packet captures, with little corroborating information like a time or ephemeral port of interest, can be annoying to work with.  One old trick is to add a column that shows "http.time" which allows you to quickly look at the quickest and slowest TTFB transactions. One shortcoming to this sort is that it doesn't show you data for requests that had no HTTP response captured. This can either be due to a prolonged hang, or bad luck at the end of a capture.  To check for these directly, you can filter on ""http.request && !http.response_in" which will show you requests that never got a response (http.response_in is used internally by the dissector when piecing together http.time values)

In a recent adventure, I became interested in measuring CPU usage on AIX.  I don't have the benefit of professional help or third-party monitoring.  I also rarely trust either of these things when they're available, because either one can have trouble describing what's actually being measured. On any system, there are lots of pitfalls to consider when a laymen starts wondering about CPU usage. On AIX, there are some unique pitfalls. What's being measured? To determine the health of a system, you might quickly check the overall CPU usage to make sure it's within some tolerance. But once a system is unhealthy, you'll often quickly need to focus on individual process usage. Sometimes, the process-level statistics are presented in ways very different from the system-wide. Sometimes this is simply because different tools are used -- whether the calculations, meanings of fields, or just the sampling intervals are different. Something as simple as ...