Sneaky performance problems in shell scripts

On many systems, the backtick or $() operator causes a temporary file to be used to capture the output of the command before shoving it into the variable.  Often times these are by design only intending to capture and process one line (or even one word!) of output.

Many of these uses can be refactored into reading from a pipeline:

NEW_PATH=`foo|grep bar | ...`
 ... $NEW_PATH ...
 ... $NEW_PATH ...
 ... $NEW_PATH ...

becomes

foo|grep bar | while read NEW_PATH; do
   ... $NEW_PATH ...
   ... $NEW_PATH ...
   ... $NEW_PATH ...
done

On a degenerate, but not at all staged shell script that I looked at, this cut the execution time by 3x (which was important because it took in excess of 90 seconds to run)

Comments

Post a Comment

Popular posts from this blog

systemd unit for WebSphere Liberty (wlp)

This is the for-dummies cheatsheet for creating a basic WLP systemd unit. Note that the paths here use a separate WLP_USER_DIR directory specified in etc/server.env Step 1: Create /etc/systemd/system/wlp.service: [Unit] Description=wlp After=network-online.target Wants=network-online.target [Service] ExecStart=/home/covener/data/apps/wlp-bin/bin/server start ExecStop=/home/covener/data/apps/wlp-bin/bin/server stop User=covener Environment=JAVA_HOME=/home/covener/java Type=forking Restart = always PIDFile=/home/covener/data/apps/wlp-usr/servers/.pid/defaultServer.pid [Install] WantedBy=multi-user.target Step 2: Stop your server manually (systemd gets confused by a define service that was started outside of systemd) Step 3: Get systemctl to pick up the new unit: systemctl daemon-reload Step 4: Start server with systemctl: systemctl start wlp Step 5: Enable for starting on reboot: systemd enable wlp Other Hints: systemctl --no-...
Read more

Showing HTTP requests with no HTTP responses in wireshark

Large packet captures, with little corroborating information like a time or ephemeral port of interest, can be annoying to work with.  One old trick is to add a column that shows "http.time" which allows you to quickly look at the quickest and slowest TTFB transactions. One shortcoming to this sort is that it doesn't show you data for requests that had no HTTP response captured. This can either be due to a prolonged hang, or bad luck at the end of a capture.  To check for these directly, you can filter on ""http.request && !http.response_in" which will show you requests that never got a response (http.response_in is used internally by the dissector when piecing together http.time values)
Read more

openssl and atexit

TIL: - linux calls atexit() callbacks when a library is unloaded, not just at process exit - AIX doesn't - openssl 1.1 adds an atexit handler - openssl 1.1 has some code to try to prevent the library from really being unloaded by inflating the dlopen reference count so its atexit will really be available at exit. - https://github.com/openssl/openssl/pull/1693 - apparently this isn't working on AIX - openssl 1.1 doesn't seem to be provided by IBM or third parties on AIX as of early 2018.
Read more