dusybox 0.2.1

Description

Simple implementations of System Ultilities in Dlang. The primary purpose is to understand Dlang and to learn system programming.

• Getting started
• List of tools
• Why I learn Dlang
• My questions on Dlang forum
• Learning resources

List of tools

Tool	TODO	Examples	Description	Subjects
free			Display system memory	Struct. Function overloading
watch	TODO	Examples	Watch command output	ncurse. Launch shell command. Getopt
plotbar	TODO	Examples	Draw 2-d bar chat	Struct. Overloading. Testing
jq	TODO	Examples	Simple json tool	JSON parser

Getting started

To use any ultilities below, you need a dlang compiler and also the dub package manager. The compiler can be installed by your system package manager, for e.g,

$ pacman -S dmd   # on ArchLinux

To intsall dub please follow this link https://github.com/dlang/dub#installation. The latest ArchLinux database has dub in the official repository and you can also install them with pacman -S dub.

Now from the top directory of project, you can start testing / running any tool as below

$ dub test -d 2 dusybox:free
$ dub run dusybox:free

When being compiled, the tool's binary is located under ./bin/ directory. For example, ./bin/dzfree here dz is the common prefix for our tools (dz sounds a bit similar to dusy, isn't it?). The testing and other build profile still generate binary files under the top directory though.

The Makefile can help to run all tests and/or to build release versions of all tools. You can override the list of tools with help of TOOLS=:

$ make releases TOOLS=watch

free

Print information about system memory.

It's similar to the free command on your Linux system.

TODO

• [x] Support different Linux versions
• [x] Print various information in a single command
• [x] Print human-readable memory size

Examples

$ dub run dusybox:free
                total        user        free      shared  buff/cache   available
Mem (kB):    16337688     3181032     6025164      759692     7131492    12487640
Mem (mB):       15954        3106        5883         741        6964       12194
Mem (gB):          15           3           5           0           6          11
Mem  (%):      100.00       19.47       36.88        4.65       43.65       76.43

watch

Execute a shell command and print its output to the standard output device every one second. This is similar to the popular watch command.

TODO

• [x] Document the external requirement (e.g, libncursesw5-dev on Ubuntu-16.04)
• [x] Do not work inside screen. Work-around: TERM=tmux dzwatch. See also https://github.com/D-Programming-Deimos/ncurses/issues/35. To fix this issue, you need to rebuild the application on the target machine.
• [ ] Support -i to work with case-insensitive regular expression
• [x] Redirect output from stderr (This works out-of-the-box)
• [x] Print time information of the last interaction
• [x] Print basic information about input command and iterator number.
• [x] Wait 1 second after every execution. No more --interval 1 :)
• [x] Specify maxium number of executions with -n <number>
• [x] Fix problem with overflow output, e.g, generated by ps xauw. Wait for https://github.com/mpevnev/nice-curses/issues/2.
• [x] Exit if output matches some regular expression
• [x] Exit if user presses q or Q. Wait for https://github.com/mpevnev/nice-curses/issues/1.
• [x] Tool doesn't work with pipe commands, e.g, ps x | grep foo: it reports command not found error. As a work-around you can use bash -c "ps x | grep ff" instead.

Examples

$ dub run dusybox:watch -- free -m
$ dub run dusybox:watch -- ps x

$ dub run dusybox:watch -- -n 10 'ps xwa | grep "f[i]r"'
:: No 4/10, Cmd ["ps xwa | grep \"f[i]r\""]
15774 ?        SNsl   3:01 /usr/lib/firefox/firefox
15776 ?        ZN     0:00 [firefox] <defunct>

...
:: Reached maximum number of interation (2) at 2017-Sep-05 18:04:47.0811478.

Watch ElasticSearch cluster status and exit if cluster is green:

$ dzwatch -e '"status"\s+:\s+"green"' 'curl -s http://elk.example.net:9201/_cluster/health | json_pp'

plotbar

This tool is inspired by https://github.com/lebinh/goplot.

It visualizes your data as a simple bar chart. goplot draws relative bars (compare bar height to the highest bar), why this tool draws absolute bars (compare bar height to the sum of all bars).

The tool reads data from STDIN (the only source so far), and fetches every entry in format

key value

It will generate error messages to STDERR in case some line doesn't match the above format and/or their value is invalid.

In this version, tab (\t) is not support. You must use space(s) between key and value.

TODO

• [ ] Option to draw relative bars
• [ ] Continuous mode (keep drawing new bar while reading from stdin)
• [x] Support tab delimeter in key value line
• [ ] Support negative data (2-direction bar chart)
• [x] Display actual value after the bar
• [x] Set the minium percent number to display (-m min)
• [ ] Display last n items (like sort | tail)
• [ ] Sort the output (if the input is sorted)
• [x] Additive mode (Sum of duplicated items)
• [x] Fix bug when parsing input data (previous value is reused.)
• [x] Move common part to a library file
• [ ] Avoid overflow (when input key is too long, and/or the bar is too high)
• [ ] Use gnuplot instead?

Examples

Find the biggest folder items, display ones consume great than 2% of total storage. (The idea for this example comes from https://github.com/lebinh/goplot.) Please note that you can't use \t character in this example: The input parser doesn't understand tab.

$ dub run dusybox:plotbar -- -m 2 < <(2>/dev/null du -s /home/* | awk '{printf("%s %s\n", $2, $1)}')

/home/pi.fast :  9 % ========= (9466072)
     /home/pi : 13 % ============= (14541032)
 /home/btsync : 64 % ================================================================ (69425660)
  /home/ebook :  8 % ======== (8600288)
 /home/backup :  2 % == (2615004)

Display the ElasticSearch indices the have most documents. Skip all indices that consumes less than 2% in the total number of documents.

$ curl -s 'elk.example.net:9201/_cat/indices?h=index,docs.count' | ./bin/dzplotbar -m 2

           aws-lambda-test-uat-test-20170824 :  9 % ========= (4986415)
api-gateway-execution-logs-test-uat-20170824 :  4 % ==== (2486179)
           aws-lambda-test-uat-test-20170824 :  2 % == (1177304)
           aws-lambda-test-dev-test-20170815 :  4 % ==== (2227446)

Display the biggest indexes (in stored size):

$ curl -s 'elk.example.net:9201/_cat/indices?h=index,store.size&bytes=k' | ./bin/dzplotbar -m 2

aws-lambda-test-uat-test-20170824 :  2 % == (2847921)
                     emr-20170904 :  2 % == (3364511)
aws-lambda-test-uat-test-20170824 :  4 % ==== (5544297)
aws-lambda-test-uat-test-20170821 :  2 % == (2853427)

Now find the biggest source (by discarding date suffixes):

$ curl -s 'elk.example.net:9201/_cat/indices?h=index,store.size&bytes=k' \
  | sed -re 's#-[0-9]{8}##g' \
  | ./bin/dzplotbar -m 5 2>/dev/null

  aws-lambda-test-uat-test :  5 % ===== (3145751)
                       emr : 11 % =========== (6974423)
 aws-lambda-test-uat-test2 : 11 % =========== (6622399)
cloudtrail-defaultloggroup : 11 % =========== (6726637)

Find the package that has most files on ArchLinux system

$ pacman -Ql | grep -vE '/$' | awk '{printf("%s 1\n", $1 );}' | ./bin/dzplotbar -m 2
            evince :  2 % == (3058)
           efl-git :  2 % == (3563)
           python2 :  3 % === (4646)
adwaita-icon-theme :  4 % ==== (5426)
              mono :  2 % == (2443)
             linux :  3 % === (3984)
     linux-headers :  9 % ========= (12296)
            python :  5 % ===== (6784)
               ghc :  4 % ==== (5728)
 claws-mail-themes :  3 % === (4689)
           openssl :  2 % == (3252)
               qt4 :  3 % === (3825)
              perl :  2 % == (2393)
            libxcb :  2 % == (2371)
           ncurses :  3 % === (3678)
             cmake :  2 % == (2267)
         man-pages :  2 % == (3491)
               gcc :  2 % == (2198)

Find the biggest packages on ArchLinux system

$ pacman -Qi | awk '
    /Name/ {printf("%s", $NF);}
    /Installed Size.+KiB/ {printf(" %s\n", $(NF-1))}
    /Installed Size.+MiB/ {printf(" %s\n", $(NF-1) * 1024)}' \
  | ./dzplotbar  -m 2
          mono :   4 % ==== (199782)
    ghc-static :  17 % ================= (843428)
       firefox :   3 % === (143370)
linux-firmware :   4 % ==== (206377)
      chromium :   4 % ==== (212572)
        python :   3 % === (131430)
           ghc :   8 % ======== (425339)
           gcc :   2 % == (119081)

jq

This is not https://github.com/stedolan/jq.

Instead, this tool reads line from STDIN and considers each line as a JSON string. This is useful as I need to process multiple JSON lines from nginx and/or ELK system.

If input line can be parsed, the result will be printed to stdout (if the tool has not any argument), or each item from arguments is looked up in the final JSON object. If the argument is

./bin/dzjq .foo bar

then the .foo is used as a lookup key, while bar is printed literally. If the program fails to query a key .foo, it prints foo instead.

TODO

• [ ] Handle delimeter
• [ ] Handle formatted string
• [ ] Handle object other than integer and/or string
• [ ] Nested key query
• [ ] Advanced query with array support
• [x] Move common methods to a library file
• [ ] Option to flush STDOUT buffer on every processed input line
• [x] Add unit tests
• [x] Literraly support
• [x] Process lines from STDIN as invidual documents. See also https://github.com/stedolan/jq/issues/744.

Examples

Print key .a and .b, print 1 literally.

$ echo '{"a": 9, "b": {"c": 1}}' | dub run dusybox:jq -- .a 1 .b
9       1       {"c":1}

Print the original JSON string

$ echo '{"a": 9, "b": {"c": 1}}' | dub run dusybox:jq --
'{"a": 9, "b": {"c": 1}}'

Generate simple statistics from nginx access log file. The format of log file is similar to this one.

$ dub run dusybox:jq -- .host 1 < /home/pi/df/acces.log | ./bin/dzplotbar -m 2
     kibana.int.example.net : 25 % ========================= (269)
    airflow.dev.example.net :  3 % === (33)
    grafana.int.example.net : 70 % ====================================================================== (755)
airflow.staging.example.net :  3 % === (28)

How about the requests or statuses?

$ dub run dusybox:jq -- .request_uri 1 < /home/pi/df/acces.log | ./bin/dzplotbar -m 2
/api/console/proxy?path=_aliases&method=GET :  4 % ==== (44)
/api/console/proxy?path=_mapping&method=GET :  4 % ==== (44)
                  /api/datasources/proxy/16 : 34 % ================================== (364)
                  /api/datasources/proxy/14 : 12 % ============ (132)
                  /api/datasources/proxy/13 :  5 % ===== (55)
                    /elasticsearch/_msearch :  4 % ==== (40)
                  /api/datasources/proxy/12 : 11 % =========== (122)

$ dub run dusybox:jq -- .status 1 < /home/pi/df/acces.log | ./bin/dzplotbar -m 2
200 : 93 % ============================================================================================= (1013)
304 :  4 % ==== (43)

Why I learn Dlang

1. To learn something cool
2. To design some maintainable DSL for my jobs
3. To teach my daugher some programming skills ;)
4. Some Dlang documentation is just awesome. See for example https://github.com/PhilippeSigaud/Pegged/wiki.

My questions on Dlang forum

1. How to list all process directories under /proc/: Problem with character range when using Dlang glob pattern. The current implementation of std_file doesn't support popular range, e.g, [a-z] or [0-9]
2. formattedRead can't work with tab delimeter input: For some reason, the function doesn't work if there is only tab in input string. If there is at least one space, then it works. Tab was not considered as a space all the time.
3. How to skip some field/word in formattedRead: Ruby has a phantom symbol _ to discard unwanted item. We can do almost the same in Dlang? At least there is a work-around.
4. Convert user input string to Regex: That's so trivial in Dlang. However, I'm not sure if that's safe.
5. Problem with std.string.strip(): Can't use result in format routine: It's important to remember that formattedRead consumes the input range, hence you can't use the consumed range for later processing.

Some other issues (not in Dlang forum):

1. Exception with nice-curses: Problem with handling overflow window.
2. Timeout feature in nice-curses: Timeout feature to work with user input
3. Error opening terminal: screen: Actual problem is library incompatibility.
4. dub: Fix anchor link in README.md when viewing package info

Learning resources

1. Ali Çehreli - Programming in D: This is a good book for Dlang newbies. There are some very basic introductions to different problems (unicode, oop, concurrency). Well, basic doesn't mean simplicity and that's a good start if you have some basic programming experience in other language I think.
2. Dlang Learning Forum: A good place to ask any question about Dlang :) The community is helpful. However, the forum may be a bit noisy and it is so basic; it's good to know that the forum was running by a Dlang application.
3. Introduction to PEG: This is an impressive documentation about PEG (you already know (e)BNF, do you?) The most useful documentation I've ever read :)