Usage

FlhCli is command line utility with behavior dictated by the internal command name specified on the command line. These commands can have different parameters to alter their behavior.

FortiGate where the outputs are collected from is specified in the flhcli.xml config file. That file also contains the SSH credentails and few other parameters.

There are several parameter groups that can be specified on different positions. For example:

flhcli --parameterA --parameterB valueB command --parameterC valueC --parameterD

• flhcli Program name.
• --parameterA Global "switch" that is applied to all commands.
• --parameterB valueB Another parameter with specified value.
• command Name of the flhcli command.
• --parameterC valueC Similar to --parameterB but this one is specific for the command.
• --parameterD Similar to --parameterA but again specific to the command in use.

Exceptionally there can be also commands with subcommands where the commands parameters's are separated from subcommand's parameters. At this moment there is only one command like this and it is the cmd command.

Global parameters

Global paramaters configure the behavior that is not related to any command. These parameters follow immediately after the binary name and all of them must be specified before the first command.

The commonly used global parameters are:

• --name Selects the right FortiGate configuration from the config file. If not specified, the first configuration found in the file is used.
• --copy Output displayed on terminal is also saved to the specified text file. Which is never overwritten and outputs are always appended to its end.
• --redirect Similar to the --copy parameter above, but the commands output is no longer displayed

When saving the output to text file with --copy or --redirect parameters, you can also add --prepend-time or --prepend-timestamp to include the time information with each line in the file.

The output file can be "rotated" when it reaches certain size in MiB with --rotate-size option (example value 10 means ten MiB) or when FlhCli uses the same file for certain time with --rotate-age (example value 24h means one day; available multiplicators are s for seconds, m for minutes and h for hours).

Be aware that both rotation options are only approximate and the real file can get slightly bigger and older. Also it can happen that the file is rotated in the middle of a command output (but not in the middle of a line). The rotated filename will contain the time range when it was collected.

For complete list of available global parameters run flhcli -h.

Config file

The default config file is flhcli.xml in the current directory, but different can be specified using --config global parameter.

It is suggested to first create a config file using setup command and then manually edit/add devices as necessary. The simplest config file can look as following:

<flhcli>
   <device name="test">
      <host>10.100.10.200</host>
      <port>22</port>
      <auth type="password">
         <username>admin</username>
         <password></password>
      </auth>
      <transcripts>/tmp/flh</transcripts>
      <connections>4</connections>
   </device>
</flhcli>

This specifies device named "test" with SSH listening on IP address 10.100.10.200 and port 22. To access the device over SSH user "admin" is used without any password. Session transcripts are stored in "/tmp/flh" directory and if necessary, the utility can create up to four parallel SSH connections.

Password can be either plain text or encrypted. To change the encrypted password for this device use flhcli setup --change-password test command.

There can be unlimited amount of devices in config file (provided that they have different names) and the global parameter --name can be used to select the right device.

Commands

Command name follows the last global parameter and it is the main part of the command deciding what exactly will be done.

Each command is completely independent on the other commands and has its own parameters (yet I try to use the same names for parameters with the same behavior in different commands). These parameters are all specified after the command name.

Parameters common for most commands

• --interval Specifies how often the screen is refreshed. For most commands it also specifies the time difference between collecting the counters. Longer intervals smooths the numbers but also filter out short time peeks. Shorter intervals can better catch the peeks, but the numbers may differ a lot from one refresh to another. Interval needs to be specified in units, for example 5 seconds is written as 5s, one minute and thirty seconds is written as 1m30s (or 90s).
• --iterations By default the output is refreshed every --interval until user stops the program with CTRL+c. If this option is bigger than zero, then only specified amount of refreshes is done. This is mainly useful if running it from some script.
• --max Maximum number of lines displayed at the same time. This is usually not counting the headers and summary lines.

Minimum packets to show

In many commands it is possible to display only lines with certain amount of packets. This is useful for example in interfaces command where a lot of interfaces (like "mgmt") can have link but the throughput is minimal and uninteresting.

For relevant commands, following parameters usually exist:

• -z Hide interfaces/counters with 0 packets.
• -zz Hide interfaces/counters with less than 50 packets.
• -zzz 12345 Specify the limit explicitly.

Selecting throughput units

In many commands there is some kind of throughput counter. The application by default selects the most appropriate units for each throughput value, but it is possible to alter this behavior using --units parameter. To get the list of possible options use something like flhcli interfaces --units help.

The for automatic decision there are two possible values: autoD (default) and autoB. They are more or less the same with only one exception: autoD uses 1000 as multiplier and autoB uses 1024. For example, in autoD 1 Mbps equals to 1000 Kbps, but in autoB 1 Mibps equals to 1024 Kibps (notice the extra "i" that is also displayed).

All other options used in --units parameter disable the automatic units selection and use the specified units for all counters instead. Be careful about differencies in upper/lower case and the "i":

• Kbps "kilo-bits-per-second" - 1 kilobit has 1000 bits and 8 kilobits makes one kilobyte.
• KBps "kilo-byte-per-second" - 1 kilobyte has 1000 bytes or 8000 bits.
• Kibps "kibi-bit-per-second" - 1 kibibit has 1024 bits and 8 kibibits makes one kibibyte.
• KiBps "kibi-byte-per-second" - 1 kibibyte has 1024 bytes or 8192 bits.

When using automatic units, the throughput is shown as a number with two decimal places. This is usually good enough, but sometimes the higher precision might be necessary. For that there usually is -p parameter which keeps using the automatic units, but uses a smaller units than would be used normally. For example, if the units used without any parameters would be Mbps, then the units with -p parameter are Kbps. This is done independently on every throughput value in the output.

cputop

Similar to diagnose sys top command from FortiGate CLI but displays also system processes, can distuinquish between the CPU spent in user and kernel mode, shows the last CPU the program was running on and how long it has been running. It also has much more advanced filters than the built‑in CLI command.

Columns description

• Columns PROCESS, PID show the process name and its ID.
• Columns PPID, PARENT show the same information about the process that started this one (i.e. the "parent" process).
• Column called S is a shortcut for state of process - it can be sleeping S which means the process doesn't have anything to do, running/runable R for active proceses, waiting for IO D most commonly hard disk, or zombie Z which signalizes some problem in process handling.
• NICE means priority and the number is between -19 and +20. Value of -19 means the highest possible priority but it still means that process gets more CPU time and not that it would run exclusively.
• Column TOTAL% is a summary of columns USER% and SYSTEM% that can can distinguish between the CPU time spent by running the code of the process (USER%) and time spent handling the system calls from the process in the kernel (SYSTEM%). However, please see the paragraph below because this can be quite tricky.
• CPU is the last CPU (core) the process was running on. There are some processes in FortiOS that are bound to one specific core and those will always show the same CPU (example of such process would be ipsengine). However, most of the processes do not have a specific CPU assigned and they can run anywhere the system decides to - for those processes the CPU might be changing with every refresh.
• The last column RUNNING FOR shows how long ago the current process was started. FortiOS automatically restarts the process if it fails for any reason, but the restarted process will get new PID and column will reset as well.

Summary section

At the bottom there is a summary for other processes that are not displayed because of the maximum lines on screen limit (--max) and the total summary. Processes not matching the command line filters (see below) are not counted.

Processes sshd and newcli are excluded from regular displaying and are only shown in the summary section because these processes are utilized by the cputop utility itself and the CPU used by them is not really the CPU spent by running the FortiOS functions.

System vs. user CPU

Each running process can be running either in user mode executing the process'es own code in which case it will be accounted in USER% column, or it can be waiting for FortiOS kernel to do something the process does not have right to do on its own, which will be accounted in SYSTEM% column.

However, the SYSTEM% column may sometimes be completely wrong. This happens because any incoming interrupt (hardware or software) will stop the code running on the same CPU and execute the interrupt handler code in the kernel. If there was a process running on this CPU, the time spent handling the interrupt will be improperly accounted to the SYSTEM% column of that process.

For that reason it can be problematic to rely on the system CPU utilization reported for the process. And since the TOTAL% is just a summary of USER% and SYSTEM%, it is affected by the same problem. (And because this column should match the CPU utilization reported by FortiGate CLI command diagnose sys top, it also means that the offical command is affected by the same problem). Under these circumstances the USER% column is the only one that can be trusted.

Parameters and filters

Except the common parameters used by many commands (like --interval, --iterations and --max - see the Commands section) following parameters are available:

• --no-system Hide the kernel threads.
• --no-user Hide the regular processes.
• --pid Show only processes with specified PIDs - it can be used multiple times or interval can be used. (like --pid 20-29).
• --name Regular expression on process name.
• --parent Regular expression on parent process name.
• --state Regular expression on process state. For example S|R to see sleeping and running processes.
• --cpu Show only processes running on the specified CPU(s) - it can be used more times to specify multiple CPUs, or interval can be used (like --cpu 0-19).
• --sort-by-time Sort by time the process is running - from youngest. To display from oldest processes first use also --sort-reverse.
• --min-age or --max-age Filter processes by their age.

cpuprofiler

When the FortiGate spends a lot of time running the kernel code (high percentage in system, irq, softirq) this command shows kernel functions the time is spent in. The function really is the name of the function in the source code and while meaning of a lot of them is quite obvious, it might be hard to be sure what exactly they do without access to the source code. This output can be quite important for developers.

This command is based on FortiOS profiler CLI command, that works in a way that it checks the name of currently executing kernel (!) function one hundred times per second. For more utilized systems this may not be very precise because there can be more functions executed during these 10 ms. Also if the CPU is not running in the kernel mode when the function name is checked, none is counted. Nevertheless this is still good enough to identify the function(s) the system is spending most time in.

There is special function called "poll_idle" which is executed when the CPU is idle. When the CPU (one core) if completely idle, the output would show just one line saying "poll_idle" with ~100 hits per second. This is multiplied by the number of cores in the system - when profiling fully idle system with 40 cores (and not limiting the profiler to specific core) the "poll_idle" would be close to 4,000 hits per second.

In the example above the profiler was running only for one specific CPU core (#18). In the second part of the output there is the approximate CPU utilization for that core for the last interval. In this example the CPU spent in the system (which is only measurable by CPU profiler) was 14 %, therefore the summary of cpuprofiler hits for this interval without "poll_idle" should be around 14. The actual summary is 3+3*2+6*1 = 15 (which is also on the line "not idle summary") which is expected because of the inaccuracies in measuring this over SSH connection.

And in the RATIO % column it can be seen that out of 14 % of CPU spent in kernel, the "nturbo_intf_user_recv_pkts" function consumed around 20 %. Time spent in "poll_idle" is not taken into account for the RATIO % column. In absolute numbers this function consumed about 2.8 % of this core (0.14*0.20).

Parameters and filters

• --cpu The CPU (core) to profile. It can be specified multiple times and/or the range like --cpu 0-19 can be used. If not specified, all CPUs in the system are profiled.

crashlog

Follows the crashlog from FortiGate. In single session (i.e. without constant logins and logouts) it shows only the new entries that appeared during the last --interval. By default it doesn't show lines that were already present when this command was started, but that can be changed with --check-old parameter.

It can be run without any additional parameters to only show changes, but its true power is in searching the content using regular expression and executing arbitrary command when match is found.

Monitoring crashlog for specific content

When parameter --search with valid regular expression is used, all matched parts will be displayed in red color. Example usage is to locate wad crashes with regular expression (?m)^.*?> application wad\s*\n.*? \*\*\* signal [0-9]+ .*? received \*\*\*.

The expression is quite simple, only the (?m) at the beginning might be confusing - it means to allow ^ and $ to match on every new line or end of line respectivelly, instead of only on the first and last one.

Another useful flags are: (?i) to make searching case-insensitive, (?s) to allow . to match also new line, or their combination (like (?mi)). For complete supported syntax, check this regexp documentation.

Acting when specific content is found

Parameter --execute can be specified to run arbitrary command when content is matched with --search parameter. The command can have its own command line options all of which must be passed as single parameter and it is parsed internally using generic shell rules (i.e. respecting spaces, quotation marks, etc. but not expanding any variables).

When the specified command is started, the matched content (the red part on screen) is passed to the command on its standard input. If command is not interested, it can ignore it.

Command is executed synchronously (i.e. FlhCli waits for it to finish before continuing) and it can be running for the time specified by --execute-timeout parameter. If it does not finish in that time, it is killed.

If the command prints anything on its standard output and terminates on its own and without any error (error code 0), the output is shown on terminal.

To prevent "command bursts" (for example when multiple processes crash at the time time), the command is executed only once within the holddown period configured with --execute-hold parameter.

Example

One of the use cases can be to monitor for wad crashes and when that happens to restart whole wad nicely using FortiOS command.

For that you would combine the --search string shown above with --execute parameter which can start another instance of FlhCli with command to simply run diagnose test app wad 99. The full crashlog command might look like this:

Linux/MacOS

./flhcli -config flhcli.xml crashlog \
    -search '(?m)^.*?> application wad\s*\n.*? \*\*\* signal [0-9]+ .*? received \*\*\*' \
    -execute './flhcli -force-pty -config flhcli.xml cmd -iterations 1 -human direct "diag deb enable" "diag test app wad 99"'

Windows

flhcli -config flhcli.xml crashlog ^
    -search "(?m)^.*?> application wad\s*\n.*? \*\*\* signal [0-9]+ .*? received \*\*\*" ^
    -execute "flhcli -force-pty -config flhcli.xml cmd -iterations 1 -human direct 'diag deb enable' 'diag test app wad 99'"

With following result:

frags

Display the number of fragmented packets coming to FortiGate together with reassembly errors and the number of packets that needed to be fragmented by FortiGate on their way out.

Columns description

• TIME Time when the numbers were calculated.

Section "INCOMING":

• FRAGMENTS Number for fragments (packets pieces) recevied.
• FULLPACKETS Number of full packets reasembed on FortiGate.
• TIMEOUT Timeouts occured when reassembling packets - that means some packet(s) was lost.
• ERRORS Total errors when reassembling packets (includes TIMEOUT).

Section "OUTGOING":

• FULLPACKETS Number of full packets that had to be fragmented by FortiGate.
• FRAGMENTS Number for fragments (packets pieces) created.
• FAILURES Number of packets that needed to be fragmented but weren't (can be because of DF flag).

Parameters and filters

• --headers After how many lines to display headers again. Set to zero to disable showing headers in the output (they are still printed at the beginning).
• --no-divide Do not calculate per-second counts. If the interval to is set to 5 seconds, then the number of fragments is displayed for these 5 seconds.
• --batch Display the output more suitable for programmatic processing. Does not display any headers or formating.

hwinfo

Dumps some hardware information into an XML file. This file can be later converted to nice HTML page using auxiliary command aux hwinfo. Currently following information is dumped:

• Total available RAM
• Fortinet chips connected to PCI(-X) (with no additional information at this moment):
  • NP4, NP6, NP7
  • CP7, CP8, CP9
• Available processors with their sockets and cores numbers + data about processor vendor model and frequency. Also for each processor shows FortiOS functions statically bound to it. Following functions are recognized at this moment:
  • IRQ handling for slow path traffic between NP6 and FortiOS
  • IRQ handling for nTurbo traffic between NP6 and nTurbo driver(s) in FortiOS
  • IRQ handling for nTurbo traffic between nTurbo driver(s) and IPSengines

Parameters

Optional parameters:

• --output Name of the file to save the XML into. If not specified it will be printed on standard output.

Example

<?xml version="1.0" encoding="UTF-8"?>
<hardwareInfo v="1" model="1500D" partnumber="P12917-05" version="6.2.3" build="1066">
    <cpus>
        <sockets>1</sockets>
        <processors>12</processors>
        <cpu id="0">
            <socket>0</socket>
            <core>0</core>
            <thread>0</thread>
            <family>6</family>
            <mhz>3200</mhz>
            <model>45</model>
            <vendor>GenuineIntel</vendor>
            <modelname>Intel(R) Xeon(R) CPU E5-1650 0 @ 3.20GHz</modelname>
            <functions>
                <np6-fortios>
                    <np6>0</np6>
                    <queue>0</queue>
                </np6-fortios>
                <np6-fortios>
                    <np6>1</np6>
                    <queue>0</queue>
                </np6-fortios>
                <nturbo-ipse>
                    <driver>0</driver>
                    <queue>0</queue>
                </nturbo-ipse>
            </functions>
        </cpu>
        <cpu id="1">
            <socket>0</socket>
            <core>0</core>
            <thread>1</thread>
            <family>6</family>
            <mhz>3200</mhz>
[...]
        <cpu id="10">
            <socket>0</socket>
            <core>5</core>
            <thread>0</thread>
            <family>6</family>
            <mhz>3200</mhz>
            <model>45</model>
            <vendor>GenuineIntel</vendor>
            <modelname>Intel(R) Xeon(R) CPU E5-1650 0 @ 3.20GHz</modelname>
            <functions>
                <np6-fortios>
                    <np6>0</np6>
                    <queue>5</queue>
                </np6-fortios>
                <np6-fortios>
                    <np6>1</np6>
                    <queue>5</queue>
                </np6-fortios>
                <nturbo-ipse>
                    <driver>1</driver>
                    <queue>3</queue>
                </nturbo-ipse>
            </functions>
        </cpu>
        <cpu id="11">
            <socket>0</socket>
            <core>5</core>
            <thread>1</thread>
            <family>6</family>
            <mhz>3200</mhz>
            <model>45</model>
            <vendor>GenuineIntel</vendor>
            <modelname>Intel(R) Xeon(R) CPU E5-1650 0 @ 3.20GHz</modelname>
            <functions>
                <nturbo-ipse>
                    <driver>1</driver>
                    <queue>4</queue>
                </nturbo-ipse>
            </functions>
        </cpu>
    </cpus>
    <ram>
        <total>16843980800</total>
    </ram>
    <fortichips>
        <np6 id="0"></np6>
        <np6 id="1"></np6>
        <cp8></cp8>
        <cp8></cp8>
    </fortichips>
</hardwareInfo>

interfaces

This command can display traffic received and sent on physical interfaces (VLANs are not supported in this utility). The numbers can be taken from ISF (if present) and/or from FortiOS kernel (depending on given parameters). This is useful to see how much traffic is really managed by the physical interface and what part of it is not offloaded and therefore appears also in the FortiOS kernel.

By default only counters from ISF are displayed together with offload percentage. With parameters --show-host-rx, --show-host-tx or -show-host more sections can be added to see the similar counters on "host interface" (kernel). Also the default sections can be removed with --no-isf-rx, --no-isf-tx or --no-isf for both directions.

Only interfaces that are up and have link are displayed.

Be careful on VMs and generally on devices without ISF - because by default only ISF counters are displayed, you will see all zeros on these types of devices. In such case following parameters are suggested --show-host --no-isf --no-offload, see below for their explanation.

With -a parameter (as "advanced") more details are displayed, like "unicast", "broadcast" and "multicast" counters (seperated where this information is available), calculated link overhead and offloading percentage also for packets.

Columns description

Column names are quite self-explaning and only following might need more details:

• D/SPEED Means "duplex" and "speed" and these are the values actually used on the link.
• #u+b+m Summary for unicast, broadcast and multicast packets.
• Offld % Percentage of offloaded traffic. With no parameters this is calculated based on the number of packets. If -a parameter was used, columns RXp RXb TXp TXb are displayed instead and they show offloading percentage based on received RX?, transfered RX? packets ??p or bytes ??b.
• phy layer overhead Shown only in advanced mode and means calculated overhead in addition to the packet data itself. It is 20 additional bytes per packet (preamble, start of frame and interpacket gap).

Parameters and filters

• --name Show only interface with names matching this regular expression.
• --no-isf Hide the ISF counters. Alternatively --no-isf-rx or --no-isf-tx can be used to hide only one direction.
• --show-host Show host counters (i.e. not offloaded traffic). Alternatively --show-host-rx- or --show-host-tx can be used to show only one direction.
• --show-direct Show direct port statistics from NP6 NIC. It is not clear (to me) what exactly this counter shows - most probably it is pulled from NP6 LIF. The counters do not appear to be completely real‑time, which is causing small inaccuracy. Avoid using this option unless the front port is connected directly to NP6 without ISF. There are also sub-options --show-direct-rx and --show-direct-tx to enable only one direction.
• --show-xaui-rx For packets incoming on interface show also counters collected on the link between ISF and NP6 for this interface. Be careful as the XAUI lane is usually shared with multiple different interfaces and these counters count the whole traffic on XAUI not just the one from the interface shown on the line. Also there are no --show-xaui-rx or --show-xaui parameters because packets use target's interface XAUI for egress which is probably different from the incoming one, hence the numbers would be meaningless and confusing when shown on the line of incoming interface.
• --no-offload Do not show the offload statistics. This is useful to save a space on narrow terminals.
• --no-vlinks Ignore the NP6 link interfaces. They are always up therefore they are displayed by default.
• --units -p Select the units to use when displaying traffic throughput. See the Selecting throughput units section for more details.
• -z, -zz, -zzz Ignore interfaces that are up but do not transmit many packets. See the Minimum packets section for more details.

inttop

Shows the number of hardware and software interrupts handled per second. By default it shows interrupts on all CPU cores, but it is also possible to filter specific core(s) when investigating high "irq" or "softirq" utilization.

By default it shows separate line for each CPU the interrupt was processed by (together with CPU utilization of that core during the last interval), but with --summarize parameter it can also show the total handler executions over all the CPU cores.

When CPU utilization is displayed, it also colorizes the lines based on the CPU load.

Columns description

• LINE Identifies the type of interrupt. For hardware interrupts it is generally the interrupt number. For software interrupts it is the name of softirq.
• SOURCE Specifies whether the interrupt is hardware (H) or software (S).
• CPU Number of CPU core handling this interrupt. While it is generally possible that multiple interrupts of the same type are handled by different cores, in FortiOS most of the important interrupts are statically bound to one specific core.
• RUNS Average number of handler executions per second during the last interval. This can be modified with --no-divide and/or --from-start parameters (see below).
• DESCRIPTION More "human‑readable" information about the interrupt.
• USR SYS NCE IRQ SWI IOW STL IDL The average CPU utilization of the same CPU core the interrupt is handled on. Shortcuts mean "user", "system", "nice", "irq", "softirq", "iowait", "steal" (for VMs) and "idle" respectively. This part of the output can be disabled with --no-cpu parameter.

Hardware vs. software interrupts

Hardware in peripheral buses (like NP6, CPx, etc.) generate hardware interrupts when they have something to tell to the CPU. Nowadays these interruts are usually MSI-X based (in-band data messages instead of pulling the physical interrupt lane) but they are still counted as hardware interrupts in FortiOS.

Handling the hardware interrupts is extremely sensitive task, because when a hardware interrupt is being serviced, no other hardware interrupt can be received and since the hardware interrupts have the uttermost priority, rest of the system might experience delays when it takes too long to service it.

For this reason there are also software interrupts aka softirqs. Those are started but kernel (usually after the hardware interrupt is acknowledged, but can be also delayed or run independently). They have slightly lower priority and run with hardware interrupts enabled, therefore they can also perform more complex tasks.

Generally, for relatively small packet rate, when hardware interrupt is triggered, it only schedules software interrupt and terminates. That is the reason why we usually don't see high CPU utilization in "irq" but we do see it in "softirq". For higher packet rates, the hardware interrupts for incoming packets can be minimal and the packets can be polled from constantly running softirq rutine.

Parameters and filters

• --no-hard Do not display hardware interrupts.
• --no-soft Do not display software interrupts.
• --cpu Show only handlers executed on the specified CPU(s). Can be used multiple time or with a range.
• --no-cpu Do not display average CPU utilization for the interrupt core.
• --summarize Summarize the handlers executed on all CPU cores for the same interrupt.
• --desc Regular expression to show only specific interrupts based on the description column.
• --ndesc Regular expression similar to --desc but showing only non-matching interrupts.
• --no-divide Do not calculate per-second executions. If the interval to collect interrupts is set to 5 seconds, then the number of hits is displayed for these 5 seconds. Together with --from-start parameter it can display the number of interrupt handlers executed since the program started.
• --from-start Without --no-divide parameter this calculates the hit counts from the very beginning of the program. It is still avereged to get per-second hits but the interval the per-second hits are calculated is getting longer and longer. This can smooth the numbers when the load on the unit is constant but probably is not really useful alone.

ipseccpu

When IPSec decryption is handled in software (on VMs or generally tunnels that cannot be offloaded), it is happening on CPU assigned to the interface (queue actually) where data is coming from. Similarly, encryption is done on the CPU assigned to interface (queue) the plan-text data is coming from.

This command shows how CPUs are utilized with IPSec encryption and decryption. At the same time it shows overall CPU utilization on the same line to allow relating the IPSec handling part to whole system utilization.

In the first section packets handled by cpu all counters are related to the CPU id shown in first column. Second section percentage of all cpus [%] shows what is the percentage of all the packets (for specific counter) handled by FortiOS that the specic CPU is handling. Last section overall cpu utilization [%] shows global CPU utilization and it is not specific (only) to IPSec handling.

Line colorization is based on overall CPU utilization unless --no-cpu-util parameter is used. In that case colors are based on the percentage of packets handled by each CPU.

Parameters and filters

• --cpu Specific CPU(s) to investigate. It can be used multiple times and/or range (like 3-6) can be used. If not specified, all CPUs are displayed.
• --sort-by-enc Sort by enc counter instead of by CPU number.
• --sort-by-dec-in Sort by dec-in counter instead of by CPU number.
• --sort-by-dec Sort by dec counter instead of by CPU number.
• --sort-by-dec-out Sort by dec-out counter instead of by CPU number.
• --show-total Show extra line with summary of packets for each counter over all CPUs.
• --no-cpu-util Omit section overall cpu utilization [%] (colorization rules change as described above).

ipsectop

This command displays bandwidth statistics for IPSec tunnels configured on device. Because real traffic flows via "phase2" tunnels, it is what this command is based on. By default it shows all tunnels that are up (phase2 established) even if there is no traffic flowing through. This can be adjusted with -z parameter to ignore tunnels with no traffic.

Traffic statistics is unusable for tunnels offloaded to hardware, because in that case the counters in FortiOS are not updated. Such tunnels are shown in blue color by default and can be completely filtered out with --no-offload parameter.

Default view with no special parameters shows packets/s and bytes/s for each direction (where RX means incoming traffic that was decrypted by FortiOS and TX outgoing traffic encrypted by FortiOS) for all established tunnels in all VLANs. Tunnels can be filtered by various parameters as decribed in section Filtering interesting tunnels below.

Offload column describes whether the traffic if partially (rx-only/tx-only), fully (both) or not at all (no) offloaded. Be aware that counters only make sense for not-offloaded traffic (can be achieved by setting npu-offload disable in phase1 configuration).

Two special lines at the end (unless disabled with --no-total parameter) show summary. Line SUM shows summary of all tunnels that are displayed above. Line ALL shows summary of all tunnels found, even if some were filtered out of view.

Advanced views can be enabled by combining parameters --show-phase1-sum, --show-phase1, --show-percentage, --show-selectors. They are described below in section Advanced views.

Filtering interesting tunnels

• -z Hide tunnels with 0 Rx and 0 Tx packets during the last period.
• -zz Similar to -z but with threshold of 50 packets.
• -zzz Similar to -z but with custom threshold.
• --name-phase1 Regular expression on phase1 name that displayed tunnels must belong to. By default matching is case sensitive, unless prefix (?i) is used (example: vpn vs '(?i)vpn').
• --name-phase2 Similar to --name-phase1 but applies directly on (phase2) tunnel name.
• --net-phase1 At least of the the IPSec envelope IPs must be inside this network. CIDR notation (example: 10.0.0.0/8).
• --ip-phase2 Show only tunnels with selectors containing this IP (example: 1.2.3.4).

Advanced views

By default only VDOM, tunnel (phase2) name, offload status and its Rx/Tx counters are displayed. With --show-* parameters other sections can be shown as well. This default section can be hidden with --no-phase2 parameter.

• --show-percentage For each tunnel traffic counter show percentage of this tunnel of all tunnels for the same phase1 and (another counster) percentage of this tunnel of all the IPSec traffic processed by the device.
• --show-selectors Show configured phase2 selectors (protocols, networks, ports on each side).
• --show-phase1 On each line show also phase1 details (name, envelope IPs and ports). Mutiple tunnels with the same phase1 will show the same content.
• --show-phase1-sum Show also summarized counters of all tunnels within the same phase1. Mutiple tunnels with the same phase1 will show the same content.

There is also one "shortcut" option --only-phase1 that transforms the view to phase1 overview, with each phase1 shown only once and with summarized statistics. This is useful to get higher level overview about the IPSec service on the device.

Sorting parameters

• --sort-by-rxp Sort by tunnel (phase2) Rx packets.
• --sort-by-rxb Sort by tunnel (phase2) Rx bytes.
• --sort-by-txp Sort by tunnel (phase2) Tx packets.
• --sort-by-txb Sort by tunnel (phase2) Tx bytes.
• --sort-by-sum-rxp Sort by summarized Rx packets of all tunnels in the same phase1.
• --sort-by-sum-rxb Sort by summarized Rx bytes of all tunnels in the same phase1.
• --sort-by-sum-txp Sort by summarized Tx packets of all tunnels in the same phase1.
• --sort-by-sum-txb Sort by summarized Tx bytes of all tunnels in the same phase1.

Other parameters

• --interval How often to collect counters. 5s (5 seconds) by default. Higher numbers make better averages but hide peeks.
• --max Maximum number of lines to show on screen. By default 30. Rather than increasing this number it might be better to sort tunnels by traffic so the most utilized are on top.
• --vdom Regular expression of VDOM to collect tunnels from. Example ^vdom1|vdom2$. If not specified, all VDOMs are used.
• --units -p Select the units to use when displaying traffic throughput. See the Selecting throughput units section for more details.

ipstop

Display utilization and various packet statistics of IPSengine processes, both for packets received from kernel via raw socket and also for packets received via nTurbo.

Be aware that this command is a little more fragile than the other commands because the output from diagnose ips ... and diagnose test application ipsmonitor ... is often scrambled when multiple ipsengines try to write the output at the same time, especially when their CPU utilization is high. There is no way how FlhCli can fix this other that running the command again when it notices a problem.

With no additional parameters it shows basic summary of nTurbo kernel driver (can be hidden with --no-nturbo-driver), nTurbo part of userland ipsengine processes (can be hidden with --no-ipse-nturbo) and generic IPSE statistics (can be hidden with --no-ipse).

Columns description

• IPSE PID Process ID of ipsengine process(es).
• nTurbo driver / DRV/Q NTurbo driver ID and the queue number (local to that driver).
• nTurbo driver / -> IPSE Number of packets sent through this queue to userland ipsengine process.
• nTurbo driver / CPU CPU id this nTurbo driver queue is using to talk to userland ipsengine.
• nTurbo driver / DROP Summary of all drop counters for this nTurbo driver queue (in both RX and TX directions).
• IPSE nTurbo engine / RX Packets received from nTurbo kernel driver.
• IPSE nTurbo engine / DROP Summary of dropped packets userland ipsengine part handling the nTurbo queue.
• IPSE packets / CPU # CPU the userland ipsengine used last time (should be statically bound to a single CPU).
• IPSE packets / Total Total packets processed by this ipsengine process.
• IPSE packets / Raw % Percentage of packets processed in this ipsengine process received through the raw socket from kernel (rest should have come through nTurbo). This is calculated from nTurbo driver / -> IPSE and IPSE packets / Total. If nTurbo driver section is not displayed, it will show ? because it is also not collected (unless --collect-all parameter is used).
• IPSE packets / CPU % Summary of user and system CPU utilization of this ipsengine process.
• Overall CPU utilization shows the utilization of the CPU the ipsengine is running on. The fields mean "user", "system", "nice", "irq", "softirq", "steal", "idle" repectively.

Advanced mode

There are many more columns displayed when -a (for "advanced") parameter is used. Most probably they won't fit on screen by default, therefore it is suggested to start with "standard" (not -a) mode and then select only one section (i.e. hide the other two) and enable "advanced" mode.

When other sections are hidden with two of these parameters --no-ipse-nturbo, --no-ipse, --no-nturbo-driver. And -a parameter is used. (Overall CPU utilization cropped from the screenshot.)

Parameters and filters

(Those not yet mentioned above)

• --no-cpu Do not show overall CPU utilization section. This is hidden automatically when only userland ipsengine nturbo counters are shown, because in that case the CPU id is not collected.
• --sort-by-cpu Sort by ipsengine summary (user + system) CPU utilization.
• --sort-by-cpu-id Sort by CPU id.
• --sort-by-pid Sort by ipsengine process id. This is default.

lograte

Show rate of logs for every log target as reported by miglog. By default, difference in counters between cycles is shown (except for counters that show current state and do not simply increase in time - such are marked with "<!>" in column name). It is also possible to show counts per second with paramer --per-second.

Automatic restart of miglogd

By default, counters are only displayed and no other action is taken. However, with following parameters, miglogd process can be automatically restarted when one or more counters crosses configured threshold:

• --counter Name of the counter to monitor. This is regular expression allowing more than one counter to match.
• --operator How the counter value is compared to the threshold:
  • eq Equal : matches when --counter == --threshold
  • ne Not equal : matches when --counter != --threshold
  • lt Less than : matches when --counter < --threshold
  • gt Greater than : matches when --counter > --threshold
  • le Less or equal : matches when --counter <= --threshold
  • ge Greater or equal : matches when --counter >= --threshold
• --threshold Threshold value for comparing. Be aware that unless --per-second is used, this is compared to the number for the whole collecting interval (which is 5 seconds by default).
• --warn Number of consecutive cycles to only warn. Only if it is still matching after this number of cycles, the miglogd will be restarted. Three cycles by default. Set to 0 to disable warning feature and restart on first match.
• --wait If miglog is restarted, wait for this time to let it stabilize before continuing to monitor. It is 20s (20 seconds) by default.

Column(s) matching the --counter will be colorized in following way:

• Green - expression not matching. No action is to be taken.
• Orange - expression is matching, but --warn is in effect. No action at this point.
• Red - expression is matching and miglogd is being restarted.

Applied settings are displayed in MONITOR[...] and WARN[...] headers.

To restart if sent counter is less than 10 (per default interval which is 5 seconds) for two consecutive intervals (1st is just warning, 2nd restart):

$ flhcli lograte -counter sent -operator lt -threshold 10 -warn 1

Other parameters and filters

• --target Regular expression matching the target name(s) that are to be shown (and monitored). Targets' names are the same ones as in config file (for example: fortianalyzer, fortianalyzer2, syslogd4, etc.). By default all targets reported by miglog are shown.
• --interval How often to check counters. By default 5s (5 seconds). Be aware that this represents number of logs per the whole interval - the longer the interval the higher will be counters.
• --per-second Switch counting from "per interval" to "per second". All counters and monitor thresholds will be calculated per second, regardless on --interval length.

np6lanes

Display utilization of all lanes between all NP6es and ISF (or "front ports" if ISF is not present). In most architectures NP6 is connected to ISF via 4 indepedent 10G lanes and this utility shows the current utilization for each of them.

Similar to interfaces command the default view show summarize packets counts (unicast + broadcast + multicast) for each direction and current throughput. With -a ("advanced") parameter the counters are shown separately and physical layer overhead is calculated (20 bytes per packet).

Parameters and filters

• -a Advanced mode where counters are separated for more detailed overview.
• --npu Select NP6 id(s) to display. It can be repeated multiple times or range (like --npu 0-7) can be used.
• --show-size Display also packet size histogram.
• --show-total Display also summary of all counters (including those that didn't fit on screen).
• --sort-by-name By default the lanes are sorted by their utilization, but with this option they are ordered by their name. This is useful when the lanes have more or less the same traffic.
• --no-rx Do not show RX direction counters.
• --no-tx Do not show TX direction counters.
• --units -p Select the units to use when displaying traffic throughput. See the Selecting throughput units section for more details.
• -z, -zz, -zzz Ignore interfaces that are up but do not transmit many packets. See the Minimum packets section for more details.

np6drops

Collects dropped packets from DCE, HRX and anomaly counters for every NP6 in the system. Most of the counters also contain simple description of the packets it drops.

Parameters and filters

• --counter Regular expression for counter name.
• --section Regular expression for section name.
• --npu Select NP6 id(s) to display. It can be repeated multiple times or range (like --npu 0-7) can be used.
• --sort-by-name Sort by drop counter name instead of the number of dropped packets.
• -zzz Ignore counters with less than specified number of dropped packets per interval (1 by default)
• --no-divide Do not calculate per-second drops. If the interval to collect interrupts is set to 5 seconds, then the number of drops is displayed for these 5 seconds.
• --summarize Summarize the drop counters across all (selected) NP6 chips.
• --batch Display the output more suitable for programmatic processing. Does not display any headers or formating. Only displays NP6 ID, drop counter name and number of packets.

policytop

Display the number of packets per second and throughput for each configured policy (IPv4 and IPv6). It also displayes these counters separately for non-offloaded traffic, fully NPU offloaded traffic and traffic that took the path via NTurbo. For each policy it also shows the number of sessions created per second.

After start this utility reads IPv4 and IPv6 policy configuration for all VDOMs (which make take some time for more complex configurations, maximum time to wait can be adjusted with --policy-list-timeout parameter) and remembers the policy id together with its name and action for certain time (1 hour by default, changeable with --cache parameter). This means that changes in policy configuration will not be reflected until this cache expires.

Parameters and filters

• --no-notoffloaded Hide "Not offloaded" columns.
• --no-full Hide "Fully NPU offloaded" columns.
• --no-nturbo Hide "NTurbo path" columns.
• --no-ipv4 Hide IPv4 policies.
• --no-ipv6 Hide IPv6 policies.
• --accept Show only policies with "accept" action.
• --deny Show only policies with "deny" action.
• --vdom Only investigate policies for a VDOM matching a regular expression.
• --name Only investigate policies with name matching a regular expression.
• --uuid Instead of simple policy ID use its UUID.
• --policy-id Show only policy(ies) with specified ID (can repeat or be a range).
• --policy-uuid Show only policy(ies) with specified ID (can repeat).
• --with-default Show also counters for implicit drop policy "0".
• --units -p Select the units to use when displaying traffic throughput. See the Selecting throughput units section for more details.
• -z, -zz, -zzz Ignore interfaces that are up but do not transmit many packets. See the Minimum packets section for more details.
• --from-start Together with --no-divide shows the total traffic transferred from the start of the utility.
• --sort-by-id --sort-by-packets --sort-by-packets-full --sort-by-packets-no --sort-by-packets-nturbo --sort-by-rate-full --sort-by-rate-no --sort-by-rate-nturbo Change the way how policies are ordered. By default summary of total rate through the policy is used. -no suffix means "Not offloaded", -full means "Fully NPU offloaded". If it does not have any such suffix, the summary of "Not offloaded", "Full NPU offload" and "NTurbo path" is used.

traffic

Collects the traffic statistics for some well known protocols (and generic IP/TCP/UDP/ICMP counters for others). It can present it either in packets per second (with --packets parameter) or in throughput (with --throughput parameter). All counters represent both the offloaded and non-offloaded traffic.

Because of the way how FortiGate creates this statistics, the collection interval should be a multiple of 2. If it is not, the outputs will be very unstable (one line with significantly higher counters, next one with significanly lower counters). Even when using the default interval of 2 seconds, it can sometimes happen that there are no new statitistics generated by FortiGate and the output shows zero or very small numbers on one line.

Columns description

• TCP, UDP, ICMP, IP These fields count all packets that didn't match any of the other protocols.

Meaning of the other fields is not absulately clear, becase it is unknown what combination of protocols and ports is matched for which application. Following description is best guess but there might be other packets matching it in addition to what is listed here:

• Browsing TCP/80 and TCP/443
• DNS TCP/53 and UDP/53

When -a parameter is used:

• Email TCP/25
• FTP TCP/21
• Gaming ?
• Im ?
• Newsgroups ?
• P2P ?
• Streaming ?
• TFTP UDP/69
• VoIP ?

When -a parameter is not used:

• Others Summary of the parameters shown when -a parameter is used.

Parameters and filters

Every time exactly one of following parameters must be used:

• --packets Show the statistics as packets per second.
• --throughput Show the statistics as data throughput. The right units are selected automatically but parameter --units can be used to override it.

Optional parameters:

• --headers After how many lines to display headers again. Set to zero to disable showing headers in the output (they are still printed at the beginning).
• --no-divide Do not calculate per-second counts. If the interval to is set to 5 seconds, then the number of fragments is displayed for these 5 seconds.
• --batch Display the output more suitable for programmatic processing. Does not display any headers or formating.
• -a Advanced output. Instead of showing a summary for all calculated but not displayed protocols in "Others" field, show every protocol in its own column.