Debugging Polycube services

The main way to debug a service in Polycube is by using debug log messages. Polycube provides the primitives that enables to print messages (from both data and control plane) in the same log file, hence facilitating the integrated debugging of both data and control plane components. However, primitives to be used are different in the above two cases and will be presented later in this document.

By default, the log file can be found in /var/log/polycube/polycubed.log and it contains all the output produced by the polycubed daemon and the running services.

Configuring logging levels

Polycube defines six log levels, listed below in increasing order of importance, following the general accepted practice proposed on StackOverflow:

  • trace: when the developer would “trace” the code and trying to find one part of a function specifically.

  • debug: information that is diagnostically helpful to people more than just developers (IT, sysadmins, etc.).

  • info (default value): information that is usually useful to log (service start/stop, configuration assumptions, etc); information that users would like to have available but usually do not care about under normal circumstances.

  • warning: anything that can potentially cause application oddities, but that can usually be automatically recovered by the framework, such as switching from a primary to backup server, retrying an operation, missing secondary data, etc.

  • error: any error which is fatal to the operation, but not the service or application, such as cannot open a required file, missing data, etc.. These errors should force user (administrator, or direct user) intervention.

  • critical: any error that is forcing a shutdown of the service or application to prevent data loss (or further data loss). These errors should be reserved only for the most heinous errors and situations where there is guaranteed to have been data corruption or loss.

Each Polycube service has a loglevel property that enables the printing of log messages filtered by the choosen level. Setting the loglevel to X means that all the messages of that loglevel and above will be printed. For instance, if loglevel=warning, all messages marked as warning, error and critical are printed.

Setting the loglevel property of a service can be achieved by creating the service with the proper property, such as in the following:

polycubectl router r0 add loglevel=debug

This creates a new instance of a router, called r0, with the highest logging level (debug) turned on.

Note 1 The loglevel of each individual service does not affect the loglevel of the polycubectl command line. In other words, polycubectl logging can be set to minimal (i.e., logging level = info), while a given service can have the logging level equal to debug. More information is available in the configuring polycubectl section.

Note 2 The loglevel of each individual service does not affect the loglevel of the polycubed system daemon. In other words, polycubed logging can be set to minimal (i.e., logging level = info), while a given service can have the logging level equal to debug. More information is available in the debugging the polycubed daemon section.

Note 3: the log file can be easily filtered on either (1) a specific loglevel or (2) a specific service by using the standard Unix grep tool.

Usage examples:

sudo polycubed | grep "debug"
sudo polycubed | grep -E "warning|error|critical"
sudo polycubed | grep "bridge"

Logging the eBPF data plane

The common eBPF primitive bpf_trace_printk(), which is widely used in other eBPF frameworks, should not be used in Polycube. Polycube offers a new pcn_log() primitive, which integrates both data and control plane logging in the same location.

Syntax:

  pcn_log(ctx, level, msg, args...)

- ``ctx``: packet being processed.
- ``level``: type of logging level: ``LOG_TRACE``, ``LOG_DEBUG``, ``LOG_INFO``, ``LOG_WARN``, ``LOG_ERR``, ``LOG_CRITICAL``.
- ``msg``: message to print
- ``args``: arguments of the message

Three special format specifiers are available:

  • %I: print an IP address

  • %M: print a MAC address

  • %P: print a TCP/UDP port

Please note the the custom specifiers expect the data to be in network byte order while standard specifiers expects it to be in host byte order.

Current limitations of pcn_log():

  • Cannot be used inside a macro

  • Maximum 4 arguments are allowed

Usage example:

pcn_log(ctx, LOG_DEBUG, "Receiving packet from port %d", md->in_port);

Logging the control plane

Custom printf() or similar primitives, which make the code difficult to debug, should not be used in the Polycube control plane. In fact, the pcn_log() primitive presented below can be used also in the control plane, as Polycube includes a powerful logging system implemented within a dedicated class.

Usage example:

logger()->info("Connected port {0}", port_name);
logger()->debug("Packet size: {0}", packet.size());

Debugging using Integrated Development Environment (IDE)

VS Code

Polycube uses cmake to create the compilation environment (Makefiles, etc), which, by default is configured to create the executables in Release mode. To debug your code, you must compile Polycube in Debug mode, with the following commands (follow also here):

mkdir Debug
cd Debug
cmake -DCMAKE_BUILD_TYPE=Debug ..
make -j

Once all the Polycube executables have been created with _debug_ information, you must install them in the default folders with the following command:

sudo make install

Now, let’s create a launch.json file, following the VS Code instructions:

  1. Run

  2. Start Debugging (or Add Configuration)

  3. Select the C ++ template (GDB/LLDB)

  4. Default configuration

  5. As “program”, add /usr/local/bin/polycubed, which represents the default location. Alternatively, if you modified the installation script, enter the path that comes out as a result of the which polycubed command

  6. As “args”, add ["--loglevel=DEBUG"] to enable Polycube debug mode logging, which provides diagnostic information useful to people more than just developers. Note that the Polycube _debug_ logging (which simply means _the highlest level of verbosity_) is independent from the build type, hence it has to be enabled explicitly.

  7. Later, also following this link, create a file called “gdb” in “home/{username}” (for example) and enter pkexec /usr/bin/gdb "$@" to make sure that you are prompted to enter the password at startup. This is because Polycube requires root permissions to run. Finally just edit the launch.json file with these 3 lines:

"externalConsole": false,
"miDebuggerPath": "/home/<username>/gdb",
"MIMode": "gdb",

The final launch.json file should be something like the following:

{
    // Use IntelliSense to learn about possible attributes.
    // Hover to view descriptions of existing attributes.
    // For more information, visit: https://go.microsoft.com/fwlink/?linkid=830387
    "version": "0.2.0",
    "configurations": [
        {
            "name": "(gdb) Launch",
            "type": "cppdbg",
            "request": "launch",
            "program": "/usr/local/bin/polycubed",
            "args": ["--loglevel=DEBUG"],
            "stopAtEntry": false,
            "cwd": "${fileDirname}",
            "environment": [],
            "externalConsole": false,
            "miDebuggerPath": "/home/pino/gdb",
            "MIMode": "gdb",
            "setupCommands": [
                {
                    "description": "Abilita la riformattazione per gdb",
                    "text": "-enable-pretty-printing",
                    "ignoreFailures": true
                }
            ]
        }
    ]
}

CLion

If you use CLion, you can debug Polycube with the following steps:

  1. Run CLion with sudo (Polycube needs root permissions)

  2. Set breakpoints and build/install Polycube (as explained above)

  3. At this point there are two equivalent ways to debug Polycube with CLion:

    1. Use the CLion debugger: at the top right select Debug ‘polycube’;

    2. (or) Run Polycube in a terminal with enabled debug logs (e.g., sudo polycubed --loglevel = DEBUG); then, from CLion, go to Run->Attach to Process.. and search for “polycubed”

  4. At this point, from another terminal, just use polycubectl to interact with Polycube.

Debugging network traffic using tcpdump/Wireshark

Debugging (i.e., sniffing) network traffic flowing between two Polycube services (e.g., on the link that connects a Polycube router to a Polycube bridge) can be achieved with the span mode.

// TODO