Performance Counters in GNU Radio

Trunk Recorder is designed around a set of recorders that tune to an active transmission and start recording. When nothing is happening, they pause. This has worked really well, but I sometimes have issues with parts of a previous call getting stuck in all of the buffers between blocks in the GNU Radio flowgraph. This talk from Matt Ettus highlights some of the issues with the latency from the buffers.

I wanted to get a handle of how many samples were getting stored in the buffers, in order to see if there was a bottleneck somewhere. It turns out that there is a built-in feature in GNU Radio called Performance Counters that gives you info about the state of your flowgraph. Currently, there are five performance counters. For each counter, the block keeps track of both its average and variance. These counters are:

noutput_items: the number of noutput_items the block is called with.
nproduced: the number of items the block produces.
input_buffers_full: the percentage of how full each input buffer is.
ouptut_buffers_full: the percentage of how full each output buffer is.
work_time: the number of CPU clock ticks during the call to general_work().
work_time_total: cumulative sum of all time spent in work.

Unfortunately, it is a little neglected and not all of the documentation is current. First off, both of these entries (1 & 2) in the GNU Radio are wrong – Performance Counters ARE enabled by default. You can see for yourself here.

option(ENABLE_PERFORMANCE_COUNTERS "Enable block performance counters" ON)

This is great – it means you can just do an apt-get install gnuradio and have access to Performance Counters. All you have to do to turn them on, is edit your gnuradio-runtime.conf file. On Ubuntu, my file is at: /etc/gnuradio/conf.d/gnuradio-runtime.conf. On my Homebrew install it is at: /opt/homebrew/etc/gnuradio/conf.d/gnuradio-runtime.conf. To enable the Performance Counters and export them, edit this section of the file and set the following to true:

[PerfCounters]
on = True
export = True
clock = thread
#clock = monotonic

[ControlPort]
on = True
edges_list = True

If you did everything correctly, when you start a GNU Radio program you should get a message like this saying which port the messages will be published to:

controlport :info: Apache Thrift: -h radiobox -p 46267

GNU Radio install 2 programs for processing messages coming off a GR program’s control port: gr-ctrlport-monitor and gr-perf-monitorx. crtlport-monitor lets you look at the state of all of the variables being published over the control port. This includes all of the Performance Counters, plus some additional ones. You can set how often it updates these values.

perf-monitor only displays information about the Performance Counters. It shows stats on how long each block takes and how full its buffers are. Both programs can be a pain to get running because they require some additional libraries and Python deps, but will fail silently without them. I modified this section of code to get those errors:

        try:
            GNURadioControlPortClient(
                args.host, args.port, 'thrift', self.run, Qt.QApplication(sys.argv).exec_)
        except Exception as e:
            sys.stderr.write(
                "gr-perf-monitorx: lost ({0}).\n".format(e))
            print("ControlPort failed to connect. Check the config of your endpoint.")
            print("\t[ControlPort] on = True")
            print("\t[ControlPort] edges_list = True")
            print("\t[PerfCounters] on = True")
            print("\t[PerfCounters] export = True")
            sys.exit(1)

To get it running I had to install the following… but it maybe different on your system:

apt-get install libgraphviz-dev
apt-get install graphviz
pip3 install pygraphviz

I also had to edit some of the function calls because the libraries had changed. And after all this… I didn’t really get any conclusive answers on the buffers. 🤷‍♂️