[Bug] bpftrace PMDA failure in QA

[kmcdonell]

Issue Description

[this is a copy of recent emails on the pcp-maintainers@groups.io list, moving the discussion here for higher visibility]

qa/1721 seems to be failing everywhere (CI and the QA Farm).

The smoking gun I can see is this from 1721.full (this is from bozo Ubuntu 24.04) [note this is not the only failure signature, see more below]:

_pmdabpftrace_wait_for_value: iteration 20

bpftrace.scripts.biolatency.probes
    value 0                <- expecting 3 here
_pmdabpftrace_wait_for_value failure @Mon 01 Jun 2026 07:24:01 AEST
Log for pmdabpftrace on bozo.localdomain started Mon Jun  1 07:23:46 2026

[Mon Jun  1 07:23:46] pmdabpftrace(604947) Info: Initializing, currently in 'notready' state.
[Mon Jun  1 07:23:46] pmdabpftrace(604952) Info: manager: started pmdabpftrace process manager
[Mon Jun  1 07:23:46] pmdabpftrace(604952) Info: manager: using bpftrace v0.20.2
[Mon Jun  1 07:23:49] pmdabpftrace(604947) Info: registering script from file /var/lib/pcp/pmdas/bpftrace/autostart/biolatency.bt...
[Mon Jun  1 07:23:49] pmdabpftrace(604952) Info: script: starting script s97bd008e31114117952bac65804c72d4 (name=biolatency, user=root)...
[Mon Jun  1 07:23:49] pmdabpftrace(604947) Info: Ready to process requests.
[Mon Jun  1 07:23:49] pmdabpftrace(604952) Info: script: started script s97bd008e31114117952bac65804c72d4 (name=biolatency, user=root, PID=604955)
[Mon Jun  1 07:23:49] pmdabpftrace(604952) Info: script: stopped script s97bd008e31114117952bac65804c72d4 (name=biolatency, user=root, PID=604955) due to error: stdin:25:9-36: ERROR: delete() requires one argument (2 provided)
        delete(@start, args.sector);
        ~~~~~~~~~~~~~~~~~~~~~~~~~~~

I've rooted around at https://bpftrace.org/doc and delete() with one argument seems to have been marked as "deprecated" in version 22 and is not described in version 23, and I can't find anything for version 20.

By the time I get to Ubuntu 26.04, bpftrace has moved to version 25 and the wheels totally fall off as in:

_pmdabpftrace_wait_for_value: iteration 40        <- I increased the timeout to be sure

bpftrace.scripts.biolatency.probes
    value 5                        <- looks like someone else is registering probes
_pmdabpftrace_wait_for_value failure @Mon Jun  1 07:17:09 AM AEST 2026
Log for pmdabpftrace on vm24 started Mon Jun  1 07:16:44 2026

[Mon Jun  1 07:16:44] pmdabpftrace(164835) Info: Initializing, currently in 'notready' state.
[Mon Jun  1 07:16:44] pcp(164850) Info: manager: started pmdabpftrace process manager
[Mon Jun  1 07:16:44] pcp(164850) Info: manager: using bpftrace v0.25.0
[Mon Jun  1 07:16:46] pmdabpftrace(164835) Info: registering script from file /var/lib/pcp/pmdas/bpftrace/autostart/biolatency.bt...
[Mon Jun  1 07:16:46] pcp(164850) Info: script: starting script sb2bc4c7432674d7a8a9ceebf4b2d6692 (name=biolatency, user=root)...
[Mon Jun  1 07:16:46] pmdabpftrace(164835) Info: Ready to process requests.
[Mon Jun  1 07:16:46] pcp(164850) Info: script: started script sb2bc4c7432674d7a8a9ceebf4b2d6692 (name=biolatency, user=root, PID=164854)
[Mon Jun  1 07:16:46] pcp(164850) Error: Error parsing bpftrace output, please open a bug report:
While reading:
'stdin:25:9-36: WARNING: Return value discarded.\n'
the following error occured:
Traceback (most recent call last):
  File "/usr/lib/pcp/pmdas/bpftrace/bpftrace/process_manager.py", line 58, in read_bpftrace_stdout
    process_bpftrace_output(self.runtime_info, script, line)
    ~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/pcp/pmdas/bpftrace/bpftrace/parser.py", line 137, in process_bpftrace_output
    obj = json.loads(line)
  File "/usr/lib/python3.14/json/__init__.py", line 352, in loads
    return _default_decoder.decode(s)
           ~~~~~~~~~~~~~~~~~~~~~~~^^^
  File "/usr/lib/python3.14/json/decoder.py", line 345, in decode
    obj, end = self.raw_decode(s, idx=_w(s, 0).end())
               ~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.14/json/decoder.py", line 363, in raw_decode
    raise JSONDecodeError("Expecting value", s, err.value) from None
json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0) 
...

[another 60 or so lines of Python traceback babble deleted]

Then I've done some more analysis on the failures ... there are 3 distinct failure cases, apparently dependent on the version of bpftrace that is installed/supported (shown as vNN.N below).

-- number of arguments for delete() ... I guess we could just not run qa/1721 if the version is smaller than 22

vm20 v9.4 Ubuntu 20.04 2:delete() requires one argument
vm04 v16.0 CentOS Stream8 2:delete() requires one argument
vm39 v16.0 RHEL 8.10 2:delete() requires one argument
vm07 v17.0 Debian 12.13 2:delete() requires one argument
vm40 v17.0 AmazonLinux 2023 2:delete() requires one argument
vm17 v19.1 openSUSE Leap 2:delete() requires one argument
bozo v20.2 Ubuntu 24.04 2:delete() requires one argument
u20 v20.2 Ubuntu 24.04 2:delete() requires one argument
u22 v20.2 Ubuntu 24.04 2:delete() requires one argument
u24 v20.2 Ubuntu 24.04 2:delete() requires one argument
u26 v20.2 Ubuntu 24.04 2:delete() requires one argument

-- unexpected value for the check variable bpftrace.scripts.biolatency.probes ... these ones need someone who knows to investigate why the number of scripts is apparently 5 instead of 3.

vm14 v22.1 CentOS Stream10 1:last value: 5 (not 3)
vm02 v23.5 openSUSE Leap 1:last value: 5 (not 3)
vm22 v23.5 Fedora 42 1:last value: 5 (not 3)
vm03 v24.2 Fedora 43 1:last value: 5 (not 3)
vm26 v24.2 CentOS Stream9 1:last value: 5 (not 3)
vm28 v24.2 RHEL 9.8 1:last value: 5 (not 3)
vm30 v24.2 RHEL 10.2 1:last value: 5 (not 3)

-- parser failure ... looks like we're completely screwed for v25 (and later?) so this needs to be on someone's TODO list

vm24 v25.0 Ubuntu 26.04 1:last value: 5 (not 3)
vm24 v25.0 Ubuntu 26.04 3:Error parsing bpftrace output
vm34 v25.0-18a5ccb8c Arch Linux 1:last value: 5 (not 3)
vm34 v25.0-18a5ccb8c Arch Linux 3:Error parsing bpftrace output

I have no idea if this is a QA-only style of failure or if the functionality of the bpftrace PMDA is really broken across all these platforms.

Operating System

All in the QA Farm and CI

Architecture

All in the QA Farm and CI

PCP Version

PCP 7.1.5

Component

bpftrace PMDA

Steps to Reproduce

$ cd qa
$ ./check 1721

Additional Information

No response