How to use the Python debugger (pdb)

We all know things can go wrong when writing programs. Syntax errors, spelling mistakes, even forgotten sections of code; it's all possible. Sometimes, issues like these are easy to detect. Here's an example:

$ python3
Python 3.9.9 (main, Nov 19 2021, 00:00:00) 
[GCC 10.3.1 20210422 (Red Hat 10.3.1-1)] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> DEBUG: bool = True
>>> 
>>> def print_var_function(val: str):
...     print(f"This is a simple dummy function that prints val={val}")
... 
>>> if __name__ == "__main__":
...     print(f"What is your name?")
...     name = input()
...     if DEBUG:
...         print_var_function(name)
... 
What is your name?
jose
This is a simple dummy function that prints val=jose

Even if you write small Python programs, you will find out soon enough that tricks like this are not enough to debug a program. Instead, you can take advantage of the Python debugger (pdb) and get a better insight into how your application is behaving.

Getting started

Before starting this tutorial, you should have:

• Working knowledge of Python (objects, workflows, data structures)
• Curiosity about how you can troubleshoot your scripts in real time
• A machine that can run Python 3 (for example, I'm using Fedora Linux)

Note: I will use a modern version of Python (3.7+) in this tutorial, but you can find the older syntax for some of the operations in the official pdb documentation.

Case study: A simple script to generate a network diagram

A friend of yours gave you a small Python script to test. He said he wrote it in a rush, and it may contain bugs (in fact, he admitted he tried to run it, but he is pretty sure the proof of concept is good). He also said the script depends on the module Diagrams. It's time for you to try his script.

First, create a virtual environment and install some dependencies:

python3 -m venv ~/virtualenv/pythondebugger
. ~/virtualenv/pythondebugger/bin/activate
pip install --upgrade pip diagrams

Next, download and install the following script:

$ pushd $HOME
$ git clone git@github.com:josevnz/tutorials.git
$ pushd tutorials/PythonDebugger

Unfortunately, when you run the script, it crashes:

(pythondebugger) $ ./simple_diagram.py --help
Traceback (most recent call last):
  File "/home/josevnz/tutorials/PythonDebugger/./simple_diagram.py", line 8, in <module>
    from diagrams.onprem.queue import Celeri
ImportError: cannot import name 'Celeri' from 'diagrams.onprem.queue' (/home/josevnz/virtualenv/pythondebugger/lib64/python3.9/site-packages/diagrams/onprem/queue.py)

So at least one import is wrong. You suspect it is Celery (which your friend spelled Celeri), so you launch the script again with the Python debugger mode enabled:

(pythondebugger) $ python3 -m pdb simple_diagram.py --help
> /home/josevnz/tutorials/PythonDebugger/simple_diagram.py(2)<module>()
-> """
(Pdb)

No crashes, and the prompt (Pdb) tells you that you are currently on line 2 of the program:

(pythondebugger) $ python3 -m pdb simple_diagram.py --help
> /home/josevnz/tutorials/PythonDebugger/simple_diagram.py(2)<module>()
-> """
(Pdb) l
  1  	#!/usr/bin/env python
  2  ->	"""
  3  	Script that show a basic Airflow + Celery Topology
  4  	"""
  5  	import argparse
  6  	from diagrams import Cluster, Diagram
  7  	from diagrams.onprem.workflow import Airflow
  8  	from diagrams.onprem.queue import Celeri
  9  	
 10  	
 11  	def generate_diagram(diagram_file: str, workers_n: int):

You can press c (continue) until there is a breakpoint:

(Pdb) c
Traceback (most recent call last):
  File "/usr/lib64/python3.9/pdb.py", line 1723, in main
    pdb._runscript(mainpyfile)
  File "/usr/lib64/python3.9/pdb.py", line 1583, in _runscript
    self.run(statement)
  File "/usr/lib64/python3.9/bdb.py", line 580, in run
    exec(cmd, globals, locals)
  File "<string>", line 1, in <module>
  File "/home/josevnz/tutorials/PythonDebugger/simple_diagram.py", line 2, in <module>
    """
ImportError: cannot import name 'Celeri' from 'diagrams.onprem.queue' (/home/josevnz/virtualenv/pythondebugger/lib64/python3.9/site-packages/diagrams/onprem/queue.py)
Uncaught exception. Entering post mortem debugging
Running 'cont' or 'step' will restart the program
> /home/josevnz/tutorials/PythonDebugger/simple_diagram.py(2)<module>()
-> """

There's the problem: The bad import won't let the script continue. So, start with n (next) and move line by line:

Uncaught exception. Entering post mortem debugging
Running 'cont' or 'step' will restart the program
> /home/josevnz/tutorials/PythonDebugger//simple_diagram.py(2)<module>()
-> """
(Pdb) step
Post mortem debugger finished. The /home/josevnz/tutorials/PythonDebugger//simple_diagram.py will be restarted
> /home/josevnz/tutorials/PythonDebugger//simple_diagram.py(2)<module>()
-> """
(Pdb) n
> /home/josevnz/tutorials/PythonDebugger//simple_diagram.py(5)<module>()
-> import argparse
(Pdb) n
> /home/josevnz/tutorials/PythonDebugger//simple_diagram.py(6)<module>()
-> from diagrams import Cluster, Diagram
(Pdb) n
> /home/josevnz/tutorials/PythonDebugger//simple_diagram.py(7)<module>()
-> from diagrams.onprem.workflow import Airflow
(Pdb) n
> /home/josevnz/tutorials/PythonDebugger//simple_diagram.py(8)<module>()
-> from diagrams.onprem.queue import Celeri
(Pdb) n
ImportError: cannot import name 'Celeri' from 'diagrams.onprem.queue' (/home/josevnz/virtualenv/pythondebugger/lib64/python3.9/site-packages/diagrams/onprem/queue.py)
> /home/josevnz/tutorials/PythonDebugger//simple_diagram.py(8)<module>()
-> from diagrams.onprem.queue import Celeri

You won't be able to proceed unless you fix the broken instruction in line 8. You need to replace import Celeri with import Celery:

(Pdb) from diagrams.onprem.queue import Celery
(Pdb) exit()

For the sake of argument, maybe you want the program to enter the debug mode if a module is missing (in fact, you insist there is a Celeri module). The change in the code is simple; just capture the ImportError and call the breakpoint() function (before you show the stack trace using traceback):

#!/usr/bin/env python
"""
Script that show a basic Airflow + Celery Topology
"""
try:
    import argparse
    from diagrams import Cluster, Diagram
    from diagrams.onprem.workflow import Airflow
    from diagrams.onprem.queue import Celeri
except ImportError:
    breakpoint()

If you call the script normally and one of the imports is missing, the debugger will start for you after printing the stack trace:

(pythondebugger) $ python3 simple_diagram.py --help
Exception while importing modules:
------------------------------------------------------------
Traceback (most recent call last):
  File "/home/josevnz/tutorials/PythonDebugger//simple_diagram.py", line 11, in <module>
    from diagrams.onprem.queue import Celeri
ImportError: cannot import name 'Celeri' from 'diagrams.onprem.queue' (/home/josevnz/virtualenv/pythondebugger/lib64/python3.9/site-packages/diagrams/onprem/queue.py)
------------------------------------------------------------
Starting the debugger
> /home/josevnz/tutorials/PythonDebugger//simple_diagram.py(21)<module>()
-> def generate_diagram(diagram_file: str, workers_n: int):
(Pdb) 

Fix the bad import and move on to see what the script can do:

(pythondebugger) $ python3 simple_diagram.py --help
usage: /home/josevnz/tutorials/PythonDebugger//simple_diagram.py [-h] [--workers WORKERS] diagram

Generate network diagrams for examples used on this tutorial

positional arguments:
  diagram            Name of the network diagram to generate

optional arguments:
  -h, --help         show this help message and exit
  --workers WORKERS  Number of workers
  
# Generate a diagram with 3 workers
(pythondebugger) [josevnz@dmaf5 ]$ python3 simple_diagram.py --workers 3 my_airflow.png

The resulting diagram looks like this:

Image

Take a closer look at the code with step, continue, args, and breakpoints

Run the script again but using a negative number of workers:

$ python3 simple_diagram.py --workers -3 my_airflow2.png

This produces a weird image with no Celery workers:

Image

This is unexpected. Use the debugger to understand what happened and also come up with a way to prevent this; start by asking to see the full source code with ll:

(pythondebugger) $ python3 -m pdb simple_diagram.py --workers -3 my_airflow2.png
> /home/josevnz/tutorials/PythonDebugger/simple_diagram.py(2)<module>()
-> """
(Pdb) ll
  1  	#!/usr/bin/env python
  2  ->	"""
  3  	Script that show a basic Airflow + Celery Topology
  4  	"""
  5  	try:
  6  	    import sys
  7  	    import argparse
  8  	    import traceback
  9  	    from diagrams import Cluster, Diagram
 10  	    from diagrams.onprem.workflow import Airflow
 11  	    from diagrams.onprem.queue import Celery
 12  	except ImportError:
 13  	    print("Exception while importing modules:")
 14  	    print("-"*60)
 15  	    traceback.print_exc(file=sys.stderr)
 16  	    print("-"*60)
 17  	    print("Starting the debugger", file=sys.stderr)
 18  	    breakpoint()
 19  	
 20  	
 21  	def generate_diagram(diagram_file: str, workers_n: int):
 22  	    """
 23  	    Generate the network diagram for the given number of workers
 24  	    @param diagram_file: Where to save the diagram
 25  	    @param workers_n: Number of workers
 26  	    """
 27  	    with Diagram("Airflow topology", filename=diagram_file, show=False):
 28  	        with Cluster("Airflow"):
 29  	            airflow = Airflow("Airflow")
 30  	
 31  	        with Cluster("Celery workers"):
 32  	            workers = []
 33  	            for i in range(workers_n):
 34  	                workers.append(Celery(f"Worker {i}"))
 35  	        airflow - workers
 36  	
 37  	
 38  	if __name__ == "__main__":
 39  	    PARSER = argparse.ArgumentParser(
 40  	        description="Generate network diagrams for examples used on this tutorial",
 41  	        prog=__file__
 42  	    )
 43  	    PARSER.add_argument(
 44  	        '--workers',
 45  	        action='store',
 46  	        type=int,
 47  	        default=1,
 48  	        help="Number of workers"
 49  	    )
 50  	    PARSER.add_argument(
 51  	        'diagram',
 52  	        action='store',
 53  	        help="Name of the network diagram to generate"
 54  	    )
 55  	    ARGS = PARSER.parse_args()
 56  	
 57  	    generate_diagram(ARGS.diagram, ARGS.workers)
(Pdb)

Going step by step here is not very efficient, so go just deep enough in the code, to line 57, to see the effect of passing the number of workers == -1 (to pretty print the ARGS variable):

> /home/josevnz/tutorials/PythonDebugger//simple_diagram.py(57)<module>()
-> generate_diagram(ARGS.diagram, ARGS.workers)
(Pdb) pp ARGS
Namespace(workers=-3, diagram='my_airflow2.png')

Also, it is very useful to know the object type. Check to see the ARGS type:

(Pdb) whatis ARGS
<class 'argparse.Namespace'>

It looks like the next logical step is to dive into the function that generates the diagram with l (list) to confirm where you are:

(Pdb) l
 52  	        action='store',
 53  	        help="Name of the network diagram to generate"
 54  	    )
 55  	    ARGS = PARSER.parse_args()
 56  	
 57  ->	    generate_diagram(ARGS.diagram, ARGS.workers)
[EOF]

Dive one s (step) inside and then move n (next) one instruction:

(Pdb) s
--Call--
> /home/josevnz/tutorials/PythonDebugger//simple_diagram.py(21)generate_diagram()
-> def generate_diagram(diagram_file: str, workers_n: int):
(Pdb) n
> /home/josevnz/tutorials/PythonDebugger//simple_diagram.py(27)generate_diagram()
-> with Diagram("Airflow topology", filename=diagram_file, show=False):
(Pdb) l
 22  	    """
 23  	    Generate the network diagram for the given number of workers
 24  	    @param diagram_file: Where to save the diagram
 25  	    @param workers_n: Number of workers
 26  	    """
 27  ->	    with Diagram("Airflow topology", filename=diagram_file, show=False):
 28  	        with Cluster("Airflow"):
 29  	            airflow = Airflow("Airflow")
 30  	
 31  	        with Cluster("Celery workers"):
 32  	            workers = []

The prompt tells you that you are inside the generate_diagram function. Confirm which (arguments) were passed:

(Pdb) a generate_diagram
diagram_file = 'my_airflow2.png'
workers_n = -3
(Pdb) 

Inspect the code again and study where it iterates workers_n for the number of times to add Celery workers to the diagram:

(Pdb) l
 33  	            for i in range(workers_n):
 34  	                workers.append(Celery(f"Worker {i}"))
 35  	        airflow - workers
 36  	
 37  	
 38  	if __name__ == "__main__":
 39  	    PARSER = argparse.ArgumentParser(
 40  	        description="Generate network diagrams for examples used on this tutorial",
 41  	        prog=__file__
 42  	    )
 43  	    PARSER.add_argument(

Lines 33-34 populate the workers list with Celery objects. You can see what happens to the range() function when you pass a negative number when trying to create an iterator:

(Pdb) p range(workers_n)
range(0, -3)

The theory says this will generate an empty iterator, meaning the loop will never run. Confirm the before and after state of the workers variable. To do that, set up two b (breakpoints):

1. After the workers variable is initialized, on line 33.
2. After the loop that populates the workers exits, on line 35.

You don't want to execute code one step at a time, so c (continue) through the b (breakpoints) instead:

(Pdb) b simple_diagram.py:33
Breakpoint 1 at /home/josevnz/tutorials/PythonDebugger//simple_diagram.py:33
(Pdb) b simple_diagram.py:35
Breakpoint 2 at /home/josevnz/tutorials/PythonDebugger//simple_diagram.py:35
(Pdb) b
Num Type         Disp Enb   Where
1   breakpoint   keep yes   at /home/josevnz/tutorials/PythonDebugger//simple_diagram.py:33
2   breakpoint   keep yes   at /home/josevnz/tutorials/PythonDebugger//simple_diagram.py:35

Now it's time to print the contents of workers and continue as promised:

(Pdb) c
> /home/josevnz/tutorials/PythonDebugger//simple_diagram.py(33)generate_diagram()
-> for i in range(workers_n):
(Pdb) p workers
[]
(Pdb) c
> /home/josevnz/tutorials/PythonDebugger//simple_diagram.py(35)generate_diagram()
-> airflow - workers
(Pdb) workers
[]

If you press c (continue) the program will exit, or r (return) will get you back to main. Use return:

(Pdb) r
--Return--
> /home/josevnz/tutorials/PythonDebugger//simple_diagram.py(35)generate_diagram()->None
-> airflow - workers
(Pdb) c
The program finished and will be restarted
> /home/josevnz/tutorials/PythonDebugger//simple_diagram.py(2)<module>()
-> """
(Pdb) exit()

You can fix the application by adding some defensive coding to the --workers argument to accept values between 1 and 10 (a diagram with more than 10 workers probably won't look very good). Next, it's time to change the code to add a validation function:

def valid_range(value: str, upper: int  = 10):
    try:
        int_val = int(value)
        if 1 <= int_val <= upper:
            return int_val
        raise ArgumentTypeError(f"Not true: 1<= {value} <= {upper}")
    except ValueError:
        raise ArgumentTypeError(f"'{value}' is not an Integer")


if __name__ == "__main__":
    PARSER = argparse.ArgumentParser(
        description="Generate network diagrams for examples used on this tutorial",
        prog=__file__
    )
    PARSER.add_argument(
        '--workers',
        action='store',
        type=valid_range,
        default=1,
        help="Number of workers"
    )

Of course, you want to learn if this works or not, so relaunch the debugger and set a breakpoint on line 39:

(Pdb) b simple_diagram.py:39
Breakpoint 1 at /home/josevnz/tutorials/PythonDebugger//simple_diagram.py:39
(Pdb) c
> /home/josevnz/tutorials/PythonDebugger//simple_diagram.py(39)valid_range()
-> try:
(Pdb) l
 34  	            for i in range(workers_n):
 35  	                workers.append(Celery(f"Worker {i}"))
 36  	        airflow - workers
 37  	
 38  	def valid_range(value: str, upper: int  = 10):
 39 B->	    try:
 40  	        int_val = int(value)
 41  	        if 1 <= int_val <= upper:
 42  	            return int_val
 43  	        raise ArgumentTypeError(f"Not true: 1<= {value} <= {upper}")
 44  	    except ValueError:
(Pdb) a valid_range
value = '-3'
upper = 10
(Pdb) p int(value)
-3
(Pdb) n
> /home/josevnz/tutorials/PythonDebugger//simple_diagram.py(40)valid_range()
-> int_val = int(value)
(Pdb) n
> /home/josevnz/tutorials/PythonDebugger//simple_diagram.py(41)valid_range()
-> if 1 <= int_val <= upper:
(Pdb) p 1 <= int_val <= upper
False
(Pdb) n
> /home/josevnz/tutorials/PythonDebugger//simple_diagram.py(43)valid_range()
-> raise ArgumentTypeError(f"Not true: 1<= {value} <= {upper}")
(Pdb) n
argparse.ArgumentTypeError: Not true: 1<= -3 <= 10
> /home/josevnz/tutorials/PythonDebugger//simple_diagram.py(43)valid_range()
-> raise ArgumentTypeError(f"Not true: 1<= {value} <= {upper}")
(Pdb) n
> /home/josevnz/tutorials/PythonDebugger//simple_diagram.py(44)valid_range()
-> except ValueError:
(Pdb) n
--Return--
> /home/josevnz/tutorials/PythonDebugger//simple_diagram.py(44)valid_range()->None
-> except ValueError:
(Pdb) n
--Call--
> /usr/lib64/python3.9/argparse.py(744)__init__()
-> def __init__(self, argument, message):
(Pdb) c
usage: /home/josevnz/tutorials/PythonDebugger//simple_diagram.py [-h] [--workers WORKERS] diagram
/home/josevnz/tutorials/PythonDebugger//simple_diagram.py: error: argument --workers: Not true: 1<= -3 <= 10
The program exited via sys.exit(). Exit status: 2
> /home/josevnz/tutorials/PythonDebugger//simple_diagram.py(2)<module>()
-> """

A lot happened here as you moved through using n and c:

1. The function was called as expected, and the breakpoint on line 39 was called.
2. You printed the arguments for the validation function.
3. You evaluated (p) the sanity checks and confirmed that -3 did not pass the range check, and an exception was raised as a result.
4. The program exited with an error.

[ Learn how to modernize your IT with managed cloud services. ]

Without the debugger you can confirm what you inspected before:

pythondebugger) $ python3 simple_diagram.py --workers -3 my_airflow2.png
usage: /home/josevnz/tutorials/PythonDebugger//simple_diagram.py [-h] [--workers WORKERS] diagram
/home/josevnz/tutorials/PythonDebugger//simple_diagram.py: error: argument --workers: Not true: 1<= -3 <= 10

# A good call
(pythondebugger) [josevnz@dmaf5 ]$ python3 simple_diagram.py --workers 7 my_airflow2.png && echo "OK"
OK

Image

Learn how to jump before running

You can skip through code by jumping with the debugger. Be aware that depending on where you jump, you may disable the execution of code (but this is useful to understand the workflow of your program). For example, set up a breakpoint on line 32, just before you create the workers list, and then j (jump) to line 36 and print the value of workers:

(pythondebugger) $ python3 -m pdb simple_diagram.py --workers 2 my_airflow4.png
> /home/josevnz/tutorials/PythonDebugger//simple_diagram.py(2)<module>()
-> """
(Pdb) b simple_diagram.py:32
Breakpoint 1 at /home/josevnz/tutorials/PythonDebugger//simple_diagram.py:32
(Pdb) c
> /home/josevnz/tutorials/PythonDebugger//simple_diagram.py(32)generate_diagram()
-> with Cluster("Celery workers"):
(Pdb) j 36
> /home/josevnz/tutorials/PythonDebugger//simple_diagram.py(36)generate_diagram()
-> airflow - workers
(Pdb) workers
*** NameError: name 'workers' is not defined

Oh! workers never initialized. Jump back to line 32, set a breakpoint to line 36, and continue to see what happens:

(Pdb) j 32
> /home/josevnz/tutorials/PythonDebugger//simple_diagram.py(32)generate_diagram()
-> with Cluster("Celery workers"):
(Pdb) b simple_diagram.py:36
Breakpoint 2 at /home/josevnz/tutorials/PythonDebugger//simple_diagram.py:36
(Pdb) c
> /home/josevnz/tutorials/PythonDebugger//simple_diagram.py(36)generate_diagram()
-> airflow - workers
(Pdb) p workers
[<onprem.queue.Celery>, <onprem.queue.Celery>]
(Pdb) 

Yes, time traveling is also confusing!

Try code with interact

If you were paying close attention, you might have noticed that the label on the Celery workers ranges from Worker 0 to Worker N-1. Having a Worker 0 is not intuitive. The fix is trivial, so see if you can "hot-fix" the code:

(pythondebugger) $ python3 -m pdb simple_diagram.py --workers 2 my_airflow4.png
> /home/josevnz/tutorials/PythonDebugger//simple_diagram.py(2)<module>()
-> """
(Pdb) b simple_diagram.py:35
Breakpoint 1 at /home/josevnz/tutorials/PythonDebugger//simple_diagram.py:35
(Pdb) c
> /home/josevnz/tutorials/PythonDebugger/simple_diagram.py(35)generate_diagram()
-> workers.append(Celery(f"Worker {i}"))
(Pdb) p i
0
(Pdb) p f"Worker {i}"
'Worker 0'
(Pdb) interact
*interactive*
>>> f"Worker {i}"
'Worker 0'
>>> f"Worker {i+1}"
'Worker 1'
>>> workers.append(Celery(f"Worker {i+1}"))
>>> workers
[<onprem.queue.Celery>]

Here's what happens:

1. Set a breakpoint on line 35, which is where you create Celery object instances.
2. Print the value of i. You see it is 0 and will go through workers_n - 1.
3. Evaluate the expression of f"Worker {i}".
4. Start an interactive session. This session inherits all the variables and context to the moment of the breakpoint. This means you have access to i and the workers list.
5. Test a new expression by adding i+1 and confirming if the fix works.

This is less expensive than restarting the program with the fix. Also, imagine the value of doing this if your function were much more complex and getting data from remote resources like a database. Pure gold!

Going back to the last fix, replace line 35 with this:

            for i in range(workers_n):
                workers.append(Celery(f"Worker {i+1}"))

Image

What did you learn?

A lot is possible with the Python debugger. In this tutorial, you:

• Ran an application in debug mode
• Executed a script step by step
• Inspected the content of variables and learned about members of a module
• Used breakpoints and jumps
• Incorporated the debugger in the application
• Applied a hot-fix in the running code

You now know that you don't need an integrated development environment (IDE) to debug and troubleshoot a Python application, and that pdb is a very powerful tool you should have in your toolset.