The /proc filesystem is a goldmine of information about running processes, and it lives entirely in RAM. It’s not a real filesystem: the kernel generates these files on the fly. Want to know what a process is doing? Don’t install another tool. Just read /proc.
ps, lsof, strace all read from /proc. You can do the same thing directly. Understanding /proc gives you a way to debug that doesn’t rely on specific tools being installed or having the right permissions.
The Structure
Every running process gets a directory named after its PID:
$ ls /proc1 cmdline devices filesystems mounts swaps10 cpuinfo diskstats interrupts partitions sys2 crypto dma iomem pressure sysvipc3 cgroups domainname ioports slabinfo thread-selfacpi cgroup_rw exec-domains kallsyms schedstat timer_list...The numeric directories are processes. Let’s look inside one:
$ ls -la /proc/1234/total 0dr-xr-xr-x 9 root root 0 Mar 1 10:45 .-r--r--r-- 1 root root 0 Mar 1 10:45 attr-rw-r--r-- 1 root root 0 Mar 1 10:45 autogroup-r--r--r-- 1 root root 0 Mar 1 10:45 cgroup--w------- 1 root root 0 Mar 1 10:45 clear_refs-r--r--r-- 1 root root 0 Mar 1 10:45 cmdline-r--r--r-- 1 root root 0 Mar 1 10:45 comm...Each file tells you something about the process.
The Essential Files
1. cmdline, The Command That Started the Process
$ cat /proc/1234/cmdline | tr '\0' ' '; echo/usr/bin/python3 /opt/myapp/server.py --config /etc/app.conf --debugThis is the exact command line. Useful when ps output is truncated or you need the full arguments.
2. environ, Environment Variables
$ cat /proc/1234/environ | tr '\0' '\n' | head -n 10PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/binHOME=/rootUSER=rootLANG=en_US.UTF-8APP_DEBUG=1DATABASE_URL=postgresql://localhost/mydbSee what environment variables the process inherited. Useful for debugging why a config setting isn’t being picked up.
3. fd/, Open File Descriptors
$ ls -la /proc/1234/fdtotal 0lrwx------ 1 root root 64 Mar 1 10:45 0 -> /dev/pts/1lrwx------ 1 root root 64 Mar 1 10:45 1 -> /dev/pts/1lrwx------ 1 root root 64 Mar 1 10:45 2 -> /dev/pts/1lrwx------ 1 root root 64 Mar 1 10:45 3 -> socket:[12345]lrwx------ 1 root root 64 Mar 1 10:45 4 -> /var/log/app.loglrwx------ 1 root root 64 Mar 1 10:45 5 -> /etc/app.conflrwx------ 1 root root 64 Mar 1 10:45 6 -> /dev/nullEach number is a file descriptor. The symlink target shows what it points to. Socket fds show socket:[inode]. This is exactly what lsof reads under the hood.
4. status, Process State and Limits
$ cat /proc/1234/statusName: python3Umask: 0022State: S (sleeping)Tgid: 1234Pid: 1234PPid: 1000TracerPid: 0Uid: 0 0 0 0Gid: 0 0 0 0FDSize: 256VmPeak: 123456 kBVmHWM: 56789 kBVmRSS: 45678 kB...Process state (sleeping, running, zombie), memory usage (VmRSS is resident set size), and more. FDSize is the number of file descriptor slots the kernel has currently allocated for the process, not a hard cap; it grows as the process opens more files, up to whatever /proc/[pid]/limits allows.
5. maps, Memory Layout
$ cat /proc/1234/maps55555555b000-55555555c000 r-xp 00000000 08:05 9999999 /usr/bin/python355555575b000-55555575c000 r--p 00001000 08:05 9999999 /usr/bin/python355555575c000-55555575d000 rw-p 00002000 08:05 9999999 /usr/bin/python37ffff7dd5000-7ffff7dff000 r-xp 00000000 08:05 8888888 /lib64/libc-2.31.so...Every memory region the process has mapped. Useful for understanding what libraries are loaded and memory layout during debugging.
6. cgroup, Control Group Info
$ cat /proc/1234/cgroup12:freezer:/11:memory:/system.slice/myapp.service10:cpuacct:/system.slice...Which cgroups the process belongs to. Useful for understanding resource limits and systemd service behavior.
7. limits, System Limits
$ cat /proc/1234/limitsLimit Soft Limit Hard Limit UnitsMax cpu time unlimited unlimited secondsMax file size unlimited unlimited bytesMax data size unlimited unlimited bytesMax stack size 8388608 unlimited bytesMax core file size 0 unlimited bytesMax resident set unlimited unlimited bytesMax processes 31767 31767 processesMax open files 1024 65536 filesThe resource limits for this process. This is what ulimit shows.
Real-World Debugging Scenarios
What files is my app actually opening?
$ ls -la /proc/$(pgrep myapp)/fd | awk 'NF>9 {print $NF}' | sort | uniqHow much memory is my process using?
$ grep VmRSS /proc/$(pgrep myapp)/statusVmRSS: 45678 kBIs my app running with the right UID/GID?
$ grep Uid /proc/$(pgrep myapp)/statusUid: 1000 1000 1000 1000What command was used to start this process?
$ cat /proc/$(pgrep myapp)/cmdline | tr '\0' ' '; echoSee all open network sockets:
$ cat /proc/net/tcpsl local_address rem_address st tx_queue rx_queue tr tm->when retrnsmt uid timeout inode0: 0A0A0A01:0050 00000000:0000 0A 00000000:00000000 00:00000000 00000000 0 0 123456Not human-readable, but it’s there. That’s what ss parses.
The Key Insight
/proc is the single source of truth for process information. Every process monitoring tool reads from /proc. When tools fail or aren’t installed, /proc is still there.
You don’t need ps to see processes; they’re in /proc. You don’t need lsof to see file descriptors; they’re in /proc/[pid]/fd. You don’t need special tools to see environment variables; they’re in /proc/[pid]/environ.
Learning /proc gives you a foundation that works on any Linux system, with or without tools. And honestly, understanding how the kernel exposes this data makes you a better systems engineer.
Next time something goes wrong, don’t reach for another tool. Start with /proc. The answers are already there.
