Linux Networking Basics

When a service stops responding in production, the first reaction is often to blame the application. However, the actual cause usually lies below the application layer: a firewall rule silently dropping a health-check probe, a DNS cache holding onto an outdated record, or a bind address set to 127.0.0.1 instead of 0.0.0.0. Linux networking for DevOps focuses on quickly identifying issues at this level, relying on three key areas: managing iptables, diagnosing DNS, and following a layered troubleshooting process before the on-call phone starts ringing nonstop.

Key Takeaways:

• Modern Ubuntu uses iptables-nft, which converts iptables commands into nftables rules internally, so the familiar commands still work.
• Set the INPUT chain default policy to ACCEPT and place an explicit DROP rule at the end, preventing a rule flush from locking you out of the server.
• Limit SSH connection attempts using the iptables recent module and log dropped packets through a dedicated, rate-limited chain to avoid overwhelming logs.
• Use dig for scripting and detailed diagnosis; nslookup is better suited for quick interactive checks. NOERROR responses with zero answers are valid, while NXDOMAIN indicates a missing domain.
• Follow a bottom-up troubleshooting process: check link, IP, routing, firewall, service, DNS, and finally packet capture.

Why the Layered Approach Wins

Most real-world network problems involve multiple layers of the TCP/IP protocol suite. A service might seem unreachable due to a misconfigured default gateway at the network layer, a disabled network interface at the data link layer, or a blocked port at the transport layer. Randomly switching between tools wastes time and can lead to incorrect conclusions. The most effective method is to start with the lowest layer and work upward, as explained in our earlier Linux networking for DevOps overview.

The sequence matters because each step is quick and eliminates a whole category of issues. Verify the interface is UP and error-free, check the IP address and subnet, confirm the route to the gateway, inspect firewall rules, ensure the service is listening, test DNS resolution, and only then capture packets for deeper analysis. This methodical process distinguishes fact-based debugging from guesswork, a point supported by the DevOpsCube command reference and practical troubleshooting resources.

For DevOps and SRE teams, this approach matters for three reasons. Outages often stem from subtle misconfigurations below the application layer rather than application bugs. Security relies on minimal firewall rules and properly configured DNS. Automation tools like Ansible, Terraform, and Kubernetes expect consistent networking, so configuration drift in the underlying network components can cause deployment failures that are difficult to trace.

iptables Management in 2026: From Basics to Production Rules

iptables is the traditional firewall framework built on the Netfilter kernel subsystem, responsible for packet filtering, NAT, and connection tracking. Although nftables and firewalld are gaining popularity, most production systems still use iptables. Understanding iptables remains important because on modern Ubuntu systems, the default iptables binary is iptables-nft, which translates iptables commands into nftables rules internally. The /usr/sbin/iptables symlink points through /etc/alternatives to iptables-nft, providing the familiar interface with a modern backend. Switching to the legacy backend using update-alternatives is possible, but the default handles most production workloads, so most teams do not need to change it.

The most critical practice involves setting the default policy. Many tutorials recommend setting the INPUT chain default to DROP. This can cause problems: running iptables -F clears all rules but leaves the default policy unchanged, so if the default is DROP, all packets (including your SSH session) are blocked. Instead, keep the default policy ACCEPT and add an explicit DROP rule at the end of the chain. If the rules are flushed, the DROP rule disappears but the ACCEPT default keeps your connection alive. This disables the firewall temporarily but prevents lockout. For safer testing, use iptables-apply, which applies rules temporarily and reverts after ten seconds unless confirmed.

Start with connection tracking to build a reliable baseline. The conntrack module maintains a table of active connections. The first rule should accept established and related traffic, then drop invalid packets. Without the established rule, responses to outgoing connections get dropped, breaking DNS resolution and HTTP responses. On busy servers, the conntrack table can grow large, so tune it via /proc/sys/net/netfilter/nf_conntrack_max and nf_conntrack_tcp_timeout_established.

# Secure baseline: allow established/related, drop invalid, open only what you need
iptables -A INPUT -m conntrack --ctstate ESTABLISHED,RELATED -j ACCEPT
iptables -A INPUT -m conntrack --ctstate INVALID -j DROP
iptables -A INPUT -i lo -j ACCEPT
iptables -A INPUT -p tcp --dport 22 -j ACCEPT # SSH
iptables -A INPUT -p tcp --dport 80 -j ACCEPT # HTTP
iptables -A INPUT -p tcp --dport 443 -j ACCEPT # HTTPS
iptables -A INPUT -j LOG_DROP # explicit drop via logging chain

SSH brute-force attacks are persistent and will target any public server within minutes of deployment. The iptables recent module can limit new SSH connections without needing external tools like fail2ban. One rule marks new connections, and another drops sources that have made four or more attempts within a short period. This effectively stops brute-force attempts and reduces log noise and CPU usage. For visibility, create a dedicated LOG_DROP chain that logs dropped packets with a prefix before dropping them, and throttle logging with the --limit option to prevent log flooding.

Docker adds complexity by automatically creating iptables rules to manage container networking and port forwarding. It creates chains like DOCKER, DOCKER-BRIDGE, DOCKER-CT, DOCKER-FORWARD, and DOCKER-INTERNAL. The DOCKER-USER chain is the only one you should modify to filter traffic to containers, since editing Docker’s other chains can disrupt container networking. If you use containers, expect to see rules you did not create and understand their purpose before changing them.

Persistence is a common pitfall. iptables rules exist only in kernel memory and disappear on reboot. On Ubuntu, install the iptables-persistent package and run netfilter-persistent save to save rules to /etc/iptables/rules.v4 and rules.v6, which a systemd service loads at startup. On Red Hat-based systems, use /etc/init.d/iptables save. Always back up your current rules with iptables-save > /tmp/rules.v4 before making changes so you can quickly restore if needed.

The iptables vs. nftables vs. firewalld Decision

For new projects, nftables provides more maintainable rule sets. For existing systems, iptables remains the default on many distributions. If you migrate, do so gradually and test each rule carefully, since differences in stateful handling can disrupt workflows. Document the purpose of every rule to avoid leftover cruft and hidden security gaps.

DNS Configuration and Troubleshooting

DNS resolution problems cause many production outages and slowdowns. Linux reads /etc/resolv.conf to configure resolvers, listing nameservers and search domains. When systemd-resolved is active, this file is usually a symlink to /run/systemd/resolve/stub-resolv.conf, which points to the local stub listener at 127.0.0.53. That explains why dig may show a loopback server even though your actual upstream servers are different, and why you should use resolvectl status to see the real upstream servers for each interface.

For diagnosis, dig is preferred because it sends queries directly to resolvers and outputs the full response in a structured format. Use dig +short for concise answers, dig @server to query a specific resolver, dig -x for reverse lookups, and dig +trace to follow resolution from root servers down and identify where failures occur. dig +dnssec verifies signed records, and dig +tcp forces TCP to check if firewalls block large DNS responses. nslookup is simpler and interactive, suitable for quick checks, but its output is harder to parse and lacks trace and DNSSEC features. The OpsCheck DNS troubleshooting guide recommends using dig for scripting and debugging, and nslookup only for quick interactive lookups.

A common confusion involves dig returning NOERROR with zero answers, which means the query was processed but the requested record type does not exist at that name. NXDOMAIN means the entire domain name does not exist. Mixing these up leads to chasing nonexistent problems. Also, a CNAME at the zone apex fails silently because RFC 1034 forbids a CNAME coexisting with SOA and NS records, causing unpredictable resolver behavior. Negative caching can cause issues too: if a resolver cached an NXDOMAIN before you created the record, the SOA minimum field controls how long that negative cache lasts, so a low TTL on the new record does not override a longer negative TTL.

For automated infrastructure, always template /etc/resolv.conf and verify it after each deployment. Avoid hard-coding it on cloud VMs that use DHCP or cloud-init, since configuration drift there causes intermittent, hard-to-trace failures. Be aware that Docker and Kubernetes inject their own resolver settings that override the host configuration. If you deploy services relying on DNS-based challenge validation for certificates, review our DNS challenge validation model to prevent propagation delays and rate-limit errors.

Networking Troubleshooting Workflows

When a service is unreachable or latency increases, follow the troubleshooting steps from the bottom up. This assumes you have SSH or console access to the affected system. Each step eliminates a layer of potential issues:

• Verify the link: run ip link show. Confirm the interface is UP with no errors, and check dmesg or syslog for dropped packets or disconnects.
• Check IP addressing: run ip addr show. Confirm the correct IP, subnet, and broadcast addresses, and watch for overlapping CIDRs.
• Test routing: run ip route show and ping the default gateway. Use traceroute or mtr to map multi-hop paths and detect routing loops or blackholes.
• Inspect the firewall: run iptables -L -n -v. Check byte counters to verify whether rules match as expected and whether traffic is dropped or rejected.
• Confirm service exposure: run ss -tulnp. Look for accidental binds to 127.0.0.1 instead of 0.0.0.0, which makes services unreachable externally even if running.
• Test DNS: use dig or nslookup. Verify hostnames resolve to correct IPs and check both IPv4 and IPv6 results.
• Capture packets: run tcpdump last, filtering by port, protocol, or source IP for detailed analysis.

Two symptoms often cause confusion. Connection refused usually means no service is listening on that port or the client used the wrong port. Connection timed out usually indicates a firewall, security group, incorrect route, or a host that is down. Remember that ping only tests ICMP reachability, not TCP; use curl or nc -vz host port from a peer in the same network zone to test actual service availability. MTU problems on VPNs cause strange partial failures that can appear as TLS timeouts, so check MTU with ping -M do -s 1472. Incorrect MTU settings disrupt VPN tunnels, overlays, and cloud interconnects.

If you suspect asymmetric routing, use conntrack -L to examine the stateful connection tables. In containerized setups, verify that the correct iptables chains, such as DOCKER-USER for Docker or KUBE-FORWARD for Kubernetes, handle traffic properly, and check both host and overlay network paths. To correlate network events with application logs, central log aggregation using tools like ELK, Loki, or Fluentd helps provide a complete view; see our log aggregation guide for details.

Common Pitfalls and How to Avoid Them

Even experienced engineers encounter recurring issues in production. The most frequent mistake is forgetting to save firewall rules: iptables rules exist only in memory and disappear after a reboot unless saved with netfilter-persistent save, leaving services exposed. Another common error is locking yourself out by setting the default policy to DROP, which is why the ACCEPT-default-plus-explicit-DROP pattern is important. Always test firewall changes inside a persistent screen or tmux session and keep out-of-band console access available via IPMI, iDRAC, or a cloud serial console.

DNS misconfiguration is another frequent cause of problems. Never hard-code /etc/resolv.conf on machines using DHCP or cloud-init, and always clear the local DNS cache with resolvectl flush-caches after updating upstream records, since stale caches can cause hours of downtime. Split DNS setups can cause confusion: in hybrid environments, different servers resolve internal and external names, so use dig @server to test against the intended resolver.

Do not overlook IPv6. Many tools default to IPv4, but dual-stack environments, especially Kubernetes, require explicit ip6tables rules. Without IPv6 firewall rules, IPv6 traffic remains unprotected. Finally, log dropped packets with a rate-limited rule before your final DROP. Continuous packet drops can fill logs quickly if unchecked, so always limit logging frequency. For containerized deployments, understand how your orchestrator manages host and overlay networking; our Docker multi-stage build guide covers related deployment details.

Mastering these areas turns networking from a source of mysterious outages into a manageable part of your infrastructure. Start with the bottom-up workflow, secure your firewall using the ACCEPT-default pattern and rate-limited SSH, and make dig your primary DNS tool. Keep this guide handy as your on-call reference, and use every incident to improve your runbook.

Related Reading

More in-depth coverage from this blog on closely related topics:

• How to Spot AI-Generated Pictures
• How to Comply with EU AI Act Article 50
• Best Hardware for Large AI Models
• iOS 26.5 update: new features, bugs
• Meta AI Content Watermarking Risks in 2026

Sources and References

Sources cited while researching and writing this article:

• 17 Best Linux Networking and Troubleshooting Commands – DevOpsCube
• How to Troubleshoot DNS Like a Pro Using dig and nslookup