Discover Top Posts Tagged with #ebpf

AI-Assisted VoidLink Rootkit Hides in Linux Systems

VoidLink, a Linux rootkit, combines Loadable Kernel Modules and eBPF programs to stealthily hide processes and network connections, with AI-assisted development observed in its iterative coding and deployment.

Source: Elastic Security Labs

Read more: CyberSecBrief

#linux #malware #rootkit #aws #ebpf #cybersecurity

Build your own scrubbing center. Understand the mitigation pipeline, handle per-CPU race conditions, and drop malicious traffic at wire-spee

Your iptables firewall is already dead.

When a 100Gbps DDoS flood hits your infrastructure, relying on iptables, ufw, or fail2ban is just an exercise in futility.

Here is the hard truth: By the time a packet reaches netfilter, the Linux kernel has already executed context switches and allocated an sk_buff memory structure. If 20 million malicious packets arrive per second, the sheer overhead of allocating and destroying those structures will cause 100% CPU starvation. Your server is dead.

The Fix: The Kernel Bypass Revolution

It’s time to drop packets before the operating system even realizes they exist. We do this using eBPF & XDP (eXpress Data Path). You attach an eBPF program directly to the NIC driver, executing an xdp_drop instruction to discard the packet instantly with virtually zero CPU overhead.

The "Cloud VM" Illusion

Beware of tutorials running XDP on a 1Gbps Cloud VM. They show you beautiful flame graphs of low CPU usage, but it’s a fatal illusion. XDP saves your CPU, but it does not save your bandwidth. If a 40Gbps flood hits your 1Gbps VM, the pipe saturates instantly. Your upstream ISP will panic and issue a Null-Route (Blackhole), completely isolating you from the internet.

Engineering a Real Scrubbing Center

To build a true wire-speed mitigation pipeline, you need:

Raw Bandwidth: 10Gbps to 100Gbps unmetered uplinks on Dedicated Bare Metal to physically absorb the volumetric attack.

BGP FlowSpec: To distribute the attack load across global datacenters.

Per-CPU LRU Maps: Writing production XDP code means handling multi-queue NIC architectures. You must use BPF_MAP_TYPE_LRU_PERCPU_HASH to prevent lock contention, race conditions, and memory exhaustion from spoofed IPs.

Stop paying the massive Cloudflare tax. Deploy raw compute and build your own scrubbing center.

📖 Read the full 100Gbps eBPF/XDP Engineering Blueprint here: 🔗 Dropping 100Gbps DDoS Attacks: The Ultimate eBPF & XDP Guide

#Cybersecurity #eBPF #XDP #DDoS #Linux #SysAdmin #DevOps #Networking #BareMetal #InfoSec #ServerInfrastructure #Tech #OpenSource #ServerMO

Observability Stack 2026: From Data Sprawl to Control Plane

The era of 'collect and keep everything' has met its financial and operational ceiling. As we move into 2026, the median enterprise spend on observability has surpassed $800,000 annually, with high-scale organizations often exceeding the $10 million mark. This surge isn't just a byproduct of more traffic; it is the result of a paradigm shift where observability has transitioned from a reactive debugging luxury into a mission-critical control plane for autonomous and agentic systems.,Modern stacks are no longer judged by the number of dashboards they host, but by their ability to provide high-cardinality insights without lighting the IT budget on fire. For the engineering leader in 2026, setting up an observability stack is less about selecting a vendor and more about architecting a unified telemetry pipeline that balances deep kernel visibility with intelligent, cost-aware data routing. Standardizing the Edge with OpenTelemetry and eBPF In 2026, OpenTelemetry (OTel) has achieved a staggering 95% adoption rate for new cloud-native instrumentations, effectively ending the era of proprietary vendor agents. The first step in a modern setup is the deployment of an OTel Collector-heavy architecture. This allows teams to decouple instrumentation from the backend, providing the flexibility to switch providers—a move 67% of IT leaders now consider within a two-year window to avoid vendor lock-in. Complementing OTel is the rise of eBPF (Extended Berkeley Packet Filter) for zero-instrumentation visibility. By 2027, it is estimated that 40% of production telemetry will be gathered at the kernel level, allowing platform teams to capture networking, syscalls, and security events without touching application code. This 'invisible' layer is crucial for monitoring the non-deterministic behaviors of agentic AI workflows that traditional tracing often misses. Breaking the Log Jam: AI-Driven Data Tiering Logs currently consume over 50% of the average observability budget, yet industry data shows that up to 80% of log volume consists of repetitive, low-value 'heartbeat' messages. The 2026 stack solves this through 'Adaptive Telemetry.' By implementing streaming aggregators like RisingWave or specialized OTel processors, organizations are now summarizing repeated patterns at the source while routing raw, high-fidelity data to low-cost object storage (S3/Azure Blob) via open formats like Apache Iceberg v3. This 'hot-cold' separation is no longer manual. AI-native observability platforms now use pattern recognition to automatically downsample 'normal' operations while instantly 'hydrating' or replaying detailed logs when an anomaly is detected. This strategy has allowed early adopters in the financial sector to reduce their annual ingestion costs by 35% without compromising their 2 am troubleshooting capabilities. From Dashboards to Decisions: The Rise of SLO-Driven AI Ops The most significant evolution in 2026 is the migration of Service Level Objectives (SLOs) from static charts to active decision-making engines. Organizations with mature observability practices now report 79% less downtime than those stuck in fragmented monitoring. This is achieved by feeding real-time telemetry directly into automated remediation loops, where AI 'collaborators' suggest or execute configuration rollbacks based on breach-risk forecasts. As we look toward 2027, observability is expanding into the 'Black Box' of GenAI. High-performing stacks now include specific instrumentation for LLM latency, token usage, and hallucination rates. By correlating these AI-specific metrics with traditional infrastructure health, teams can finally pinpoint whether a slow response is a failure of the model, the vector database, or a simple TCP timeout in the underlying Kubernetes cluster. Building an observability stack in 2026 is an exercise in strategic restraint and architectural foresight. The goal is no longer to see everything, but to ensure that the signals you do see are actionable, cost-effective, and standardized. By centering your strategy on OpenTelemetry and eBPF, you aren't just fixing today’s bugs; you are building the infrastructure required to govern the increasingly autonomous digital ecosystems of tomorrow.,As systems grow in complexity and the line between human and machine agency blurs, your telemetry will be the only source of truth that matters. The question for 2027 isn't whether you have enough data, but whether your data is smart enough to let your engineers focus on innovation rather than fire-fighting. Read the full article

#andAI-drivendatatieringareslashingMTTRandcontrollingspiralingcloudcosts.#eBPF #Masterthe2026observabilitystack.LearnhowOpenTelemetry

Modern Observability: Engineering the 2026 Telemetry Stack

The transition from simple monitoring to deep observability represents a fundamental shift in how we interpret the digital ghost in the machine. In early 2026, as distributed microservices move toward autonomous orchestration, the old guard of 'pre-defined dashboards' has crumbled under the weight of sheer complexity. Engineering teams are no longer asking if a system is up; they are hunting for the 'why' behind transient 400ms spikes that only affect 0.5% of users in specific geographic clusters.,True observability isn't a tool you buy—it's a capability you build through a telemetry-first culture. By unifying traces, metrics, and logs into a single, high-cardinality data stream, organizations can finally move past the fragmented 'war room' culture. This guide deconstructs the architectural blueprint required to gain total visibility into the modern cloud-native stack, where the distance between a code commit and a production outage is measured in microseconds. Standardizing the Foundation with OpenTelemetry and eBPF The heartbeat of any modern stack starts with OpenTelemetry (OTel), which has effectively ended the era of vendor lock-in. In the current 2026 landscape, the OTel Collector acts as the universal translator, decoupling the application's instrumentation from the backend storage. By deploying collectors as sidecars or daemonsets, SREs are capturing rich context—metadata like container IDs, commit hashes, and cloud provider regions—ensuring that every span tells a complete story. To reach the next level of granularity without the 20-30% performance tax of traditional agents, industry leaders like Netflix and Datadog have pivoted toward eBPF-based instrumentation. This 'kernel-level' visibility allows for zero-code instrumentation of the network stack and system calls. For instance, recent benchmarks show that eBPF-powered probes can capture distributed traces with less than 2% CPU overhead, making it feasible to observe high-throughput environments where every clock cycle is a precious commodity. Solving the Cardinality Crisis in Metrics Storage As systems scale, the traditional Prometheus-style metrics model often hits the 'cardinality wall'—where the sheer number of unique label combinations explodes the memory requirements of the time-series database. By 2027, the volume of telemetry data is projected to grow by 400% across enterprise DevOps teams. To survive this, the stack must incorporate sophisticated aggregators like Mimir or VictoriaMetrics, which utilize advanced sharding and persistent storage on S3-compatible backends to keep costs manageable. Smart sampling is the secret weapon of the data scientist in this space. Instead of capturing every single successful 200 OK response, elite stacks utilize 'tail-based sampling' to keep 100% of errors and high-latency traces while discarding the noise of healthy traffic. This strategy ensures that when a P99 latency spike occurs at 3 AM, the investigator has the full diagnostic trace available, rather than a thinned-out representation that hides the root cause. The Rise of Trace-to-Log Correlation The historical silos between logging and tracing are dissolving. In a high-functioning observability stack, a single 'TraceID' acts as the connective tissue that binds a distributed request across twenty different microservices. When an engineer clicks on a span in a Jaeger or Honeycomb UI, the stack should immediately surface the specific log lines emitted by that exact execution thread. This 'pivoting' capability reduces the Mean Time to Identification (MTTI) from hours to seconds. Moving into 2027, we are seeing the integration of vector-based log searching. By applying machine learning to log patterns, the stack can automatically group millions of lines of text into 'clusters' of behavior. If a new deployment introduces a subtle logic flaw, the system doesn't just alert on the error; it highlights that the error pattern is unique to the latest container image hash, effectively automating the first stage of the investigation. Operationalizing Insight with SLO-Driven Alerting The final layer of the stack is the human interface: how we consume these millions of data points without succumbing to alert fatigue. The most successful organizations have abandoned 'static threshold' alerts—like firing a page when CPU hits 80%—in favor of Service Level Objectives (SLOs) based on the user experience. By measuring the 'Error Budget,' teams can mathematically determine if a system's instability is significant enough to warrant waking up an engineer. These SLOs are powered by the very same high-cardinality data captured at the base of the stack. When the burn rate of an Error Budget accelerates, the stack automatically generates a 'snapshot' of the system state, including relevant traces, recent logs, and infrastructure metrics. This provides the 'incident responder' with a pre-packaged investigative brief, allowing them to focus on remediation rather than data gathering in the heat of a production crisis. Building an observability stack is an iterative journey toward radical transparency. As we look toward the horizon of 2027, the focus is shifting from simply seeing the system to understanding its intent. The architecture we have explored—rooted in open standards, kernel-level efficiency, and smart data sampling—is what separates the organizations that merely survive outages from those that use them as catalysts for architectural evolution.,The future belongs to the 'observable' enterprise, where the telemetry stream is treated with the same rigor as the primary application database. By investing in this foundation today, engineers gain the freedom to innovate at velocity, confident that no matter how complex the system becomes, the light of observability will always find the path back to the truth. Read the full article

#Beyondmonitoring:Adata-drivenguidetoarchitectinghigh-cardinalityobservabilitystacksusingOpenTelemetry #eBPF

Modern Observability Stack Setup: 2026 Engineering Guide

The era of 'monitoring' as a fragmented collection of dashboards is dead. In 2026, the complexity of distributed systems and the explosion of machine-scale workloads have turned observability into a core survival metric rather than a luxury. With the median annual downtime for enterprises still hovering around 77 hours, and high-impact outages costing upwards of $1 million per hour, the stakes for a perfectly tuned telemetry pipeline have never been higher. Engineers no longer just watch systems; they architect the very data flows that allow those systems to be understood.,As we move into 2027, the shift toward 'Unified Observability' is driven by a stark reality: 97% of organizations have faced significant cost surprises in their telemetry spend. This guide moves beyond the basics to explore how a modern stack—built on OpenTelemetry and eBPF—can collapse data silos and provide a single source of truth that translates technical health into business resilience. Standardizing the Foundation with OpenTelemetry The first pillar of a 2026-grade stack is the total adoption of OpenTelemetry (OTel). Gone are the days of proprietary vendor agents that lock your data into a specific ecosystem. OpenTelemetry has become the undisputed industry standard, with a 445% surge in library downloads over the last two years. By implementing OTel, you decouple your instrumentation from your backend, allowing you to route traces, metrics, and logs through a unified collector that handles processing and export with surgical precision. To maximize efficiency, leading engineering teams are deploying the OTel Collector as a Gateway pattern. This centralized approach allows for advanced tail-based sampling—crucial when you consider that by 2027, observability costs are projected to consume 15% of the total IT budget for many firms. By filtering out the noise at the edge and only sending high-value 'interesting' traces to your expensive storage tiers, you can maintain deep visibility while curbing the linear cost growth that breaks traditional budgets. The eBPF Revolution: Visibility Without Intrusion While OTel handles application-level insights, the 'blind spot' of 2026 remains the infrastructure layer. This is where eBPF (Extended Berkeley Packet Filter) has revolutionized the stack. By running directly in the Linux kernel, eBPF provides deep, low-overhead visibility into network traffic, security events, and system calls without requiring code changes. This 'zero-instrumentation' approach is essential for modern Kubernetes environments where pods are ephemeral and manual instrumentation of every microservice is a logistical nightmare. Integrating eBPF-based tools like Cilium or Coroot into your stack allows you to capture service-to-service communication and TLS-encrypted traffic metrics automatically. In 2026, this level of 'Auto-Observability' is the only way to keep up with the scale of Agentic AI workflows. It provides the ground truth needed to validate AI-driven decisions, ensuring that when an autonomous agent scales a service, your observability layer captures the 'why' behind the action in real-time. Collapsing the Silos: Moving to Unified Backends The 'LGTM' stack (Loki, Grafana, Tempo, Mimir) remains a powerful open-source choice, but the operational tax of managing four separate databases is driving a shift toward unified backends like ClickHouse. In 2026, the most mature 4% of organizations have moved toward a single-store architecture where logs, metrics, and traces are joined via standard SQL. This eliminates 'swivel-chair analysis'—the time-wasting process of jumping between different tools to correlate a metric spike with a specific log error. By 2027, it is estimated that 84% of organizations will pursue tool consolidation to solve this integration gap. A unified database allows for native cross-signal correlation, enabling your engineers to run complex queries that ask, 'Which specific database trace caused this 5% spike in p99 latency for my order service?' This capability is what reduces the Mean Time to Resolution (MTTR) from hours to minutes, directly protecting the bottom line in an era where every second of latency impacts user retention. The Rise of AgenticOps and AI-Driven Remediation The final layer of the 2026 observability stack isn't a data store, but an intelligence layer. 'AgenticOps' is moving beyond simple anomaly detection to autonomous troubleshooting. Modern platforms are now capable of simultaneously validating multiple hypotheses and suggesting deterministic remediations. For instance, if a memory leak is detected in a Canary deployment, the observability system can automatically trigger a rollback while providing a full root-cause analysis report to the developer's Slack channel. However, this autonomy requires strict 'Governance as Code.' As we approach 2027, the focus is shifting toward creating human-in-the-loop boundaries. Organizations are implementing policy engines that define exactly which actions an AI agent can take—such as restarting a pod—versus which require a senior engineer's approval. This balanced approach ensures that while the system scales and heals at machine speed, the human operator remains the ultimate pilot of the digital infrastructure. Building a world-class observability stack in 2026 is no longer about just gathering data; it is about engineering clarity from chaos. By standardizing on OpenTelemetry, leveraging the invisible power of eBPF, and consolidating telemetry into unified backends, organizations can transform their observability from a cost center into a strategic engine for growth. Teams that achieve full-stack visibility today are seeing 79% less downtime, a statistic that translates directly into market dominance.,The window to modernize is narrowing as the 'energy wall' and data explosion of 2027 loom on the horizon. Would you like me to generate a specific OpenTelemetry Collector configuration or a Prometheus recording rule to help optimize your current stack's data ingestion? Read the full article

#andunifiedtelemetryforreducingdowntimeandscalingROI.#eBPF #Masterthe2026observabilitystack.Adata-drivenguidetoOpenTelemetry

서버를 아무리 늘려도 트래픽 병목이 해결되지 않는 진짜 이유

갑자기 서비스에 엄청난 트래픽이 몰려서 서버가 뻗어버리는 현상. 인프라 다루는 분들이라면 정말 자다가도 벌떡 일어날 악몽이죠. 보통 이런 상황이 오면 가장 먼저 하는 조치가 오토스케일링(Auto-scaling)으로 서버 대수를 무식하게 늘리는 겁니다.

그런데 가끔 이런 답답한 구간에 부딪힙니다.

"컴퓨팅 자원을 아무리 쏟아부어도 응답 속도(Latency)가 전혀 개선되지 않네?"

왜 그럴까요? CPU나 메모리가 부족해서가 아닙니다. 서버의 네트워크 스택 자체가 패킷을 처리하다가 지쳐버렸기 때문입니다.

문제는 끊임없는 '컨텍스트 스위칭'

우리가 흔히 쓰는 프록시나 방화벽 툴들은 대부분 '유저 스페이스(User Space)'에서 동작합니다. 외부에서 데이터 패킷이 하나 들어올 때마다 내부에서는 이런 복잡한 과정이 일어납니다.

패킷 도착

커널 스페이스에서 유저 스페이스로 데이터 복사

보안 및 라우팅 로직 처리

다시 커널 스페이스로 데이터 전달

트래픽이 초당 수만 건 단위로 치솟으면 이 컨텍스트 스위칭(Context Switching) 비용 자체가 눈덩이처럼 불어납니다. 결국 시스템 전체를 마비시키는 주범이 되는 거죠.

이제는 커널 단에서 직접 쳐내야 할 때: eBPF

이 한계를 돌파하려면 패킷이 유저 스페이스로 올라오기도 전에, 아예 OS 커널의 가장 낮은 단에서 불필요한 트래픽을 드랍시키거나 라우팅해버려야 합니다. 최근 글로벌 테크 기업들이 앞다투어 인프라에 도입하고 있는 eBPF(Extended Berkeley Packet Filter) 기술이 바로 그 역할을 합니다.

eBPF를 활용하면 무겁고 느린 사이드카 프록시 대신, 네트워크 인터페이스 카드(NIC) 수준에서 실시간으로 데이터를 통제할 수 있습니다. 성능 차이는 굳이 말할 필요도 없이 압도적입니다.

단순한 개념을 넘어서 실제 프로덕션 환경에서 어떻게 eBPF를 구현하고 레이턴시를 0.1ms 단위로 최적화하는지 궁금하시다면, 최근에 읽었던 아래 가이드를 꼭 한번 참고해 보시길 권합니다.

👉 커널 레벨의 실시간 분산 시스템 확장 아키텍처 가이드 (C언어 XDP 후킹 코드 포함)

무조건 서버만 늘리는 시대는 지났습니다. 이제는 네트워크의 가장 깊은 곳, 커널의 효율성을 의심하고 최적화해야 할 타이밍입니다.

#서버개발 #백엔드 #트래픽처리 #네트워크 #eBPF #인프라엔지니어링 #분산시스템 #시스템아키텍처