#distributed computing

[utopicwork]

The groundwork is laid for DPUN/DiSCompute (distributed public utility network and distributed super computer respectively), I have written a generic async task system that locally and remotely networked workers can pull tasks from and run across across multiple cpu cores/cpus if allowed as well as a system for notifying when tasks are done and can be removed. There is a simple locking system with staleness for if a worker goes offline or is otherwise unable to finish the tasks they pulled so that the system can clean up dangling tasks. I'll implement Daisy for data soon so there will be a distributed data backend.

[blubberquark]

Share Your Anecdotes: Multicore Pessimisation

I took a look at the specs of new 7000 series Threadripper CPUs, and I really don't have any excuse to buy one, even if I had the money to spare. I thought long and hard about different workloads, but nothing came to mind.

Back in university, we had courses about map/reduce clusters, and I experimented with parallel interpreters for Prolog, and distributed computing systems. What I learned is that the potential performance gains from better data structures and algorithms trump the performance gains from fancy hardware, and that there is more to be gained from using the GPU or from re-writing the performance-critical sections in C and making sure your data structures take up less memory than from multi-threaded code. Of course, all this is especially important when you are working in pure Python, because of the GIL.

The performance penalty of parallelisation hits even harder when you try to distribute your computation between different computers over the network, and the overhead of serialisation, communication, and scheduling work can easily exceed the gains of parallel computation, especially for small to medium workloads. If you benchmark your Hadoop cluster on a toy problem, you may well find that it's faster to solve your toy problem on one desktop PC than a whole cluster, because it's a toy problem, and the gains only kick in when your data set is too big to fit on a single computer.

The new Threadripper got me thinking: Has this happened to somebody with just a multicore CPU? Is there software that performs better with 2 cores than with just one, and better with 4 cores than with 2, but substantially worse with 64? It could happen! Deadlocks, livelocks, weird inter-process communication issues where you have one process per core and every one of the 64 processes communicates with the other 63 via pipes? There could be software that has a badly optimised main thread, or a badly optimised work unit scheduler, and the limiting factor is single-thread performance of that scheduler that needs to distribute and integrate work units for 64 threads, to the point where the worker threads are mostly idling and only one core is at 100%.

I am not trying to blame any programmer if this happens. Most likely such software was developed back when quad-core CPUs were a new thing, or even back when there were multi-CPU-socket mainboards, and the developer never imagined that one day there would be Threadrippers on the consumer market. Programs from back then, built for Windows XP, could still run on Windows 10 or 11.

In spite of all this, I suspect that this kind of problem is quite rare in practice. It requires software that spawns one thread or one process per core, but which is deoptimised for more cores, maybe written under the assumption that users have for two to six CPU cores, a user who can afford a Threadripper, and needs a Threadripper, and a workload where the problem is noticeable. You wouldn't get a Threadripper in the first place if it made your workflows slower, so that hypothetical user probably has one main workload that really benefits from the many cores, and another that doesn't.

So, has this happened to you? Dou you have a Threadripper at work? Do you work in bioinformatics or visual effects? Do you encode a lot of video? Do you know a guy who does? Do you own a Threadripper or an Ampere just for the hell of it? Or have you tried to build a Hadoop/Beowulf/OpenMP cluster, only to have your code run slower?

I would love to hear from you.

[sciencesideanswers]

can you explain why there are some posts which are like a week old and out of nowhere they get a note and it shows as 1 favourite and when you open that its just empty?

Hi! I have two main hypotheses about this.

First hypothesis: Themes.

It is entirely possible that some themes just hide the notes that a particular post has. So, if you’re seeing a post page, you might be looking at the blog theme, and the theme might just not be showing the posts. This does not apply if you’re looking at the Tumblr Dashboard or the Tumblr mobile version of a post -- those don’t have custom themes.

Sometimes the theme design just didn’t consider notes, or sometimes they only considered a certain amount of them. Or it might just be coded weirdly. Furthermore, the Tumblr theme coding allows you to show any arbitrary subset of notes. I am not exactly sure why you would choose to only show some notes, but that’s something that you can absolutely do.

Here are Tumblr Docs on the code that themes use. Right now the notes are put on the page all or none (or loaded by AJAX, meaning that they will be loaded after the page has loaded). Still, custom JavaScript (code) or CSS (styles) on the page is something that you control and that gives you, as a coder, power to hide any of them.

Second hypothesis: Eventual consistency.

I wrote a bit about it in an old answer about the no-notes post, but here goes another try.

Tumblr is a really big platform. As such, it is impossible for Tumblr to keep everything running from a single server. This means that they need to provide the content from different servers at the same time. This also means that some of that information (such as the notes for a post) must be present in multiple places at the same time… and that means, in turn, that when a new note comes in, it will be sent to one of those first. If you look at the post, and the second server is the one showing you the info, then you won’t see the notes.

(Disclaimer 1: It is a lot more complicated than this, I’m simplifying how it works.)

(Disclaimer 2: I’m making several assumptions -- only Tumblr employees know exactly what’s going on.)

Similar to Youtube and the views not being updated in real-time, the servers, later on, will catch up and all of them will show the right notes. That’s why the mechanism is called eventual consistency: it might be consistent at times, but it eventually settles down the differences.

The strategy, put simply, is a trade-off to be able to handle really big systems with a lot of data.

– Alpha

Support us on Patreon

[ph4z0n]

It’s time to collaborate, communicate and co-create the tools we need the most

There’s a special feeling at the start of something new. Excitement. Hope. That glimmer of what might be.

We felt it in 2016 at the Internet Archive’s first Decentralized Web Summit.  Two years later, we’re gathering to celebrate the working code that hints at the true potential of the Decentralized Web. Register here to secure your spot at the Decentralized Web Summit: Global Visions/Working Code, July 31-August 2.  You’ll be joining the founders and builders of decentralized protocols from around the world, along with lawyers, human rights activists, artists, and journalists. We’re all united by one thing—the desire for a Web that is more private, secure, censorship-resistant, and open—this time for good.

[utopicwork]

GitHub - deepseek-ai/3FS: A high-performance distributed file system designed to address the challenges of AI training and inference workloa

A high-performance distributed file system designed to address the challenges of AI training and inference workloads. - GitHub - deepseek-

Now this is interesting

"The Fire-Flyer File System (3FS) is a high-performance distributed file system designed to address the challenges of AI training and inference workloads. It leverages modern SSDs and RDMA networks to provide a shared storage layer that simplifies development of distributed applications."

[pai3ai]

How PAI3.AI Power Nodes Help Reduce Barriers to Advanced AI Access

Artificial intelligence is transforming the way people work, create, and solve problems. From automating business processes to supporting research and development, AI has become an important tool across industries. However, access to advanced AI technologies is often limited by high infrastructure costs, technical complexity, and centralized systems that can be difficult for smaller organizations to use.

PAI3.AI is working to address these challenges through its Power Node network. By creating a decentralized AI infrastructure, PAI3.AI aims to make advanced AI resources more accessible to businesses, developers, and individuals. Power Nodes play a key role in helping the platform deliver AI services efficiently while reducing common barriers to adoption.

What Are PAI3.AI Power Nodes?

PAI3.AI Power Nodes are specialized network participants that contribute computing resources to the PAI3.AI ecosystem. These nodes help process AI workloads, support network operations, and provide the computational power needed for various AI applications.

Instead of relying on a single centralized data center, the network distributes tasks across multiple Power Nodes. This approach helps create a more flexible and scalable infrastructure that can support growing demand for AI services.

The Challenges of Accessing Advanced AI

Many organizations face obstacles when trying to adopt advanced AI solutions. Some of the most common challenges include:

High costs associated with AI infrastructure and hardware

Limited access to powerful computing resources

Complex setup and maintenance requirements

Dependence on centralized service providers

Scalability concerns as AI workloads increase

These barriers can prevent startups, small businesses, independent developers, and educational institutions from fully benefiting from AI technologies.

How Power Nodes Help Reduce These Barriers

1. Supporting a Distributed Computing Network

Power Nodes contribute computing resources to a decentralized network. By distributing workloads across multiple participants, PAI3.AI can help reduce the need for expensive centralized infrastructure.

This approach allows AI services to be delivered through shared resources, making advanced technology more accessible to a broader audience.

2. Improving Scalability

As demand for AI applications grows, infrastructure must be able to scale effectively. Power Nodes help expand the network's computing capacity by allowing additional resources to join the ecosystem.

This scalable model enables the platform to support increasing workloads without relying solely on large centralized facilities.

3. Lowering Entry Barriers

Traditional AI systems often require significant investments in hardware and technical expertise. Through its distributed network model, PAI3.AI seeks to provide easier access to AI capabilities.

Businesses and developers can focus on building solutions rather than managing complex infrastructure from the ground up.

4. Enhancing Network Reliability

A decentralized network can help improve operational resilience. Since computing resources are distributed across multiple Power Nodes, the network is less dependent on a single point of failure.

This structure helps maintain service availability and supports consistent performance across the ecosystem.

5. Encouraging Wider Participation

Power Nodes allow more participants to contribute to the AI ecosystem. This creates opportunities for collaboration while helping expand the overall computing capacity available to the network.

A broader participation model supports innovation and helps make AI resources available to a larger community.

Benefits for Businesses and Developers

Organizations using AI technologies can benefit from the Power Node ecosystem in several ways:

Access to scalable AI infrastructure

Reduced dependency on centralized providers

Improved resource availability

Support for diverse AI applications

Greater flexibility for future growth

These advantages can help businesses adopt AI solutions more efficiently while focusing on innovation and operational goals.

Conclusion

As artificial intelligence becomes increasingly important across industries, improving access to AI infrastructure remains a key challenge. PAI3.AI Power Nodes help address this issue by supporting a decentralized network designed to make advanced AI resources more accessible, scalable, and reliable.

By distributing computing power across a growing ecosystem of participants, PAI3.AI is helping reduce traditional barriers to AI adoption. This approach creates new opportunities for businesses, developers, and individuals to access the tools they need to explore and benefit from the future of artificial intelligence.

[markandsparkpriya]

Analyzing the Financial Trajectory of Edge Data Center Market Insights

The financial outlook for the Edge Data Center Market is characterized by steady, predictable growth. Starting from a valuation of USD 27,300 million in 2025, the industry's climb to an estimated USD 72,656 million by 2033 underscores its status as the bedrock of the modern digital sector. This momentum is supported by broad industrial acceptance and a clear, well-defined demand for low-latency processing that continues to expand globally alongside population growth and industrialization.

Navigating Structural Edge Data Center Market Trends

The Edge Data Center Market Trends are being redefined by public policy initiatives that prioritize digital self-sufficiency and the expansion of domestic communication supply chains. Governments are increasingly recognizing the strategic necessity of robust local infrastructure, leading to increased funding for factory modernization and specialized connectivity clusters. These systemic investments create a rising tide of demand that benefits hardware producers, telecommunication providers, and software developers alike.

Overcoming Industrial Barriers

The industry is working diligently to overcome the historic issues of high initial capital expenditure and the complexity of regulatory compliance across different jurisdictions. By emphasizing the integration of digital twin technology and predictive maintenance in edge node management, the narrative is shifting from "high-cost infrastructure" to "high-reliability processing." This change in perspective is instrumental in driving greater capital engagement and ensuring that infrastructure facilities remain competitive and profitable in an increasingly crowded global landscape.

Strategic Importance of Collaboration

Partnerships between hyperscalers and local telecom providers represent the next frontier of growth. As non-traditional sectors enter the digital space, the role of edge data centers will evolve to incorporate complex electronic integration and software-driven process control within the nodes themselves. Companies that foster this kind of collaborative, cross-functional ecosystem will be best positioned to capitalize on the diversifying digital landscape of the next decade.

[technology01281120643]

{

"@context": "https://schema.org",

"@type": "Article",

"headline": "",

"description": "",

"articleSection": "technology",

"keywords": "\"Unlocking the Potential of Edge Computing: How to Optimize IoT Devices for Real-Time Data Processing\"",

"datePublished": "2026-02-02T19:47:27.406768",

"dateModified": "2026-02-02T19:47:27.406768"

}

{

"@context": "https://schema.org",

"@type": "HowTo",

"name": "",

"description": ""

}