STORJ.IO

DISTRIBUTED CLOUD STORAGE

[storjblog]

Development Update #5 from Storj Labs

We are just one week away from the Linux Foundation's Open Source Summit where Ben Golub and Shawn Wilkinson will be giving a keynote! They will make some major announcements on stage so our team has been focusing on finishing everything they need. The V3 network has made tremendous progress since our last development update. Here is what we have been working on:

Recent development accomplishments:

We added bucket support in V3 so you can now create and organize your data in buckets! This puts us one step closer to full AWS S3 compatibility.

We created a simple CLI (command line interface tool) with basic commands, like create buckets and upload/download objects. For a full list of available commands see our readme in GitHub.

We added secure node ID generation which is a proof of work defense against sybil attacks on the network.

We finished the bandwidth accounting protocol framework. This protocol will be used to keep track of how much bandwidth is used for payments.

Bugs, bugs, and more bugs! We squashed about a dozen bugs since our last development update. This really makes the v3 network feel polished.

Fixed the office fridge to prevent it from staying open and spoiling our coffee creamer. This will help us stay properly caffeinated to continue development on V3.

Improved error messages and the overall user experience.

In our next update, we plan to share updates on:

How the Open Source Summit conference went and a recap of the announcements we made.

Improvements to the CLI tool.

Enhancements to the pointer DB so that it only tracks pieces that are still valid.

Improving the user experience and fixing bugs.

If you want to know what else we have been working on, take a look at our previous development update or visit our GitHub repository. If you have questions, feel free to reach out by emailing [email protected] or through our community forum.

- Brandon Iglesias, Product Manager

[storjblog]

Join the Storj Labs Town Hall on September 12

Our team is excited to announce that we’ve selected a date for the next community town hall. The meeting will take place on September 12, 2018 at 1:00 p.m. EDT (see below for other time zones).

Please register for the town hall meeting here.

In the town hall meeting, you will hear Storj Labs leaders share updates about V3 development progress, new partnerships and much more. Presenters will include Ben Golub, our executive chairman and interim CEO; Shawn Wilkinson, co-founder and chief strategy officer; JT Olio, director of engineering; and John Gleeson, Vice President of Operations. Similar to the format used during our previous town hall meeting, there will be a short presentation followed by an extensive Q&A session. Please send questions you’d like us to address to [email protected].

If you cannot attend the meeting, the replay will later be available on the Storj Labs YouTube channel.

Storj Town Hall Details

Where: The town hall meeting will be live-streamed on our YouTube channel and on the Zoom.us platform. Livestream links will be emailed, along with a reminder, the day of the event. Please use this link to pre-register on Zoom.us. Be prepared to join the meeting by downloading the Zoom.us software before the meeting starts.

Date: September 12, 2018

Time: 1:00 p.m. EDT, 5:00 p.m. GMT, 8:00 p.m. MSK and 10:00 a.m. PST. Please check your local time.

Town hall format: 20-minute presentation and a 40-minute Q&A.  

Stay in touch with Storj on social media:

Twitter

YouTube

Rocket.Chat

Reddit

- The Storj Labs Team

[storjblog]

Planned Network Testing Ahead of V3 Launch

As we approach the release of the new Storj V3 platform, we will be performing some tests on the current network related to farmer reputation.

The purpose of these tests is to verify and validate enhanced network metrics and the reputations of long-term, highly available and reliable farmers. The data gathered throughout the process will be used as part of the evaluation criteria for granting early V3 network access to our current farmer base. During the course of this test, we will only upload first-shard data to locations with a maximum of 5 nodes per IP address. In case you are running more nodes, we advise you to shut down the excess ones to achieve higher per-node-usage and efficiency. We will also be conducting audits and periodic downloads in conjunction with our storage upload testing.  

As we begin the transition from the current network to our new V3 network, we aim to prioritize farmers with a proven track record of availability, reliability and consistently high reputation.

We’ll be sending out more communications on the V3 network as we approach public alpha in the coming months.  

In the meantime, here are a few things to get you up to speed about what we’re building:

Check out this overview for the V3 network from JT Olio, our Director of Engineering

Join the V3 waitlist so we can keep you up to date on our release cycle

Keep a close eye on our GitHub repo and see all the updates on the current master branch

Thank you for your continued support of the network!

- John Gleeson, VP of Operations

[storjblog]

Development Update #4 from Storj Labs

Since our last development update, a lot has happened here at Storj Labs. Two of our leaders, Ben Golub and Shawn Wilkinson, will be giving a keynote at The Linux Foundation's Open Source Summit in Vancouver on August 29th-31st. We have been spending a lot of time getting ready for some major announcements that we will be making at the event so stay tuned for that news! Aside from that, we have made a lot of great progress on V3. Check out our latest accomplishments below.

Recent development accomplishments:

We are now able to upload and download data on the V3 network!

We made it possible to run an entire V3 network locally with just a few simple commands. This will be great for anyone who wants to contribute to the project or test the network locally.

We finished creating the stat database, which keeps track of simple node metrics, like audit success ratios, latency, amount of data stored, etc. This data will be an integral part of our reputation system in the future.

We updated the GitHub readme for the V3 repo.

We increased the test coverage and uncovered some bugs, which we fixed.

In our next update, we plan to share updates on:

Bug fixes and overall improvements in V3.

Preparations for Open Source Summit and the announcements we will be making.

Adding furnishing to support ‘buckets’ in V3. This will let you organize and store your data in different buckets as you please.

Adding the finishing touches to our bandwidth accounting protocol, which is being used to keep track of how much bandwidth farmers have used.

If you want to know what else we have been working on, take a look at our previous development update or visit our GitHub repository. If you have questions, feel free to reach out by emailing [email protected] or through our community forum.

- Brandon Iglesias, Product Manager

[storjblog]

Development Update #3 from Storj Labs

We hope you are staying cool during these hot summer months, here at Storj we are just starting to heat up! We have continued to make great progress towards V3 and want to keep you in the loop on the latest updates.

Recent development accomplishments:

We created a wrapper around the object store interface for the Minio gateway so that if you want to use Minio, you have the ability to do CRUD operations.

We added configuration handling to the heavy client. This allows you to set your ports, file destination, options for services, and keys.

We combined the network state, reputation system, and DHT cache into a single service to make it easier to run. For piece storage, we created a start/ stop method which handles creating a database and starting all the services for you.

We added “errcheck” linting to the V3 repo via Travis (the bot) to ensure we do not overlook any reported error in our code.

We increased our overall test coverage in the V3 repo.

We added gRPC for DHT node to node communication.

We added a bunch of sections to the white paper. It is taking shape and will be shared with the community as soon as possible.

In our next update, we plan to share our progress on these projects:

Creating a simple command line tool (CLT) for starting an S3 gateway and CRUD capabilities.

Creating a developer tool to run a mini-network on your development computer for easier testing.

Adding support for buckets in the pointer database so that you can organize your data into however many buckets you please.

Implementing our bandwidth allocation protocol, which will be used to keep track of how much bandwidth farmers have used so that they can be paid for it. The protocol is nearly complete we are just adding some finishing touches.

Increasing the test coverage in our code base.

If you want to know what else we have been working on, take a look at our previous development update or visit our GitHub repository. If you have questions, feel free to reach out by emailing [email protected] or through our community forum.

- Brandon Iglesias, Product Manager

[storjblog]

Ask Storj: Dealing with Crypto Scams

Q: What do you think about all the scams in our industry and can you offer some guidance? There are all kinds of scams on Twitter and other communities online.

A: This is a rampant problem throughout the entire crypto/blockchain industry. People are impersonating companies on Twitter, Slack, Rocket.Chat and Telegram, and sending phishing emails—it's a constant and every day challenge. Our community leaders do their best to notify people and users of these scams, ban malicious users from our communities, and stay on top of the latest scams, but there’s still a constant deluge of scammers trying to impersonate Storj Labs and other companies online.

Nearly every crypto company has experienced the same thing. Just this week, Chris McClure and the Svandis team issued an open letter to Twitter and Jack Dorsey, urging them to address the huge issue with bots, scammers and abusive behavior on the platform. We also signed the letter, as we have experienced issues with bots attempting to scam our community via Twitter accounts impersonating Storj Labs.

We try our best to work with the platforms to combat abusive behavior. We made some changes on Rocket.Chat to reduce the flow of fraud there. We highly encourage anyone, if you see impersonators on Twitter, to report them. We've done our best to ban and remove abusive accounts on Reddit. The problem resembles whack-a-mole and I don't think there is a perfect solution.

We will never do any Storj giveaways that require you to send us any private key. Users should never send their crypto to Storj Labs, unless they are paying for services as a user. Farmers should never pay any Ethereum, Storj, or other cryptocurrency to join our community.  In short, it helps to have a healthy dose of common sense. Just like anything else, if it sounds too good to be true, it probably is. Watch out for yourself and we will do our best to watch out for you and keep these scammers out of our various forums and online communities.

If you ever receive any communication from Storj Labs, it will always come from some domain at storj.io. If you are ever in doubt about an email that claims to be from Storj Labs, please forward it to [email protected] and we will verify its authenticity.

—Shawn Wilkinson

[storjblog]

Announcing the Network V3 Farmer Waitlist

Today, the Storj Labs team is excited to announce we have launched a waitlist for farmers who would like early access to the next version of the Storj network. Community members who would like to join the waitlist can visit the Storj Labs website to reserve their slot as one of the first people to share their hard drive on the V3 network. We previously opened a V3 waitlist for developers, companies and other people who would like to store data on the new network. This waitlist is for farmers who would like to share their excess hard drive capacity with the network in exchange for STORJ tokens.

At the launch of the alpha and beta of the new network, we will be limiting both farmer and renter access to ensure we can maintain a healthy balance of both supply and demand. This will improve stability and testing, while also limiting any impact to reliability, performance or file durability as we fine-tune the platform for production launch.

Farmers who join the waitlist will be given early access to set up a storage node on the network.

The Latest on Network V3  

Our team has made significant progress building V3 of the Storj network. Just this week, our engineering team released an update that highlights the latest development milestones. We also recently released a V3 overview video with Storj Labs Director of Engineering JT Olio, where he explains how the new network will overcome many of the limitations we’ve uncovered in the current system to improve reliability and scalability, while also achieving AWS S3 compatibility.

Stay Up to Date

To keep up with the latest updates from Storj, join our community chat, follow us on Twitter and subscribe to our newsletter.

[storjblog]

Development Update #2 from Storj Labs

Since our last development update, our engineering team has completed several key pieces of the V3 platform. Here are some of the highlights; we hope you are as excited as we are about this new functionality!

Recent development accomplishments:

We finished adding pagination support to our pointer database so that the system can return specific results given a starting offset and limit. The pointer database keeps track of node IPs and ports.

We automated some of our deployment practices. For example, when something is merged to master, we automatically deploy some test farmers to look for bugs. If the merge affects the heavy client, we will automatically rebuild the heavy client Docker image and deploy it to begin generating test coverage.

We implemented an interface (transport client) that handles all of the gRPC dial out complexities so that establishing connections between nodes is very simple. The transport client would be utilized by our S3 gateway (Minio).

We added tests in pkg/kademlia.

We created a few diagrams of some of the services we are building to help explain them in the V3 white paper. The V3 white paper will be published in the near future.

We set up a CLA bot (Contributor License Agreement) so that in the future, if someone wants to contribute code to the repository, they will automatically be prompted to sign our CLA.

In our next update, we plan to share our progress on these projects:

Implementing our bandwidth accounting protocol, which will be used to keep track of how much bandwidth farmers have used so that they can be paid for it.

Creating a stat database to keep track of farmer stats such as uptime, audit success ratios, latency, and data stored.

Continuing work on our V3 white paper so that is can be published in the near future.

Increasing the test coverage in our code base.

Continuing work on the heavy client to combine the network state, reputation and DHT cache into a single process. Along with this effort, we are also adding the ability to update the configs for anyone who wants to run a heavy client in the future.

If you would like to take a look at everything we have been working on please visit our GitHub repository. If you have questions, feel free to reach out by emailing [email protected] or through our community forum.

- Brandon Iglesias, Product Manager

[storjblog]

Announcing a Campaign for a New Logo

Leading up to the launch of Storj network V3, we feel the time is right to update the brand to better reflect the resilient, distributed nature of our platform. The V3 network will deliver a massive improvement in durability, scalability, performance and ease of integration. It will also support streaming, drop-in S3 compatibility and achieve a significant improvement in both performance and reliability.

The original Storj logo was designed in our early years, shortly after Shawn Wilkinson, our founder and chief strategy officer, won the Texas Bitcoin Hackathon in 2014 and before our first token sale that same year. We still love the logo and plan to employ it in certain use cases moving forward, however with V3’s launch being such a pivotal moment for our business, the time is right to refine our brand so it better communicates our values and encapsulates the vision of decentralized cloud storage.

Tardigrade is Storj Grade

For the redesign process, we’ve selected the tardigrade as our theme. Whether you call them water bears, moss piglets or tardigrades, we view these microscopic creatures as the perfect embodiment of the Storj network.

Tardigrades are highly distributed and inhabit every part of the world, from hot deserts to rainforests to the arctic. Storj is also highly distributed, with thousands of storage nodes across the entire world, including more than 200 countries and territories (we aren’t in Antarctica yet). Tardigrades are also highly resilient and can survive the vacuum of space, extremely hot and cold temperatures, radiation and drought - being able to live for more than 30 years without food or water. So is decentralized storage, and our V3 network will be even more resilient through the incorporation of new technologies and erasure encoding approaches.  

Stay tuned for more details! 

[storjblog]

Development Update #1 from Storj Labs

Since our town hall earlier this month, our engineering team has made significant progress towards building V3 of the Storj network. To continue our goal of transparency, we will provide regular development updates to keep the community apprised of our progress and the milestones we meet. You can also follow our GitHub, where we regularly commit updates to our repositories. 

Recent development accomplishments:

Our engineering team has created a way to track a file’s TTL (time-to-live). This enables us to execute certain tasks, like automatically deleting files from the network when their TTL expires. This differs from the current network, which did not utilize TTL-style semantics and simply deleted files automatically every 90 days. The old approach was more in-line with other decentralized storage providers, however this new approach is better suited for users who are used to file management in traditional cloud environments.

We added gRPC for storage operations to storage nodes. This is how different nodes (both heavy clients and farmers) across the network communicate with one another and exchange various information, including portions of files.

Our team also automated the deployment of the heavy client when code is successfully merged to master branch for internal staging tests.

We continued our work on the V3 white paper, including creating a collection of technical diagrams that will be incorporated for reference.

Our continuous integration tool (Travis), is now measuring our test coverage so that we can ensure it does not fall below a certain threshold. This reduces the chances of introducing bugs into the code.

In case you missed it, we announced another milestone a few weeks ago. We successfully completed a streaming test on the V3 decentralized cloud storage network where we uploaded and streamed a buffered HD video off of a series of nodes. This would allow companies to utilize the Storj network to stream video files directly from the cloud to their users, similar to commonly used cloud video platforms like YouTube and Vimeo. This is a vast improvement over other video streaming approaches, where a large portion of the file must be downloaded to begin streaming, the video can only be streamed to the machine where it was uploaded from, or the video can only be streamed to other users participating in the decentralized storage network.

In our next update, we plan to share our progress on these projects:

Creating TLS-based identity signing, which enables nodes to cryptographically verify one another

Building a 'Transport Client' which will abstract away details about finding and contacting a storage node by ID

Incorporating pagination support for file listing, so it will have the ability to return partial results

Continuing work on the heavy client to combine the network state, reputation and DHT cache into a single process

Designing and creating additional technical diagrams that will be incorporated into the V3 white paper

We are very excited about the functionality we were able to build into the platform during the last several weeks. If you have questions, feel free to reach out by emailing [email protected] or through our community forum.

- Brandon Iglesias, Product Manager

[storjblog]

Ask Storj: Company Goals and Business Targets

Q: Can you share some of your corporate goals to help the community see Storj's long-term vision?

A: We have many goals that we use internally to measure our success as a company. We reached over 100 petabytes of data stored on the current iteration of the Storj network. We want to be at least 10 times that by the end of 2019. We want the egress, or bandwidth usage, as a percentage of storage to roughly double. Once we have V3 launched, we'll be providing consistent metrics on a regular basis so the community can gauge our progress. We hope to see payouts-per-farmer go up by roughly a factor of five. With V3, we will also see significant improvements in metrics like file durability and availability. With V3, we should be adding nines to the end of our current metrics.

Another key metric that I'm focused on is economic value that we're creating for a community; and that's economic value for our farmers, but also economic value for people and partners who build platforms on top of Storj. One of the things that really attracted me to this company, having come from two open-source companies prior to this, is our opportunity to provide a meaningful way of monetizing open-source projects. That's something that is fairly difficult to do with traditional clouds. In our case, we want to take a portion of the net revenue (the difference after we pay farmers), and make sure that's available to people who help generate more demand for the system, which are largely open-source projects, like FileZilla. Once we get a firmer handle on what the V3 metrics are in all of those areas, we'll be sharing more.

Have a question for the Storj team? Send it to [email protected]. You can also check out our frequently asked questions.

—Ben Golub

[storjblog]

A look at Storj Labs’ decentralized cloud storage architecture with JT Olio

In our recent town hall meeting, we took a quick look at our development team’s progress towards building V3 of the Storj Labs network. We kept it short because we realized that not everyone is interested in the fine, technical details of the platform’s inner workings. However, we know that many developers within our community love this kind of stuff. Because of that, we committed to delivering a longer presentation on network V3 that dives into how the new network will make improvements over the current version.

We hope you enjoy this video. If you have any follow-up questions, please send them to [email protected]. You can also view the slides for this presentation on SlideShare.

- JT Olio

[storjblog]

Ask Storj: Token Lock-up Update

Welcome to Ask Storj, a new section on the Storj Blog where we answer your questions about our technology and our business. 

Q: What is the plan for the 245 million tokens that are currently locked up? 

A: After our token sale last year, we put 245 million tokens in a smart contract as detailed in this blog post. The 245 million STORJ tokens are currently locked up until July 29th of this year. As you may know, we sold a certain amount of tokens in the token sale. We have another bucket of tokens that are available for farmer payouts, community payments, bounties, etc. We believe that we currently have enough tokens outside of those locked up so we will be renewing the lock-up for another six months, meaning these tokens will remain locked in a smart contract until the first part of 2019. Prior to these tokens unlocking, we will release a detailed STORJ token distribution plan, and that will happen post-V3 launch. Once V3 launches, we expect to see a lot more demand on the network, and a lot more need for liquidity to pay out Storj tokens to users. Once we have a better sense of what those metrics are going to be, we'll provide more detail. We believe the additional time will allow us to make a more accurate and data-driven token distribution plan. 

Have a question for the Storj team? Send it to [email protected]. You can also check out our frequently asked questions.

—Shawn Wilkinson

[storjblog]

Quarterly Update: Storj Town Hall #3

With so much to discuss about the last few months, and only an hour to do so, we focused on two key topics for our Q2 2018 town hall: farmer payouts and V3 development progress.

Establishing a process for consistent monthly payments is one of our top goals. John Gleeson, vice president of operations, described the following schedule:

John said we are committed to making payments more transparent and efficient, fixing any remaining issues from March and April, and making sure May payments go out on schedule. John introduced a new app that we expect will improve the payout process. Learn more about it in New Farmer Payout App Live on the Storj Blog.

JT Olio, director of engineering, gave an update on V3 development. Quoting Google’s Jeff Dean, JT pointed out that the “right design at X may be very wrong at 10X or 100X.” We need to design for 10X growth, he said, and that necessitates a redesign. At a high level, what we’re building with V3 is a decentralized cloud storage network that can scale to the exabyte range. V3 will feature an Amazon S3-compatible gateway that customers download and run locally that uses a heavy client for metadata, and stores data on farmer nodes. “It takes data, breaks it up, uses Reed Solomon erasure coding, stores it on farmers, and keeps track of metadata on that customer’s account,” JT said.

Ben Golub, Executive Chairman and interim CEO, summed up the state of Storj:  “We have the opportunity to make the storage of data dramatically more secure, more performant, more reliable, more private, and more economical than any of the current centralized clouds can provide, and we can do it in a way that provides meaningful economic value for farmers, for open source projects and the broader community,” Ben said.

The town hall included about 15 minutes of presentations and 40 minutes of live questions and answers. If you missed it, you can watch the replay below.

https://youtu.be/2ebP-MAxnSY

[storjblog]

New Farmer Payout App Live

To improve the payout process for Storj farmers, we have launched a new service to track payments each month. Instead of searching in the monthly Google sheet, you can visit https://payout.storj.io/ and enter your payment destination address. The app will display gigabyte hours, bytes downloaded, terabytes stored, terabytes downloaded, and the payout amount in dollars.

Try the New Payout Tool

The new app also lets you easily see your payment history. We hope it will speed up the payment process in the future. We will be making future improvements to the tool, as we receive feedback from the community. This app is an iterative step toward providing better visibility and better information for farmers. With better access to information anticipated with the release of the V3 network, we’ll be able to incorporate performance data and benchmarking, as well as recommendations for improvement.

We’ve also created a new process for disputing payout stats, starting with May 2018.* If you believe our stats for your payout address as displayed on https://payout.storj.io/  are incorrect, you can let us know by filling out the Storj Share Dashboard Support Request Form. In order to start a dispute, you will need to have the logs for the corresponding month for all your nodes. This will allow our scripts to estimate your gigabyte hour (GBh) usage and downloaded bytes for each of your nodes during the month in question. Our request form will guide you through the process of filing your dispute. This app is an iterative step toward making the interface with our product more convenient and user-friendly.

Many thanks to Stefan Benten, one of our community leaders, and our team members Atikh Bana and Faris Huskovic for creating the app. Let us know what you think in the comments!

*The Storj Share Dashboard Request Form lets you dispute payouts for the current month only.

**Gigabyte hours is a measure of disk space consumption.

—The Storj Labs Team

[storjblog]

Last Chance to Convert SJCX to STORJ This Year

If you are still holding StorjcoinX (SJCX), it’s time to convert the deprecated* coins to STORJ tokens.

On June 29, 2018, Poloniex, the main exchange that supports SJCX, will permanently shut down all withdrawals of SJCX tokens still held by their customers. Poloniex already disabled SJCX trading on October 26, 2017, when they listed our new STORJ token. Starting May 16, we will open the STORJ converter app, located at https://sjcxto.storj.io, for the last time. The last date to request conversion is June 15, and the converter will close on June 30. There may be an opportunity to convert SJCX to STORJ again next year.

Please be careful when you share your conversion information. Scammers have targeted our community via Twitter, email, and other social media. Do not send your SJCX to any BTC address you receive by direct message on RocketChat, reddit, Telegram, or any SJCX deposit address displayed on any social media post sent to you via email. Be careful to check the URL of the converter app and make sure it is the one stated in this post. Start at the official Storj website and click on https://sjcxto.storj.io.

To convert tokens from SJCX to STORJ, take the steps described in our STORJ token conversion guide for reference.

—The Storj Team

*SJCX is a deprecated coin, replaced by STORJ, an ERC20 token on the Ethereum blockchain. When we made the move from Counterparty to Ethereum, we agreed to do a one for one conversion of existing tokens.

[storjblog]

Save the Date for the Storj Q2 Town Hall

Our next quarterly town hall is just around the corner. Everyone in our community is welcome to join the meeting, which we will live stream and make accessible worldwide.

In this meeting, hear from Ben Golub, our executive chairman and interim CEO, Shawn Wilkinson, co-founder and chief strategy officer, and JT Olio, director of engineering. Topics include an update on Storj, product development news, progress toward releasing the updated V3 network architecture, and farmer payouts. Presentations will last about 20 minutes, and there will be lots of room for Q&A. Please send questions you’d like us to address to [email protected].

This is your chance to give us feedback and discuss topics related to Storj and our community. We’ll record the town hall meeting. If you can’t join us live, you can watch the replay later on the Storj Labs YouTube channel.

Storj town hall details

Where: The town hall meeting will be live streamed on our YouTube channel and on the Zoom.us platform. Livestream links will be emailed along with a reminder the day of the event. Please use this link to pre-register on Zoom.us. Be prepared to join the meeting by downloading the Zoom.us software before the meeting starts.

Date: June 5, 2018

Time: 1:00 p.m. EST, 6:00 p.m. GMT, 9:00 p.m. in MSK and 10:00 a.m. PST. Please check your local time.

Town hall format: 20-minute presentation and a 40-minute Q&A.

Stay in touch with Storj on social media:

Twitter YouTube Rocketchat Reddit

—The Storj Team