*This is the first article of my new DESIGN RULEZ series. [Subscribe](http://feeds.feedburner.com/vishalshah) to stay up-to-date on future articles & commentary.* ## Life can be quite complicated Life’s complicated. We generate & collect more data than ever before and our insatiable thirst for data only grows. Our devices are getting more and more sophisticated and thus complicated. There is more trickery and gadgetry packed in an iPhone 5 than your 70’s Camaro. Everything is connected. Even our power systems are now connected. Mobile devices are spreading like viruses and hence mobile usage is sky rocketing, generating all kinds of data. Our systems and architectures are getting more and more complicated by the day as we support more users, produce and process more data. Our systems are getting also increasingly intelligent and sophisticated. And all of this is on the rise. Imagine the world just 20 years from now. You get the idea.. ## Emphasize on Design & more importantly Continuous-Design So, how we can we sustain & thrive in this environment? The only way in my opinion is by employing “good design”. Specifically those who put design-first & more importantly employ continuous-design. ## What do you mean by Design? Design in this context is simply put, putting some thought into and about the problem you are trying to solve, and how you are going to attempt to solve it. Sometimes this is merely planning. Sometimes though it is whiteborading, sketching, talking, communicating, collaborating, researching and trying to put something together that will help you get early visibility on what you are trying to build and how and why. You might be a person who at the first sign of something, might go off and prototype something quickly and that’s great. But you can be at the same time, a lot more effective, if you can take a step back and try to “visualize” the problem and what you are trying to do. Its amazing, how critical this simple, “take a step back” is. For me, its so important that I have tried to solve most of my problem solving, that way - the forest. ## Why Design? Design can have such an impact to the what you try to build, that in my opinion it will dictate the product’s mere survival over time. Survival from competition, from new/fresh ideas from startups & entrepreneurs, from radical thinkers, and from just sheer expecations from users. Users have learned to expect more and more from software & hardware systems. Just read some of the app reviews on the iTunes App store for example and you will get a sense of how demanding users can get! Design is also something, that’s very fundamental. It will drive nearly all aspects of whatever project you are/will be working on. It scales from what you are trying to build to “how” you will try to prototype or build, how you will be involved, how work is shared across team members, how tasks will be shared, etc. ## What about Prototyping? I recommend you spend some cycles designing/whiteboarding/taking a step back, even before a quick prototype. It doesn’t have to be a lot. Often, just enough to get a sense of what you are trying to do and how, some sort of picture you have in mind... Prototyping all of sudden, becomes a lot more effective. Just try it, and let me know. ## What about Continuous Design? This one is obvious. As you work towards a problem, things often change - requirements change, new things come to light, time line changes, expectations change. Hence, if you are not employing continuous design in even the smallest of your personal or professional projects, you are missing out on a ton of opportunities, to the point where you might be jeopardizing the project survival over time. For some projects, this is just fine - An example is a lego set. Say, you bought a lego set and you are just excited to put something together. You might just pick up the blocks, and stack them and build something random out of them. And that’s Ok. You are just trying to make sense of it and experience them. But the minute, you want to do something more interesting with them, clearly the above strategy won’t work, even though you have something in mind in terms of what you want to try building. Imagining it, picturing it and trying to figure out some sort of strategy, will not save a ton of time and frustration down the road, it will help you actually build, what you are trying to! ## DESIGN RULEZ Series I am starting this DESIGN RULEZ series to share and promote my ideas, hoping it will help somebody :) The hope is that these ideas & principles could serve as a helpful tool to evaluate a new or existing, internal or external software project for a variety of reasons. Or more importantly to build a sustainable product/system architecture and processes. Some of the ideas could also reveal interesting insights into a project’s health & quality. There is no exact science here, so don’t take these literally. I present them as guiding principles that are common among my favorite systems developed. So here we go... > What do I mean by a “Software Project”? Its a project worked upon by a group of engineers. In some cases, for large applications/frameworks, a project in this context boils down to one or many of its components. ## How to Build/Spot Sustainable, Well-Designed Software Architectures - Part One In part one of this series, I am going to address this question I have had for a while by starting with source control, one of the best places to start while evaluating or building software projects. Yes, even something like source control can use some design love. And I will talk about some of the reasons as to why so... ### Source control Obviously source control is essential, but inspecting the source control tells a lot about a project. For small projects, a single repo is enough. For larger projects, I really think projects should be neatly broken down into multiple repos. A repo for a logical component. With git, this becomes a lot easy, hence I highly recommend git for all projects if choosing an open source technology. Perforce rocks as an awesome commercial source control technology. Ideally, all the repos should be using the same source control technology. A varied set of repos generally infers the existence of legacy code. That doesn’t necessarily mean a bad thing, but it does :) Something to keep an eye out... But in my book its a clear concern for most active projects. Next up, the the directories & files within source control should be meticulously named and organized. Lets take the example of the open source [scala programming language project](https://github.com/scala/scala). scala/ +--build/ Build products output directory for ant. +--build.xml The main Ant build script. +--dist/ The destination folder for Scala distributions. +--docs/ Documentation and sample code. +--lib/ Pre-compiled libraries for the build. | +--fjbg.jar The Java byte-code generation library. | +--scala-compiler.jar The stable reference ('starr') compiler jar | +--scala-library.jar The stable reference ('starr') library jar | +--scala-library-src.jar A snapshot of the source used to build starr. | ---ant/ Support libraries for ant. +--pull-binary-libs.sh Pulls binary artifacts from remote repository. +--push-binary-libs.sh Pushes new binary artifacts and creates sha. +--README.rst The file you are currently reading. +--src/ All the source files of Scala. | +--actors/ The sources of the Actor library. | +--compiler/ The sources of the Scala compiler. | +--library/ The sources of the core Scala library. | ---swing/ The sources of the Swing library. +--target/ † Build products output directory for sbt. +--test/ The Scala test suite. ---tools/ Developer utilities. You can tell that the directory structure is well-designed and consistent. Note the naming conventions, all very consistent - concise, lower case, use hyphens to separate words. However, even a nicely designed project and repo, there are still some things I am not terribly happy with it. That speaks a little bit about myself :) You can imagine how I react if I see regular project structures out there... #### Consistency & Well-Designed Consistency is the first item I want to discuss. When I first check out source control, the folder structure and files should look consistent, and well designed. All file names and folders should follow the same case convention (minus some exceptions like README). They should use the same word separator - CamelCase or “-“. I should be able to see all the files in a folder in my screen without scrolling. If I have to scroll, break up the files or folders into folders. This might sound crazy, but you will be surprised how effective this can be. Just give it a try and let me know your thoughts. #### Documentation For one, I personally do not like the idea of including documentation (docs) as part of the the core codebase. Few reaons why - you are seriously including the size of the repo clone, checkout, merge etc. It gets worse as the documentation grows as more pages are added including things like localization, etc. Now how do I spot this inconsistency? To me, it just doesn’t seem coherent enough. If you are a scala developer you might disagree and mention that docs are maintained by developers who commit to the codebase. Well, that’s Ok, you can easily include projects from multiple git repositories yet work with them together using [git submodules](http://git-scm.com/book/en/Git-Tools-Submodules). That’s why choosing the right source control so incredibly important. It sets the tone for future development and build processes! Other source control technologies have similar concepts to work with multiple repos at the same time. The advantage now for developers is that cloning, etc are much faster and frankly I am strong believer in seeing only the things that matter to my work. Even scanning through excess crud slows us down. Note, source control is terrific, but sometimes the added cost and complexity might not be worth it. For many projects, documentation might better sit in a CMS or wiki as long as they have history support and are properly organized with project versions in mind. Imagine if the project attracts technical writers. Instead of forcing them to check out and check in updates, a better CMS might be much more efficient. Also, for documentation, I believe in one thing - eliminating as many barriers as possible to get someone to author. The more the constraints, the more likelikhood it will get snoozed on. If I could update the documentation from my MBP, iPad or iPhone, I am more inclined to update it often. Besides, most projects will have some web documentation anyways. Why not combine the web and project documentation in one CMS. I also like CMS that support easier integration of diagrams, pictures, etc. ##### Documentation Templates It’s important to have documentation templates, to have the documentation look ridiculously consistent. Just remember when you last ran into a project with awesome and beautiful documentation. What was your first impression? That’s the kind of impression we are looking for from new folks who join the team or folks who check out our project. ##### Documenation Categories Don’t lump all documentation on one top-level page. Check out [ruby-lang.org](http://www.ruby-lang.org/en/documentation/). Notice documentation is broken down in to getting started, manuals, reference, editors and further reading. Its very clear to me where to go for the appropriate piece of documentation I am looking for. Notice, docs within each category share the same template for consistency. #### Developer Documentation Like most good projects, it makes sense to author a how-to developer documentation explaining who to get started, author code, build, code conventions, design, code review & submission processes. Developer documentation should not be checked in but in the project’s CMS like discussed above. And developer documentation should be separate than the project’s documenation. #### Consumer Documentation Some projects have consumers who are interested in building it for other projects. They are not necessarily interested in the project itself for the most part, other than just how to build it. This should be documented in a separate section, making it easy for others to integrate your project with others and become popular! Little things like this help spread the word, and help get viral. People remember such things and like to share good experiences with others... #### Who’s Who? Clearly noting who the lead and contributors are encourages better communication. Its simple, just try to consistently keep it updated. ##### Really? Frankly, I will be honest here. If you think this is too much and choose to just ignore this, in my opinion, you should not make the project open to others. If you want to work with others, pay some attention to this. #### Architecture Diagrams I also recommend associating an architecture diagram at the top level of all components and if necessary, the sub-components as well. Just imagine this - your project is nicely organized and divided into components across directories and repos, as discussed above. And at every level, as necessary, I can just open a diagram to visualize how the component is structured, designed and how it fits in the larger picture. This does not have to be fancy. Even an ASCII/text file will do it, as long as the representation is visual. I do not want to be reading text and paragraphs at that time. #### Sub-Projects Well, I think the whole project can be broken down and split in multiple sub-projects as repos! This directory structure and repo is frankly intimidating to me! You might feel right at home, but frankly, when I started diving in the codebase, I was lost, as to where to start! But, I am blaming anybody. This is a very typical structure for projects like these. I have seen it over and over and it works. But I am disappointed, it has not evolved as I would have liked. Here’s how I would further restructure the scala project. Scala has a core codebase for the language and a separate set of large libraries exist to work with the core language - compiler, actors and swing. These are not 3rd party libraries and hence I think, they are better classified as dedicated projects that can evolve on their own and better yet, as a scala core developer I can choose to not worry about if I don’t need to. And I can always pull in these libraries as submodules if I need them. If these libraries were small and very tightly knit to this project, there might be value to not create a different project, however I recommend at least creating a scala-libraries repo where all the libraries go, so in the future if you have more libraries, they fit right there. For 3rd party libraries that the project does not own, the project’s build system should configure them so the build scripts pull them before building and the documenation point to the appropriate project sites as a courtesy to others. Another subtle benefit of the above approach which I bet most people won’t realize at first is that, creating separate projects for libraries allows for better, distributed ownership model. I consider this an important aspect the leads should think about that often. Code/Project ownership drives the project’s direction and evolution. Also, this enables the developers to support multitude of developers without incorporating all of them in the core scala repo. Such a relief. It allows opportunities for 3rd party developers to create their own libraries/dependencies and fork/extend the core source code to take advantage of, without always going through the project lead, an incredibly powerful idea. Also, this allows for the libraries and dependencies to have their own code conventions, and dependencies, essentially creating a tree of projects and their dependencies. A good example is the facebook [scribe project](https://github.com/facebook/scribe) hosted on github which depends on the [thrift framework](http://thrift.apache.org) which is hosted on apache.org. #### Dev Tools Finally developer utilities and tools, I am not so sure. I think there is value in having tools associated with a project, to automate etc, but often times these tools end up related to building the project or setting up the runtime/OS environment. I also dislike that all the tools are in a flat directory, as opposed to organized in folders. Maybe they started as a just a small bunch, but that’s what differentiates a good project from great. Continuous improvement, relentless effort to keep the project structure and codebase beautiful. As the tools grew, someone/lead should have noticed it and created a folder structure to organize them. #### Bug Tracking Bug tracking is something I think should be very well integrated with source control and this can be sometimes tricky, but I encourage you try hard to make sure that is the case, as it can be very rewarding. Knowing a change is one thing, but why it happened is something that’s a lot more powerful. And sure you can check out the checkin comment to get an idea but how many checkins involve changes to just one file. Most checkins involve many files and how do I connect what changed. Integrating with bug tracking allows me to simply tie a bug tracking id with my checkin, and really that’s the most important part of the checkin message. I should just be able to look that tracking id and find out all about the change, why we made it, who requested it, who filed it and what was the priority, what release(s) it made it in, etc... Now, that’s powerful. I am disappointed, so many project leads don’t require tracking id’s as part of the checkins. Its a must. There is another subtle but important benefit here. This gets people in the habit of filing bug tickets before checking in, making sure we have some of the important meta data for the change! This is part of the cultural change I had described above. It does not work if just one individual follows this. The power comes when the entire team operates like this. And contrary to popular belief, this can be applied to open source projects as well, not just closed source. For example github has an integrated bug tracking system with unique bug ids that can be associated with code checkins. #### Designing for Source Control Performance Often times, leads, managers ignore the performance of the source control server. I can not emphasize how important this is. Low performance and latency on source control servers, generally will still get the work done, but what most don’t realize is that developers will adjust their development processes and style with it. For example, there are some source control best practices that become difficult to follow if you don’t have a high performance source control solution. * Checking in code often * Cloning, forking codebases and submitting patches * Create dev/feature branches as opposed to working off the main branch * Deleting old branches as part of regular cleanup * Regular merging on branch updates * Tagging releases and milestones frequently * Setting up notifications on state changes in the source control You inhibit developer productivity and unknowingly are creating and environment that encourages potentially poor source control practices just because the source control system is not awesome enough. Its not that high of a cost compared to other systems we maintain anyway, and the benefits can be huge. Worst case, developers will have one less thing to worry about and complain. So, unless you are strapped startup, please spend some time designing a high performance source control solution and ideally have one or more team members focused on maintaining and evolving the system. #### Designing for Source Control Tools This is directly related to source control performance.There are some really nice tools out that one should invest in. As codebase grows and more and more projects are being developed and sustained, it becomes increasingly important to build and support a bunch of tools, to simplify a developer’s life. Tools such as scripts for common/mundane tasks, scripts to integrate with commit hooks and project lifecycle, visualization tools like [Gource](http://code.google.com/p/gource/), tools to merge code, etc.. Often times individual developers have to come up with their own solution to these problems. Its not optimal as the org ends up with many different styles of development. And if you are a large organization, you often don’t let the entire company know, you have built a cool source control utility to solve something. Hence, I highly recommend putting some thought and design into the various tools developers will need and build a system to share these tools with others. #### It’s Cultural A lot of what I discussed is cultural and more effective if everybody in the team embraces as opposed to one person relentlessly trying hard. Its unfortunate but there are organizations where this might not be a hit, but we are talking about building and spotting some of the best built software projects around, and yes it takes some effort to get there. But the rewards are equally amazing. ### That’s All Folks? That’s not all! In future articles, I will be covering a lot of sticky topics that I consider important such as Versioning, Continuous Integration (CI) & Design principles such as Less is more, high level client, API, middle tier and backend server software architectures, team process and culture best practices and other interesting tidbits such as RESTful vs Binary APIs, Synchronous vs Asynchronous (non-blocking) IO, etc... As you can imagine “design” is loosely referenced here and that’s the idea. It essentially translates to putting some thought in these matters, and I think they in turn drive the success of the team, the products, the organization. [V][1] [@goldenv][2] [1]: http://www.vishalshah.org [2]: https://twitter.com/goldenv
gracieabrams
gummygunk
vaporwave
s-u-f-j-a-n-s-t-e-v-e-n-s
prguitarman
stormborns
origamiaround
theartofmadeline