#unionfs

[dummdida]

overlay filesystem

http://lkml.iu.edu/hypermail/linux/kernel/1410.3/01094.html

[jeciblog]

Alfresco : Test d’upgrade avec une grosse volumétrie - 1

Comment effectuer une recette exhaustive avec une très grosse volumétrie, sans dupliquer les très gros serveurs ?

C’est un problème récurrent avec Alfresco : un serveur de production contient plusieurs To de données sur le “Content Store”. Je souhaite effectuer un upgrade de mon serveur (vers Alfresco 4.2.1 par exemple) en effectuant un test grandeur nature au préalable. Cependant je n’ai pas un serveur de test avec plusieurs To d’espace libre ^_^.

Je vous propose une série d’articles sur ce thème détaillant 3 solutions mises en oeuvre chez des clients :

Solution 1 : montage NFS & UnionFS

Solution 2 : créer un “Content Store” fantôme

Solution 3 : créer un “Content Store” partiel

Solution 1 : Montage NFS & UnionFS

Principe

Cette première solution est idéale si vous pouvez faire communiquer votre serveur de production (ou la baie de stockage) avec votre serveur.

L’idée est simple : sur le serveur de test on charge un backup de la base de données et des indexes lucene. Ensuite on effectue un montage (en lecture seule) du Content Store de production sur le serveur de test, puis on définit un point de montage en écriture au dessus de ce premier montage.

Pour cela on a deux possibilités : soit faire un montage à la main, soit utiliser UnionFS.

Première solution : montage à la main

La première solution est très simple à mettre en oeuvre mais n’est pas très flexible.

Nous avons besoin d’un nouveau serveur alfresco. Vous pouvez utiliser une machine virtuelle préconfigurée pour gagner du temps.

Dans un premier temps on restaure un backup de la base de données de production sur notre serveur de test.

mysql -u alfresco -p alfresco < backup-alfresco-2014-04-01.sql

Considérons que nous souhaitons effectuer notre test le 01/04/2014. Un montage NFS du Content Store de production a été configuré sur notre serveur. Ce montage doit obligatoirement être en lecture seule pour protéger les données du serveur de production. Point de montage : /mnt/cs_prod/

On configure un point de montage en écriture pour le jour du test :

mkdir /tmp/cs_test_20140401 mount --bind /tmp/cs_test_20140401 /mnt/cs_prod/contentstore/2014/04/01/

Puis on définit le bon espace dans alfresco-global.properties

dir.root=/srv/alf_data dir.contentstore=/mnt/cs_prod/contentstore/

Idéalement on utilise un backup des indexes lucene pour éviter une réindexation complète sur notre nouveau serveur.

Enfin on peut lancer Alfresco. Vous voila en présence d’une copie parfaite de votre serveur de production.

Si vous souhaitez travailler plusieurs jours avec ce serveur, il est nécessaire de monter un dossier par jours utiles : ./2014/04/02 ou plus simplement utiliser UnionFS.

Seconde solution : UnionFS

Lorsque les live-cd étaient à la mode (je pense en particulier à Knoppix), il a été mis au point un mécanisme de système de fichiers compressés - squashfs - qui était accessible en lecture seule (car sur un cd-rom). Pour supporter la modification des fichiers, un système de fichiers - UnionFS - est configuré au dessus du premier. Plusieurs implémentations existent, la plus évoluée est aufs.

C’est cette technologie que nous utilisons.

Comme dans la solution précédente, on réalise un backup de la base de données de production sur notre serveur de test puis montage NFS du Content Store de production (point de montage : /mnt/cs_prod/). Ensuite nous configurons aufs au dessus de notre point de montage NFS.

sudo apt-get install aufs-tools mkdir /tmp/cs_test sudo mount -t aufs -o br:/tmp/cs_test:/mnt/cs_prod none /srv/alf_data/contentstore/

Puis on définit le bon espace dans alfresco-global.properties

dir.root=/srv/alf_data dir.contentstore=/srv/alf_data/contentstore/

[paultag]

my sbuilder setup (sb)

I've expanded a bit on my idea (that is, the sbuilder wrapper to help aid with managing more then one sbuild chroot), and it got out of hand very quickly. It's gone from one small shell script to a set of 8, and growing. Quickly. Until I spin these off into their own repo, they'll be in my [dot-bin](https://github.com/paultag/dot-bin) Before getting too far, please relize this is only how I'm using sbuild. There are many other (more) sane configurations that you can (and should) use. This was not made to be useful for anyone other then me :) I've added the ability to use unionfs-fuse, set up a way to login interactively and a script to create new chroots without having them register as a global schroot. Firstly, forgive me for my sins. This is all a huge hack, but it's use isn't the *worst* thing in the world. As of the time of this writing, I now have a gcc and clang chroot for each of stable, testing and unstable. Now -- the big question -- how do you use this. Rather, this is more of a how do I use sbuilder :) Firstly, I've created a file for each schroot set that I'm using. Here's my `/etc/schroot/chroot.d/sid`. [sid] type=directory description=Debian sid/amd64 autobuilder directory=/var/sbuild/sid groups=root,sbuild root-groups=root,sbuild script-config=sbuild/config It's very similar to the one that sbuilder-createchroot creates. If you're just looking for aufs / unionfs support, keep in mind schroot has some great support. check `man 5 schroot.conf`. Next, create a `dist.d` directory in `/var/sbuild` -- such as `/var/sbuild/sid.d`. In the future, this might get automagically created by the `sb-create` script. For now, it just expects that exists. The arguments for all the sb* commands is always in the order of variant, dist. variant defaults to "generic", which I've set as an alias to the generic chroot, and the dist defaults to "sid". At some point, I'll change the default to be the last UNRELEASED entry in the changelog. So, to create a new chroot to become your clang chroot, you may choose to invoke it as `sb-create clang`, or more explicitly `sb-create clang sid`. To create a new chroot to become your stable / gcc chroot, you may create it with `sb-create gcc stable`. After you have a chroot, you can start to use the `sb` or `sbo` scripts. `sbo` is just a wrapper around the `sb` script that sets an envvar to tell the `sb-prehook` and `sb-posthook` to set up the unionfs overlay. The unionfs overlay routine replaces the dist symbolic link (such as `/var/sbuild/testing`) with a folder, and mounts the real chroot (such as `/var/sbuild/testing.d/testing-gcc`) under the unionfs divert point (which is something like `/var/sbuild/testing.overlay.d/testing-gcc`). This lets us make sure the chroot is never affected by a routine build, even if it's designed to trash a chroot (unless, of course, it uses any number of ways to get outside the chroot and trash the pristine directory -- this is to prevent mistakes, not malicious scripts). By the way, for those interested, there's a small issue with what I'm describing it's very subtle. Basically, there's a few assumptions made by how schroot operates, and handles "sessions". Since it'll keep stuff mounted if the session is still active, it makes purging all this a real ... pain. The magic behind the scripts is in the `sb-endsessions` routine -- which will properly end all the schroot sessions for that dist, and since we have our own high-level locking system, we shouldn't have a situation where this kills off another build. The code (which might have issues) looks like: SESSIONS=`schroot --all-sessions -l | grep "^session:$DIST-"` for session in $SESSIONS; do echo "W: Ending session $session" schroot -e -c $session done By backing of schroot's mounts, we can do our mounting / removing without worrying about klobbering schroot. Now, there's also the case of wanting to interactively log into a chroot. For that, there's `sb-login` and `sb-maint`, both take the usual variant / dist arguments. `sb-login` logs in as the current user, and `sb-maint` logs in as the root user. Eventually, I'll be adding some scripts to upgrade all the build chroots for a dist at once.

[ishcabittle]

Split Disk MacBook

Rumors are swirling about the next generation MacBook Pro and what it might contain, how thin it may be, and the processor architecture moving to SandyBridge.  The latest rumor, coming from MacGeneration, basically confirms the most intriguing of the rumored specs, the 16GB mSSD that would house the OS.  

It's pretty exciting, a super fast boot volume that houses the OS, allowing for large capacity hard drives as well as all of the MacBook Air features introduced in the last revision: instant on, fast boot time, and incredible responsiveness.  Anyone who has used a MacBook Air will tell you that it feels very fast, faster than the 1.4GHz processor would lead you to believe.  You could feasibly add a 1TB 2.5" drive to a 15" MacBook Pro and with this mSSD still boot in 12 seconds.

What fascinates me about this idea also begs the most questions - how do they intend to handle the split in the Unix level?  There are a couple of ways that I can see Apple swinging this:

The most probable solution is simple linking.  The folder /Users/<user home>/ is linked to the secondary volume, along with /Applications/.  Any new software installed goes to the large volume, with the smaller SSD volume reserved for CoreServices, the Finder (which benefits greatly from the boost), etc.  I already do this on my hackintosh, mapping my home folder to a large RAID array and booting off of fast 10,000RPM drives.  This method is clunky to novice users, though, as two volumes are visible on the desktop.

Apple adds a dramatic new technology to OS X, unionfs.  Linux users have had the benefits of this tech for a while, and it allows for dramatic speed increases by separating various parts of the OS to be placed on different media.  By integrating this technology, Apple will be able to unify the file system across two volumes, thereby creating a seamless experience where users double click on "the hard drive" and actually see the contents of two separate devices working in tandem.

Apple has duplicated or replicated the basic idea and functionality of unionfs, but made it there own, creating something we haven't considered yet.  As likely as I think the first scenario is, if I had to bet money I would play the long odds and bet on Apple creating something new entirely.

These guesses are predicated on the idea that the integrated SSD is actually in the works, and these rumors aren't just rumors.  We should know pretty soon, the shipping times slipping as they have.

The other big rumor, Light Peak, I can see happening.  It was demoed two years ago, which is more than enough time to solidify the standard.  It would certainly justify Apple staying away from USB3.0; why support a standard that is about to be blown away by a factor of three?  The fact that the demo was performed on non-standard hardware running OS X kinda leads you to believe that Apple gave Intel their blessing from an engineering standpoint as well.  It's been a while since the laptop space has been shaken up with something new, and with the MacBook Airs moving forward, it's likely the Pros will follow suit.

The last thing I'll touch on is something mentioned by MacGeneration, the option to remove the optical drive and add more storage.  I would take that option, personally, but I find it curious that they are reporting it as an option.  I feel it more Apple-like to simply remove the optical drive altogether, and give you the option to purchase a $99 external Superdrive.  Imagine a MacBook Pro with an i5 processor, 1TB of spinning platter storage, a 16GB mSSD boot volume, 8GB of RAM, and half the total thickness of the current MacBook Pro.  Oh, and a Light Peak Superdrive for when you want to practice the archaic method of consuming entertainment, from optical media.