Will's Random Tumblr

[ivebeenlinuxed]

Dediserve Issues

If you are on Lon2 - the main issue is a low Disk I/O Speed, which backlogs the processors in I/O wait, slowing everything down and exhausting RAM if you don't have good limits on apache. You can check this for yourself using "hdparm -t /dev/xvda1", "dd if=/dev/zero of=test bs=64k count=64k conv=fdatasync", or "latencytop". I found at times my disk speed went down to 1MB/s. Read the "client friendly version" of this at http://bluelightstudios.co.uk/dublin-migration/

If you need any help moving - or tying over services until the refit, let me know.

Do not ask them to IP forward - they can't do it.

---------------- SUPPORT TICKET TO DEDISERVE ------------

Hi,

Please read this query again:

The service bondsworldwide.co.uk and naturana.co.uk are assigned to old IPs 109.104.119.111 and 109.104.118.122 these IPs were meant to be forwarded to 217.78.0.61 and 217.78.0.62 respectively.

Please also note:

Changing the password of the VM was not needed. Looking at the problem, the IPs 217.78.0.61-64 has a good SSH, and therefore the server configuration was not the issue. Furthermore Apache was listening (before it was reconfigured) properly on 217.78.0.62:80, and Varnish was listening on 217.78.0.61:80, as confirmed by a simple "netstat -plt". This means that the broken Apache configuration tonight (along with the addition of a superfluous "/etc/apache2/httpd.conf" was not needed. The only part of the system that needed checking was the underlying network protocols, and their configured programs such as "iptables". Furthermore, I would ask, how does changing the password affect the service?! I have since reset the root password to it's original setting.

As of the first point, during the course of this evening my Apache Server was reconfigured to listen on 109.104.118.111 - hence when it was restarted by yourselves the service went offline. My logs have told me this happened at 02/09/2013 07:18:10PM - during your login time. Logs show this was for 7 minutes. After this time, I fixed the configuration myself, as I knew the response to such an incident could be a long time.

Your plans for the refit, however great the improvements will be, were not thought out. There was no thought to the impact on the SANs and whether or not "hdparm" is a valid real-value for the Disk I/O, it was not acceptable - even by a comparison to VMs, or using the "dd" suggested on your knowledge wiki. A warning should have been issued about this. Furthermore, blaming the outage on a program clearly not using any resources (tmux) to try and cover up the issue is not helpful. I would prefer to know about downtime up front, how long this would last, and for you to update me, as a fellow technician, on how long you are expecting to experience problems for. "latencytop" clearly showed that disk access was the issue.

The IP forwarding has not, and is not, being tested thoroughly. This is a major issue currently, and has been since Thursday (07/02/2013 10:19:19) when this service was offered to me. HTTP/SSH is not being forwarded on 109.104.118.111. It would have been better, like you did to begin with, not to offer me this service if you check it sufficiently. This is still an issue now on 109.104.118.111 which needs fixing.

Tickets are being sorted during office hours. My clients require high availability, and this is why I work through the night to get the services sorted. Your autoresponses are misleading when you say "Many thanks for your update, we are reviewing this for you and one of my colleagues will update you shortly." at 02:37:36 and your response isn't until 08:46:11. If a critical machine went down for this long I would have severe problems. Not having an ETA also makes it very hard to schedule periods of downtime for my clients. On performing my upgrade operations I schedule a maintenance window - normally in the early hours between 1am and 3am. My clients also know about what is happening to the service beforehand.

Furthermore, when I agree to maintenance being done on my server through a ticket response, I expect it to be carried out promptly, so I can monitor it. Furthermore, I expect the service to be fully operation at the end. I cannot have a ticket response from yourselves half way through the work asking me if I want this or that, then it being left in an unusable fashion until I respond. I suggest a possible solution to this to be using the instant messaging chat you have on your support website to better connect your technician with your client (me).

The backup you took was 24 hours in advance of the move and therefore meant that, had I not intervened, I would have lost 24 hours worth of server data. After you offered to delete my machine, I took my own backup, which I have since used to restore those 24 hours worth of emails. Had I lost this it would have been completely unacceptable.

I am extremely unhappy that your web support helpdesk panel keeps on redirecting me on refresh and cutting out due to redirection loops. Pingdom has logged 502 downtime periods with your panel since 05/02/2013 17:28:19  If you need to oursource the work to someone, I am more than willing to fix it at my normal rate, but you can't just leave it any longer.

My clients are now looking for compensation - and the only reason they aren't asking for more is because I stuck a caching proxy on the old server when I first found your SAN problem, with a TTL page of 5 days, so that the hard disk use was kept as low as possible. As such I have kept my uptime at 97% (currently below my 99% SLA), however I have calculated that without these measures I added I would have been at 90% and falling, due to continued outages that would have related to your SAN upgrades.

Please sort this out. If you need to outsource some work, then feel free, but these issues have been running on for days because, I suspect, the technical staff are at their limits with all the retrofit (and the issues it is causing to clients). If you don't have 24 hour technicians this is a major issue. I understand you don't want all your staff touching the routing configuration, but you should always have someone on hand to be able to if needed.

Please give me a proper ETA for a response on this email, and a complete the fix for 109.104.118.111, which does not affect the downtime of my server. I think through a passive look at the current configuration on the VM you should be able to see that the server is not the issue. I do not wish to see more downtime because of this IP forwarding problem. Please advise me a time window should you need to disable, reconfigure or reboot my server where these changes will take longer than 2 minutes offline.

Will Tinsdeall

[ivebeenlinuxed]

Secularist States

After looking back through my Twitter posts it seems that such a small status hardly does justice to such a large problem. So I explain myself a little bit more thoroughly and to the point here:

The idea of a secular state is one which draws much positive discussion, however the principals, when looked at it's implementation leave it without any viable route to the real world. It also leaves Britain afraid of it's history.

"Secularism is the principle of separation between government institutions and the persons mandated to represent the State from religious institutions and religious dignitaries" (Source Wikipedia). Theologians call this the SSD Spirital Secular Divide. Where matters of public affairs, and jobs, are separated from the spiritual matters.

One of the first problems with this ideal is that of who runs such a state? Whether Athiest, Christian or Muslim everyone believes in a set of moral ideals. The mere principal of separating religious dignitaries from government is abserd, as, by being the Prime Minister (or any other office holder), it will make you a dignitary, and as a person believing either in a God, or not in a God, it will make you a representative of that faith. Faith is a belief in a set of principals, as every human has some sort of principals it seems difficult for any human to look on a problem purely from a "secular" view.

My next problem in this ideal is the religions oppose this secular divide. Whether in a Church or a Mosque, believers consider it a moral duty to share God with others, and to live their life fully for that God. In Christianity there is the term a "Sunday Christian", a frowned on term about someone who goes to Church once a week for the hour, says his prayers, then goes and lives the rest of his week like any other person would. Christians are taught to be "salt and light", and to "make disciples of all nations" (granted something Christians have sometimes taken too far in our history). Likewise, I am sure Muslims have the same sort of analogy (though they would have a harder time, as one of the five pillars of Islam is to pray five times a day). Further to this point I would even argue that atheism is a faith, as they argue that worshipping a God is absurd with some sects considering it their job to show others this "truth". This issue leaves the question, should we ban people from sharing their faith and restrict freedom of speech when in public office? Should no-one of any faith run an office unless they denounce the main principals of their belief? In the end, this wouldn't make all faiths happy - instead it would upset every faith!

One of the examples of where the state has tried to become completely secular is "Winterval". Birmingham City Council (UK) decided that one year it was not going to put up "Happy Christmas" in case it caused offence to Muslims. As many are discussing here, a secular country would have to dispense with Christmas! However, by doing this not only did it outrage Christians, but also Muslims - the ones that the council were trying not to outrage!

"This year, though, the defenders of Christmas aren't only invoking the fear that nebulous Muslim forces might be about to obliterate Britain's traditional religion. Simultaneously, they have also aligned themselves with Muslim groups, arguing that the real enemy is secularisation. It's a position well-crafted for the historical moment, and for the currently fashionable notion of Britain as comprised of groups defined above all by their faith (even though barely 10% of us regularly attend any kind of religious service). "Any repetition of public bodies or local authorities renaming Christmas, so as not to offend other faith communities, will tend, as in the past, to backfire on the Muslim community in particular," the Christian Muslim Forum warned in a letter to councils last month." - http://www.guardian.co.uk/world/2006/dec/08/religion.communities

And here, my friends, my argument lies: Secularism in the end is trying to please everyone by giving everyone nothing. In our state faith lies our culture, heritage and festivals. It defines are holidays, and our commercial seasons. In the end, as with any state, it's beliefs hold it firm.

Religious groups don't want secularism, because they all know it doesn't work. They want to be accepted and understood. People come to Britain for it's culture, it's landmarks and it's history. A truly secular society would be ashamed of it's past, and afraid of it's future. It's desire for a true equality it ties everyone's hands.

What is next? Maybe mosques should not be allowed domes, as to stop religion impacting the landscape? Should the Queen be made to renounce her faith? Christians should not be allowed to wear crucifixes (http://www.personneltoday.com/articles/2010/02/12/54101/christian-british-airways-crucifix-worker-loses-discrimination-appeal-against-airline.html) or Sikh's from wearing turbans (http://news.bbc.co.uk/1/hi/world/europe/4782420.stm)? This is secularism. Is this the way we want it?

For those which say we need an evidence based rules: what evidence is there murder is wrong? This is not an evidential question, it is a moral question. Some would say a moral absolute - something we just know. Therefore, this is not an evidence based law, this is a law based on beliefs! We still do not have an answer to if euthanasia is OK from a secular point of view! Should we just not legislate at all on this?

I would argue who looses out on a tolerant, Christian State? Who is being protected by secularism?

[ivebeenlinuxed]

Server Upgrades: 26 Jan 2012

Hi All,

We have now taken snapshots of our production servers and will begin the process of readying for the upgrade from Ubuntu 10.10 to 11.10. We have decided to move away from this version, but not to an LTS, for a number of reasons as we hope to explain below (Please note I am only describing changes most likely to affect to the service):

PHP

PHP is currently at version 5.3.3 in the Ubuntu 10.10 repo, giving us a few minor issues with php-fpm, the new CGI process to help handle some of the high load sites on our server. Let me explain:

A few months ago we trialled a system known as mod_fcgid which we had major problems with. As we rolled it out to a wider range of our customers we noticed the increased strain on the server we were getting. This, very simply was because the server could not handle the number of mod_fcgi instances we were throwing at it (resulting in 500 errors).

FCGI works by starting up processes beforehand, then when a CGI request comes through assigning that request to an already running CGI process. This meant that at times of low traffic we could handle a lot more requests than standard mod_cgi. However, as the amount of traffic increased the server could not deal with the high load PHP puts on the server. Furthermore as the FCGI scripts run as their host user (so each client can have r/w permissions to only themselves and not a shared www-data group creating security issues) and therefore each user had to have their own set of idle FCGI tasks - a heavy burden which made its way into swap space. There also were some design flaws as well with FCGI which we were not expecting, for example requests were buffered in RAM, rather than being directed straight through to the CGI process.

So, with this new upgrade what are we trying to do? First of all we are changing how the FCGI processes are bundled. php-fpm bundles the FCGI interface with PHP, rather than bundling it with Apache. First of all this will improve shared opcode caching (using php5-apc), keeping a shared cache for longer. Secondly the php-fpm interface allows adoptive rates of spawning, so in times of less traffic less processes exists. Finally, as a design feature, the interface is attached to PHP and therefore is subject to the max_execution_time PHP variable - very handy.

Secondly, we will be rolling this out to only the most active sites. Although this might seem unbalanced, in the end it will improve everyone's performance. By having a larger opcache for the most active sites it frees up both processing power and RAM for use on other sites, thereby benefiting everyone. Furthermore by only creating php-fpm pools where needed it will increase the overall resources for other requests.

So why the upgrade? PHP 5.3.6 is now available in the stable package archive of Ubuntu 11.10, amongst other things containing bug fixes for the php-fpm status page (some bad headers conflicted with Apache on PHP 5.3.3). This will allow us to monitor this process more carefully.

SSL Certificates

As you may have noticed we have started putting on new SSL certificates onto our dashboards. We hope with this latest upgrade to extend this to our mail interfaces.

As you may have found out, we are very keen to ensure your data is secure and as part of our policy at the moment we refuse to run a standard FTP service. Security should be implemented as default, and when securing something is as easy and leaving it vulnerable it seems only sensible to take the more secure option. We believe by disabling the FTP interface we force people to take a secure route, via SFTP (FTP over SSH), without any extra effort. This helps people not make insecure decisions in the first place!

Our SSL policy is starting to follow the same path now - the extra work for securing protocols such as POP3, IMAP and SMTP rests with us. The extra work to configure this for you - nothing. At the moment our development plan involves opening secure and valid SSL secured mail connections to all our servers, then in the future looking at a pain free way of moving over to SSL secured connections only. After all, who wants to give away all their passwords, have their emails pried upon in transit, or have their identity stolen - with their own account details! It's logical isn't it?

Linux Kernel

Ubuntu 11.10 comes with the next major release Linux Kernel 3.0. Although if you are at home and reading this you may want to upgrade due to, most notably, the disk features I suspect these advantages will be limited as our servers are connected to a SAN (Storage Area Network), who's main source of communication is fibreoptics. (Editors Note for Home PC: Unity is horrible! Just as they've worked out a nice set of 2D graphics drivers they push a 3D platform without the full native driver support...)

Anyhow, for a server the JIT Berkley Packet filter (this used to be an interpreter) may make some minor improvements to throughput and load - We'll have to see.

For more info on the Linux Kernel check out: http://kernelnewbies.org/Linux_3.0

MySQL Server

Archive differs with minor version updates: Maverick sits at version 5.1.49 while, Oneiric is at 5.1.48. These are security fixes and a couple of cluster replication fixes (the latter of which does not affect the service)

Apache 2

The Apache 2 server will be upgraded from 2.2.16 to 2.2.20. This itself is mainly minor fixes, however you may have noticed increase speeds on the server. This was due to changing over to the apache2-mpm-worker from apache2-mpm-prefork. Not only does this allow quicker spawning, but also reduces some of the overheads. We are currently carefully examining the experimental apache2-mpm-event module and have been testing in private. This upgrade however will not make it to our production servers until it is marked as stable.

Our deployment plan is to start the server up with fewest clients available, and benchmark the service with various CGI's in order to tune to the setup to stay within RAM, and by not moving into swap we will increase server performance.

Upgrade Target

Our previous upgrades gave us a 40% increase in server response times on average, with approximately 450% increase in capacity, measured by counting the requests per second. With the latest round we hope to improve this again by 5-10%. However, as well as this we also move our server to being the most secure and cutting edge. As Maverick will soon be unmaintained (April 2012) we will have to make this update soon anyway.

We believe customers will see patchy outages over 3 hrs. We hope that these will be below 5 mins per service. Please also note that it is unlikely all our services to go out together (Mail and Website for example) except during the final reboot, and the likelyhood is ICMP ping requests will be honoured throughout.

[ivebeenlinuxed]

VPS.NET Open Letter

Dear Sir/Madam,

Yet again this month I am looking already at nearly 12hrs of downtime on my London-I server. Therefore, as last month, I would like to claim for this downtime. It is yet again unacceptable.

Let me take you through the search I have performed, around the downtime in your service:

LON-I/London-I issues:

May 3

May 10

July 15

August 28

Sept 6

Sept 7

Sept 17

November 9

London-F issues:

Oct 17

July 14

July 10

March 17

LON-C issues:

Jul 21

Aug 23

Aug 22

Sept 1

Sept 2

Sept 4

Sept 6

Nov 9

Out of the last 6 months, only two on LON-I have been without error. Furthermore most of these are SAN issues, and if I remember rightly LON-F Oct 2 incident was a near escape - where I could have quite easily lost everything, and at that time I was 2 months behind on my daily backups - again a service I *PAY* for!

I am seriously considering moving all my virtual machines and the ones I manage to another hosting provider as, although your customer service is good, your ability to keep the technical equipment on-line is lacking.

Furthermore advertising 100% uptime (http://www.vps.net/forum/topic/4133-really/) on your homepage is seriously misleading when sometimes I'm lucky to get 90%. As you can tell from the above link many others face this same problem - some claiming they have been down for over 30hrs in a month!

I am incredibly disappointed, along with my clients, at what seems to be systematic failures.

Yours Sincerely

Will Tinsdeall