HTTP 503: Service unavailable for holiday got stuck on highway mqueue
503 is a quite common http error. It points out a “Service unavailable“ issue. Basically it is just like a notice on your coffee shop door saying “We’ll be right back”. But they don’t tell you why, they don’t tell you when, they don’t tell you if. Problem is, you can think of changing your coffee provider, but you can’t think of switching to a different BI platform, for it will cost you much more than planning a reasonable revenge, burning the coffee shop and look for another one.
So, when your SAS Visual Analytics is not available, where is he? He is on holiday, I think. I make this assumption because most of the times, this issue is caused by your VA mqueue getting stuffed like a turkey on the 4th of July. Therefore, when they say “your SAS Visual Analytics is stucked in a queue”, I just imagine my VA on his new Camaro, running for holiday and getting stuck in a traffic jam of lazy VAs running to the seaside.
Basically This problem occurs because the broadcastTopic messages in ActiveMQ do not expire. As a result, the dead-letter queue (DLQ) fills up,leading to filled KEBAB db_*.log files. This broadcast topic is used by the distributed in-process (DIP) scheduler, the DIP job runner, and the job execution services (JES) job runner.
SAS has come up with a SAS note ( http://support.sas.com/kb/52/315.html ) explaining some workarounds for solving the problem :
1. Spill napalm on the DLQ: basically it is about stopping the services and deleting the mqueue files (SAS-configuration-directory/Levn/Web/activemq/data/*). Don’t forget to stop the services or your application server will evoke Satan.
2. Use the Discarding DLQ Plug-in: you are warned, this plugin is buried deep in an ancient Egyptian tomb, and very few have survived the curse. None survived the installation instead.
3. Stop the DIP Scheduler Process. Gently asking won’t stop the service, so you’d better try shooting his legs.
Option one has always been working for me, but I can understand you do not always have napalm available.
(Special thanks to Mr. Faccin, our soul savior: he gave us this advice one year ago, and we brought back to life up to 20 VA corpses. Beware: the living-dead Visual Analytics era is about to come!)