Note: This is a long and technical piece about Development Operations: all the things that happen to make Furvilla a technical success when we launch.
Furvilla runs on the Heroku platform using a Postgres database for records and a Redis instance for short-term storage of things like controlling our feature rollout ability. We’ve got a benchmark rate of around 10,000 simultaneous players right now.
We use Memcachier for cache storage and push as much as possible to external services (e.g. asset serving and user uploaded images to Amazon S3, emails via Mandrill) and use worker processes (Sidekiq) to handle things like API interaction for payment transactions and other requests that aren’t directly related to the response cycle.
We use a variety of services to monitor all aspects of our infrastructure including New Relic and UpDown. These enable us to keep average response times on the site to 0.5 seconds and alert us to any issues that might be creeping up on us.
We also use a variety of logging services to capture application errors, user-generated errors, and separately, record a bunch of interesting and important data about our players and how they’re using the game. We capture this for audience profiling (our adverts) and essential site operation (e.g. logging actions Moderators take on your account, logging activity sessions, preventing multiple account abuse, and account restoration, etc.)
Our Customer Facing Tools are the things you see around the site, like our Help widget (GrooveHQ) in the lower right corner of the browser window. They’re tools we use to communicate about or with you. For instance, we have a custom internal API that enables our Customer Service Team to look up relevant account information when you contact us, without needing to log into our Admin Tools.
One recent addition is our status page (soon to be unveiled) which is powered by an external partner, SorryApp. This service is absolutely critical as it enables us to communicate important downtime information to you all even when the servers are down or the database is borked.
Our internal tools include the Admin Tools (enabling staff to operate the site) as well as our moderation and profanity filtering services: we pre-moderate all player-uploaded images (paintie designs and Fur Idol entries), and profanity filter all player-submitted content (comments, posts, profiles) to remove content that we feel is unsuitable for our family-friendly site.
The hardest part is keeping it all running. The more external dependencies there are, the more issues can crop up. Right now, I’m set to receive email alerts whenever someone triggers our cute Serious Error page or an external service goes down for more than 5 minutes, but eventually this’ll expand into alerts sent straight into our team chatroom and SMS alerts, phone calls, etc. and we’ll have more robust monitoring of our external service use.
We’re fortunate enough to be able to use a lot of automation in this business. An example of this is that Git deploys are automatically run through our test suite before being uploaded to Heroku, and we benefit from a robust, scalable code architecture that means any future developer can jump right in, knowing that everything is tested and features can be added or extended easily.
dailybridgerton
annaxmalina
origamiaround
gummygunk
pixelskylines
baratheonpilled
tangentwithtanner
dededos
snowghoul
