Bluff Avenue Blog

Updates and news from the Bluff Avenue trenches.

Game server outage

Saturday, May 9, 2020 6:15 PM

Hey all, sorry if you had trouble playing this afternoon/evening.  The game server crashed and I didn't notice for a few hours.  I have an alarm set up to text me if the website itself goes down, but not the game server.

 

Several of you had messaged me on Twitter to let me know.  Thanks--I will keep a closer eye on my messages in the future.

 

It looks like we had a ton of games created yesterday, and it may have eaten up the game server's memory.  I deleted a bunch of games that hadn't been played in 2 days to help clear up space.  Sorry if this nuked a game you use regularly, but not in the last 2 days.

Visibility: Anyone
1. Posted by

Thanks for the post. We appreciate you and all the work you've put into this website

5/10/2020 6:37 AM
 
2. Posted by

Yes this site is amazing thank you Kevin!

5/10/2020 1:22 PM
 

Well that was fun.

Friday, May 1, 2020 12:18 AM

If you tried to play on Bluff Ave today, you might have noticed we had a rather lengthy outage.  My apologies for that.  Given the big increase in traffic on the site, I've been trying to help keep you bored quarantined people sufficiently entertained by keeping a closer eye on the server and fixing crashes as soon as possible.

 

I had been talking to Daniel yesterday and it came up that the site has been doing pretty well since tweaking the load balancer settings a few weeks ago.  Maybe we got a little too self-congratulatory, because the poker gods apparently took offense and decided to throw a whole new kind of outage at us.

 

There was basically a mixup where we had our internet cut off at our host in the early morning hours.  The soonest they could restore service would be Sunday. <gulp> A 4 day outage?  What would all you folks do?  There's only so many times you can rewatch Tiger King.

 

So we decided to improvise and move the physical server to a new location that had Internet connectivity.  In other words, under my home office desk where I'm currently typing this. :)  This required several hours of work on the part of Daniel, investigating the connectivity issue, picking up the server, reconfiguring it for a new network, helping me get my network set up properly, and then shuttling it over here, all while maintaining social distance!

 

To his credit, it worked without a hitch.  I plugged it in, turned it on, and bam, service restored around 10:30 PM.  Let's give Daniel a hand for his heroics.

 

Long term, this event points to the need for us to move Bluff Ave into the cloud (AWS).  It's something I'm going to start looking into.

 

P.S. I've been neglecting Bluff Ave's Twitter account for the longest time.  I've decided to dust it off, and in the future will use it for incidents like this, so follow us there at @bluffave.

Visibility: Anyone

Thank you! (Oh, and search is fixed!)

Friday, April 24, 2020 12:07 AM

The latest news: as I mentioned in my last post, the site has kind of blown up since the pandemic hit.  We're seeing 10 to 50 times the traffic we did in February.  We've made some tweaks to settings that we think have helped with site stability, but spikes in traffic have still been taking the site down every few days.  I've been keeping a much closer eye on it lately than I had been in recent years, so hopefully downtimes are limited to a couple of hours rather than days.

 

Now to the topic of this post.

 

Along with the huge surge in coronavirus traffic we've gotten in the last month has come a flood of blog comments and DMs from people who have asked to help contribute with donations (and even work) to keep the site running.  I just wanted to let you all know I have been reading them (even if I haven't replied to all), and that I'm incredibly appreciative of the offers.  A heartfelt "thank you" to you all.

 

All of the renewed interest has gotten me excited about the site again, and I'm mulling getting back to work on it in my spare time.  Stay tuned, I may have something to announce shortly.

 

Oh, and by the way, I fixed my first bug on the site in years! :) Group and user search had been broken for the longest time. I was poking around on the database and discovered it apparently lost a few of the required indexes during a server migration a while back.  I added them back and bam, we have search again. OK, so not a real bug fix, but something works now that didn't before, so let me have my moment. :)

Visibility: Anyone

PLEASE READ: Game Clean-up

Wednesday, March 18, 2020 11:12 PM

Apparently due to coronavirus, we've hit the big time...or something.  After a month of February where typically some double-digit number of games would get created in a day, in the last week, we've had 5 straight days of more than a thousand games created per day.

 

You folks must be bored at home!

 

Well, sadly, that "BETA" in the logo is no joke.  We stopped active development on this site years ago, and that was before we made the thing scalable.  So if enough games get created and don't get deleted, the server goes boom.

 

If the last few days is the new normal, we're going to have to get aggressive about cleaning up unused games, since they're sucking up precious server memory.  Starting today, games that are older than 2 days and have never had anyone sit down at the table are subject to getting deleted. Scheduled tournaments that haven't started yet will get an exception.

 

Sorry about the inconvenience, but this site's for playin', not for setting up empty tables!

Visibility: Anyone

Deleted games older than 6 months

Thursday, May 7, 2015 12:03 AM

You may have noticed lag issues or other game flakiness with the site the last few weeks.  It's been a mystery to us, as nothing in our server or network configuration has changed recently, and up till a few weeks ago the site had been running without a hitch for months.

 

I was starting to suspect it may be the sheer number of games created on the site in the last 6 years.  So long as they were active and showing up in a group game lobby, they were loaded in memory.  I counted almost 7000 games that were older than 6 months.

 

So our latest attempt to solve the site performance problems was to delete all games 6 months old or older.  If that was indeed the problem, that should probably buy us another 5 years of runway before we run out of memory again. :)

Visibility: Anyone

Sign Up to Play!

 

Log In

 

Forgot your password?