News & Events
April 21st, 2010
Server is down, enroute to recover
One of our servers has gone down and the backup system has failed to recover it gracefully. We are enroute to the datacenter to recover the system. We will update our customers as we process the situation. Thank you for your patience.
January 7th, 2010
Server Downtime from about 11am - 1pm
Our hosting facility is removing power from our rack space to consolidate servers (seems like this happens every 2 years or so). During that interval, we will be performing hardware maintenance to replace a failing CPU cooling fan and upgrade memory in one of our servers. We apologize for the inconvenience for the time of day, and if were purely our choice, we would have likely picked a more "off-hours" time-frame. We will perform the maintenance in such a way as to minimize the window in which email cannot be delivered by staggering the down time for our primary email server and the backup server.
November 5th, 2009
Network slow or unreachable
Due to circumstances outside of our direct influence, we are very sorry to report a service outage. The co-location service provider where we host our equipment suffered a networking problem. Our equipment was running fine, and all processes were functional, but it was as if they were cut-off from the network. The service interruption started at approximately 6:00am and lasted until approximately 12:30pm EST. All network traffic to our servers was affected. Some web and email traffic was making it through periodically, but all services were intermittent and slow. Our co-location service provider has routed all their traffic (which includes our machines and your applications) onto a spare circuit for the time being, which is not quite as fast as their regular circuit. Hopefully they will repair their primary circuit as quickly as possible. We will provide further updates as we know them. Thank you.
Update 2:20pm
It appears that the data center has found the right switches, plugged back in all the equipment, blown the dust out, taken out the trash, painted the walls and cleaned the dishes. The bandwidth and accompanying higher speeds seems to have been restored.
August, 2009
Backup MX
We have instituted our backup MX server, yeah! What this protects us from (and by extension, all of our customers) is losing email due to an outage on our main mail server. Our secondary mail server will cache all incoming email and delivery it when our main service is back up. Spammers are currently trying to take advantage of the (usually) less protected back up mail service we just put up, but rest assured that we are still heavily denying their ability to send spam to any mailbox we host. Enjoy the greater reliability this brings.
March 23rd & 24th, 2009
Large DNS Outage cascades to us
Most of our services experienced an outage this evening due to a failure of an upstream provider of DNS. DNS or Domain Name Services allow names like "binarystudios.net" to resolve to an IP address, which in that example points at our servers. We purchase our DNS from a company which distributes their services through several geographical separate data centers to avoid a problem where any single failure causes an outage. It would seem something very unlikely happened on their end which caused their services to fail at every location. After about a hour, their service was restored, and it turn, ours has been restored. We will update this section if new information becomes available of if someone has a question where everyone might benefit from the answer. Thank you.
Update March 24th
It seems that our upstream DNS provider is having more trouble today. Some web sites and services we host may be slow to respond, or might not respond at all. We have already taken steps to distribute core services to other DNS locations so that email and our local DNS mirror can continue to function. We will keep updating here as news occurs. Thank you.
October 25th, 2008
WebMail Upgrade
We have updated to the latest version of our web based email client, added in all the plugins and modules we like, made our specific modifications, and added a few new things along the way. You'll automatically get the new version when accessing email at our webmail interface. If you have a problem using the new interface, please let us know, and access the previous webmail interface. Please logout and log back in before deciding it does not work for you, also add in a shift-reload within your browser to refresh the style sheets.
We have added a few new features, and updated several other ones to their latest version. Look under your options menu to configure your look and how these features work. Here's a list of them and a brief description:
- Archive: Check the Options panel and use the button in the lower right corner to pull all the selected messages off our service into an archive. Use this method to back up your email if you would like.
- Select Range: Look to the right of the Toggle All switch for this link. The Select Range feature allows you to pick a range of items by clicking once on the first item, then the last and clicking Select Range picks all the items in-between. You can also trigger this behavior by clicking on an item, then shift-clicking the second, all the items between the two will be selected.
- Small Notify: Look for this link in the left hand folder pane, it pops up a small browser window to monitor when you receive email.
- Quick Save: This feature operates silently in the background until you need it. Every 10 seconds, the email you're working on is saved and if there is a problem you can recover it when you log in again.
- Preview Pane: You can get a more "traditional" view of your email with the message list above and the message contents at the bottom. This is configurable left-to-right also.
- Address book: Additional support has been added to make it easier to add addresses (look near the top of a message when reading it). CSV import and export is also available.
- Notes & Calendar: Don't forget to check out these features, using the smal icons near the upper right corner. The Calendar feature is iCal compliant.
October 12th, 2008
Significant server update
We had to physically go out to our machines today because the provider was having us switch racks. They were consolidating several of their customers to cut down on their rack usage. This appeared to be a good time to install some of the upgrades that we have been waiting for: 25% faster CPU's, 2GB more ram, 5x faster backup machine, OS Patching, larger RAID 5, larger hot backup disks, and upgraded server firmware.
It went well, but took a lot longer then we anticipated. In order to upgrade the RAID 5 cluster, it had be backed-up, then un-made (read as: destroyed) then restored onto the new larger cluster. I would not do this unless I had 2 solid backups, your data (and ours) is too important to risk. During the machine shutdown prior to moving racks, one of the backups become unresolvable - so I had to create another one from the original data. That took over 2 hours.
After finally getting the equipment moved over to the new racks (which had it's own issues, especially around a set of server rack rails which did not fit), the new RAID set had to be initialized. That took about an hour. All the while, the new service cross-connects between our machines would not work properly (these allow us to do what's called "out-of-band" remote system management). After that, the RAID set needed to be restored, which took almost 4 hours.
Adding up all that work equals just under 10 hours of down-time for us, which is the most we've ever had. Prior to today, our uptime was 259 days. There are still a few things left for us to do, but they can wait for a bit. Glad to be back and running, we apologize for how long it took. In the end, we are trying to create a better service platform for all of us, and position us for growth. These upgrades proactively increase our performance capabilities.