==== 10:41 pm UPDATE ====
The issue was in fact a networking conflict and it has been cleared.
All sites are serving as expected.
==== Initial Post ====
At about 6pm EST tonight (Feb 16), the Linux web server Wallyworld stopped serving websites. Our server admins are working to determine the cause to be a possible network conflict. They are working to address the conflict.
Online Merchants and folks interested in getting into selling online will want to review this…
Host Merchant Services offers credit card processing services, payment gateways, and mobile commerce solutions with a focus on outstanding customer service.
Take advantage of a special offer for a free PCI Compliance Analysis, a $100 value!
Hello,
[Update: 26.Jan.2012 @ 5:20 PM]
Webserver Win002 is back up and running.
[Original Post: 26.Jan.2012 @ 5PM]
The webserver Win002 needs to be rebooted to fix an issue with Active Directory. It should be back online in fifteen minutes after being rejoined to the domain. We apologize for the inconvenience, the downtime should be brief.
Last week, customers on Mail77 experienced multiple performance issues which included major delays in sending and receiving email. Any amount of time without email is unacceptable – and the extended time to resolution was especially frustrating and painful for all of the customers who were affected.
Please find a detailed overview of the nature of the issue, the root cause, resolution and the steps we are taking to mitigate this in the future by going to the following URL.
http://www.hostmysite.com/docs/RFO-HostMySite-Mail77_Issue_01102012.pdf
- Final Update 13.Jan.2012 @ 4:00 PM:
We have been monitoring the server Barricade (mail77.safesecureweb.com) over the last 24-hours and we have seen no further issue with mail. The spooling issues are no longer a problem and mail delays are within normal expectations of sending and receiving email.
All connection issues that were experienced by 3rd party mail applications such as Microsoft Outlook should no longer be receiving error messages and mail should be sent and received through the system without issue.
We do appreciate your patience in this matter and as always, if you do have any further questions, please feel free to contact our support staff.
——————————————————————————————-
- Update 12.Jan.2012 @ 3:30 PM:
We seem to have found a few hard drives from the same batch which were previously not known to be bad and the spool grew rapidly around 10:30 AM. We pulled out the drives from the suspected bad batch and performance on the disks improved quite rapidly. The spool was halved in about 15 minutes and has been fairly stable most of the day. Mail has been sending, albeit slowly.
The connection issues are still plaguing the server and in an attempt to fix this, we switched network traffic to a NIC that was installed (but not enabled) last night. We didn’t switch it then because at the time, mail77 seemed more stable. If you are finding MS Outlook, Mail or your mail program of choice getting connection errors please try logging into webmail. Webmail can be accessed by going to http://mail77.safesecureweb.com or to http://mail.[your_domain.com].
We have a robocopy running on the live hardware to copy the mail folders, etc to the new hardware. We also placed a new drive from a different batch of hard drives into the live hardware to begin rebuilding one of the arrays and that is progressing at a reasonable pace, but not hyper-fast because the server is live and e-mail takes a significant amount of read/writes to a hard disk. The RAID arrays on the new hardware are also rebuilding. Both are expected to finish overnight. If the arrays on the live server are stable, the server will remain on the live hardware until next Thursday when the version of SmarterMail is scheduled to be upgraded. If it still proves to be an issue, we will be cutting over to the new hardware tomorrow.
——————————————————————————————-
- Update 11.Jan.2012 @ 9:00 PM:
We have done a RAM upgrade to mail77 and made some suggested changes to the BIOS.
All spooled mail from earlier today was put back and sent out within 5 minutes.
——————————————————————————————-
- Update 11.Jan.2012 @ 5:54 PM:
Performance is still degraded on the old, live hardware. The spool is still a bit high but the server is sending messages. We are going to do a copy of the live data to the new hardware tonight, verify the RAID card has updated firmware to ensure it can handle the new drives and switch over to the new hardware overnight.
——————————————————————————————-
- Update 11.Jan.2012 @ 2:27 PM:
The spool has been growing but the mail server is processing messages, albeit slowly. We’re looking at what settings we can tweak temporarily to help with the I/O. We’ll be restarting SmarterMail a few times. Customers using POP3 are reporting errors connecting with mail.[your domain name here] in MS Outlook / Apple Mail, we’re finding changing mail.[your domain name here] to mail77.safesecureweb.com is allowing connections to affected customers.
——————————————————————————————-
- Update 11.Jan.2012 @ 12:21 PM:
The old hardware has been online since approximately 11AM and sending mail. The spool is holding steady.
——————————————————————————————-
- Update 11.Jan.2012 @ 10:45 AM:
“Hey, what’s going on with my e-mail??”
Mail77 was migrated to new hardware last night as previously reported. Mail service on mail77 is increasing load to the point where I/O is overloading a previously unreported hard disk error. That disk was causing the raid array to throttle down in the original hardware so the disks in the array could keep parity of data, i.e. stay synchronized. Because the drive was not reporting as failing or even erroring in the old hardware, when drives were chosen to move to the new hardware and rebuild the RAID array, the poorly performing drive was unknowingly moved. This means indexing, rebuilding of the RAID array and the very high I/O of a mail server have been causing performance on the new server to slow/stop basically every 10 – 20 minutes.
“Ok, so what are you going to do to get my mail back up quickly?”
The old server still has synchronized data and we are going to put the old server back online. Some users will not see mail from the past 10 – 12 hours. We are going to put that server back online, let the new hardware rebuild its RAID array, then synchronize/merge data. This means people logging into webmail and using IMAP on mail77 will not see messages from the past 10 – 12 hours because IMAP and webmail leave a copy of the messages on the mail server. POP3 users will be unaffected because POP3 downloads e-mail to your computer and does not leave a copy of e-mail on the mail server unless otherwise specified.
“What about tomorrow and going forward?”
The above decision was not made lightly and we are making every effort to get mail service up and running on the new hardware because it is impacting for everyone on that server. With the old server up and running, performance will be slow but it will work. It is a big upgrade for hardware, much better processor/more ram and what we thought at the time were better disks. Basically, our shared admin team hasn’t been working on anything else other than this for the past two days.
——————————————————————————————-
- Update 10.Jan.2012 @ 10:30 PM:
During the migration of Mail77 to new hardware, all mail was delivered to the original server. Once the new server was brought online and functioning correctly, we cut over to it. We are currently migrating all of the spooled messages from the old server to the new one at a rate of about 1000/10 minutes.
——————————————————————————————-
- Update 10.Jan.2012 @ 9:00 PM:
Mail75 has had all spooled mail fed back through and sent out. This server is now functioning normally.
Mail77 has been fully migrated to new hardware and spooled messages have been fed into the active spool for the last ~1 hour and will continue until all messages have been delivered
——————————————————————————————-
- Update 10.Jan.2012 @ 5:15pm:
spooled mail on Mail75 is being slowly fed back into the spool for delivery. new mail is delivering normally.
——————————————————————————————-
- Update 10.Jan.2012 @ 4:42 PM:
We are migrating mail77 to new hardware.
——————————————————————————————-
- Update 10.Jan.2012 @ 4:05 PM:
We are rebooting mail77. This should take a few minutes.
——————————————————————————————-
- Update 10.Jan.2012 @4:02 PM EST:
To verify if your mail is currently on one of these servers, please do the following…
Ping your mail IP address by opening a command prompt and type
ping mail.yourdomain.xyz
If the results return with either “208.112.71.220″ (mail75) or “204.12.14.36″ (mail77), you are on one of these servers. Mail on mail75 is currently delivering as we are moving into mail back into the spool to deliver. Mail77 is still performing in an underwhelming manner. These are definitely our top priorities. Unfortunately, the disk I/O on mail77 is taking a long time to narrow down to one or two single causes.
We do thank you for your patience thus far. Additional updates will be posted as soon as we have more information.
——————————————————————————————-
- Original Post 10.Jan.2012:
Hello,
The mail spools on mail75 and mail77 are abnormally high and while they are sending mail, there is a significant delay. Our shared admin team is working on resolving the spooling with these servers. We will provide an update when one is available.
===== Update: 2:26PM =====
Hello,
The server is back up and running. After the site was rebooted, the external network interface did not come back up. This was resolved by one of the shared admin team at console. Sites should be serving without an issue.
===== Original Post =====
Hello,
The shared webserver California encountered a system error which required it to be rebooted. The reboot is taking a while because it is virtualized on Hyper-V and the entire node required reboot. The restart times for the virtualized servers are far above normal reboot times. We have our network operations team and admin team looking into this right now and anticipate the server will be back up soon. We will post an update once the server is back online.
===== Resolved 5:15pm =====
We have completed the migration from the compromised physical server to the newly deployed vm. Please contact support if you have any issues with your site.
===== Update 8:35pm =====
FTP permissions have been corrected and FTP is now running as expected.
Please remember that the FTP passwords were all changed and you will need to update them by following these steps:
In order to reset your FTP password(s), please do the following:
***NOTE: We do ask and recommend that you use strong passwords for each FTP user. This means to use both upper and lower-case letters as well as numbers. Please do not use a password that is the same or similar to your previous FTP password.***
1) Login to your control panel at https://cp.hostmysite.com
2) Click on “Website Administration” and select “FTP Manager” from the left left-side tabs.
3) Select each individual user from the drop-down box at the top of the page.
4) Enter the new password and confirm it.
5) Click “Update user”.
===== Update 5:19 PM =====
Spoke too soon. FTP is not working. We will provide an update with more information soon.
===== Update 4:56 PM =====
FTP is now working on Dotnet1. We did need to set new passwords and you will need to change your FTP user password. Instructions can be found here:
http://www.hostmysite.com/support/cpanel/changepasswd/#ftp
The server should be completely back online. We expect to be releasing a formal RFO sometime in the next five to seven business days. Please contact support by phone (877-215-HOST) if you are still having issues with your site.
===== Update 12:35 PM =====
Websites are serving again. FTP is still not working but that is our next task. We will make another update once we have more information about the FTP users. It’s possible password resets will need to occur but if we do that, we will let you know the new password.
===== Update 10:49 AM =====
No sites are are serving because there are still some permissions issues on which we are working. An attempted reinstall of IIS was not successful at fixing all the issues with permissions so we are going to be fixing them manually. We will post an update when a reasonable ETA can be given. This has absolute top priority among our shared admin team.
===== Update 7:30 AM =====
Websites are now serving pages. FTP is still down, and we are working on correcting that.
===== Update 6:00 AM =====
The migration of content has completed. We are currently working on configuration and permission changes, and some sites may be down during this period.
===== Update 8:45pm =====
We have been able to bring the new vm online and sites are starting to serve as their content is copied over to it. There is no way to expedite the copy nor change the priority of a specific folder.
Unfortunately, we have not ruled out the FTP as the cause of the compromise and are not enabling FTP until further evaluation is done.
If you find that your site is loading but have other issues, please submit a ticket to support otherwise sites will come live on their own.
===== Update 6:20 pm =====
Data migration is still running. We will update next when we have more information.
===== Update 1:00 pm =====
Server configuration is nearing completion. Data migration is currently ongoing.
===== Update 9:00 am =====
The data migration is ongoing. We are currently configuring the new server with the required website and system configuration settings.
===== Update 3:30 am =====
The migration continues at this time.
===== Update 1:00 am =====
We have started the migration of content to the replacement server.
===== Update 10:25 pm =====
A new vm has been deployed and we are configuring the network to begin migration of content.
===== 9:39 pm =====
The web server named Dotnet1 has been found to be compromised.
We have been forced to pull the server off the network due to nature of the compromise.
The plan is to deploy a replacement virtual machine and migrate all web content over to it.
Updates will come when available
=====3:00am Update=====
We have completed the migration and all websites are up!
======10:36 PM======
The shared Linux Builder MGMGrand was found this evening to be root level compromised. A new server has been deployed and the content is being migrated over. Due to the nature of the compromise however, the server has been taken offline until the migration can be completed and the new server brought live.
=====6:30am Update=====
Server performance has stabilized at this time. We are working to correct the issue with the old mail subfolders.
We will continue to provide updates as they become available.
=====4:00am Update=====
Mail server performance has improved dramatically. You may still experience some slight slowness with webmail, and this is due to the indexing that is occurring. Additionally, for now your old mail can be found under the Mail –> Mail subfolder. New mail will arrive in the Inbox as normal. We are working on correcting this and will let you know when we have.
We will continue to provide updates as they become available.
=====2:00am Update=====
We have completed the chassis swap of the server and are working hard to get performance back to 100%.
We will continue to provide updates as they become available.
=====12:00am Update=====
Per the suggestion of the software vendor, we are performing a chassis swap of the server. The reason is to mitigate some the disk i/o that is occurring.
Will update with more info when available.
===== 10:30pm Update =====
We have been working throughout the day to resolve this issue and at this point have opened a support ticket with the vendor. We are working directly with their support staff at this time. Will provide update when available.
Early this morning mail21.safesecureweb.com was upgraded from Smartermail 5.5 to Smartermail 8. The installation of the software itself successfully completed. Joining the server to the active directory services was not as successful–a few internal services do not function properly. To resolve, the server required a few reboots around 9AM to get it rejoined to the active directory services. Indexing in SmarterMail 8 was left running. While indexing may be beneficial in the long run, we found it to cause severe performance issues while it indexes data. We have stopped the SmarterMail indexing services for now and are seeing some improvement. Pages load after a while rather than error.
Please configure outlook or your preferred e-mail program to receive mail if you need access. Our support pages for e-mail has a “How Do I” section at the top with links to instructions how to configure common e-mail programs: http://www.hostmysite.com/support/email/.
We will provide an update when we have more of an idea what the root cause is and will engage our vendor if the issues persist.
Our Autumn Newsletter provides valuable offers for HostMySite.com customers:
Up to $200 of combined Advertising on Google credits provided by Google, Bing and Yahoo!Review the Autumn Newsletter at http://www.hostmysite.com/aboutus/newsletters/Oct2011/index.shtml
