indieCart Seller's Blog

Hi everyone,

I know it’s been an upsetting, confusing and frustrating month on Hyena Cart for many of you. There have been a lot of questions and I’ve tried to answer most of them via blog, emails and facebook posts. At this point, I thought it would be a good idea to post an FAQ so we have all of the questions and answers in one place. OK? So here we go:

What the heck has been going on on Hyena Cart lately?

On Monday, July 16th, and part of the weekend before, a hacker caused Hyena Cart to redirect peoples’ browsers to a site that installs the malware program Live Security Platinum on Windows computers. Our host put several ‘band-aids’ in place but ultimately could not find the source of the breach. Two more attacks occurred, one on July 18th, and the next on August 8. By the third attack, on Aug 8, the host was able to find the source of the breach (they had installed monitoring software earlier, but had to wait for an attack in order to find the entry point). The breach was patched and we have been running without issue since then.

Why couldn’t you fix the issue between July and Aug? There were several weeks in there — should be plenty of time.

We took that time to harden and patch different areas of the site. However, since the host had been unable to detect the exact weakness being exploited, it was impossible to tell if the issue had definitely been fixed. It wasn’t until the most recent attack (Aug 8) that the host was able to tell exactly how the hackers were redirecting users away from the site. As mentioned previously, this weakness has now been patched.

Was any information on Hyena Cart (passwords, addresses, etc.) breached?

No, the hacker was interfering with the server, not the actual Hyena Cart code and databases, so there was no information breach.

Why were sellers not notified by email in every instance?

When every seller signs up, they have an opt-in option into the Hyena Cart blog. This has been the main conduit for communicating all code updates, news and any issues on the site. Our Privacy policy states that: “Hyena Cart will not send promotional, service-related or information emails of any kind. If you wish to stay abreast of the latest Hyena Cart news, you may request email notification from the HC Blog. You can also request to be removed from this mailing list at any time.” It is extremely important to us that no Hyena Cart users, whether they are shoppers or sellers, receive unwanted email from HC.

That said, there has been one time (this past spring) when we did notify each and every seller about a password change initiated to enhance security.

And today, we are sending this note to every seller with an active account (past 2 months) to let you know a few specific things:

1. We have sunsetted the Hyena Cart blog, which was ultimately not the most efficient way to reach you (it wasn’t available if the server was crashed or had issues, and it wasn’t possible to read posts if the site was blocked for malware)
2. We have started a Google group, which will be the new way to keep in touch with Hyena Cart updates and issues. To join, please send an email to hyena-cart+subscribe@googlegroups.com
3. We are switching hosts (more on that later) and this will entail some downtime for the site. We are currently estimating that this will happen Tues / Wed this week, but we will use the Google group and our Facebook page to give everyone notice: http://www.facebook.com/hyenacart

Is Hyena Cart safe now?

It is safe at the moment. We have consistently had over 10,000 unique users on the site every day and have had no recent reports of issues. As mentioned, the root cause of the recent attacks has been found and patched. However, hackers are constantly looking for new ways in, so there’s no 100% guarantee of future safety.

If our server is safe now, why are we switching hosts?

It simply took too long for our current host to isolate and fix the issue. The problem was at the operating system level — this is a service I pay for, not a job I manage myself. I depend on the host to manage the server and the operating system, come to me proactively if they find security weaknesses or breaches, and fix them. As you all pay me to provide a service — a safe, dependable, functional website to sell your items, I depend on them to provide to me a safe, dependable server from which to host my site. They failed in their job, causing me to fail in mine. I had to escalate issues multiple times before the source of the breach was found. Not only that, but it took three breaches of the site, over the course of nearly a month, to find it. One other point — multiple servers on the current host were attacked, which makes it seem that 1) that host is being targeted for some reason, 2) the attacks have not been targeted specifically at Hyena Cart (we’re just a site that happens to be hosted on their servers). So moving away from that host makes a lot sense at this point.

Why didn’t we just switch hosts back in July?

At that time, I didn’t know whether the breach was because of the server, or because of the Hyena Cart application code. If it has been the application code, then switching hosts would have made no difference whatsoever. The hacker could have just continued breaching the site, regardless of where the site was hosted. Without knowing the source of the breach, there was no way to know whether switching hosts would fix the issue or not, so I did not choose to move us at that time. It wasn’t until last week, when the host finally found the root cause, and I realized it was not related to the application code, that I decided we needed to switch (for the reasons given in the answer to the previous question.)

Will we be 100% safe on the new host?

Unfortunately, there are simply no guarantees of safety. Just like, no matter how sophisticated and up-to-date your security system is, your home may still be robbed. What I can say is that the new host offers specific site security and monitoring which we aren’t getting at our current host. At this point, we are certainly much more educated, aware, and defended against these types of attacks than we ever were before.

What is the process for switching to a new host? Will any of my information be lost?

No information should be lost. Here is the order of events:

• Migrate all files and database information to new server <– this is currently in progress
• Test site functionality on new server – this may be quick, or it might take longer, depending on how many things we need to fix in the application code and the server configuration to make sure things are working properly

The two previous things can happen while the site is live and running off of the old server

• Freeze the site – once everything’s up and working on the new server, we’ll be ready to do a final sync with the old server, However, we’ll need to take the site offline at that point, to ensure no more updates are made in the database (so no information is lost). This will be the beginning of the downtime, and we will do our best to give at least 24 hours notice before this happens.
• Final sync of old server to new server
• Change DNS settings so hyenacart.com runs off of the new server
• Site will now be live and running off the new server, but DNS settings take ~24 hours to propagate across the web, so it will become available to shoppers and sellers in waves, depending on their geography

So the site will be unavailable to you for somewhere around 12-24 hours, depending on how long the final sync takes, and how long the DNS settings take to update in your area. I will use the google group and the facebook page to let people know when the downtime will begin.

How are you letting all the shoppers and Spots’ Corner users know about the downtime?

Once we know exactly when the downtime will be, we’ll post an announcement on the the site and on Spots’ Corner, letting everyone know. We’ll also post to the Facebook page and Google group.

What happens to our paid time with this downtime?

Once we’re up and running on the new server, I will ensure that all sellers (store and Spots’ Corner) are comp’ed for the downtime (i.e. I will extend everyone’s paid time at least 24 hours, but probably more, just to be absolutely fair). I’ll do it as quickly as possible after we’re up and running, but please give me at least 24 hours.

What if I want a refund for time I’ve already paid for?

Please contact us at support@hyenacart.com for a refund if you feel it’s needed.

They are building our server now, and we’ll begin the migration process over the weekend. There won’t be any downtime right now. Once the new server’s up, I’ll be doing testing and modification with them to ensure all the code functions properly and settings are correct. Then we’ll do the final sync and transfer the site to the new server. At that point, we’ll be down

First off, because I don’t want you to miss it, I want to tell you that we are sunsetting the Hyena Cart blog. We’ll leave it up for reference, but will be moving our communications to this Hyena Cart google group in the future. Why? A few reasons:

• Crisis communication has obviously been an issue in the past few months, sometimes because people feel they are not getting the direct communication they need, and sometimes because it’s actually impossible to send communications via the server because it is down or needs to be taken off-line immediately. Using a google group greatly speeds and simplifies the process of notifying everyone of updates.
• Google groups is an opt-in option. Many have asked for an email to all vendors in recent situations and I have struggled with this. We explicitly state in our Privacy Policy that “Hyena Cart will not send promotional, service-related or information emails of any kind. If you wish to stay abreast of the latest Hyena Cart news, you may request email notification from the HC Blog. You can also request to be removed from this mailing list at any time.” and this reflects our belief that everyone has their own comfort level with quantity of communication and should be able to control it.

So, if you would like to receive updates on Hyena Cart code updates, and, most importantly, if you would like to be emailed directly during server issues (crashes, malware, hacking, etc.), please go to this page: https://groups.google.com/d/forum/hyena-cart and click Join. Alternatively, you can send an email to hyena-cart+subscribe@googlegroups.com (you can leave the subject line and body blank.)

That said, we had another malware attack late last night into early this morning. The good news is, based on the issues a few weeks back, the host had put some tracking measures in place and was able to identify the security gap the hacker was exploiting. The gap has been closed, so we are protected from that particular exploit. You’ll note that I still can’t say that the site is 100% safe from here on in. Unfortunately, it doesn’t work that way. No site is 100% safe, and there is a group of people that is constantly developing new ways to attack websites. I can say that the different hacking methods used this past April, last month and today have all been blocked, and we have implemented some changes that will hopefully prevent other methods in the future. I can also say that I will be working with a third party specialist to strengthen site security even further.

This has been difficult for our businesses, and our customers. We’re working behind the scenes to prevent issues in the future. For those of you who have lost faith and trust, I hope you will consider returning after a period of security. If not, I understand. For those who are continuing to use Hyena Cart, thank you and thank you again. You’re the best group of sellers on the internet, and I’m grateful to play a small role in helping you showcase your items and reach your customers.

Thanks, and don’t forget to join the google group for future updates!

Hi all,

Just wanted to give you another update on the status of the server. We had some down time on Saturday night due to a denial-of-service attack on the host. There was no breach of our databases or information, or risk to users’ computers, but the site was unavailable for about an hour and a half. The host has put measures in place to prevent this from happening in the future.

As it happens, the malware issue last week also affected more than one server in our host’s facility. I honestly don’t know if this is just a string of bad luck for the host or a more deliberate attack on their system, but they and I have been taking the server’s and site’s security to a much higher level.

Thanks for your patience as we work through these issues. We will continue to work with the host to increase security and, at the same time, will do research into alternate hosts. Changing hosts isn’t a trivial move and would likely need a few days’ downtime, so it’s not something to be done lightly. Additionally, no host is invulnerable, so switching hosts wouldn’t be a guarantee of 100% uptime or an end to all malware attacks.

I won’t be making any other major changes to the site in the next few weeks, so we should be in a relatively stable state. That said, if you are experiencing any issues with Forbidden page errors or anything else (e.g., one of our security measures broke the facebook app, but that’s fixed now), please let us know at the helpdesk: support@hyenacart.com

ETA ~ steps we’ve taken so far to increase security include hardening PHP in various ways (disabling unneeded functions, changing other PHP settings), using files on the server to prevent certain types of attacks (hence the Forbidden pages that pop up sometimes — again, let me know if they are coming up in error), installation of a cron job to constantly monitor for malware, and router level prevention of denial-of-service attacks. Future steps include more extensive modification of the code to deter hackers, as well as the hiring of a security professional to examine the entire site for vulnerabilities.

Thanks again for your understanding!

Hi everyone,

I’m posting another update on the malware issue. As of now, the site is clean, but we did have reports of a second wave of infections on Wednesday night. They didn’t last long (about 4 hours), and the server / site were scanned multiple times Wednesday and Thursday night (I closed the site temporarily Wednesday morning as scans were being performed).

At this point, the host has narrowed down the second attack to weaknesses in php. They and I have spent yesterday and today tightening security settings, and combing the code for security gaps and plugging them. If you notice the site acting strangely (especially if it gives you a “Forbidden” page, or if external images aren’t displaying), please let us know at support@hyenacart.com and we’ll fix it as soon as we can. There are a few larger changes that need to be made, and will require fairly extensive code rewrites, so I will update you when those are going out so you’ll understand if you see unusual error messages.

For now, the site is up. As a facebook poster mentioned, we feel it’s best to have it running and watch for further attacks as it sharpens our strategy on how to block them. If we kept the site offline, it would not be possible to determine where to focus our efforts to increase security. If you visit the site *please* ensure that your operating system is completely up to date and you have strong antivirus software installed. Things can change moment to moment. As mentioned, at this particular moment, the site is clean, and we are making changes throughout the course of the day to tighten security, but that does not guarantee there will not be a virus in the next moment. I know that’s not totally confidence inspiring, but it’s actually the case for any site you currently visit on the web. There are no guarantees of security on the internet, but please know that we are doing our best to make Hyena Cart as secure as we possibly can.

Also, note that more frequent updates are being made on the Facebook page.

Hi everyone,

I just wanted to post another update as well as more information. The update is that the site has been reevaluated by google and given a clean bill of health so we are up and running again (yay!). If you are still seeing the warning, it is because the change in status has not propagated to your part of the web yet, but it will soon.

I’ve pieced together the order of events and wanted to clarify the steps that were taken on this end and steps that are being taken to improve our security and communication in the future. So — what happened over the past few days:

• Apparently, sometime Sunday evening, a hacker began injecting malicious code via several servers in our host’s server bank using url’s. The hacker did not access any of Hyena Cart’s databases or information that is stored on the site. Here is a quote from our host:

  “The core issue behind the issue was due to the attack from the IP which we have blocked yesterday and it had affected multiple servers. Once we blocked that IP, no more sites were blacklisted in google. In most of the servers, issue was due to vulnerable wordpress plugins. But we haven’t find any such occurrence in hyenacart.com. Also no traces of malware injection has reported in last 24 hours http request.

  Since we haven’t find out the exact issue, we will monitor the server for next few days to see whether injection attempts occur again. As a preventive measure we have updated modsecurity to the latest version.”

• By Monday morning, google had noticed the malware on the site and had blacklisted Hyena Cart. Tickets and facebook posts began to trickle in regarding the issue.
• By 11:12 am, I was aware of the issue and contacted the host about it. By 11:40, they had isolated the issue and blocked the IP that was injecting the malware. At this point, the site was secure and no viruses were being installed by the site.
• At noon, I replied to the few wall posts regarding the issue. From my computer, all issues were resolved and no warning messages were showing in my browsers.
• For the rest of the day, I had very limited access to the internet, but monitored my email (which gets notification of facebook wall posts — usually) to ensure things were still under control. Since I did not see much activity, I assumed things were fine. This would have been the point in time to issue a broader communication to everyone about what was going on, but because 1) I knew the site was secure, 2) I didn’t realize that warnings were still being shown, and 3) I did not have the time to sit down and compose a well worded and well informed note, I did not do this. I sincerely sincerely apologize for this, but please understand that I was making the best decision I could, balancing all of the priorities I had, and based on the information I had at the time.
• At this point, the site was safe, but google had not yet changed its status. Thus, people were still getting warnings, even though the site was secure.
• At about 9:30pm, I was able to get back online and realized that there had been many people still unsure about whether to access the site and a *ton* of conversation I hadn’t realized was happening. I have done my best since to respond to everyone in as timely a manner as possible.
• As of this morning, google has removed HC from its blacklist and the warning messages have been going away as this change in status propagates across the web.

For anyone who contracted the virus, it was the “Live Security Platinum” virus. Please see this page for more information.

If you would like compensation for featured time or paid time because of yesterday’s outage, please contact us at support@hyenacart.com.

Obviously, my response to the issue yesterday was not perfect. It is not a simple task, deciding when and how broadly to communicate with the group. There are multiple ways of contacting you, including direct email, this blog, and facebook. I tend to try to contact only the people who are affected and, based on what I knew on Monday morning, I tried to do just that — replying to the facebook wall posts, emails, and support tickets that had come in before noon. If I had known later in the day how widely people were being affected, I would have prioritized making a wider communication, but the lack of emails in my in-box reassured me that things were fine.

As mentioned in the quote from the host, steps have been taken to increase security on the server. As for communication, I am trying to figure out how best to manage and prioritize communications. This is my proposal:

• Direct email — I will reserve this for what I feel are “Code Red” situations (where passwords, addresses, or other personal information stored in HC databases are at risk.)
• Blog — I will continue to use this to update sellers on software updates and other critical information that is important for you to know. You do need to opt-in to receive emails of blog updates.
• Facebook page — I will use this for more ‘everyday’ communications — cool products, fun tidbits. I’ll also use it for broader communication in the event of site outages.

As I’ve mentioned on the facebook page, I understand that some of you may feel the site is undersupported for your business needs. But for those of you who weigh the pros and cons and find that HC is still right for you business, I am profoundly grateful for your understanding and so happy to have you on the site. Thanks again for all your support!

Hi all,

Sorry for the delayed update on this topic. We have had a day of making sure we’ve identified and isolated the issue. A hacker was injecting malware via url injection on the site. They have been blocked and the host is monitoring the site to ensure that it does not happen again. In the meantime, google has blacklisted the hyenacart domain, so it is still being identified as a site with malware (even though the malware is no longer present). We have applied to google to reevaluate the site, but are not sure when this will be complete.

In the meantime, it is safe to access the site, despite the warnings. We are extremely sorry for everyone’s inconvenience. We are also sorry for the delayed communication. We did make a few updates on our facebook page, but did not have a chance to complete a blog post until now.

Well, there has been a lot of change on the site lately, from a total redesign through to a move to an upgraded server, and lots in between. All of this change has been intended to make Hyena Cart even better for the vendors and shoppers and we’d like to take this opportunity to stop, regroup, and ask you your opinion of the site. Please consider answering the survey found on the welcome page of your vendor pages, as well as the Favorites page of your shopping account. We really appreciate your feedback and look forward to continually updating and improving our little corner of the web.

Thanks everyone!

Please be on the lookout this afternoon for an email being sent out to all single-user store owners. The text of it will look like this:

Dear [your name],

As you know, privacy and security are both extremely important on our site. In the past week, we have experienced the first security breach in our 8 years of operation.

A hacker was able to access login id’s and passwords to an unknown number of Hyena Cart single-user seller (HC Multi) accounts by exploiting the code used in the stores. This vulnerability has been eliminated, but the hacker may still hold login information to some seller accounts.

The hacker was using this seller login information to access vendor pages and change the Paypal account email address, redirecting shopper payments to his own account.

As mentioned, due to changes made in the code last night, the hacker no longer has access to current seller passwords. However he may have saved login information during the breach. To completely shut the hacker out, we have decided to proactively change the password to every seller’s account.

Your password has been changed to [your new password]. Please log in to your account with this new password and confirm that your Paypal information is correct (on the Store Settings >> Profile Page.) You can also modify your password, but please do *not* use your previous password.

You can log in to your account via this link:

http://hyenacart.com/HCmulti/admin/login.php?admusername=[your login id]&admpassword=[your new password]

There has been no breach to multi-user seller accounts (like congos), or to shopping accounts. In addition, we have added the following safety feature: whenever the email or Paypal account address are changed in a seller’s account, an email is immediately sent to the seller to confirm that it was an intended, authorized change.

We are currently compiling documentation of the hacking and will be notifying the authorities of the scope and identities of the known suspects (identified via Paypal information).

If any of the payments in your store were misdirected, please contact the shopper and ask that they file a Paypal dispute. In many cases, the payment has already been refunded. Once they receive a refund, they can send the funds to your account. We can also help with contacting shoppers if needed.

Thank you very much and we apologize for the inconvenience. We sincerely apologize if you received this message twice. If so, please disregard the first message.

If you have any questions or concerns, please contact us at support@hyenacart.com. You can verify that this is a genuine email from the Hyena Cart site as it is referenced in this blog post.

Thanks,

Karen

HC Admin

Thanks for your understanding. Also, please don’t be concerned if you did not receive an email notification of this blog post. We chose not to send one out since we are emailing all affected parties directly.

There have been several issues with Paypal payments made via Hyena Cart and I’d like to make everyone aware of them.

First, several accounts have been hacked via compromised passwords, and the Paypal address on the Store Settings >> Profile page was changed so that payments went to someone else’s account. Everyone, *please* log in to your account and confirm that your Paypal address is set correctly. Also, please increase the strength of your password. It should be a combination of letters, numbers and symbols, and longer than 7 characters. We’ll also be looking into installing more measures to help prevent this from happening in the future. If it does happen to you, please send the hacker’s Paypal email address to support@hyenacart.com.

Second, we have some reports of payments going to the wrong Paypal account even when the seller address is set correctly. This does not seem to be malicious as the receivers of the misdirected funds have all refunded the incorrect payments. We are unsure if this is a glitch on the Hyena Cart or Paypal side. If it does happen though, please contact the helpdesk with the Paypal address of the incorrect recipient of the funds. We will be in contact with Paypal to resolve this issue.

Thanks everyone! Your security is extremely important to us and we will work to resolve these issues as soon as possible!