Sandbox

We mainly think it is due to Google's "deep crawl".  Once that is done, things should lighten up.

This site isn't the only site that is having issues with Google, other sites are complaining too.  There are multiple posts all over the blogsphere ranting about how Google's crawling is taking sites offline.

I have converted a table in the database that should help increase the performance of the forum, especially when posting new messages.

Seconds ago:

Connection Problems
Sorry, SMF was unable to connect to the database. This may be caused by the server being busy. Please try again later.

robots.txt => disallow Google!

Tom wrote:It should be ok now.

Please note that this server is processing somewhere around 1500 requests with around 75 requests a second, a lot more than it has in the past.  Already it has used 4 GB of system RAM and is using 1.9 GB of swap.

Still happening.  Another thing is when you post a message  the  posting page remains and sometimes ther are double post. Its driving me sane

Still recieving some problems within the past hour.

The "Google storm" seemed to have passed, but I am still making it so that Google can't crawl this site.  It can crawl the other sites, like the main site, developers, and the extensions site, but not the forums.  There is just too much data to collect and would cause this issue to crop up again.

I also made a small tweak to MySQL so that it can store more data in RAM.  We have 4 GB of RAM, lets use it for all that it is worth!

Wow, 75 transactions per second.. I'm also continuing to see slow response and many page not found errors.

I would think past a certain point that upgrading the server wouldn't help. Is Joomla using a single server? Has clustering/load balancing been discussed. It's a big step but if the traffic continues it may be necessary.

This will be an interesting case study for anyone out there that has to deal with their sites becomming popular and traffic going through the roof! It would be great to summarize the troubleshoting steps took to resolve the problem (when it get's resolved).

I think the important thing to point out here is that most of the traffic is Google Bot and will likely die off in a few days. Everything seems to be running fine since the above tweaks were made though. We will continue to monitor the server closely.

Although one server is being used at the moment, it is an extremely powerful and well tuned Dual Opteron box with 4GB system RAM. Should we decide that another box is needed then we will split things up, but we would prefer to scale the existing server as it makes things easier to manage.

Rochen, I totally agree and did not mean to imply for a second that you guys aren't doing a bangup job! I can't imagine adminstering a box with that kind of traffic but it's a problem lot's of people would like to have 

Are there suggested guidelines to be followed when determining that one server is simply not enough? At that point, are there some reference platforms/docs that you would use to implement a clustered approach. I know that's a bit off topic here but I'd be interested in knowing how you proceed in handling this kind of problem should traffic continue to increase.

We would look at several factors including load averages, active concurrent connections, CPU usage, RAM usage etc. and make a decision based on all of that. There are many variables to take into account before going ahead and moving into a clustered environment.

Like I said, we would prefer to keep everything on one server and scale that if possible, purely because it makes administration etc. a lot easier. We have handled clustered setups in the past and we deploy solutions in a way that they are full scalable.

Generally in load balanced setups we use Alteon 180E’s from Nortel and then scale the web servers sitting behind the unit as required. The Alteon units can handle literally millions of requests so the solution is very scaleable and we just add more web servers as required.

Please feel free to drop me an email if you would like to discuss any of this in more detail

How has performance of the sites been these past two days?

Tom,

Only had once a slowdown (don't even know it was at my site or not). Seems like it's under controle.

Fonny

My impression is that the sites are loading perfectly fast again.

Thanks for taking such great care of us!

fonny wrote:Only had once a slowdown (don't even know it was at my site or not).

Yup, yesterday there was a slowdown for abt 10-15mins,

Today no hiccups at all

Great job, Tom and the others

500% improvement. Great work guys!

Perfectly smooth  A real joy today to be working in the Joomla world!

Glad to hear that. :whew:

Now go get some sleep Tom .

Yup, things seem back to normal speed again. Good work team!

the forum now is much more responsive to so many requests. good work!

but still the community of Joomla! users and developer are growing  maybe after 6months the users now will double and more requests to the its servers.

wyzemoro wrote:
but still the community of Joomla! users and developer are growing  maybe after 6months the users now will double and more requests to the its servers.

What a wonderful problem to have!

That brings up an interesting question though. Bandwidth and servers are not cheap so who pays the bills for all this? I imagine that going to a clustered solution would be much more expensive.  Is Joomla.org totally dependent on donations? I've often wondered how this side of the OS world works. This might make an interesting news article at some point. We know how Microsoft pays for it! How does Joomla?

From what I've gathered Rochen has been hosting the Joomla site 'at-cost' for some time now. It wasn't until recently that donations could be accepted as the necessary paperwork wasn't filed yet. Have a look at this forum section and you'll see a few other threads on donations. It was just setup.

what does this mean?

Rochen has been hosting the Joomla site 'at-cost' for some time now

wyzemoro wrote:what does this mean?

Rochen has been hosting the Joomla site 'at-cost' for some time now

From what I understand (someone please take the liberty of correcting me if I'm wrong) Rochen Hosting (several of the core Joomla developers are members of this company) have been hosting the joomla.org website on their own dedicated server and paying all of the hardware, bandwidth and other related costs out of their own pockets since the launch of the project. I have no idea how much or how little the real world cost of running the site is, but it can't be all that cheap considering the traffic that passes through.

Even if they paid for their own hardware and are using their own facilitates, they still have to pay their Tier 1 provider for access to internet backbones. (The fiber optic and satellite links that connect the internet as a whole entity.) Everything from the Forge, forums, extentions, news, help files and other misc. media has to be hosted and available for immediate access. There aren't any public figures on how much this costs, and this isn't including the other expenses such as conference accommodations for promoting the Joomla brand.

Dev team,

Please chirp in and fill in any details that I missed or are inaccurate in the post above. Thanks!

P.S. Joomla.org is one of the top 2,000 sites on the internet in regards to traffic according to Alexa.com:

http://www.alexa.com/data/details/traff ... joomla.org

Another interesting stat:

http://www.urltrends.com/viewtrend.php? ... joomla.org

In summary: We're popular! Really popular!   

talking about popularity. joomla statistics at alexa is pushing high! and mambo is slowly going down and losing users?

TheSaint wrote:From what I understand (someone please take the liberty of correcting me if I'm wrong) Rochen Hosting (several of the core Joomla developers are members of this company)

You are partially right there The Saint.  Brad is the only one of the Rochen employees that is part of the Joomla Core Dev Team.  I'm not part of the Joomla Core Dev Team, but I still watch this forum like a hawk and make sure that the server is performing as fast as it can be.

TheSaint wrote:I have no idea how much or how little the real world cost of running the site is, but it can't be all that cheap considering the traffic that passes through.

I can't quote the exact numbers since it has been a long time since I have looked at it, but I will tell you this.  It is more than $500 a month.

Why?  Simple.  The server is granted around 1000 GB of bandwidth a month, but this isn't some cheap bandwidth here.  This isn't Cogent.  This bandwidth is provided by Tier 1 bandwidth providers, several of them.  The server is multi-homed, which basically means that it has more than one connection to a major backbone.  It has about five.

Then you have to look at the hardware.  It has two AMD Opteron 244 chips running at 1.7 GHz (featuring 1 MB of L2) with 4 GB of RAM and two heavy-duty SCSI drives.

All of this adds up to some major amounts of money spent on behalf of the Joomla Project.

RochenHost is happy to be helping out in our own small way to support this project.

Thanks Tom for filling in the blanks and keeping us online.

Thanks Rochen! hope you won't stop supporting Joomla! in the future and continue the good deeds helping the community to grow thru your hosting.